azure-mgmt-network is huge #11149

yoava333 · 2020-04-30T15:20:28Z

Package Name: azure-mgmt-network
Package Version: 10.1.0
Operating System: alpine-3.11.6
Python Version: 3.8

Describe the bug
azure-mgmt-network is the biggest dependency by far for my project, the size on disk is 110MB, which means that it slows down everything else - like building & publishing my application.

To Reproduce
Steps to reproduce the behavior:

pip install azure-mgmt-network
inspect the site-packages/azure/mgmt/network folder

Expected behavior
I would expect the SDK to be much lighter, for comparison the entire GCP python SDK is ~5MB and boto3+botocore is ~50MB (which includes the entire SDK).
If I were to install azure-mgmt-compute & azure-mgmt-resource I would get another 30MB, in one of my applications the azure folder is 250MB bigger than all the rest of the filesystem combined.

The text was updated successfully, but these errors were encountered:

chenjianfei2017 · 2020-05-01T03:18:05Z

hi @yoava333 , i checked the pkg size, it is huge because network has multi versions, when you download this pkg, it will download pkgs of all version, we will discuss this on our meeting, thx.

RodgeFu · 2020-05-06T08:46:49Z

@lmazuel , do you have any suggestion here? is it possible to optimize the codegen to reduce the size? thanks.

ghost · 2020-05-15T10:49:36Z

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @aznetsuppgithub.

RodgeFu · 2020-05-15T10:51:08Z

We did some investigation and found that the current network package still include very old api-version. Add "Service Attention" label to include service team to confirm whether we still need to support all the api-versions, it will be helpful to reduce the size if some old api-version can be removed from the package. thanks.

ghost · 2020-05-19T03:41:53Z

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @aznetsuppgithub.

ghost · 2020-08-12T16:35:42Z

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @vnetsuppgithub.

talmachani · 2020-10-04T07:31:22Z

Any update on this?

ghost · 2020-10-05T17:35:59Z

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @vnetsuppgithub.

yoav-orca · 2020-11-12T14:45:01Z

Could you update on the status of this issue?

Azure right now is ~330MB I cannot use the sdk it in an AWS Lambda function anymore

sodul · 2021-08-16T20:42:07Z

We have written a script that deletes the unreferenced older APIs. It does delete a good chunk of the API folders but not all of it. With the script the Azure directory is now just under 300MB instead of over 700MB. It is compatible with most, but not all, third party packages, as long as they do not point to a version that is trimmed.

https://github.com/clumio-code/azure-sdk-trim

skuethe · 2021-09-27T08:01:43Z

Any update on this? Azure CLI size is getting out of hand :(

xiangyan99 · 2021-11-30T00:48:02Z

Do we have any decisions?

We have similar problem when we ship conda packages.

ghost · 2021-12-15T08:29:34Z

Hi @yoava333. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.

yoav-orca · 2021-12-15T08:43:11Z

/unresolve
@RAY-316 the issue not addressed/resolved the issue is only getting bigger, the package size of the network along (azure-mgmt-network) is now 286MB, which is almost 3 times larger than the original issue.

ghost · 2021-12-15T08:43:14Z

Hi yoav-orca, only the original author of the issue can ask that it be unresolved. Please open a new issue with your scenario and details if you would like to discuss this topic with the team.

yoava333 · 2021-12-15T08:44:15Z

/unresolve

travcunn · 2022-03-11T00:10:01Z

Any updates? I'd like to use the SDK in AWS Lambda but the package is too large

sodul · 2022-03-11T00:38:38Z

@travcunn you can try my tool, which will bring the azure packages down to something less massive (300-400MB), but that might still be too large for Lambda.
https://github.com/clumio-code/azure-sdk-trim

As far as I can tell, this is something that was reported years ago and each release is still growing bigger, so it does not seem to be a priority at Microsoft.

@jsntcy you did communicate almost a year ago, can you share if anything came out of your meetings? You can share with your product teams that some decision makers are using this bug as an example as to why Azure should be avoided in favor of AWS.

jsntcy · 2022-03-11T07:19:06Z

@sodul, I'm not in SDK team now. @msyyc, please share the update for this issue.

msyyc · 2022-03-11T07:31:02Z

HI, we have gradually reduced the size for several release: #21301.
What size do you expect?

jiasli · 2022-03-11T07:56:56Z

According to #21301 (comment), 21.1MB is still too large. Azure CLI 2.34.1 (https://azcliprod.blob.core.windows.net/msi/azure-cli-2.34.1.msi) is 52.3MB, which means azure-mgmt-network is almost half the size of whole Azure CLI.

msyyc · 2022-03-14T03:07:04Z

Let me paste the data in e-mail to raise more attention for architect

prawalgangwar · 2022-05-11T09:09:48Z

Facing issues with pylance in VSCode. The python language server crashes as soon as I installed azure-mgmt-network via pip3 install azure-mgmt-network

[Info  - 12:24:51 PM] (4436) [BG(1)] Long operation: parsing: XXX/dev_env/lib/python3.10/site-packages/azure/mgmt/network/v2019_12_01/operations/_operations.py (2541ms)
[Info  - 12:24:56 PM] (4436) [BG(1)] Long operation: parsing: XXX/dev_env/lib/python3.10/site-packages/azure/mgmt/network/v2020_04_01/operations/_operations.py (4044ms)
[Info  - 12:25:04 PM] (4436) [BG(1)] Long operation: parsing: XXX/dev_env/lib/python3.10/site-packages/azure/mgmt/network/v2020_05_01/operations/_operations.py (7680ms)
[Error - 12:25:21 PM] (4436) Error occurred on background thread: {"code":"ERR_WORKER_OUT_OF_MEMORY"}```

See Azure/azure-sdk-for-python#11149 and Azure/azure-sdk-for-python#17801 The size of the azure sdk is ridiculous because they keep old (and unused APIs) around. We run this azure_sdk_trim.py to prune the unused APIs while still maintaining a good compatibility level. This shaves off about 0.5GB from the image. The full images is now around ~1GB.

* Stop installing pip from pip The UBI9 image is recent enough, that we can just rely on the system-packaged pip. Should the need arise we can always backtrack on that, but for now let's stick to the pip rpm we're being provided. * Remove some unneeded collections Namely we remove "fortinet cisco dellemc f5networks junipernetworks mellanox netapp" This saves us ~400MB Before: localhost/utility-container latest 4cd658d42edf 28 seconds ago 1.97 GB After: localhost/utility-container latest 685c976bab41 22 seconds ago 1.54 GB * Run pip install in a single command No need now to have multiple invocations. * Reduce the Azure sdk size impact See Azure/azure-sdk-for-python#11149 and Azure/azure-sdk-for-python#17801 The size of the azure sdk is ridiculous because they keep old (and unused APIs) around. We run this azure_sdk_trim.py to prune the unused APIs while still maintaining a good compatibility level. This shaves off about 0.5GB from the image. The full images is now around ~1GB. * Install git-core only This avoids us installing perl-libs. The fundamental git commands (clone) are in git-core and those are the only ones we are likely going to need for the foreseeable future. * Make kubectl a symlink to oc This won't have an impact on the final size of the image or on du calculations. But it makes the intent clearer as the two commands have the exact same content. * Add ARG variabls to remove some rpms We have some rpms we remove via microdnf (fonts) and some that we forcefully remove as we'll likely never ever need those via rpm -e --nodepts (cracklib-dict) * Move the openshift_client version to a variable So it will be easier to change and there is less clustter in the Containerfile * Fix some small hadolint warnings Quote any >= sign and move the -y as a first argument * Fix package names indentation in README and add vi

lmazuel · 2023-01-09T17:36:56Z

Duplicate of #17801, where we're posting most update on the topic

lilyjma added the Network label Apr 30, 2020

ghost removed the needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. label Apr 30, 2020

lilyjma added the Mgmt This issue is related to a management-plane library. label Apr 30, 2020

allenjzhang added the feature-request This issue requires a new behavior in the product in order be resolved. label May 1, 2020

lmazuel added this to the Backlog milestone May 4, 2020

lmazuel assigned zikalino May 4, 2020

lmazuel removed the feature-request This issue requires a new behavior in the product in order be resolved. label May 5, 2020

RodgeFu assigned chenjianfei2017 and unassigned zikalino May 6, 2020

lmazuel removed this from the Backlog milestone May 11, 2020

RodgeFu added the Service Attention Workflow: This issue is responsible by Azure service team. label May 15, 2020

KarenHammons added Network - Virtual Network and removed Network labels May 19, 2020

timMSFT added Network - Virtual Network and removed Network - Virtual Network labels Aug 12, 2020

timMSFT removed the Network - Virtual Network label Oct 5, 2020

timMSFT assigned allegradomel Oct 5, 2020

timMSFT added the Network - Virtual Network label Oct 5, 2020

jiasli mentioned this issue Jul 14, 2021

azure-cli package could lose some weight Azure/azure-cli#7387

Open

jiasli mentioned this issue Oct 13, 2021

[Network] bump network to 2021-03-01 Azure/azure-cli#19590

Merged

3 tasks

RAY-316 added the issue-addressed Workflow: The Azure SDK team believes it to be addressed and ready to close. label Dec 15, 2021

ghost added needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team and removed issue-addressed Workflow: The Azure SDK team believes it to be addressed and ready to close. labels Dec 15, 2021

jsntcy assigned msyyc and unassigned jsntcy Mar 11, 2022

jiasli mentioned this issue Mar 11, 2022

Install time must be radically improved Azure/azure-cli#13028

Open

jiasli mentioned this issue May 9, 2022

Install Azure CLI on Alpine Linux Azure/azure-cli#19591

Open

lmazuel mentioned this issue Jan 9, 2023

[Python] reduce size for azure-mgmt-network #21301

Closed

lmazuel closed this as completed Jan 9, 2023

jiasli mentioned this issue Feb 16, 2023

{Packaging} Trim mgmt SDKs Azure/azure-cli#23946

Merged

github-actions bot locked and limited conversation to collaborators Apr 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

azure-mgmt-network is huge #11149

azure-mgmt-network is huge #11149

yoava333 commented Apr 30, 2020

chenjianfei2017 commented May 1, 2020

RodgeFu commented May 6, 2020

ghost commented May 15, 2020

RodgeFu commented May 15, 2020

ghost commented May 19, 2020

ghost commented Aug 12, 2020

talmachani commented Oct 4, 2020

ghost commented Oct 5, 2020

yoav-orca commented Nov 12, 2020

sodul commented Aug 16, 2021

skuethe commented Sep 27, 2021

xiangyan99 commented Nov 30, 2021

ghost commented Dec 15, 2021

yoav-orca commented Dec 15, 2021

ghost commented Dec 15, 2021

yoava333 commented Dec 15, 2021

travcunn commented Mar 11, 2022 •

edited

Loading

sodul commented Mar 11, 2022

jsntcy commented Mar 11, 2022

msyyc commented Mar 11, 2022

jiasli commented Mar 11, 2022

msyyc commented Mar 14, 2022

prawalgangwar commented May 11, 2022 •

edited

Loading

lmazuel commented Jan 9, 2023

azure-mgmt-network is huge #11149

azure-mgmt-network is huge #11149

Comments

yoava333 commented Apr 30, 2020

chenjianfei2017 commented May 1, 2020

RodgeFu commented May 6, 2020

ghost commented May 15, 2020

RodgeFu commented May 15, 2020

ghost commented May 19, 2020

ghost commented Aug 12, 2020

talmachani commented Oct 4, 2020

ghost commented Oct 5, 2020

yoav-orca commented Nov 12, 2020

sodul commented Aug 16, 2021

skuethe commented Sep 27, 2021

xiangyan99 commented Nov 30, 2021

ghost commented Dec 15, 2021

yoav-orca commented Dec 15, 2021

ghost commented Dec 15, 2021

yoava333 commented Dec 15, 2021

travcunn commented Mar 11, 2022 • edited Loading

sodul commented Mar 11, 2022

jsntcy commented Mar 11, 2022

msyyc commented Mar 11, 2022

jiasli commented Mar 11, 2022

msyyc commented Mar 14, 2022

prawalgangwar commented May 11, 2022 • edited Loading

lmazuel commented Jan 9, 2023

travcunn commented Mar 11, 2022 •

edited

Loading

prawalgangwar commented May 11, 2022 •

edited

Loading