Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add download support for modelscople. #2032

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

pass-lin
Copy link

Due to relevant legal restrictions of the Chinese government, Chinese users find it quite inconvenient to use HF.
Therefore, Chinese users often use the Chinese version of HF, which is ModelScope.
Considering this aspect, I think we should add download support for ModelScope.
Generally, Chinese users use it as a cloud disk for storing hf data. Therefore, we only need to make slight changes in the download section, and the other parts are exactly the same as hf.
The usage is similar to hf.

modelscope://username/bert_base_en

Copy link

google-cla bot commented Dec 25, 2024

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@mattdangerw
Copy link
Member

Thanks for the PR! We are checking this out, and also considering whether we should just build a general method to allow downloads from different sources (e.g. register a new provider).

@pass-lin
Copy link
Author

pass-lin commented Jan 10, 2025

Thanks for the PR! We are checking this out, and also considering whether we should just build a general method to allow downloads from different sources (e.g. register a new provider).

Thank you. The main reason is that we are limited by the risk of the Chinese government's control over download sources from other regions. For example, hf was initially allowed to be used in China, but later we couldn't connect to it directly, and we could only use the unstable mirror sites.
There is also a situation where Kaggle seems to have no servers in China, so the download is unstable.
Therefore, we hope to have a Chinese version available for selection, to make it more convenient for Chinese Keras users to use. We hope you can understand this.
But if your general download source still needs to connect to servers outside of China, there are still relevant risks involved.

@mattdangerw
Copy link
Member

@pass-lin yes definitely! For sure we want there to be a way to use models with modelscope. And advantage of a "plugin" style api would be that people could also use this for third own private hosting (e.g. s3 bucket, or some custom remote file storage, etc).

Sorry the process on this one might be slightly longer since it's adding a new (soft) dependency, but we are checking this out!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants