Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streaming large file uploads #9493

Open
maximemulder opened this issue Nov 29, 2024 · 0 comments
Open

Streaming large file uploads #9493

maximemulder opened this issue Nov 29, 2024 · 0 comments
Labels
Area: API PR or issue related to the API Project: C-BIG Issue or PR related to the C-BIG project Proposal PR or issue suggesting an improvement that can be accepted, rejected or altered

Comments

@maximemulder
Copy link
Contributor

Although I tested #9154 successfully with Raisinbread data, I did not manage to do so with some BIC pilot data that is too large (MRI sessions larger than 10GBs). This is not a blocker to merge the PR IMO, but it does make it impossible to use in its entirety for some projects with large MRI sessions.

I did my tests using the requests Python library. The reason why the upload failed is that the client loads the full tar file in memory, which is too large for that. The solution seems to be to use file streaming, which I tried to with the requests_toolbelt library and the streaming MultipartEncoder class. Unfortunately, I did not manage to successfully call the API while doing so.

I think there should be a discussion on streaming large file uploads with the LORIS API. Is it currently supported by the framework ? Is it compatible with existing endpoints ? How can a client use this feature ? Documentation ?

@maximemulder maximemulder added Area: API PR or issue related to the API Project: C-BIG Issue or PR related to the C-BIG project Proposal PR or issue suggesting an improvement that can be accepted, rejected or altered labels Nov 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: API PR or issue related to the API Project: C-BIG Issue or PR related to the C-BIG project Proposal PR or issue suggesting an improvement that can be accepted, rejected or altered
Projects
None yet
Development

No branches or pull requests

1 participant