Streaming large file uploads #9493
Labels
Area: API
PR or issue related to the API
Project: C-BIG
Issue or PR related to the C-BIG project
Proposal
PR or issue suggesting an improvement that can be accepted, rejected or altered
Although I tested #9154 successfully with Raisinbread data, I did not manage to do so with some BIC pilot data that is too large (MRI sessions larger than 10GBs). This is not a blocker to merge the PR IMO, but it does make it impossible to use in its entirety for some projects with large MRI sessions.
I did my tests using the
requests
Python library. The reason why the upload failed is that the client loads the full tar file in memory, which is too large for that. The solution seems to be to use file streaming, which I tried to with therequests_toolbelt
library and the streamingMultipartEncoder
class. Unfortunately, I did not manage to successfully call the API while doing so.I think there should be a discussion on streaming large file uploads with the LORIS API. Is it currently supported by the framework ? Is it compatible with existing endpoints ? How can a client use this feature ? Documentation ?
The text was updated successfully, but these errors were encountered: