Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate KiwixStorage #223

Open
rgaudin opened this issue Nov 26, 2024 · 0 comments
Open

Integrate KiwixStorage #223

rgaudin opened this issue Nov 26, 2024 · 0 comments
Labels
enhancement New feature or request question Further information is requested
Milestone

Comments

@rgaudin
Copy link
Member

rgaudin commented Nov 26, 2024

Ours scrapers all depend on S3 and use KiwixStorage for it. That wrapper and repo are mostly untouched (yet working) and would great benefit from being integrated into scraperlib: tests, usage, dependencies update, better API, etc.

Now that we had experience using it, it probably make sense to reduce its API to what's actually used.

It's true that some non-scraper projects are using it (cardshop and zimfarm to upload to S3), but those are also using scraperlib and we've discussed making scraperlib's dependencies more modular via extras anyway.

@rgaudin rgaudin added enhancement New feature or request question Further information is requested labels Nov 26, 2024
@benoit74 benoit74 added this to the backlog milestone Dec 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants