Allow generic S3 endpoints for alternative services

[ChimneySwift]

Overview

This is a possible implementation for a previously mentioned feature (#3691, #3973), which is custom S3-compatible endpoints in order to allow the use of non-AWS block storage.

Many of these services offer significantly cheaper alternatives to AWS (with backblaze being almost 4x cheaper).

I understand there was some work on this in the s3rework branch, but I was more interested in having something soon to see if my plan to use block storage is feasible, and this is potentially a good stop-gap until someone has the time to fully rework the s3 implementation.

I haven't performed much testing on this release, but it appears to work with Backblaze B2. I had some issues trying to get it to work with Cloudflare R2, but I'm not sure if this is more of a Cloudflare-related issue.

Changes

The main blocker for implementing this is the lack of custom endpoint support within the s3-lite version dynmap uses. There is a new version, 0.0.2-SNAPSHOT, which does support custom endpoint URLs and is a mostly drop-in replacement. Though it's not an official release, there has been no movement on that repo for about a year, so that's as good as it gets without a rewrite.

Anyway, in order to implement this I updated the s3-lite package to 0.0.2-SNAPSHOT and I have added an override_endpoint option to manually set the S3 endpoint. If override_endpoint is blank (the default), it is assumed AWS S3 is being used, and the endpoint is derived from the region.

I have released jars for this version on my fork for anyone interested in testing. I have only tested spigot on 1.20.2 thus far.

I have also cloned the wiki onto my fork, and have updated the AWS S3 page to include documentation for third-party providers.

[mikeprimm]

This looks promising - I'll give it a proper review this weekend, and merge if it looks good. I am not set up to confirm this in any meaningful way, and am not planning on spending money to do so, so other than making sure this doesn't break the AWS S3 (which I use routinely for work), this will be otherwise untested.

[ChimneySwift]

Sounds good, this was mainly so I could test some options before pursuing further, but it does seem to work. As long as it doesn't break AWS S3, I don't see a reason not to merge.

I'm not sure I will personally end up using it, not every S3-like service is really geared toward hosting map tiles with frequent gets. I've just got Bunny CDN set up which is working really well (fast and you can replicate in multiple regions for low latency around the world) but unfortunately it's not S3 compatible. At the moment I have a custom python script syncing the tiles up to bunny, but if I want to use this long term I might consider integrating directly into dynmap, even if only for a personal fork.

An interesting note is that the current S3 implementation is more trying to solve storage constraints, however it could also help bandwidth constrained users, and maybe even help improve performance depending on the service. This can be done by pushing tiles to a CDN and then setting the url.tiles setting to point to the CDN instead of the server. The website (and dynmap_world.json) is then served from the server, while the tiles are served from the CDN.

[obsd-guru]

thanks for adding this feature. was looking for it already. if you need some beta testing pls let me know. i am planning to use it with a minio s3 backend.

[mikeprimm]

Merging this - will see how it goes!