Alternative way of serving region files

[t83714]

Alternative way of serving region files

Our indexer module and frontend map-based filter require region files to supply information on all supported regions.

We offer a list of default region files via config here:

magda/magda-scala-common/src/main/resources/common.conf

Line 92 in 9876e04

regionSources = {

Users can configure those default region files via indexer Helm chart config (via appConfig, any config field can reached).

Those files are served from our AWS storage bucket.

We have the following issues with the current setup:

• There is a cost involved when serving those files
• We need to turn off the storage bucket service due to internal account consolidation

Proposed New Solution

• Move default region files to a public repo and serve from the GitHub release
  • according to here and here, there is no limit on the total size of a release, nor bandwidth usage.
  • The only limit is "Each file included in a release must be under 2 GiB"
  • Free plan repo itself will have 10GB storage and 10GB transfer (repo transfer)
• internal premium projects use GCS

Tasks

• Create a public repo and manage region files using Git Large File Storage (Git LFS) due to size of each file
• Create CI workshop to auto-release default region files manually release for now
• Update Indexer Helm Chart document to make it clearer and encourage users to use their own region files that served from their own infrastructure
• Update default region files URL for indexer & release a new version

[t83714]

first release here: https://github.com/magda-io/magda-regions/releases/tag/v1.0.0

[staffordsmith83]

Working to create new repo for default region mapping files

[t83714]

closed via PR: #3542