With the No Code metadata initiative, we've introduced various major changes:
- New Ebean Aspect table (metadata_aspect_v2)
- New Elastic Indices (entityNameindex_v2)
- New edge triples. (Remove fully qualified classpaths from nodes & edges)
- Dynamic DataPlatform entities (no more hardcoded DataPlatformInfo.json)
- Dynamic Browse Paths (no more hardcoded browse path creation logic)
- Addition of Entity Key aspects, dropped requirement for strongly-typed Urns.
- Addition of @Entity, @Aspect, @Searchable, @Relationship annotations to existing models.
Because of these changes, it is required that your persistence layer be migrated after the NoCode containers have been deployed.
For more information about the No Code Update, please see no-code-modeling.
We are merging these breaking changes into the main branch upfront because we feel they are fundamental to subsequent changes, providing a more solid foundation upon which exciting new features will be built upon. We will continue to offer limited support for previous verions of DataHub.
This approach means that companies who actively deploy the latest version of DataHub will need to perform an upgrade to continue operating DataHub smoothly.
It is important that the following containers are pulled and deployed simultaneously:
- datahub-frontend-react
- datahub-gms
- datahub-mae-consumer
- datahub-mce-consumer
From the docker directory:
docker-compose down && docker-compose pull && docker-compose -p datahub up --force-recreateDeploying latest helm charts will upgrade all components to version 0.8.0. Once all the pods are up and running, it will run the datahub-upgrade job, which will run the above docker container to migrate to the new sources.
The easiest option is to execute the run_upgrade.sh script located under docker/datahub-upgrade/nocode.
cd docker/datahub-upgrade/nocode
./run_upgrade.sh
In both cases, the default environment variables will be used (docker/datahub-upgrade/env/docker.env). These assume
that your deployment is local. If this is not the case, you'll need to define your own environment variables to tell the
upgrade system where your DataHub containers reside.
You can either
- Change
docker/datahub-upgrade/env/docker.envin place and then run one of the above commands OR - Define a new ".env" file containing your variables and
execute
docker pull acryldata/datahub-upgrade && docker run acryldata/datahub-upgrade:latest -u NoCodeDataMigration
To see the required environment variables, see the (datahub-upgrade)[../../docker/datahub-upgrade/README.md] documentation
Upgrade to latest helm charts by running the following after pulling latest master.
helm upgrade datahub datahub/ --values datahub/quickstart-values.yaml
This will upgrade all pods to version 0.8.0, and once all pods are up and ready, datahub-upgrade job will start, running the above docker image to migrate to the new stores.
This step involves removing data from previous versions of DataHub. This step should only be performed once you've validated that your DataHub deployment is healthy after performing the upgrade. If you're able to search, browse, and view your Metadata after the upgrade steps have been completed, you should be in good shape.
In advanced DataHub deployments, or cases in which you cannot easily rebuild the state stored in DataHub, it is strongly advised that you do due diligence prior to running cleanup. This may involve manually inspecting the relational tables ( metadata_aspect_v2), search indices, and graph topology.
The easiest option is to execute the run_clean.sh script located under docker/datahub-upgrade/nocode.
cd docker/datahub-upgrade/nocode
./run_clean.sh
In both cases, the default environment variables will be used (docker/datahub-upgrade/env/docker.env). These assume
that your deployment is local. If this is not the case, you'll need to define your own environment variables to tell the
upgrade system where your DataHub containers reside.
You can either
- Change
docker/datahub-upgrade/env/docker.envin place and then run one of the above commands OR - Define a new ".env" file containing your variables and execute
docker pull acryldata/datahub-upgrade && docker run acryldata/datahub-upgrade:latest -u NoCodeDataMigrationCleanup
To see the required environment variables, see the (datahub-upgrade)[../../docker/datahub-upgrade/README.md] documentation
TODO
The easiest option is to execute the run_upgrade.sh script located under docker/datahub-upgrade/nocode.
cd docker/datahub-upgrade/nocode
./upgrade.sh
In both cases, the default environment variables will be used (docker/datahub-upgrade/env/docker.env). These assume
that your deployment is local. If this is not the case, you'll need to define your own environment variables to tell the
upgrade system where your DataHub containers reside.
You can either
- Change
docker/datahub-upgrade/env/docker.envin place and then run one of the above commands OR - Define a new ".env" file containing your variables and
execute
docker pull acryldata/datahub-upgrade && docker run acryldata/datahub-upgrade:latest -u NoCodeDataMigration
To see the required environment variables, see the (datahub-upgrade)[../../docker/datahub-upgrade/README.md] documentation
TODO
The Acryl team will be on standby to assist you in your migration. Please do not hesitate to reach out to the following folks if you find trouble with the upgrade or have feedback on the process. We will work closely to make sure you can continue to operate DataHub smoothly.
- John Joyce
- Gabe Lyons
- Dexter Lee
If there is feedback on the process, please