feat(upgrades) Create new DataHubUpgrade + Restore Glossary Entities Bootstrap step #5099
Conversation
|
|
||
| private int getAndRestoreTermAspectIndices(int start, AuditStamp auditStamp, AspectSpec termAspectSpec) throws Exception { | ||
| SearchResult termsResult = _entitySearchService.search(Constants.GLOSSARY_TERM_ENTITY_NAME, "", null, null, start, BATCH_SIZE); | ||
| List<Urn> termUrns = termsResult.getEntities().stream().map(SearchEntity::getEntity).collect(Collectors.toList()); |
There was a problem hiding this comment.
Let's end early if termUrns is empty. Same with function below
| log.info("Successfully restored glossary index"); | ||
| } catch (Exception e) { | ||
| log.error("Error when running the RestoreGlossaryIndices Bootstrap Step", e); | ||
| _entityService.deleteUrn(GLOSSARY_UPGRADE_URN); |
There was a problem hiding this comment.
@dexter-mh-lee @chriscollins3456 This is probably where a retry-policy for upgrades should come into the framework! :p
For later!
...service/factories/src/main/java/com/linkedin/metadata/boot/steps/RestoreGlossaryIndices.java
Show resolved
Hide resolved
| for (Urn nodeUrn: nodeUrns) { | ||
| EntityResponse nodeEntityResponse = nodeInfoResponses.get(nodeUrn); | ||
| if (nodeEntityResponse == null) { | ||
| log.info("Node not in set of entity responses {}", nodeUrn); |
There was a problem hiding this comment.
minor: consider logging a WARN log here
| } | ||
| GlossaryNodeInfo nodeInfo = mapNodeInfo(nodeEntityResponse); | ||
| if (nodeInfo == null) { | ||
| log.info("Received null nodeInfo for urn {}", nodeUrn); |
There was a problem hiding this comment.
same consider a warn -- higher level of log
...ice/factories/src/test/java/com/linkedin/metadata/boot/steps/RestoreGlossaryIndicesTest.java
Show resolved
Hide resolved
Totally agreed - I think especially once we have another one of these changes, it'll be much easier to create shared logic. I tried thinking that through this time but with a lot of this stuff being newer I was really just focusing on getting things to work well for this step. |
|
Once we pass CI we are good to merge! Thanks Chris |
Creates a new entity called DataHubUpgrade which will be used in order to keep track of who has run what upgrade steps.
I'm creating a specific instance for
glossary-upgradewhere we will restore indices of Glossary Nodes Info aspects and Glossary Term Info aspects. Once a user boots up GMS, if they haven't run this bootstrap step they will get all of their Terms and nodes and restore their indices. All following times on bootup they will not run this step if there is already aglossary-upgradeDataHubUpgrade entity in mysql.Checklist