Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement upgrade from 3.1 to 4.0 #4587

Closed
keith-turner opened this issue May 22, 2024 · 5 comments · Fixed by #4712, #4713 or #4717 · May be fixed by #4719 or #4718
Closed

Implement upgrade from 3.1 to 4.0 #4587

keith-turner opened this issue May 22, 2024 · 5 comments · Fixed by #4712, #4713 or #4717 · May be fixed by #4719 or #4718
Assignees
Milestone

Comments

@keith-turner
Copy link
Contributor

The elasticity branch does not currently have functioning upgrade code. Need to implement upgrade from 3.1 to 4.0.

@keith-turner keith-turner converted this from a draft issue May 22, 2024
@dlmarion
Copy link
Contributor

dlmarion commented May 28, 2024

This is already started, in class Upgrader12to13, it's just that the upgrade is disabled in AccumuloDataVersion.

@cshannon cshannon changed the title Implement updgrade from 3.1 to 4.0 Implement upgrade from 3.1 to 4.0 Jun 1, 2024
@keith-turner keith-turner self-assigned this Jun 26, 2024
@keith-turner
Copy link
Contributor Author

We have issues open for some upgrade task. Plan to do a survey of the code and see if any other issue need to be opened and find all of the issues that are currently open.

@keith-turner
Copy link
Contributor Author

Some of the issues currently open re upgrade are #4692, #4652, #4637, #4315. For the survey of what may need to be done for upgrade, plan to look over what is currently done for upgrade, metadata schema changes, zookeeper schema changes, and prop changes.

@keith-turner
Copy link
Contributor Author

keith-turner commented Jun 27, 2024

Compared the files Constants.java, MetadataSchema.java, and Property.java across elasticity and main branches. Analyzing Constants.java resulted in opening #4712 and #4713. Looking at MetadataSchema.java did not see anything to do, did notice #4637.

Looking at Property.java the only table property that was removed was table.compaction.selector. Wondering if we want to delete this if it exists on upgrade. There are also some properties that could have been set at the system level that were deleted. This is related to #4315, but more general.

@keith-turner
Copy link
Contributor Author

The upgrade code in elasticity currently has no tests. Will need to add test. Was pondering doing the following for testing upgrade.

  • Create an IT in main/3.1 branch that :
    • runs mini accumulo
    • creates data in zookeeper and metadata table that upgrade would need to delete
    • stops all processes
    • then creates a zip file of the mini accumulo directory
  • In the elasticity branch do the following :
    • copy the zip file created in the 3.1 branch to test resources
    • create an IT that initializes a mini accumulo instance using this zip file
    • verify upgrade happened
    • add a comment to the IT about running the test in the other branch to update the zip file

The IT in elasticity could use this MAC functionality

public MiniAccumuloConfigImpl useExistingInstance(File accumuloProps, File hadoopConfDir)

kevinrr888 added a commit to kevinrr888/accumulo that referenced this issue Jul 3, 2024
Deletes the bulk file columns from the metadata table on upgrade to 4.x, if they exist.
closes apache#4637, related to apache#4587
keith-turner pushed a commit that referenced this issue Jul 5, 2024
Deletes the bulk file columns from the metadata table on upgrade to 4.x, if they exist.
closes #4637, related to #4587
@ctubbsii ctubbsii added this to the 4.0.0 milestone Jul 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment