Move Acquia DB backups to cloud v2. #218

acidaniel · 2023-02-28T22:34:12Z

https://palantir.atlassian.net/browse/DEV-32

Test Intructions:

Add this to an existing project and run composer install
Ensure in the build.yml from the existing project you have the following variables:
```
# Configuration for targets/acquia.xml
acquia:
 backups: 'artifacts/backups'
 site: 'palantirnet'
 database: 'palantirnet'
 env: prod (default one if not defined)
```
- Ensure there is a .env with acquia clould credentials specified in the 1Password
- Run ddev restart
- Run ddev phing acquia-get-backup
- Ensure backup was downloaded inside the backups directory specified in backups
- Run ddev phing drupal-load-db and ensure database dump downloaded is imported correctly.

composer.json

jesconstantine · 2023-03-02T14:37:40Z

src/TheBuild/Acquia/GetLatestBackupTask.php

+  protected function getLatestBackup($client, $environment_uuid) {
+    $backup = new DatabaseBackups($client);
+    $backups = $backup->getAll($environment_uuid, $this->database);
+    $filepath = $this->dir . '/' . $this->env . '_' . $this->database . '_' . 'sql.gz';


@acidaniel would it make sense to also set the file to load from backups based on this value?

https://github.com/palantirnet/the-build/pull/218/files#diff-339072f52845d316656969cbac2815305c0ccea0ce9de789ba73d78849336067L141-L145

@jesconstantine The load from backups task will load any *sql.gz file that exists in the artifact/backups directory regardless the name.

@acidaniel I think that's my concern. I think manual backups have a different file patterns but could get loaded instead of the one that we download from prod. To me this is the same issue as what we were discussing in slack here - but am I think about it correctly?

I added this to our build.yml to ensure the correct backup was loaded: https://github.com/palantirnet/wisdhs-public/pull/566/files#diff-0fad5e53ffd9ff1b0af504ff6c4d0961af03f34af69d9e00de527f2f29c9cb2bR50-R56

@jesconstantine I would love to see your commit, but I got 404 message :'( (you know why. Rub salt in the wound).

@jesconstantine I just changed here the extension, I had a typo, please let me know if there something else to do here.

@jesconstantine I think I found a way to improve performance by 96% of loading database with only using Linux commands. Here is a bit of background:

Using drupal-load-db was using a gunzip -c command that when I run it it took around 10 minutes to finish the import

Using <exec executable="ddev"... took around 56 seconds to load, but this is not an option since we need to ensure we are outside the container. So this option is discarded.

Figured out and reading that by using native Linux commands like gzip -dc in combination with a direct mysql command takes only 46 seconds

You can see how I am implementing it really straightforward. So following the general formula about performance improvement in percentage which is:

( new - old ) / old * 100%

We have:

(42 - 600)/600*100= -93

What do you think about this?

I'm so sorry about posting that link, I wasn't thinking. 😦

Here's the snippet:

build: # Configuration for the database loading utility. load_db: # Pattern to match gzipped dump files export_pattern: "./artifacts/*.sql.gz" file: "./artifacts/dhsinternet_d9_db_backup.sql.gz"

src/TheBuild/Acquia/GetLatestBackupTask.php

…ed if acquia is selected as hosting during the project setup.

acidaniel · 2023-03-07T05:28:09Z

@jesconstantine This is ready to be reviewed now. Since the package was removed now I'm getting issues in CircleCI, do you think we need to skip this file to be linted?

targets/install.xml

jesconstantine · 2023-03-07T16:45:38Z

targets/install.xml

+                <!-- Copy .env.example file to store Acquia Cloud Credentials -->
+                <copy file="${build.thebuild.dir}/defaults/install/.env.example" todir="${application.startdir}" overwrite="true"/>


I know the .env.example currently only has Acquia data in it, but I could see it being useful for non-acquia hosted projects too, what do you think?

Should we copy the file somewhere more globally and add the platform specific data to it here instead?

jesconstantine · 2023-03-07T17:06:48Z

@acidaniel you asked:

Since the package was removed now I'm getting issues in CircleCI, do you think we need to skip this file to be linted?

Can we either:

ensure the task that installs the acquia cloud API SDK package is run before running the tests?
exclude src/Acquia here, then run the new phing task to install the acquia cloud API SDK package, and then run through the code review commands specifically on src/Acquia

What do you think?

…lected as hosting provider and remove copy of the .env.example to move it some more globally.

…to improve performance by 93%

…st time.

…, we don't need this.

byrond · 2023-03-30T15:27:16Z

defaults.yml

@@ -110,7 +113,7 @@ drupal:
      # Database connection for Drupal.
      database:
        # This value is required, but a default is set in the-build.xml.
-        # database: drupal
+        database: db


Do we need to change the default in the-build.xml instead of overriding it by default here?

Instead of setting a default value here for the database name (which could break multisite behavior), I updated the default for the first database that gets copied with defaults/install/the-build/build.yml. See 90b7d7d

Now, when a multisite is added, and that default is not present, it defaults to the site directory name (as it did before this PR).

defaults.yml

byrond · 2023-04-12T13:15:12Z

I confirmed the testing steps work on an existing project using this branch. I think we just need to address the issues mentioned above, and this will be good to go.

defaults.yml

byrond · 2024-03-13T21:10:42Z

src/TheBuild/Acquia/GetLatestBackupTask.php

@@ -361,10 +196,15 @@ protected function validate() {
    if (empty($this->database)) {
      $this->database = $this->site;
    }
+    // Check .env file exists.
+    if (!file_exists(".env")) {
+      throw new \BuildException(".env file is needed in the root of project with credentials see .env.example.");


This is checking for the .env file in the root, but it actually needs to exist in the .ddev directory for the environment variables to be created. I'm going to remove this, because the API already checks for the presence of these variables and fails accordingly:

example > acquia-get-backup: [getacquiabackup] Couldn't find ACQUIA_CLOUD_API_KEY env variable. [getacquiabackup] Couldn't find ACQUIA_CLOUD_API_SECRET env variable. BUILD FAILED

…2-backup

byrond · 2024-10-14T17:05:56Z

This was successfully tested on another client project and ready to merge.

WIP, Move Acquia DB backups to cloud v2.

43bad59

acidaniel added the WIP - do not merge label Feb 28, 2023

acidaniel added 3 commits March 1, 2023 10:18

Adding Download Methods and removing dump messages.

395c45a

Removing wrong return FALSE.

6fc0dd1

Adding Exeptions and cleaning code.

fa54ae9

acidaniel changed the title ~~WIP, Move Acquia DB backups to cloud v2.~~ Move Acquia DB backups to cloud v2. Mar 1, 2023

acidaniel requested review from agentrickard, jesconstantine and becw March 1, 2023 18:29

jesconstantine reviewed Mar 1, 2023

View reviewed changes

composer.json Outdated Show resolved Hide resolved

jesconstantine reviewed Mar 2, 2023

View reviewed changes

jesconstantine reviewed Mar 6, 2023

View reviewed changes

src/TheBuild/Acquia/GetLatestBackupTask.php Outdated Show resolved Hide resolved

acidaniel added 2 commits March 6, 2023 23:06

Moving acquia package to suggested and ensure this is going to be add…

5ae6bc5

…ed if acquia is selected as hosting during the project setup.

Adding task to copy .env.example file if acquia is selected.

5f87378

acidaniel added Ready for Review and removed WIP - do not merge labels Mar 7, 2023

acidaniel requested a review from jesconstantine March 7, 2023 05:26

jesconstantine reviewed Mar 7, 2023

View reviewed changes

targets/install.xml Outdated Show resolved Hide resolved

jesconstantine reviewed Mar 7, 2023

View reviewed changes

acidaniel added 8 commits March 7, 2023 17:38

Moving suggest package and add it as a normal require if acquia is se…

3ec3f03

…lected as hosting provider and remove copy of the .env.example to move it some more globally.

Fixing the extension of the backups.

768f54a

Fixing the command to decompress file and importing it without drush …

47f9626

…to improve performance by 93%

Adding hosting packages as new phing task.

4466d7f

Adding PHPStan.neo file to exclude Acquia refeences to be scanned fir…

4e1beeb

…st time.

Removing the check if we are in CIrcleCI to add composer dependencies…

8472626

…, we don't need this.

Fixing foreach typo in clossing tag.

1364c34

Installing specific packages per hosting default task tag.

4fc7395

acidaniel requested a review from byrond March 24, 2023 18:29

byrond reviewed Mar 30, 2023

View reviewed changes

byrond reviewed Apr 12, 2023

View reviewed changes

defaults.yml Show resolved Hide resolved

byrond reviewed Apr 12, 2023

View reviewed changes

defaults.yml Show resolved Hide resolved

byrond and others added 5 commits November 17, 2023 13:16

Merge branch 'develop' into cloud-api-version-2-backup

c7bfa88

allow newer versions of phpmd

3e11faf

revert change to version constraint so we can fix it elsewhere

564b04c

Merge branch 'develop' into cloud-api-version-2-backup

a696c86

sort use statements in GetLatestBackupTask.php

90cee16

byrond reviewed Jan 15, 2024

View reviewed changes

defaults.yml Show resolved Hide resolved

byrond added 2 commits March 13, 2024 14:37

fix the-build installer for host packages

eb12d5c

remove inheritall from foreach in install.xml

d9d5cea

byrond reviewed Mar 13, 2024

View reviewed changes

byrond added 2 commits March 13, 2024 17:12

remove check for .env since there is already a check for the env vars

4a4f630

Merge remote-tracking branch 'origin/develop' into cloud-api-version-…

f77c56e

…2-backup

byrond self-assigned this Mar 13, 2024

byrond added 3 commits October 11, 2024 15:08

Merge branch 'develop' into cloud-api-version-2-backup

dc83879

revert change to default database name

be6fe11

default first database name in ddev is "db"

90b7d7d

byrond approved these changes Oct 14, 2024

View reviewed changes

patrickfweston approved these changes Oct 14, 2024

View reviewed changes

byrond merged commit 7f7be9e into develop Oct 14, 2024
1 check passed

byrond deleted the cloud-api-version-2-backup branch October 14, 2024 17:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move Acquia DB backups to cloud v2. #218

Move Acquia DB backups to cloud v2. #218

acidaniel commented Feb 28, 2023 •

edited

Loading

jesconstantine Mar 2, 2023

acidaniel Mar 7, 2023

jesconstantine Mar 7, 2023 •

edited

Loading

acidaniel Mar 8, 2023

acidaniel Mar 9, 2023

acidaniel Mar 9, 2023

jesconstantine Mar 9, 2023 •

edited

Loading

acidaniel commented Mar 7, 2023

jesconstantine Mar 7, 2023

jesconstantine commented Mar 7, 2023 •

edited

Loading

byrond Mar 30, 2023

byrond Oct 14, 2024

byrond commented Apr 12, 2023

byrond Mar 13, 2024

byrond commented Oct 14, 2024

		<!-- Copy .env.example file to store Acquia Cloud Credentials -->
		<copy file="${build.thebuild.dir}/defaults/install/.env.example" todir="${application.startdir}" overwrite="true"/>

Move Acquia DB backups to cloud v2. #218

Move Acquia DB backups to cloud v2. #218

Conversation

acidaniel commented Feb 28, 2023 • edited Loading

Test Intructions:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jesconstantine Mar 7, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jesconstantine Mar 9, 2023 • edited Loading

Choose a reason for hiding this comment

acidaniel commented Mar 7, 2023

Choose a reason for hiding this comment

jesconstantine commented Mar 7, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

byrond commented Apr 12, 2023

Choose a reason for hiding this comment

byrond commented Oct 14, 2024

acidaniel commented Feb 28, 2023 •

edited

Loading

jesconstantine Mar 7, 2023 •

edited

Loading

jesconstantine Mar 9, 2023 •

edited

Loading

jesconstantine commented Mar 7, 2023 •

edited

Loading