gdpr-dump generates inconsistent data/snapshot across tables

[sherifabdlnaby]

For Big Databases ( > 600GB ) we noticed that the resulting dumps are inconsistent across tables. Some tables have rows that were inserted after the dump was initiated, which result in inconsistent data between tables.

We traced this to how gdpr-dump is running the dumping queries, it starts a transaction per table instead of a single transaction for the entire dump. Which result in an inconsistent snapshot.

After tracing the code, it seems to be an issue with MySQLDump-PHP and not gdpr-dump itself. But I am opening the issue here to make sure if this is a known issue/limitation.

We'll be happy to contribute to this project to get this fixed 🙏🏻

Preconditions

GdprDump Version: 3.0.0

PHP Version: 7.4

Database Version: Mysql 5.7 (AWS RDS)

Steps to reproduce

1. Choose a Database with enough data so that the dumping command would take at minimum 2~3 minutes.
2. Create two empty tables a_test and z_test.
3. Run gdpr-dump with configuration with table_whitelist so that

tables_whitelist:
 - a_test
 - <a bigger table that takes 2~3 minutes to process>
 - z_test

Note that Tables are dumped in alphabetical order.
6. While gdpr-dump is running, insert some data into a_test and z_test.
7. Restore the generated dump.

Expected result.

1. a_test and z_test should be empty.

Actual result

1. a_test is empty, z_test is not.

[guvra]

Hi,

Noted, I'll wait for the fix from upstream.

The switch to druidfi was because the main repo had been inactive for a few years, but the maintainer went active again recently 😅

[guvra]

It was finally released in druidfi package (v1.0.3), I updated the phar file of the 4.0.2 release.