Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1.1.0 features #38

Merged
merged 15 commits into from
Mar 3, 2024
Merged

1.1.0 features #38

Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
0fac1bd
Inline escape() method to improve performance
staabm Feb 20, 2024
7563fab
Remove unnecessary call_user_func() to be faster
staabm Feb 20, 2024
c003fb9
Faster dump by less tiny writes()
staabm Feb 20, 2024
c15b5b8
move isEnabled() call out of hot loop
staabm Feb 20, 2024
2fe1492
simplify
staabm Feb 21, 2024
178d8c6
Merge branch 'fast-invoke' of github.com:staabm/mysqldump-php into 1.…
back-2-95 Feb 25, 2024
9165606
Merge branch 'less-writes' of github.com:staabm/mysqldump-php into 1.…
back-2-95 Feb 25, 2024
847dec0
Trigger the dump info hook when max buffer length is reached
guvra Feb 28, 2024
00c193e
micro-optimize / improve readability
staabm Feb 29, 2024
9795f14
Fix typo
staabm Feb 29, 2024
2ececab
Merge branch 'inline3' of github.com:staabm/mysqldump-php into 1.1.0-…
back-2-95 Mar 1, 2024
6e7314b
Merge branch '1.1.0-features' of github.com:guvra/mysqldump-php into …
back-2-95 Mar 1, 2024
ca955c5
Merge branch 'patch-3' of github.com:staabm/mysqldump-php into 1.1.0-…
back-2-95 Mar 1, 2024
4ec89ac
Revert "Fix typo"
back-2-95 Mar 1, 2024
4725eaa
Merge branch 'main' of github.com:druidfi/mysqldump-php into 1.1.0-fe…
back-2-95 Mar 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 44 additions & 33 deletions src/Mysqldump.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -758,38 +758,40 @@ private function prepareColumnValues(string $tableName, array $row): array
$columnTypes = $this->tableColumnTypes[$tableName];

if ($this->transformTableRowCallable) {
$row = call_user_func($this->transformTableRowCallable, $tableName, $row);
$row = ($this->transformTableRowCallable)($tableName, $row);
}

$dbHandler = $this->conn;
$hexBlobEnabled = $this->settings->isEnabled('hex-blob');
foreach ($row as $colName => $colValue) {
if ($this->transformColumnValueCallable) {
$colValue = call_user_func($this->transformColumnValueCallable, $tableName, $colName, $colValue, $row);
$colValue = ($this->transformColumnValueCallable)($tableName, $colName, $colValue, $row);
}

$ret[] = $this->escape($colValue, $columnTypes[$colName]);
}
if ($colValue === null) {
$ret[] = "NULL";
continue;
}

return $ret;
}
$colType = $columnTypes[$colName];
if ($hexBlobEnabled && $colType['is_blob']) {
if ($colType['type'] == 'bit' || $colValue !== '') {
$ret[] = sprintf('0x%s', $colValue);
} else {
$ret[] = "''";
}
continue;
}

/**
* Escape values with quotes when needed.
*/
private function escape(?string $colValue, array $colType)
{
if (is_null($colValue)) {
return 'NULL';
} elseif ($this->settings->isEnabled('hex-blob') && $colType['is_blob']) {
if ($colType['type'] == 'bit' || !empty($colValue)) {
return sprintf('0x%s', $colValue);
} else {
return "''";
if ($colType['is_numeric']) {
$ret[] = $colValue;
continue;
}
} elseif ($colType['is_numeric']) {
return $colValue;

$ret[] = $dbHandler->quote($colValue);
}

return $this->conn->quote($colValue);
return $ret;
}

/**
Expand Down Expand Up @@ -837,47 +839,56 @@ private function listValues(string $tableName)
$ignore = $this->settings->isEnabled('insert-ignore') ? ' IGNORE' : '';
$count = 0;

$isInfoCallable = $this->infoCallable && is_callable($this->infoCallable);
if ($isInfoCallable) {
($this->infoCallable)('table', ['name' => $tableName, 'completed' => false, 'rowCount' => $count]);
}

$line = '';
foreach ($resultSet as $row) {
$count++;
$values = $this->prepareColumnValues($tableName, $row);
$valueList = implode(',', $values);

if ($onlyOnce || !$this->settings->isEnabled('extended-insert')) {
if ($this->settings->isEnabled('complete-insert') && count($colNames)) {
$lineSize += $this->write(sprintf(
$line .= sprintf(
'INSERT%s INTO `%s` (%s) VALUES (%s)',
$ignore,
$tableName,
implode(', ', $colNames),
$valueList
));
} else {
$lineSize += $this->write(
sprintf('INSERT%s INTO `%s` VALUES (%s)', $ignore, $tableName, $valueList)
);
} else {
$line .= sprintf('INSERT%s INTO `%s` VALUES (%s)', $ignore, $tableName, $valueList);
}
$onlyOnce = false;
} else {
$lineSize += $this->write(sprintf(',(%s)', $valueList));
$line .= sprintf(',(%s)', $valueList);
}

if (($lineSize > $this->settings->getNetBufferLength())
if ((strlen($line) > $this->settings->getNetBufferLength())
|| !$this->settings->isEnabled('extended-insert')) {
$onlyOnce = true;
$lineSize = $this->write(';' . PHP_EOL);
$this->write($line . ';' . PHP_EOL);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we are buffering rows into batches with this PR, it might make sense to emit a event at this point in time, so a possible progress indicator could also update when a table dump is in-flight.

otherwise dumping huge tables will not see any progress updates for - in our case - 20-30 minutes.

see also #39

I would propose something like

Suggested change
$this->write($line . ';' . PHP_EOL);
$this->write($line . ';' . PHP_EOL);
($this->infoCallable)('table', ['name' => $tableName, 'rowProgress' => $count]);

Copy link

@guvra guvra Feb 28, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another call before the foreach loop is also needed, otherwise the dump info will display an incorrect table name until the 1st million characters were processed.

Also maybe adding a state instead of rowProgress, e.g.:

Before the start of the foreach:
['name' => $tableName, 'completed' => false, 'rowCount' => 0]

Within the foreach loop:
['name' => $tableName, 'completed' => false, 'rowCount' => $count]

After the foreach loop:
['name' => $tableName, 'completed' => true, 'rowCount' => $count]

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

valid points. I like your idea more than mine

$line = '';

if ($isInfoCallable) {
($this->infoCallable)('table', ['name' => $tableName, 'completed' => false, 'rowCount' => $count]);
}
}
}

$resultSet->closeCursor();

if (!$onlyOnce) {
$this->write(';' . PHP_EOL);
if ($line !== '') {
$this->write($line. ';' . PHP_EOL);
}

$this->endListValues($tableName, $count);

if ($this->infoCallable && is_callable($this->infoCallable)) {
call_user_func($this->infoCallable, 'table', ['name' => $tableName, 'rowCount' => $count]);
if ($isInfoCallable) {
($this->infoCallable)('table', ['name' => $tableName, 'completed' => true, 'rowCount' => $count]);
}

$this->settings->setCompleteInsert($completeInsertBackup);
Expand Down