Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement a max jobs per worker budget #4965

Merged
merged 9 commits into from
Sep 10, 2023
Merged

Implement a max jobs per worker budget #4965

Changes from 2 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
c6d7457
Implement a max jobs per worker budget
staabm Sep 10, 2023
500a9a4
Update ParallelFileProcessor.php
staabm Sep 10, 2023
e76edbc
fix
staabm Sep 10, 2023
377b254
docs
staabm Sep 10, 2023
8ad9388
reduce default job size
staabm Sep 10, 2023
be59729
better naming
staabm Sep 10, 2023
999bb9b
Update ParallelFileProcessor.php
staabm Sep 10, 2023
6ecaaab
remove defaults from rector config
staabm Sep 10, 2023
14724bf
Update ParallelFileProcessor.php
staabm Sep 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 43 additions & 7 deletions packages/Parallel/Application/ParallelFileProcessor.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,10 @@ final class ParallelFileProcessor
* @var int
*/
private const SYSTEM_ERROR_LIMIT = 50;
/**
* @var int
*/
private const MAX_JOBS_PER_WORKER = 10;
staabm marked this conversation as resolved.
Show resolved Hide resolved

private ProcessPool|null $processPool = null;

Expand Down Expand Up @@ -137,12 +141,24 @@ public function process(
};

$timeoutInSeconds = SimpleParameterProvider::provideIntParameter(Option::PARALLEL_JOB_TIMEOUT_IN_SECONDS);

for ($i = 0; $i < $numberOfProcesses; ++$i) {
// nothing else to process, stop now
if ($jobs === []) {
break;
}
$fileBudgetPerProcess = [];

$processSpawner = function() use (
&$systemErrors,
&$fileDiffs,
&$jobs,
$postFileCallback,
&$systemErrorsCount,
&$reachedInternalErrorsCountLimit,
$mainScript,
$input,
$serverPort,
$streamSelectLoop,
$timeoutInSeconds,
$handleErrorCallable,
&$fileBudgetPerProcess,
&$processSpawner
): void {

$processIdentifier = Random::generate();
$workerCommandLine = $this->workerCommandLineFactory->create(
Expand All @@ -153,6 +169,7 @@ public function process(
$processIdentifier,
$serverPort,
);
$fileBudgetPerProcess[$processIdentifier] = self::MAX_JOBS_PER_WORKER;

$parallelProcess = new ParallelProcess($workerCommandLine, $streamSelectLoop, $timeoutInSeconds);

Expand All @@ -167,7 +184,9 @@ function (array $json) use (
&$systemErrorsCount,
&$collectedDatas,
&$reachedInternalErrorsCountLimit,
$processIdentifier
$processIdentifier,
&$fileBudgetPerProcess,
&$processSpawner
): void {
// decode arrays to objects
foreach ($json[Bridge::SYSTEM_ERRORS] as $jsonError) {
Expand Down Expand Up @@ -195,12 +214,20 @@ function (array $json) use (
$this->processPool->quitAll();
}

if ($fileBudgetPerProcess[$processIdentifier] <= 0) {
// kill the current worker, and spawn a fresh one to free memory
$this->processPool->quitProcess($processIdentifier);

($processSpawner)();
return;
}
if ($jobs === []) {
$this->processPool->quitProcess($processIdentifier);
return;
}

$job = array_pop($jobs);
$fileBudgetPerProcess[$processIdentifier]--;
samsonasik marked this conversation as resolved.
Show resolved Hide resolved
$parallelProcess->request([
ReactCommand::ACTION => Action::MAIN,
Content::FILES => $job,
Expand All @@ -226,6 +253,15 @@ function ($exitCode, string $stdErr) use (&$systemErrors, $processIdentifier): v
);

$this->processPool->attachProcess($processIdentifier, $parallelProcess);
};

for ($i = 0; $i < $numberOfProcesses; ++$i) {
// nothing else to process, stop now
if ($jobs === []) {
break;
}

($processSpawner)();
}

$streamSelectLoop->run();
Expand Down