Migration hitting rate limit with very few entries showing up in target environment

[hartmagj]

Expected Behavior

All entries (approximately 217 in total) are pulled from one environment and updated in another environment when running a migration script.

Actual Behavior

anywhere from 2-100 entries are being updated in the target environment, with the following errors being displayed with great frequency in the console:

Oh no! Some errors occurred! Error [ContentfulMultiError]: Errors occured 11:20:68 - Rate limit error occurred. Waiting for 1624 ms before retrying... 11:20:45 - Connection error occurred. Waiting for 1978 ms before retrying...

Possible Solution

I believe it has to do with a rate limit, as every 15 minutes to an hour it tends to migrate content in numbers closer to 100 and then subsequent runs dip back down to only 1 or 2 entries, leading me to believe it has to do with this. A batched upload system might work, but I am worried this is either not the problem or the batching of less than 100 entries every 15 minutes at best would be too slow and cause connection timeouts.

Steps to Reproduce

relevant section of the migration script:

const exportOptions = expOpts;
exportOptions.queryEntries = ['sys.id=0000'];
exportOptions.queryAssets = [sys.id=${id}];
contentfulExport(exportOptions)
.then((result) => {
contentfulImport({
content: result,
...impOpts,
});
})
.catch((err) => {
console.log('Oh no! Some errors occurred!', err);
});

Context

My team and I have been regularly running this script in an attempt to merge environments, it was structured largely from code in this repository but may be outdated.

Environment

it has been run on node 12.19.1 and a couple distributions of node 16 with the same results

• Node Version: 12.19.1
• Package Manager Version: 6.14.8
• Operating System: 64-bit windows 10
• Package Version: 7.41.0