This repository has been archived by the owner on Sep 28, 2022. It is now read-only.
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Don't use batch size for channel buffers
Channel buffers are there to let workers do a bit more work while things are running, but batch size is for the amount of stuff we want to accumulate before dropping a whole batch of computed results off with the server. There's no reason for the channel buffer to be that size, and having it be that size (which is typically at least 65536) means that each import worker is typically sitting on a channel that has about a million interface-flavored objects containing pointers to records in it, and suddenly we have memory pressure.
- Loading branch information