Skip to content
This repository has been archived by the owner on Sep 28, 2022. It is now read-only.

Commit

Permalink
Don't use batch size for channel buffers
Browse files Browse the repository at this point in the history
Channel buffers are there to let workers do a bit more work
while things are running, but batch size is for the amount of stuff
we want to accumulate before dropping a whole batch of computed
results off with the server. There's no reason for the channel buffer
to be that size, and having it be that size (which is typically at
least 65536) means that each import worker is typically sitting on
a channel that has about a million interface-flavored objects containing
pointers to records in it, and suddenly we have memory pressure.
  • Loading branch information
seebs committed Oct 25, 2019
1 parent b8f364f commit 6683b21
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion import_manager.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ func (rim recordImportManager) run(field *Field, iterator RecordIterator, option
}

for i := range recordChans {
recordChans[i] = make(chan []Record, options.batchSize)
recordChans[i] = make(chan []Record, 16)
recordBufs[i] = make([]Record, 0, 16)
chans := importWorkerChannels{
records: recordChans[i],
Expand Down

0 comments on commit 6683b21

Please sign in to comment.