Skip to content
This repository has been archived by the owner on Sep 28, 2022. It is now read-only.

client.go ExperimentalReplayImport() races against client.go logImport.func1() #264

Open
jaten-molecula opened this issue Jun 22, 2020 · 1 comment
Open

client.go ExperimentalReplayImport() races against client.go logImport.func1() #264

jaten-molecula opened this issue Jun 22, 2020 · 1 comment

Comments

@jaten-molecula
Copy link

jaten-molecula commented Jun 22, 2020 •
edited
Loading

go version go1.14.4 darwin/amd64

At tip, 28cb67f, running against a tip pilosa/pilosa server (at 9dc1775b93464f78acc8573cfad2f405b1175fb5), make test-all-race detected the following race:

(base) jaten@Jasons-MacBook-Pro ~/go/src/github.com/pilosa/go-pilosa (master) $ make test-all-race
PILOSA_BIND=http://:10101 /Applications/Xcode.app/Contents/Developer/usr/bin/make test-all TESTFLAGS=-race
PILOSA_BIND=http://:10101 go test -count=1 ./... -race
==================
WARNING: DATA RACE
Write at 0x00c00012caa0 by goroutine 11:
  bytes.(*Buffer).Read()
      /usr/local/go/src/bytes/buffer.go:297 +0x4a
  io.ReadAtLeast()
      /usr/local/go/src/io/io.go:310 +0x98
  io.ReadFull()
      /usr/local/go/src/io/io.go:329 +0x93
  encoding/gob.decodeUintReader()
      /usr/local/go/src/encoding/gob/decode.go:120 +0x40
  encoding/gob.(*Decoder).recvMessage()
      /usr/local/go/src/encoding/gob/decoder.go:81 +0xa7
  encoding/gob.(*Decoder).decodeTypeSequence()
      /usr/local/go/src/encoding/gob/decoder.go:143 +0x1f2
  encoding/gob.(*Decoder).DecodeValue()
      /usr/local/go/src/encoding/gob/decoder.go:211 +0x17f
  encoding/gob.(*Decoder).Decode()
      /usr/local/go/src/encoding/gob/decoder.go:188 +0x236
  github.com/pilosa/go-pilosa.(*Client).ExperimentalReplayImport()
      /Users/jaten/go/src/github.com/pilosa/go-pilosa/client.go:1302 +0x396
  github.com/pilosa/go-pilosa.TestImportWithReplayErrors.func1()
      /Users/jaten/go/src/github.com/pilosa/go-pilosa/client_internal_it_test.go:159 +0x5b

Previous write at 0x00c00012caa0 by goroutine 77:
  bytes.(*Buffer).Write()
      /usr/local/go/src/bytes/buffer.go:169 +0x42
  encoding/gob.(*Encoder).writeMessage()
      /usr/local/go/src/encoding/gob/encoder.go:82 +0x41a
  encoding/gob.(*Encoder).EncodeValue()
      /usr/local/go/src/encoding/gob/encoder.go:253 +0x881
  encoding/gob.(*Encoder).Encode()
      /usr/local/go/src/encoding/gob/encoder.go:176 +0x5b
  github.com/pilosa/go-pilosa.(*Client).logImport.func1()
      /Users/jaten/go/src/github.com/pilosa/go-pilosa/client.go:1237 +0x2b9

Goroutine 11 (running) created at:
  github.com/pilosa/go-pilosa.TestImportWithReplayErrors()
      /Users/jaten/go/src/github.com/pilosa/go-pilosa/client_internal_it_test.go:158 +0x929
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:991 +0x1eb

Goroutine 77 (finished) created at:
  github.com/pilosa/go-pilosa.(*Client).logImport()
      /Users/jaten/go/src/github.com/pilosa/go-pilosa/client.go:1225 +0xfb
  github.com/pilosa/go-pilosa.(*Client).importRoaringBitmap()
      /Users/jaten/go/src/github.com/pilosa/go-pilosa/client.go:885 +0x98a
  github.com/pilosa/go-pilosa.(*Client).importColumnsRoaring()
      /Users/jaten/go/src/github.com/pilosa/go-pilosa/client.go:606 +0x5fb
  github.com/pilosa/go-pilosa.(*Client).importColumns()
      /Users/jaten/go/src/github.com/pilosa/go-pilosa/client.go:533 +0x946
  github.com/pilosa/go-pilosa.(*Client).importColumns-fm()
      /Users/jaten/go/src/github.com/pilosa/go-pilosa/client.go:521 +0xca
  github.com/pilosa/go-pilosa.importRecords()
      /Users/jaten/go/src/github.com/pilosa/go-pilosa/import_manager.go:204 +0x1be
  github.com/pilosa/go-pilosa.recordImportWorker()
      /Users/jaten/go/src/github.com/pilosa/go-pilosa/import_manager.go:166 +0x5eb
==================
==================
WARNING: DATA RACE
Read at 0x00c00012ca80 by goroutine 11:
  bytes.(*Buffer).empty()
      /usr/local/go/src/bytes/buffer.go:69 +0x5f
  bytes.(*Buffer).Read()
      /usr/local/go/src/bytes/buffer.go:298 +0x94
  io.ReadAtLeast()
      /usr/local/go/src/io/io.go:310 +0x98
  io.ReadFull()
      /usr/local/go/src/io/io.go:329 +0x93
  encoding/gob.decodeUintReader()
      /usr/local/go/src/encoding/gob/decode.go:120 +0x40
  encoding/gob.(*Decoder).recvMessage()
      /usr/local/go/src/encoding/gob/decoder.go:81 +0xa7
  encoding/gob.(*Decoder).decodeTypeSequence()
      /usr/local/go/src/encoding/gob/decoder.go:143 +0x1f2
  encoding/gob.(*Decoder).DecodeValue()
      /usr/local/go/src/encoding/gob/decoder.go:211 +0x17f
  encoding/gob.(*Decoder).Decode()
      /usr/local/go/src/encoding/gob/decoder.go:188 +0x236
  github.com/pilosa/go-pilosa.(*Client).ExperimentalReplayImport()
      /Users/jaten/go/src/github.com/pilosa/go-pilosa/client.go:1302 +0x396
  github.com/pilosa/go-pilosa.TestImportWithReplayErrors.func1()
      /Users/jaten/go/src/github.com/pilosa/go-pilosa/client_internal_it_test.go:159 +0x5b

Previous write at 0x00c00012ca80 by goroutine 77:
  bytes.(*Buffer).tryGrowByReslice()
      /usr/local/go/src/bytes/buffer.go:108 +0x196
  bytes.(*Buffer).Write()
      /usr/local/go/src/bytes/buffer.go:170 +0x8f
  encoding/gob.(*Encoder).writeMessage()
      /usr/local/go/src/encoding/gob/encoder.go:82 +0x41a
  encoding/gob.(*Encoder).EncodeValue()
      /usr/local/go/src/encoding/gob/encoder.go:253 +0x881
  encoding/gob.(*Encoder).Encode()
      /usr/local/go/src/encoding/gob/encoder.go:176 +0x5b
  github.com/pilosa/go-pilosa.(*Client).logImport.func1()
      /Users/jaten/go/src/github.com/pilosa/go-pilosa/client.go:1237 +0x2b9

Goroutine 11 (running) created at:
  github.com/pilosa/go-pilosa.TestImportWithReplayErrors()
      /Users/jaten/go/src/github.com/pilosa/go-pilosa/client_internal_it_test.go:158 +0x929
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:991 +0x1eb

Goroutine 77 (finished) created at:
  github.com/pilosa/go-pilosa.(*Client).logImport()
      /Users/jaten/go/src/github.com/pilosa/go-pilosa/client.go:1225 +0xfb
  github.com/pilosa/go-pilosa.(*Client).importRoaringBitmap()
      /Users/jaten/go/src/github.com/pilosa/go-pilosa/client.go:885 +0x98a
  github.com/pilosa/go-pilosa.(*Client).importColumnsRoaring()
      /Users/jaten/go/src/github.com/pilosa/go-pilosa/client.go:606 +0x5fb
  github.com/pilosa/go-pilosa.(*Client).importColumns()
      /Users/jaten/go/src/github.com/pilosa/go-pilosa/client.go:533 +0x946
  github.com/pilosa/go-pilosa.(*Client).importColumns-fm()
      /Users/jaten/go/src/github.com/pilosa/go-pilosa/client.go:521 +0xca
  github.com/pilosa/go-pilosa.importRecords()
      /Users/jaten/go/src/github.com/pilosa/go-pilosa/import_manager.go:204 +0x1be
  github.com/pilosa/go-pilosa.recordImportWorker()
      /Users/jaten/go/src/github.com/pilosa/go-pilosa/import_manager.go:166 +0x5eb
==================
--- FAIL: TestImportWithReplayErrors (0.43s)
    testing.go:906: race detected during execution of test
go-pilosa 2020/06/22 15:51:20 invalidating shard node cache, uri mismatch at 0 old: [zzz://:0], new: [http://localhost:10101]
go-pilosa 2020/06/22 15:51:58 299 pilosa/2.0 "FAKE WARNING: Deprecated PQL version: PQL v2 will remove support for SetBit() in Pilosa 2.1. Please update your client to support Set() (See https://docs.pilosa.com/pql#versioning)." "Sat, 25 Aug 2019 23:34:45 GMT"
FAIL
FAIL	github.com/pilosa/go-pilosa	44.363s
ok  	github.com/pilosa/go-pilosa/csv	0.051s
?   	github.com/pilosa/go-pilosa/examples/multicol-csv-import	[no test files]
?   	github.com/pilosa/go-pilosa/gopilosa_pbuf	[no test files]
ok  	github.com/pilosa/go-pilosa/gpexp	2.752s
?   	github.com/pilosa/go-pilosa/lru	[no test files]
FAIL
make[1]: *** [test-all] Error 1
make: *** [test-all-race] Error 2
(base) jaten@Jasons-MacBook-Pro ~/go/src/github.com/pilosa/go-pilosa (master) $ git log|head
commit 6bc638d761338d5189736e9beed8546a4bc6e5ce
Merge: d55c16e 28cb67f
Author: Travis Turner <[email protected]>
Date:   Sat Nov 30 22:00:22 2019 -0600

    Merge pull request #262 from travisturner/groupby-having
    
    add having clause support to GroupByBuilder

commit 28cb67f61c4a7db69c0907c64d8b3363b587ad9f
(base) jaten@Jasons-MacBook-Pro ~/go/src/github.com/pilosa/go-pilosa (master) $
@jaten-molecula
Copy link
Author

jaten-molecula commented Jun 22, 2020 •
edited
Loading

NB (after a quick chat with Travis) Note that this occurs in an experimental method, Client.ExperimentalReplayImport() which doesn't have a counterpart in enterprise version. Given the likelihood of some of the enterprise code replacing and deleting this code at some point in the future, this seems very low priority.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jaten-molecula