Disable rocksdb concurrent writes #126
Conversation
|
This might totally kill ingestion performance, need to test. |
|
the new gcc-6 dependency is also a serious impediment. |
|
Doing a test ingestion, the slowdown for parallel ingestion seems to be about 7%. Significant, but not a total catastrophe. |
| @@ -49,6 +49,7 @@ func (ds *RocksDS) Open(home string) error { | |||
| } | |||
| } | |||
| opts.OptimizeForPointLookup(uint64(bcmb)) | |||
| opts.SetAllowConcurrentMemtableWrites(false) | |||
There was a problem hiding this comment.
I would very much like to have a comment here.
| - gcc-4.8 | ||
| - g++-4.8 | ||
| - gcc-6 | ||
| - g++-6 |
There was a problem hiding this comment.
So it seems to build fine with gcc-5 in the test node, which is a more reasonable compiler for required dependency.
|
So my impression is that the concurrent writes were never supported, and the upstream change simply made attempting to use them a fatal error whereas it was not previously. It's also possible that the engine used in optimizeForPointLookup (which I tried to track down the code for at some point but gave up after a bit) maybe actually does support it but needs some flags set or smth? |
|
I would say keep it working for the moment and make an issue to investigate this |
See mediachain/gorocksdb#1
Basically, the latest rocksdb release (used by the embedded build in gorocksdb as of a few days ago), throws if you try to use
OptimizeForPointLookupwithout setting theallow_concurrent_memtable_writeoption tofalse. I added a setter for that option to our fork (and made a PR against the upstream repo: tecbot/gorocksdb#75), and call it fromRocksDS.Opento avoid the crash.I haven't measured how this might impact ingestion performance, but it seems that the unsupported concurrent writes could lead to unpredictable crashes, so I guess it's a good thing to disable them.