Dgraph Crashes Randomly While Running In GCE

[willcj33]

panic: runtime error: index out of range

goroutine 244714 [running]:
github.com/dgraph-io/dgraph/algo.Difference(0xc44fa0e820, 0xc4534e41a0)
	/home/travis/gopath/src/github.com/dgraph-io/dgraph/algo/uidlist.go:242 +0x26d
github.com/dgraph-io/dgraph/query.ProcessGraph(0x1739f00, 0xc433e188d0, 0xc43cf2bb00, 0xc43cf2b200, 0xc44d294c60)
	/home/travis/gopath/src/github.com/dgraph-io/dgraph/query/query.go:1729 +0x1c93
created by github.com/dgraph-io/dgraph/query.ProcessGraph
	/home/travis/gopath/src/github.com/dgraph-io/dgraph/query/query.go:1701 +0x18ea

[manishrjain]

@tzdybal : Can you look into it immediately?

[tzdybal]

It seems that bug is caused by a race condition, when multiple filters are applied concurrently to the same subgraph. I'll try to reproduce this.

@willcj33 could you provide the problematic query?

[tzdybal]

Detailed problem description:
When multiple filters are used, filter expression tree is build.
for example filter: @filter(not(allofterms(name@en, "a")) and not(allofterms(name@en, "b")) and not(allofterms(name@en, "c"))) the three looks like this:

and                                                                                                 
|                                                                                                   
+-> not                                                                                             
|    |                                                                                              
|    +-> allofterms(name@en, "a")                                                                   
|                                                                                                   
+-> and                                                                                             
    |                                                                                               
    +-> not                                                                                         
    |    |                                                                                          
    |    +-> allofterms(name@en, "b")                                                               
    |                                                                                               
    +-> not                                                                                         
         |                                                                                          
         +-> allofterms(name@en, "c")

Partial results (UID lists) are shared by all filters. At each level filters are executed concurrently. The problem occurs when and and not from the same level are scheduled to execute in the same time. Intersection done by and may alter the UID list accessed by not operator, causing the crash.

To eliminate this data race, algorithms should work on separate copies of pointers to UID list (fortunately copying entire lists is not required).