Parallel access to same file locked

[Tremax]

Setup Leofs cluster with 3 storage nodes and upload large file(1GB) to cluster -- all works just fine for uploading/downloading with 1 client. But once 2 separate downloads started of same file(i use s3cmd), 2nd one waiting until 1st one not finish downloading, no upload process running while downloading.
Noting strange in log. I also try to sniff traffic for 2nd client and it is hangs in GET request to file until s3cmd timeout occurred.

Config of cluster:

 Basic/Consistency level
-----------------------------------+----------
                    system version | 1.3.7
                        cluster Id | leofs_1
                             DC Id | dc_1
                    Total replicas | 2
          number of successes of R | 1
          number of successes of W | 2
          number of successes of D | 2
 number of rack-awareness replicas | 0
                         ring size | 2^128
-----------------------------------+----------

On storage side almost all config by default.

Parallel access to same file possible without blocking ?

[windkit]

This is because of the attempt to fix the issue when multiple clients reading the same object without disk cache #433

9:9:29.121005 <0.16467.10> leo_large_object_get_handler:terminate({badarg,
    [{erlang,phash2,[<<"test/testf">>,0],[]},
     {leo_cache_api,put_begin_tran,2,
         [{file,"src/leo_cache_api.erl"},{line,251}]},
     {leo_large_object_get_handler,put_begin_tran_with_retry,2,
         [{file,"src/leo_large_object_get_handler.erl"},{line,308}]},
     {leo_large_object_get_handler,handle_call,3,
         [{file,"src/leo_large_object_get_handler.erl"},{line,122}]},
     {gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,629}]},
     {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,661}]},
     {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, {state,<<"test/testf">>,
       {transport_record,ranch_tcp,#Port<0.3073>,5242880},
       undefined})

I will try to fix the issue, fallback to parallel fetching when there is no disk cache.
what do you think @yosukehara @mocchira

[mocchira]

@windkit

I will try to fix the issue, fallback to parallel fetching when there is no disk cache.
what do you think @yosukehara @mocchira

SGTM.

[yosukehara]

@windkit

fallback to parallel fetching when there is no disk cache.

I agree with your idea.

[Tremax]

Yes, got it, thank you!

Could you also clarify if i understand correctly. If disk cache enabled and empty, 2 clients(1 with slow connection and 2nd with fast connection) at same time try to obtain same large file, and slow client start downloading first, does it mean that 2nd(fast) client now limited in downloading speed that has 1st client and need to wait until 1st client fill up cache with this file, right ?

[windkit]

@Tremax Yes that's true, it depends on the rate of the 1st client.

Just to clarify, 2nd client would not wait until the whole file is cached, it would read from the disk cache as long as there is new data.

But yes, if 1st client has a slow connection, it would be the bottleneck. Something we may consider optimizing it.

[windkit]

@Tremax Please help to test it again with the develop branch and see if the problem persists.

[Tremax]

Will check fix this weekend.

[Tremax]

Just checked with develop branch and was able to download same file from separate client instances. Seems to be resolved.

[windkit]

@Tremax thanks for testing