Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[all][leo_redundant_manager] Keep dumping the same error in case mq worker process died #971

Closed
mocchira opened this issue Jan 17, 2018 · 0 comments

Comments

@mocchira
Copy link
Member

mocchira commented Jan 17, 2018

This can happen in all components (leo_manager, leo_storage, leo_gateway).

[W]     gateway_0@127.0.0.1     2018-01-17 11:25:00.20034 +0900 1516155900      null:null       0       {module,"leo_membership_cluster_local"},{function,"compare_with_remote_chksum/3"},{line,405},{body,{'storag
[email protected]',timeout}}
[E]     gateway_0@127.0.0.1     2018-01-17 11:25:00.26088 +0900 1516155900      ets:tab2list    710     gen_server mq_worker_node_2 terminated with reason: bad argument in call to ets:match_object(leo_object_storage_containers, '_') in ets:tab2list/1 line 710
[E]     gateway_0@127.0.0.1     2018-01-17 11:25:00.26673 +0900 1516155900      ets:tab2list    710     CRASH REPORT Process mq_worker_node_2 with 0 neighbours exited with reason: bad argument in call to ets:mat
ch_object(leo_object_storage_containers, '_') in ets:tab2list/1 line 710
[E]     gateway_0@127.0.0.1     2018-01-17 11:25:00.27767 +0900 1516155900      ets:tab2list    710     Supervisor leo_mq_sup had child mq_worker_node_2 started with leo_mq_server:start_link(mq_worker_node_2, 2, {mq_properties,mq_worker_node,undefined,leo_membership_mq_client,leveldb,3,1,"./work/queue/membe...",...}) at <0.1575.0> exit with reason bad argument in call to ets:match_object(leo_object_storage_containers,
'_') in ets:tab2list/1 line 710 in context child_terminated
[E]     gateway_0@127.0.0.1     2018-01-17 11:25:01.670040 +0900        1516155901      null:null       0       {module,"leo_backend_db_eleveldb"},{function,"first_n/2"},{line,293},{body,{badarg,[{eleveldb,async_iterator,[#Ref<0.0.2.4181>,<<>>,[]],[]},{eleveldb,iterator,2,[{file,"src/eleveldb.erl"},{line,200}]},{leo_backend_db_eleveldb,fold,4,[{file,"src/leo_backend_db_eleveldb.erl"},{line,386}]},{leo_backend_db_eleveldb,first_n,3,[{file,"src/leo_backend_db_eleveldb.erl"},{line,284}]},{leo_backend_db_server,handle_call,3,[{file,"src/leo_backend_db_server.erl"},{line,338}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"
},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,647}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}
[E]     gateway_0@127.0.0.1     2018-01-17 11:25:01.671479 +0900        1516155901      null:null       0       {module,"leo_mq_server"},{function,"handle_call/3"},{line,294},{body,{badarg,[{eleveldb,async_iterator,[#Ref<0.0.2.4181>,<<>>,[]],[]},{eleveldb,iterator,2,[{file,"src/eleveldb.erl"},{line,200}]},{leo_backend_db_eleveldb,fold,4,[{file,"src/leo_backend_db_eleveldb.erl"},{line,386}]},{leo_backend_db_eleveldb,first_n,3,[{file,"src/leo_backend_db_eleveldb.erl"},{line,284}]},{leo_backend_db_server,handle_call,3,[{file,"src/leo_backend_db_server.erl"},{line,338}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{lin
e,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,647}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}
[E]     gateway_0@127.0.0.1     2018-01-17 11:25:07.121052 +0900        1516155907      null:null       0       {module,"leo_backend_db_eleveldb"},{function,"first_n/2"},{line,293},{body,{badarg,[{eleveldb,async_iterator,[#Ref<0.0.1.1383>,<<>>,[]],[]},{eleveldb,iterator,2,[{file,"src/eleveldb.erl"},{line,200}]},{leo_backend_db_eleveldb,fold,4,[{file,"src/leo_backend_db_eleveldb.erl"},{line,386}]},{leo_backend_db_eleveldb,first_n,3,[{file,"src/leo_backend_db_eleveldb.erl"},{line,284}]},{leo_backend_db_server,handle_call,3,[{file,"src/leo_backend_db_server.erl"},{line,338}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"
},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,647}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}
[E]     gateway_0@127.0.0.1     2018-01-17 11:25:07.122604 +0900        1516155907      null:null       0       {module,"leo_mq_server"},{function,
... keep dumping the same error above

There seems to be something wrong (probably file handle leak) in termination processing.

@mocchira mocchira changed the title [all] [all][leo_redundant_manager] Keep dumping the same error in case mq worker process died Jan 17, 2018
@mocchira mocchira added this to the 1.4.0 milestone Jan 17, 2018
@mocchira mocchira self-assigned this Feb 16, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant