Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cleaning logic for discovery_nodes table #4045

Closed
wants to merge 1 commit into from
Closed

Cleaning logic for discovery_nodes table #4045

wants to merge 1 commit into from

Conversation

arcusfelis
Copy link
Contributor

@arcusfelis arcusfelis commented Jun 29, 2023 •
edited
Loading

This PR addresses "cleaning expired rows from the discovery table" MIM-1942

Proposed changes include:

  • Add mongoose_time_drift to raise alarms if system clocks are not correct
  • Add cleaning logic into mongoose_cets_discovery_rdbms
  • Test cases in cets_disco_SUITE

@mongoose-im
Copy link
Collaborator

mongoose-im commented Jun 29, 2023 •
edited
Loading

small_tests_24 / small_tests / b0df691
Reports root / small

small_tests_25_arm64 / small_tests / b0df691
Reports root / small

small_tests_25 / small_tests / b0df691
Reports root / small

ldap_mnesia_24 / ldap_mnesia / b0df691
Reports root/ big
OK: 2224 / Failed: 0 / User-skipped: 841 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / b0df691
Reports root/ big
OK: 4201 / Failed: 0 / User-skipped: 90 / Auto-skipped: 0

ldap_mnesia_25 / ldap_mnesia / b0df691
Reports root/ big
OK: 2224 / Failed: 0 / User-skipped: 841 / Auto-skipped: 0

pgsql_mnesia_24 / pgsql_mnesia / b0df691
Reports root/ big
OK: 4574 / Failed: 0 / User-skipped: 99 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / b0df691
Reports root/ big
OK: 4201 / Failed: 0 / User-skipped: 90 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_25 / odbc_mssql_mnesia / b0df691
Reports root/ big
OK: 4198 / Failed: 0 / User-skipped: 93 / Auto-skipped: 0

dynamic_domains_mysql_redis_25 / mysql_redis / b0df691
Reports root/ big
OK: 4175 / Failed: 0 / User-skipped: 116 / Auto-skipped: 0

internal_mnesia_25 / internal_mnesia / b0df691
Reports root/ big
OK: 2370 / Failed: 0 / User-skipped: 695 / Auto-skipped: 0

elasticsearch_and_cassandra_25 / elasticsearch_and_cassandra_mnesia / b0df691
Reports root/ big
OK: 2730 / Failed: 0 / User-skipped: 674 / Auto-skipped: 0

mssql_mnesia_25 / odbc_mssql_mnesia / b0df691
Reports root/ big
OK: 4571 / Failed: 0 / User-skipped: 102 / Auto-skipped: 0

pgsql_cets_25 / pgsql_cets / b0df691
Reports root/ big
OK: 4550 / Failed: 0 / User-skipped: 123 / Auto-skipped: 0

pgsql_mnesia_25 / pgsql_mnesia / b0df691
Reports root/ big
OK: 4574 / Failed: 0 / User-skipped: 99 / Auto-skipped: 0

mysql_redis_25 / mysql_redis / b0df691
Reports root/ big
OK: 4560 / Failed: 0 / User-skipped: 113 / Auto-skipped: 0

@codecov
Copy link

codecov bot commented Jun 29, 2023 •
edited
Loading

Codecov Report

Patch coverage: 91.30% and project coverage change: +0.07 🎉

Comparison is base (b83dc98) 82.21% compared to head (a09f07c) 82.28%.

Additional details and impacted files 
@@               Coverage Diff                @@
##           feature/cets    #4045      +/-   ##
================================================
+ Coverage         82.21%   82.28%   +0.07%     
================================================
  Files               541      542       +1     
  Lines             34012    34073      +61     
================================================
+ Hits              27962    28036      +74     
+ Misses             6050     6037      -13
Impacted Files Coverage Δ
src/mongoose_time_drift.erl 84.37% <84.37%> (ø)
src/mongoose_cets_discovery_rdbms.erl 91.30% <97.22%> (+8.37%) ⬆️
src/ejabberd_sup.erl 79.06% <100.00%> (+0.49%) ⬆️

... and 10 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@arcusfelis
Cleaning logic for discovery_nodes table
a09f07c 
Add mongoose_time_drift to raise alarms if system clocks are not correct
Add cleaning logic into mongoose_cets_discovery_rdbms
Test cases in cets_disco_SUITE
@arcusfelis arcusfelis force-pushed the mu-cets-disco-table-cleaning branch from b0df691 to a09f07c Compare June 29, 2023 12:16
@mongoose-im
Copy link
Collaborator

mongoose-im commented Jun 29, 2023 •
edited
Loading

small_tests_24 / small_tests / a09f07c
Reports root / small

small_tests_25 / small_tests / a09f07c
Reports root / small

small_tests_25_arm64 / small_tests / a09f07c
Reports root / small

ldap_mnesia_24 / ldap_mnesia / a09f07c
Reports root/ big
OK: 2231 / Failed: 1 / User-skipped: 842 / Auto-skipped: 0

pubsub_SUITE:dag+node_config:notify_config_test 
{error,{{badmatch,false},
    [{pubsub_tools,check_response,2,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
            {line,491}]},
     {pubsub_tools,receive_response,3,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
            {line,481}]},
     {pubsub_tools,receive_and_check_response,4,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
            {line,471}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1292}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1224}]}]}}

Report log

dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / a09f07c
Reports root/ big
OK: 4201 / Failed: 0 / User-skipped: 90 / Auto-skipped: 0

ldap_mnesia_25 / ldap_mnesia / a09f07c
Reports root/ big
OK: 2224 / Failed: 0 / User-skipped: 841 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / a09f07c
Reports root/ big
OK: 4201 / Failed: 0 / User-skipped: 90 / Auto-skipped: 0

dynamic_domains_mysql_redis_25 / mysql_redis / a09f07c
Reports root/ big
OK: 4175 / Failed: 0 / User-skipped: 116 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_25 / odbc_mssql_mnesia / a09f07c
Reports root/ big
OK: 4185 / Failed: 7 / User-skipped: 93 / Auto-skipped: 6

mod_blocking_SUITE:manage:add_user_to_blocklist_with_white_spaces 
{error,
  {timeout_when_waiting_for_stanza,
    [{escalus_client,wait_for_stanza,
       [{client,
          <<"alice_unnamed_2347_add_user_to_blocklist_with_white_spaces_2351@domain.example.com/res1">>,
          escalus_tcp,<0.8290.2>,
          [{event_manager,<0.8242.2>},
           {server,<<"domain.example.com">>},
           {username,
             <<"alicE_unnamed_2347_add_user_to_blocklist_with_white_spaces_2351">>},
           {resource,<<"res1">>}],
          [{event_client,
             [{event_manager,<0.8242.2>},
            {server,<<"domain.example.com">>},
            {username,
              <<"alicE_unnamed_2347_add_user_to_blocklist_with_white_spaces_2351">>},
            {resource,<<"res1">>}]},
           {resource,<<"res1">>},
           {username,
             <<"alice_unnamed_2347_add_user_to_blocklist_with_white_spaces_2351">>},
           {server,<<"domain.example.com">>},
           {host,<<"localhost">>},
           {port,5222},
           {auth,{escalus_auth,auth_plain}},
           {wspath,undefined},
           {username,
             <<"alicE_unnamed_2347_add_user_to_blocklist_with_white_spaces_2351">>},
           {server,<<"domain.example.com">>},
           {host,<<"localhost">>},
           {password,<<"matygrysa">>},
           {stream_id,<<"8e9833eaca8baed4">>}]},
        5000],
       [{file,
          "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_client.erl"},
        {line,136}]},
     {mod_blocking_SUITE,
 ...

Report log

mod_blocking_SUITE:manage:remove_user_from_blocklist 
{error,{{assertion_failed,assert_many,false,
              [is_iq_result,#Fun<mod_blocking_SUITE.32.78585296>],
              [],[]},
    [{escalus_new_assert,assert_true,2,
               [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_new_assert.erl"},
                {line,84}]},
     {mod_blocking_SUITE,'-remove_user_from_blocklist/1-fun-0-',2,
               [{file,"/home/circleci/project/big_tests/tests/mod_blocking_SUITE.erl"},
                {line,201}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1291}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1223}]}]}}

Report log

mod_blocking_SUITE:manage:add_many_users_to_blocklist 
{error,{{assertion_failed,assert_many,false,
              [is_iq_result,#Fun<mod_blocking_SUITE.32.78585296>],
              [],[]},
    [{escalus_new_assert,assert_true,2,
               [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_new_assert.erl"},
                {line,84}]},
     {mod_blocking_SUITE,'-add_many_users_to_blocklist/1-fun-0-',4,
               [{file,"/home/circleci/project/big_tests/tests/mod_blocking_SUITE.erl"},
                {line,190}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1291}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1223}]}]}}

Report log

mod_blocking_SUITE:manage:remove_many_user_from_blocklist 
{error,{{assertion_failed,assert_many,false,
              [is_iq_result,#Fun<mod_blocking_SUITE.32.78585296>],
              [],[]},
    [{escalus_new_assert,assert_true,2,
               [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_new_assert.erl"},
                {line,84}]},
     {mod_blocking_SUITE,'-remove_many_user_from_blocklist/1-fun-0-',3,
               [{file,"/home/circleci/project/big_tests/tests/mod_blocking_SUITE.erl"},
                {line,211}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1291}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1223}]}]}}

Report log

mod_blocking_SUITE:manage:clear_blocklist 
{error,
  {{badmatch,
     {error,
       {connection_step_failed,
         {{escalus_session,session},
          {client,
            <<"[email protected]/res1">>,
            escalus_tcp,<0.8320.2>,undefined,
            [{event_client,
               [{event_manager,<0.8277.2>},
              {server,<<"domain.example.com">>},
              {username,
                <<"kate_unnamed_2347_clear_blocklist_2356">>},
              {resource,<<"res1">>}]},
             {resource,<<"res1">>},
             {username,<<"kate_unnamed_2347_clear_blocklist_2356">>},
             {server,<<"domain.example.com">>},
             {host,<<"localhost">>},
             {port,5222},
             {auth,{escalus_auth,auth_plain}},
             {wspath,undefined},
             {username,<<"kate_unnamed_2347_clear_blocklist_2356">>},
             {server,<<"domain.example.com">>},
             {host,<<"localhost">>},
             {password,<<"makrowe;p">>},
             {stream_id,<<"ba8de012da7cae67">>}]},
          [{compression,false},
           {starttls,true},
           {stream_management,true},
           {advanced_message_processing,true},
           {client_state_indication,false},
           {sasl_mechanisms,[<<"SCRAM-SHA-256">>,<<"PLAIN">>]},
           {caps,undefined}]},
         {timeout,session_reply}}}},
   [{escalus_story,'-start_ready_clients/2-fun-0-',3,
      [{file,
         "/home/circleci/project/big_tests/_build/default/lib/escalus...

Report log

mod_blocking_SUITE:manage:add_user_to_blocklist 
{error,
  {timeout_when_waiting_for_stanza,
    [{escalus_client,wait_for_stanza,
       [{client,
          <<"[email protected]/res1">>,
          escalus_tcp,<0.8291.2>,
          [{event_manager,<0.8235.2>},
           {server,<<"domain.example.com">>},
           {username,
             <<"alicE_unnamed_2347_add_user_to_blocklist_2350">>},
           {resource,<<"res1">>}],
          [{event_client,
             [{event_manager,<0.8235.2>},
            {server,<<"domain.example.com">>},
            {username,
              <<"alicE_unnamed_2347_add_user_to_blocklist_2350">>},
            {resource,<<"res1">>}]},
           {resource,<<"res1">>},
           {username,
             <<"alice_unnamed_2347_add_user_to_blocklist_2350">>},
           {server,<<"domain.example.com">>},
           {host,<<"localhost">>},
           {port,5222},
           {auth,{escalus_auth,auth_plain}},
           {wspath,undefined},
           {username,
             <<"alicE_unnamed_2347_add_user_to_blocklist_2350">>},
           {server,<<"domain.example.com">>},
           {host,<<"localhost">>},
           {password,<<"matygrysa">>},
           {stream_id,<<"7f6ef0eb1d96396a">>}]},
        5000],
       [{file,
          "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_client.erl"},
        {line,136}]},
     {mod_blocking_SUITE,'-add_user_to_blocklist/1-fun-0-',2,
       [{file,
          "/home/circleci/project/big_te...

Report log

mod_blocking_SUITE:manage:add_another_user_to_blocklist 
{error,
  {timeout_when_waiting_for_stanza,
    [{escalus_client,wait_for_stanza,
       [{client,
          <<"alice_unnamed_2347_add_another_user_to_blocklist_2355@domain.example.com/res1">>,
          escalus_tcp,<0.8288.2>,
          [{event_manager,<0.8270.2>},
           {server,<<"domain.example.com">>},
           {username,
             <<"alicE_unnamed_2347_add_another_user_to_blocklist_2355">>},
           {resource,<<"res1">>}],
          [{event_client,
             [{event_manager,<0.8270.2>},
            {server,<<"domain.example.com">>},
            {username,
              <<"alicE_unnamed_2347_add_another_user_to_blocklist_2355">>},
            {resource,<<"res1">>}]},
           {resource,<<"res1">>},
           {username,
             <<"alice_unnamed_2347_add_another_user_to_blocklist_2355">>},
           {server,<<"domain.example.com">>},
           {host,<<"localhost">>},
           {port,5222},
           {auth,{escalus_auth,auth_plain}},
           {wspath,undefined},
           {username,
             <<"alicE_unnamed_2347_add_another_user_to_blocklist_2355">>},
           {server,<<"domain.example.com">>},
           {host,<<"localhost">>},
           {password,<<"matygrysa">>},
           {stream_id,<<"d683d7c6c8bac5f8">>}]},
        5000],
       [{file,
          "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_client.erl"},
        {line,136}]},
     {mod_blocking_SUITE,'-add_another_user_to_blocklist/1-fun-0-',2,
       ...

Report log

mod_blocking_SUITE:effect:init_per_group 
{'EXIT',{{badrpc,timeout},
     [{escalus_rpc,call_with_cookie_match,
             [mongooseim@localhost,ejabberd_admin,register,
            [<<"alicE_unnamed_2358">>,<<"domain.example.com">>,
             <<"matygrysa">>],
            3000,mongooseim],
             [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_rpc.erl"},
            {line,34}]},
      {lists,foreach_1,2,[{file,"lists.erl"},{line,1442}]},
      {escalus_ejabberd,create_users,2,
              [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_ejabberd.erl"},
               {line,211}]},
      {escalus_fresh,create_users,2,
             [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_fresh.erl"},
              {line,62}]},
      {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
      {test_server,run_test_case_eval1,6,
             [{file,"test_server.erl"},{line,1379}]},
      {test_server,run_test_case_eval,9,
             [{file,"test_server.erl"},{line,1223}]}]}}

Report log

pgsql_mnesia_24 / pgsql_mnesia / a09f07c
Reports root/ big
OK: 4574 / Failed: 0 / User-skipped: 99 / Auto-skipped: 0

internal_mnesia_25 / internal_mnesia / a09f07c
Reports root/ big
OK: 2370 / Failed: 0 / User-skipped: 695 / Auto-skipped: 0

elasticsearch_and_cassandra_25 / elasticsearch_and_cassandra_mnesia / a09f07c
Reports root/ big
OK: 2730 / Failed: 0 / User-skipped: 674 / Auto-skipped: 0

pgsql_mnesia_25 / pgsql_mnesia / a09f07c
Reports root/ big
OK: 4574 / Failed: 0 / User-skipped: 99 / Auto-skipped: 0

pgsql_cets_25 / pgsql_cets / a09f07c
Reports root/ big
OK: 4550 / Failed: 0 / User-skipped: 123 / Auto-skipped: 0

mssql_mnesia_25 / odbc_mssql_mnesia / a09f07c
Reports root/ big
OK: 4571 / Failed: 0 / User-skipped: 102 / Auto-skipped: 0

mysql_redis_25 / mysql_redis / a09f07c
Reports root/ big
OK: 4560 / Failed: 0 / User-skipped: 113 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_25 / odbc_mssql_mnesia / a09f07c
Reports root/ big
OK: 4198 / Failed: 0 / User-skipped: 93 / Auto-skipped: 0

@arcusfelis arcusfelis mentioned this pull request Jun 30, 2023
Merged
@arcusfelis
Copy link
Contributor Author

Closing because #4049 has better implementation (shorter).

@arcusfelis arcusfelis closed this Jun 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@arcusfelis @mongoose-im