Mongoose mam id rework #4412

NelsonVides · 2024-12-03T14:06:42Z

Here we rid of all native code, and most critically

Benchmark:

  def next_id() do
    Benchee.run(
      %{
        "native code" => fn list ->
          Enum.map(list, fn t -> :mongoose_mam_id.next_unique(t) end)
        end,
        "pure erlang" => fn list ->
          Enum.map(list, fn t -> :mongoose_mam_id_2.next_unique(t) end)
        end,
      },
      inputs: %{
        "1M monotonic" => monotonic_list(100_000_000)
      },
      formatters:
      [
        {Benchee.Formatters.Console, extended_statistics: true}
      ],
      parallel: 1,
      time: 5,
      warmup: 0
    )
  end

  defp monotonic_list(size) do
    fun = fn(_, [head | _] = acc) ->
      next = head + :rand.uniform(20) - 1
      [next | acc]
    end
    :lists.reverse(:lists.foldl(fun, [:rand.uniform(20)], :lists.seq(1, size)))
  end

returns

Operating System: macOS
CPU Information: Apple M3 Pro
Number of Available Cores: 12
Available memory: 36 GB
Elixir 1.17.3
Erlang 27.1.2
JIT enabled: true

Benchmark suite executing with the following configuration:
warmup: 0 ns
time: 5 s
memory time: 0 ns
reduction time: 0 ns
parallel: 1
inputs: 1M monotonic
Estimated total run time: 10 s

Benchmarking native code with input 1M monotonic ...
Benchmarking pure erlang with input 1M monotonic ...
Calculating statistics...
Formatting results...

Name                  ips        average  deviation         median         99th %
pure erlang          0.31         3.23 s    ±10.49%         3.23 s         3.47 s
native code          0.28         3.54 s     ±6.26%         3.54 s         3.70 s

Comparison:
pure erlang          0.31
native code          0.28 - 1.10x slower +0.32 s

Extended statistics:

Name                minimum        maximum    sample size                     mode
pure erlang          2.99 s         3.47 s              2                     None
native code          3.39 s         3.70 s              2                     None

codecov · 2024-12-03T14:18:14Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.30%. Comparing base (33fe8d6) to head (761b6ee).
Report is 6 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4412      +/-   ##
==========================================
- Coverage   85.31%   85.30%   -0.01%     
==========================================
  Files         550      550              
  Lines       33903    33905       +2     
==========================================
  Hits        28923    28923              
- Misses       4980     4982       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

```elixir def next_id() do Benchee.run( %{ "native code" => fn list -> Enum.map(list, fn t -> :mongoose_mam_id.next_unique(t) end) end, "pure erlang" => fn list -> Enum.map(list, fn t -> :mongoose_mam_id_2.next_unique(t) end) end, }, inputs: %{ "1M monotonic" => monotonic_list(100_000_000) }, formatters: [ {Benchee.Formatters.Console, extended_statistics: true} ], parallel: 1, time: 5, warmup: 0 ) end defp monotonic_list(size) do fun = fn(_, [head | _] = acc) -> next = head + :rand.uniform(20) - 1 [next | acc] end :lists.reverse(:lists.foldl(fun, [:rand.uniform(20)], :lists.seq(1, size))) end ``` returns ``` Operating System: macOS CPU Information: Apple M3 Pro Number of Available Cores: 12 Available memory: 36 GB Elixir 1.17.3 Erlang 27.1.2 JIT enabled: true Benchmark suite executing with the following configuration: warmup: 0 ns time: 5 s memory time: 0 ns reduction time: 0 ns parallel: 1 inputs: 1M monotonic Estimated total run time: 10 s Benchmarking native code with input 1M monotonic ... Benchmarking pure erlang with input 1M monotonic ... Calculating statistics... Formatting results... Name ips average deviation median 99th % pure erlang 0.31 3.23 s ±10.49% 3.23 s 3.47 s native code 0.28 3.54 s ±6.26% 3.54 s 3.70 s Comparison: pure erlang 0.31 native code 0.28 - 1.10x slower +0.32 s Extended statistics: Name minimum maximum sample size mode pure erlang 2.99 s 3.47 s 2 None native code 3.39 s 3.70 s 2 None ```

mongoose-im · 2024-12-04T11:59:10Z

elasticsearch_and_cassandra_27 / elasticsearch_and_cassandra_mnesia / 761b6ee
Reports root/ big
OK: 473 / Failed: 0 / User-skipped: 49 / Auto-skipped: 0

small_tests_27 / small_tests / 761b6ee
Reports root / small

small_tests_26 / small_tests / 761b6ee
Reports root / small

small_tests_27_arm64 / small_tests / 761b6ee
Reports root / small

ldap_mnesia_26 / ldap_mnesia / 761b6ee
Reports root/ big
OK: 2350 / Failed: 1 / User-skipped: 912 / Auto-skipped: 1

cluster_commands_SUITE:clustered:init_per_group

{'EXIT',
  {{verify_result,[],
     [{times,20,
        [{#{node => mongooseim2@localhost},[mongooseim@localhost],true},
         {#{node => mongooseim@localhost},
        [mongooseim2@localhost],
        true}]}],
     ok},
   [{mongoose_helper,do_wait_until,2,
      [{file,
         "/home/circleci/project/big_tests/../test/common/mongoose_helper.erl"},
       {line,359}]},
    {distributed_helper,verify_result,2,
      [{file,
         "/home/circleci/project/big_tests/../test/common/distributed_helper.erl"},
       {line,72}]},
    {distributed_helper,add_node_to_cluster,2,
      [{file,
         "/home/circleci/project/big_tests/../test/common/distributed_helper.erl"},
       {line,36}]},
    {cluster_commands_SUITE,init_per_group,2,
      [{file,
         "/home/circleci/project/big_tests/tests/cluster_commands_SUITE.erl"},
       {line,96}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1390}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1234}]}]}}

Report log

cluster_commands_SUITE:join_successful_prompt

{error,
  {{verify_result,[],
     [{times,20,
        [{#{node => mongooseim2@localhost},[mongooseim@localhost],true},
         {#{node => mongooseim@localhost},
        [mongooseim2@localhost],
        true}]}],
     ok},
   [{mongoose_helper,do_wait_until,2,
      [{file,
         "/home/circleci/project/big_tests/../test/common/mongoose_helper.erl"},
       {line,359}]},
    {distributed_helper,verify_result,2,
      [{file,
         "/home/circleci/project/big_tests/../test/common/distributed_helper.erl"},
       {line,72}]},
    {cluster_commands_SUITE,join_successful_prompt,1,
      [{file,
         "/home/circleci/project/big_tests/tests/cluster_commands_SUITE.erl"},
       {line,215}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1302}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1234}]}]}}

Report log

ldap_mnesia_27 / ldap_mnesia / 761b6ee
Reports root/ big
OK: 2352 / Failed: 0 / User-skipped: 912 / Auto-skipped: 0

dynamic_domains_mysql_redis_27 / mysql_redis / 761b6ee
Reports root/ big
OK: 4724 / Failed: 0 / User-skipped: 154 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_26 / pgsql_mnesia / 761b6ee
Reports root/ big
OK: 4759 / Failed: 0 / User-skipped: 119 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_27 / pgsql_mnesia / 761b6ee
Reports root/ big
OK: 4759 / Failed: 0 / User-skipped: 119 / Auto-skipped: 0

internal_mnesia_27 / internal_mnesia / 761b6ee
Reports root/ big
OK: 2494 / Failed: 0 / User-skipped: 770 / Auto-skipped: 0

pgsql_cets_27 / pgsql_cets / 761b6ee
Reports root/ big
OK: 4849 / Failed: 0 / User-skipped: 188 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_27 / odbc_mssql_mnesia / 761b6ee
Reports root/ big
OK: 4754 / Failed: 0 / User-skipped: 124 / Auto-skipped: 0

cockroachdb_cets_27 / cockroachdb_cets / 761b6ee
Reports root/ big
OK: 4848 / Failed: 1 / User-skipped: 188 / Auto-skipped: 0

adhoc_SUITE:adhoc:disco_info_sm_commands

{error,
  {timeout_when_waiting_for_stanza,
    [{escalus_client,wait_for_stanza,
       [{client,<<"alice_disco_info_sm_commands_43@localhost/res1">>,
          escalus_tcp,<0.2530.0>,
          [{event_manager,<0.2513.0>},
           {server,<<"localhost">>},
           {username,<<"alicE_disco_info_sm_commands_43">>},
           {resource,<<"res1">>}],
          [{event_client,
             [{event_manager,<0.2513.0>},
            {server,<<"localhost">>},
            {username,<<"alicE_disco_info_sm_commands_43">>},
            {resource,<<"res1">>}]},
           {resource,<<"res1">>},
           {username,<<"alice_disco_info_sm_commands_43">>},
           {server,<<"localhost">>},
           {host,<<"localhost">>},
           {port,5222},
           {auth,{escalus_auth,auth_plain}},
           {wspath,undefined},
           {username,<<"alicE_disco_info_sm_commands_43">>},
           {server,<<"localhost">>},
           {password,<<"matygrysa">>},
           {stream_id,<<"d8ad8b5a67a6a3fd">>}]},
        5000],
       [{file,
          "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_client.erl"},
        {line,136}]},
     {adhoc_SUITE,'-disco_info_sm_commands/1-fun-0-',1,
       [{file,"/home/circleci/project/big_tests/tests/adhoc_SUITE.erl"},
        {line,131}]},
     {escalus_story,story,4,
       [{file,
          "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
        {line,72}]},
     {test_server,ts_tc...

Report log

pgsql_mnesia_27 / pgsql_mnesia / 761b6ee
Reports root/ big
OK: 5150 / Failed: 0 / User-skipped: 128 / Auto-skipped: 0

mssql_mnesia_27 / odbc_mssql_mnesia / 761b6ee
Reports root/ big
OK: 5145 / Failed: 0 / User-skipped: 133 / Auto-skipped: 0

pgsql_mnesia_26 / pgsql_mnesia / 761b6ee
Reports root/ big
OK: 5150 / Failed: 0 / User-skipped: 128 / Auto-skipped: 0

mysql_redis_27 / mysql_redis / 761b6ee
Reports root/ big
OK: 5129 / Failed: 0 / User-skipped: 149 / Auto-skipped: 0

small_tests_26 / small_tests / 761b6ee
Reports root / small

ldap_mnesia_26 / ldap_mnesia / 761b6ee
Reports root/ big
OK: 2352 / Failed: 0 / User-skipped: 912 / Auto-skipped: 0

cockroachdb_cets_27 / cockroachdb_cets / 761b6ee
Reports root/ big
OK: 4849 / Failed: 0 / User-skipped: 188 / Auto-skipped: 0

arcusfelis

looks ok

This comment was marked as outdated.

Sign in to view

NelsonVides force-pushed the remove_dead_tools branch from 31d8d79 to 4864740 Compare December 3, 2024 15:50

NelsonVides force-pushed the mongoose_mam_id branch from 395aa57 to bb1ba36 Compare December 3, 2024 15:50

This comment was marked as outdated.

Sign in to view

NelsonVides force-pushed the remove_dead_tools branch from 4864740 to cd5e55f Compare December 4, 2024 10:22

NelsonVides force-pushed the mongoose_mam_id branch from bb1ba36 to 1a17bba Compare December 4, 2024 10:22

This comment was marked as outdated.

Sign in to view

NelsonVides force-pushed the remove_dead_tools branch from cd5e55f to 33fe8d6 Compare December 4, 2024 11:47

NelsonVides added 2 commits December 4, 2024 12:48

Remove port-compiler as we have no more local native code

761b6ee

NelsonVides force-pushed the mongoose_mam_id branch from 1a17bba to 761b6ee Compare December 4, 2024 11:50

Base automatically changed from remove_dead_tools to master December 4, 2024 16:49

NelsonVides marked this pull request as ready for review December 4, 2024 16:50

arcusfelis approved these changes Dec 4, 2024

View reviewed changes

arcusfelis merged commit d8bb43d into master Dec 4, 2024
4 checks passed

arcusfelis deleted the mongoose_mam_id branch December 4, 2024 16:53

jacekwegr added this to the 6.3.1 milestone Dec 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mongoose mam id rework #4412

Mongoose mam id rework #4412

NelsonVides commented Dec 3, 2024 •

edited

Loading

This comment was marked as outdated.

codecov bot commented Dec 3, 2024 •

edited

Loading

This comment was marked as outdated.

This comment was marked as outdated.

mongoose-im commented Dec 4, 2024 •

edited

Loading

arcusfelis left a comment

Mongoose mam id rework #4412

Mongoose mam id rework #4412

Conversation

NelsonVides commented Dec 3, 2024 • edited Loading

This comment was marked as outdated.

codecov bot commented Dec 3, 2024 • edited Loading

Codecov Report

This comment was marked as outdated.

This comment was marked as outdated.

mongoose-im commented Dec 4, 2024 • edited Loading

arcusfelis left a comment

Choose a reason for hiding this comment

NelsonVides commented Dec 3, 2024 •

edited

Loading

codecov bot commented Dec 3, 2024 •

edited

Loading

mongoose-im commented Dec 4, 2024 •

edited

Loading