-
Notifications
You must be signed in to change notification settings - Fork 2
Benchmark results
Olli Lyytinen edited this page Jan 9, 2019
·
2 revisions
This page contains results of running all the benchmarks using different adapters. The values reported are iterations per second, thus higher numbers indicate better performance.
The tests were run in two batches. One batch connected to localhost back ends (adapters not marked _net) while the other connected to remote back ends (adapters marked _net). Rather than changing the back end and running the tests again, instead the machine that ran the tests was changed and the back end was kept the same desktop Ubuntu machine. This was done because these benchmarks are more about the back end and the connection to it rather than the performance of the code running on the front end.
All back ends were left at the default settings they come in in Ubuntu. Fcrepo_wrapper was used for Fedora.
Note that in nearly all cases the standard deviation of iteration duration is quite high so small changes may be attributable to randomness rather than any real difference in performance.
These results are from tests with Valkyrie 1.2.1.
For detailed descriptions, see the source code in lib/valkyrie_benchmark/tests.
| Test | Notes |
|---|---|
| basic_metadata_tests create_stub | Create a single small resource with a title but no other data. |
| basic_metadata_tests create_with_text | Create a single resource with a title and a long text field but no other data. |
| basic_metadata_tests create_with_many_fields | Create a single resource with many different data fields present. |
| basic_metadata_tests update_stub | Update a small rerource. |
| basic_metadata_tests update_with_text | Update a resource with a long text field. |
| basic_metadata_tests update_with_many_fields | Update a resource with many data fields. |
| basic_metadata_tests delete_stubs | Delete a small resource. |
| basic_metadata_tests delete_with_many_fields | Delete a resource with many data fields. |
| member_tests create_parent | Create a resource with 100 member_ids. Creating members not inluded. |
| member_tests reload_parent | Reload the parent resource which has 100 member_ids. Does not include loading the members, only the parent resource with a list of member_ids. |
| member_tests update_child | Update the title of one of the members of the parent resource. |
| member_tests update_parent | Update the title of the parent resource. |
| member_tests one_more_page | Add one more member to the parent resource that already has 100 member_ids. |
| member_tests remove_page | Remove one of the members from the parent. Does not remove the member resource itself, only its id from the member_ids of the parent. |
| member_tests find_parents | Find the single parent resource of one of the members. |
| member_tests find_members | Find the 100 members of the parent resource. |
| alternate_id_tests create | Create a resource with two alternate_ids. |
| alternate_id_tests find | Find a resource using one of its alternate_ids. |
| Adapter | iterations/s | std.dev. | iterations | time (s) |
|---|---|---|---|---|
| basic_metadata_tests create_stub | ||||
| active_record_mysql | 35.725 | (±39.8%) | 358 | 10.021 |
| active_record_mysql_net | 14.672 | (±37.1%) | 147 | 10.019 |
| active_record_postgres | 63.501 | (±29.7%) | 637 | 10.031 |
| active_record_postgres_net | 41.284 | (±30.3%) | 420 | 10.173 |
| active_record_sqlite | 19.716 | (±31.8%) | 198 | 10.043 |
| postgres | 87.306 | (±12.3%) | 880 | 10.079 |
| postgres_net | 73.497 | (±21.1%) | 742 | 10.096 |
| fedora | 101.302 | (± 8.8%) | 1020 | 10.069 |
| fedora_net | 87.715 | (± 7.9%) | 880 | 10.032 |
| solr | 281.265 | (± 7.1%) | 2835 | 10.079 |
| solr_net | 188.585 | (±13.5%) | 1890 | 10.022 |
| redis | 3686.666 | (± 9.6%) | 37044 | 10.048 |
| memory | 5754.085 | (±12.8%) | 57618 | 10.013 |
| basic_metadata_tests create_with_text | ||||
| active_record_mysql | 31.366 | (±36.8%) | 318 | 10.138 |
| active_record_mysql_net | 13.159 | (±27.1%) | 132 | 10.031 |
| active_record_postgres | 64.194 | (±27.2%) | 645 | 10.048 |
| active_record_postgres_net | 37.783 | (±44.0%) | 378 | 10.005 |
| active_record_sqlite | 17.240 | (±36.8%) | 173 | 10.035 |
| postgres | 75.948 | (±18.4%) | 760 | 10.007 |
| postgres_net | 42.853 | (±22.0%) | 429 | 10.011 |
| fedora | 85.873 | (±16.4%) | 864 | 10.061 |
| fedora_net | 80.774 | (± 7.4%) | 812 | 10.053 |
| solr | 226.975 | (±10.7%) | 2280 | 10.045 |
| solr_net | 163.235 | (±12.1%) | 1638 | 10.035 |
| redis | 3147.589 | (±21.1%) | 31730 | 10.081 |
| memory | 5482.670 | (±16.2%) | 55620 | 10.145 |
| basic_metadata_tests create_with_many_fields | ||||
| active_record_mysql | 33.780 | (±33.9%) | 338 | 10.006 |
| active_record_mysql_net | 14.271 | (±36.4%) | 143 | 10.020 |
| active_record_postgres | 61.575 | (±31.9%) | 616 | 10.004 |
| active_record_postgres_net | 33.532 | (±27.4%) | 336 | 10.020 |
| active_record_sqlite | 15.797 | (±37.2%) | 159 | 10.065 |
| postgres | 75.383 | (±31.9%) | 755 | 10.016 |
| postgres_net | 40.589 | (±28.6%) | 408 | 10.052 |
| fedora | 33.890 | (±23.3%) | 352 | 10.386 |
| fedora_net | 41.433 | (± 7.2%) | 416 | 10.040 |
| solr | 170.094 | (±16.6%) | 1701 | 10.000 |
| solr_net | 131.149 | (± 7.6%) | 1320 | 10.065 |
| redis | 2313.812 | (±12.2%) | 23098 | 9.983 |
| memory | 3013.171 | (±15.7%) | 30060 | 9.976 |
| basic_metadata_tests update_stub | ||||
| active_record_mysql | 36.999 | (±41.7%) | 371 | 10.027 |
| active_record_mysql_net | 14.630 | (±34.1%) | 147 | 10.048 |
| active_record_postgres | 66.713 | (±42.8%) | 668 | 10.013 |
| active_record_postgres_net | 32.343 | (±31.4%) | 324 | 10.018 |
| active_record_sqlite | 19.541 | (±32.2%) | 196 | 10.030 |
| postgres | 85.792 | (±25.0%) | 858 | 10.001 |
| postgres_net | 56.802 | (±31.9%) | 568 | 10.000 |
| fedora | 42.376 | (±18.5%) | 424 | 10.006 |
| fedora_net | 46.423 | (± 6.4%) | 468 | 10.081 |
| solr | 127.289 | (±29.0%) | 1273 | 10.001 |
| solr_net | 82.838 | (±24.0%) | 832 | 10.044 |
| redis | 5158.887 | (± 8.8%) | 51388 | 9.961 |
| memory | 8315.620 | (± 8.1%) | 82580 | 9.931 |
| basic_metadata_tests update_with_text | ||||
| active_record_mysql | 36.444 | (±39.2%) | 365 | 10.015 |
| active_record_mysql_net | 14.554 | (±31.1%) | 146 | 10.032 |
| active_record_postgres | 78.298 | (±38.4%) | 783 | 10.000 |
| active_record_postgres_net | 31.226 | (±31.2%) | 314 | 10.056 |
| active_record_sqlite | 17.284 | (±35.1%) | 173 | 10.009 |
| postgres | 70.973 | (±39.9%) | 713 | 10.046 |
| postgres_net | 42.622 | (±20.0%) | 430 | 10.089 |
| fedora | 41.496 | (±19.0%) | 415 | 10.001 |
| fedora_net | 44.374 | (± 6.7%) | 448 | 10.096 |
| solr | 86.431 | (±13.4%) | 865 | 10.008 |
| solr_net | 61.657 | (±11.2%) | 618 | 10.023 |
| redis | 3250.157 | (±14.4%) | 33578 | 10.331 |
| memory | 7410.607 | (±14.0%) | 73780 | 9.956 |
| basic_metadata_tests update_with_many_fields | ||||
| active_record_mysql | 29.283 | (±48.1%) | 293 | 10.006 |
| active_record_mysql_net | 13.649 | (±31.0%) | 137 | 10.038 |
| active_record_postgres | 64.193 | (±30.8%) | 642 | 10.001 |
| active_record_postgres_net | 32.485 | (±29.1%) | 327 | 10.066 |
| active_record_sqlite | 15.778 | (±39.3%) | 160 | 10.141 |
| postgres | 78.652 | (±33.4%) | 789 | 10.032 |
| postgres_net | 41.206 | (±28.8%) | 420 | 10.193 |
| fedora | 16.642 | (±17.2%) | 168 | 10.095 |
| fedora_net | 15.548 | (± 6.4%) | 156 | 10.034 |
| solr | 46.084 | (±27.6%) | 461 | 10.003 |
| solr_net | 32.587 | (±18.0%) | 328 | 10.065 |
| redis | 3015.899 | (± 8.2%) | 30081 | 9.974 |
| memory | 5035.339 | (± 8.6%) | 49759 | 9.882 |
| basic_metadata_tests delete_stubs | ||||
| active_record_mysql | 37.392 | (±43.4%) | 374 | 10.002 |
| active_record_mysql_net | 17.640 | (±39.9%) | 177 | 10.034 |
| active_record_postgres | 97.272 | (±38.5%) | 973 | 10.003 |
| active_record_postgres_net | 46.634 | (±42.5%) | 471 | 10.100 |
| active_record_sqlite | 19.096 | (±32.9%) | 191 | 10.002 |
| postgres | 62.159 | (±41.2%) | 622 | 10.007 |
| postgres_net | 69.326 | (±25.4%) | 695 | 10.025 |
| fedora | 58.376 | (±18.6%) | 584 | 10.004 |
| fedora_net | 53.326 | (± 9.3%) | 535 | 10.033 |
| solr | 38.785 | (± 7.7%) | 388 | 10.004 |
| solr_net | 35.781 | (± 2.8%) | 360 | 10.061 |
| redis | 12920.103 | (±54.5%) | 128638 | 9.956 |
| memory | 22538.412 | (±139.8%) | 223032 | 9.896 |
| basic_metadata_tests delete_with_many_fields | ||||
| active_record_mysql | 34.842 | (±44.0%) | 349 | 10.017 |
| active_record_mysql_net | 17.488 | (±43.9%) | 175 | 10.007 |
| active_record_postgres | 94.522 | (±36.5%) | 945 | 9.998 |
| active_record_postgres_net | 39.417 | (±48.8%) | 399 | 10.122 |
| active_record_sqlite | 17.869 | (±36.5%) | 179 | 10.017 |
| postgres | 77.900 | (±36.8%) | 779 | 10.000 |
| postgres_net | 42.907 | (±47.4%) | 435 | 10.138 |
| fedora | 38.456 | (±17.6%) | 385 | 10.011 |
| fedora_net | 40.628 | (± 7.3%) | 408 | 10.042 |
| solr | 38.667 | (± 7.7%) | 387 | 10.009 |
| solr_net | 35.835 | (± 2.8%) | 360 | 10.046 |
| redis | 12247.333 | (±53.1%) | 122002 | 9.962 |
| memory | 20831.860 | (±183.0%) | 206876 | 9.931 |
| member_tests create_parent | ||||
| active_record_mysql | 11.899 | (±16.4%) | 119 | 10.001 |
| active_record_mysql_net | 5.706 | (±17.1%) | 58 | 10.166 |
| active_record_postgres | 12.293 | (±26.7%) | 123 | 10.006 |
| active_record_postgres_net | 4.953 | (±19.4%) | 50 | 10.094 |
| active_record_sqlite | 4.501 | (±58.1%) | 46 | 10.220 |
| postgres | 60.369 | (±27.2%) | 604 | 10.005 |
| postgres_net | 34.343 | (±25.2%) | 351 | 10.221 |
| fedora | 3.369 | (±27.8%) | 34 | 10.091 |
| fedora_net | 3.391 | (± 0.0%) | 34 | 10.028 |
| solr | 67.393 | (±14.2%) | 677 | 10.046 |
| solr_net | 56.934 | (± 7.0%) | 570 | 10.012 |
| redis | 1271.680 | (± 8.8%) | 12705 | 9.991 |
| memory | 2003.266 | (±12.0%) | 20002 | 9.985 |
| member_tests reload_parent | ||||
| active_record_mysql | 1108.477 | (±17.1%) | 11075 | 9.991 |
| active_record_mysql_net | 350.627 | (± 5.4%) | 3535 | 10.082 |
| active_record_postgres | 1245.528 | (±12.4%) | 12443 | 9.990 |
| active_record_postgres_net | 317.207 | (±11.2%) | 3196 | 10.075 |
| active_record_sqlite | 1372.456 | (±13.2%) | 13710 | 9.989 |
| postgres | 1072.369 | (±16.2%) | 10715 | 9.992 |
| postgres_net | 284.285 | (± 7.3%) | 2871 | 10.099 |
| fedora | 3.862 | (±25.4%) | 39 | 10.098 |
| fedora_net | 3.846 | (± 0.0%) | 39 | 10.140 |
| solr | 187.011 | (±14.0%) | 1870 | 9.999 |
| solr_net | 76.444 | (±11.6%) | 776 | 10.151 |
| redis | 4860.895 | (±17.4%) | 48459 | 9.969 |
| memory | 450946.020 | (±14.5%) | 4132300 | 9.164 |
| member_tests update_child | ||||
| active_record_mysql | 27.529 | (±51.1%) | 276 | 10.026 |
| active_record_mysql_net | 16.362 | (±28.2%) | 164 | 10.023 |
| active_record_postgres | 75.433 | (±43.5%) | 755 | 10.009 |
| active_record_postgres_net | 36.269 | (±28.8%) | 364 | 10.036 |
| active_record_sqlite | 18.231 | (±34.9%) | 183 | 10.038 |
| postgres | 82.180 | (±28.2%) | 822 | 10.002 |
| postgres_net | 64.654 | (±26.1%) | 654 | 10.115 |
| fedora | 48.558 | (±18.0%) | 486 | 10.009 |
| fedora_net | 51.383 | (± 5.8%) | 515 | 10.023 |
| solr | 104.087 | (±12.1%) | 1041 | 10.001 |
| solr_net | 71.153 | (±11.1%) | 714 | 10.035 |
| redis | 5271.741 | (±10.8%) | 52512 | 9.961 |
| memory | 9051.413 | (±11.8%) | 89054 | 9.839 |
| member_tests update_parent | ||||
| active_record_mysql | 15.899 | (±29.3%) | 159 | 10.001 |
| active_record_mysql_net | 12.593 | (±15.3%) | 126 | 10.005 |
| active_record_postgres | 50.135 | (±35.7%) | 503 | 10.033 |
| active_record_postgres_net | 27.482 | (±33.3%) | 276 | 10.043 |
| active_record_sqlite | 9.829 | (±35.6%) | 99 | 10.073 |
| postgres | 69.451 | (±18.9%) | 695 | 10.007 |
| postgres_net | 37.940 | (±19.6%) | 380 | 10.016 |
| fedora | 1.147 | (± 0.0%) | 12 | 10.464 |
| fedora_net | 1.272 | (± 0.0%) | 13 | 10.224 |
| solr | 62.078 | (±12.2%) | 621 | 10.004 |
| solr_net | 49.507 | (± 8.0%) | 500 | 10.099 |
| redis | 1700.916 | (±11.4%) | 16987 | 9.987 |
| memory | 3725.912 | (± 6.9%) | 37099 | 9.957 |
| member_tests one_more_page | ||||
| active_record_mysql | 10.672 | (±25.7%) | 107 | 10.026 |
| active_record_mysql_net | 7.532 | (±12.9%) | 76 | 10.090 |
| active_record_postgres | 25.552 | (±32.8%) | 256 | 10.019 |
| active_record_postgres_net | 16.270 | (±36.8%) | 163 | 10.018 |
| active_record_sqlite | 6.305 | (±28.0%) | 64 | 10.151 |
| postgres | 60.865 | (±24.1%) | 609 | 10.006 |
| postgres_net | 36.595 | (±19.9%) | 368 | 10.056 |
| fedora | 1.314 | (± 0.0%) | 14 | 10.658 |
| fedora_net | 1.262 | (± 0.0%) | 13 | 10.303 |
| solr | 60.016 | (±14.0%) | 601 | 10.014 |
| solr_net | 48.074 | (± 8.3%) | 484 | 10.068 |
| redis | 1529.556 | (± 7.8%) | 15278 | 9.989 |
| memory | 3129.050 | (± 6.6%) | 31136 | 9.951 |
| member_tests remove_page | ||||
| active_record_mysql | 13.978 | (±51.6%) | 140 | 10.016 |
| active_record_mysql_net | 7.566 | (±25.3%) | 76 | 10.044 |
| active_record_postgres | 36.896 | (±34.3%) | 369 | 10.001 |
| active_record_postgres_net | 19.177 | (±32.9%) | 192 | 10.012 |
| active_record_sqlite | 7.767 | (±22.8%) | 78 | 10.043 |
| postgres | 81.269 | (±29.7%) | 813 | 10.004 |
| postgres_net | 40.939 | (±24.6%) | 410 | 10.015 |
| fedora | 1.448 | (± 0.0%) | 15 | 10.359 |
| fedora_net | 1.403 | (± 0.0%) | 15 | 10.691 |
| solr | 93.567 | (±12.3%) | 936 | 10.004 |
| solr_net | 63.749 | (±13.8%) | 640 | 10.039 |
| redis | 3489.415 | (± 8.7%) | 34782 | 9.968 |
| memory | 5998.075 | (± 8.1%) | 59459 | 9.913 |
| member_tests find_parents | ||||
| active_record_mysql | 422.106 | (± 9.2%) | 4220 | 9.997 |
| active_record_mysql_net | 179.551 | (± 6.7%) | 1802 | 10.036 |
| active_record_postgres | 374.019 | (± 8.8%) | 3739 | 9.997 |
| active_record_postgres_net | 193.848 | (± 5.7%) | 1957 | 10.096 |
| active_record_sqlite | 494.997 | (± 7.1%) | 4948 | 9.996 |
| postgres | 1867.333 | (± 8.6%) | 18645 | 9.985 |
| postgres_net | 606.951 | (± 8.8%) | 6102 | 10.054 |
| fedora | 4.443 | (±22.0%) | 45 | 10.129 |
| fedora_net | 4.289 | (± 0.0%) | 43 | 10.026 |
| solr | 532.160 | (±10.1%) | 5318 | 9.993 |
| solr_net | 251.998 | (±10.6%) | 2525 | 10.020 |
| redis | 0.257 | (± 0.0%) | 3 | 11.662 |
| memory | 1.147 | (± 0.0%) | 12 | 10.458 |
| member_tests find_members | ||||
| active_record_mysql | 42.495 | (±13.0%) | 425 | 10.001 |
| active_record_mysql_net | 30.666 | (± 9.7%) | 308 | 10.044 |
| active_record_postgres | 43.637 | (±14.5%) | 438 | 10.037 |
| active_record_postgres_net | 25.224 | (±11.8%) | 254 | 10.070 |
| active_record_sqlite | 40.584 | (±13.6%) | 406 | 10.004 |
| postgres | 65.886 | (±12.2%) | 659 | 10.002 |
| postgres_net | 48.289 | (±12.2%) | 484 | 10.023 |
| fedora | 0.748 | (± 0.0%) | 8 | 10.688 |
| fedora_net | 0.617 | (± 0.0%) | 7 | 11.349 |
| solr | 43.021 | (±13.0%) | 431 | 10.018 |
| solr_net | 32.824 | (± 6.1%) | 330 | 10.054 |
| redis | 169.611 | (±10.7%) | 1696 | 9.999 |
| memory | 6910.329 | (± 5.3%) | 68327 | 9.888 |
| alternate_id_tests create | ||||
| active_record_mysql | 30.971 | (±38.6%) | 312 | 10.074 |
| active_record_mysql_net | 16.075 | (±29.3%) | 161 | 10.016 |
| active_record_postgres | 66.726 | (±25.4%) | 670 | 10.041 |
| active_record_postgres_net | 35.546 | (±29.3%) | 356 | 10.015 |
| active_record_sqlite | 16.489 | (±34.7%) | 165 | 10.007 |
| postgres | 78.760 | (±19.4%) | 792 | 10.056 |
| postgres_net | 39.199 | (±29.1%) | 393 | 10.026 |
| fedora | 13.941 | (± 7.1%) | 140 | 10.042 |
| fedora_net | 10.845 | (± 9.2%) | 109 | 10.050 |
| solr | 101.836 | (±23.3%) | 1026 | 10.075 |
| solr_net | 81.808 | (±27.4%) | 825 | 10.085 |
| memory | 3737.772 | (±10.2%) | 37673 | 10.079 |
| alternate_id_tests find | ||||
| active_record_mysql | 944.415 | (± 6.1%) | 9486 | 10.044 |
| active_record_mysql_net | 286.443 | (± 8.7%) | 2886 | 10.075 |
| active_record_postgres | 956.204 | (± 6.4%) | 9595 | 10.034 |
| active_record_postgres_net | 331.951 | (± 8.4%) | 3325 | 10.017 |
| active_record_sqlite | 1185.238 | (± 6.2%) | 11918 | 10.055 |
| postgres | 1894.218 | (± 6.4%) | 19089 | 10.078 |
| postgres_net | 595.796 | (±11.1%) | 5995 | 10.062 |
| fedora | 37.753 | (±10.5%) | 378 | 10.012 |
| fedora_net | 31.489 | (± 6.3%) | 316 | 10.035 |
| solr | 326.415 | (± 8.2%) | 3300 | 10.110 |
| solr_net | 134.253 | (±11.8%) | 1352 | 10.071 |
| memory | 11.822 | (± 8.4%) | 119 | 10.066 |