Skip to content
This repository has been archived by the owner on Jul 19, 2023. It is now read-only.

Add ability to query local blocks #127

Merged
merged 6 commits into from
Aug 24, 2022
Merged

Add ability to query local blocks #127

merged 6 commits into from
Aug 24, 2022

Conversation

simonswine
Copy link
Collaborator

@simonswine simonswine commented Jul 27, 2022 •
edited
Loading

The block querier allows to query parquet files locally (and by using a BucketReader, also from a object store bucket)

The current implementation will load all profiles from the matching series finger prints in full, before checking if the timestamps matches on RowGroup or Profile level. (This is something I would like to look at outside of this PR).

This also adds a CLI firetool to list blocks. (I also plan on integrating parquet-tool in it):

$ go run ./cmd/firetool blocks list
+----------------------------+---------------------------+---------------------------+--------------+-------------------------+--------------------------+--------------------------+--------------------------+--------------------------+
|          BLOCK ID          |          MINTIME          |          MAXTIME          |   DURATION   |        PROFILES         |       STACKTRACES        |        LOCATIONS         |        FUNCTIONS         |         STRINGS          |
+----------------------------+---------------------------+---------------------------+--------------+-------------------------+--------------------------+--------------------------+--------------------------+--------------------------+
| 01GAHA0DN8J7DGH2TA6R603Y76 | 2022-08-15T18:49:45+01:00 | 2022-08-15T18:49:59+01:00 | 14.525s      | 38 (12 kB in 1 RGs)     | 264 (30 kB in 1 RGs)     | 709 (31 kB in 1 RGs)     | 578 (24 kB in 1 RGs)     | 835 (50 kB in 1 RGs)     |
| 01GAHA0WA07SFVE39T2Y37F2J6 | 2022-08-15T18:49:59+01:00 | 2022-08-15T18:50:14+01:00 | 15.244s      | 37 (11 kB in 1 RGs)     | 330 (38 kB in 1 RGs)     | 783 (34 kB in 1 RGs)     | 612 (25 kB in 1 RGs)     | 874 (53 kB in 1 RGs)     |
| 01GAHA1AYRKVPRHFSM70BRQQ0R | 2022-08-15T18:50:15+01:00 | 2022-08-15T18:50:29+01:00 | 14.526s      | 38 (12 kB in 1 RGs)     | 343 (40 kB in 1 RGs)     | 780 (34 kB in 1 RGs)     | 606 (25 kB in 1 RGs)     | 868 (53 kB in 1 RGs)     |
| 01GAHA1SKGEG4DG2SAXT5TN5PA | 2022-08-15T18:50:29+01:00 | 2022-08-15T18:50:44+01:00 | 15.245s      | 37 (11 kB in 1 RGs)     | 389 (44 kB in 1 RGs)     | 859 (37 kB in 1 RGs)     | 656 (27 kB in 1 RGs)     | 935 (57 kB in 1 RGs)     |
| 01GAHA2888JG3FQW3C24JFTTJ0 | 2022-08-15T18:50:45+01:00 | 2022-08-15T18:50:59+01:00 | 14.523s      | 38 (12 kB in 1 RGs)     | 401 (48 kB in 1 RGs)     | 867 (38 kB in 1 RGs)     | 662 (27 kB in 1 RGs)     | 936 (56 kB in 1 RGs)     |
| 01GAHA2PX0P85FJ2BRDEVTFC7R | 2022-08-15T18:50:59+01:00 | 2022-08-15T18:51:14+01:00 | 15.243s      | 37 (12 kB in 1 RGs)     | 434 (51 kB in 1 RGs)     | 911 (40 kB in 1 RGs)     | 684 (28 kB in 1 RGs)     | 964 (58 kB in 1 RGs)     |
| 01GAHA35HRNN9XF6EWXSM47VSJ | 2022-08-15T18:51:15+01:00 | 2022-08-15T18:51:18+01:00 | 3.244s       | 10 (5.0 kB in 1 RGs)    | 396 (46 kB in 1 RGs)     | 803 (35 kB in 1 RGs)     | 612 (25 kB in 1 RGs)     | 878 (55 kB in 1 RGs)     |
| 01GAHE0FA8SX9FSN4ZGBZBEV9T | 2022-08-15T19:59:46+01:00 | 2022-08-15T19:59:59+01:00 | 12.851s      | 34 (26 kB in 7 RGs)     | 207 (37 kB in 42 RGs)    | 621 (124 kB in 125 RGs)  | 538 (92 kB in 108 RGs)   | 795 (106 kB in 159 RGs)  |
| 01GAHE11W0JPC0SPBE4CDGT1K1 | 2022-08-15T19:59:59+01:00 | 2022-08-15T20:04:59+01:00 | 4m59.244s    | 750 (586 kB in 150 RGs) | 1430 (295 kB in 286 RGs) | 2280 (456 kB in 456 RGs) | 1228 (210 kB in 246 RGs) | 1618 (209 kB in 324 RGs) |
| 01GAHEA6V06PQNNTKGGZ9G8A0W | 2022-08-15T20:04:59+01:00 | 2022-08-15T20:09:59+01:00 | 4m59.245s    | 750 (600 kB in 150 RGs) | 1887 (409 kB in 378 RGs) | 2664 (534 kB in 533 RGs) | 1341 (229 kB in 269 RGs) | 1757 (228 kB in 352 RGs) |
| 01GAHEKBT0FW8YF6751BEJVZCR | 2022-08-15T20:09:59+01:00 | 2022-08-15T20:14:59+01:00 | 4m59.243s    | 750 (596 kB in 150 RGs) | 2090 (458 kB in 418 RGs) | 2757 (553 kB in 552 RGs) | 1346 (230 kB in 270 RGs) | 1758 (227 kB in 352 RGs) |
| 01GAHEWGS0FCFTEW3VCXV03SX4 | 2022-08-15T20:14:59+01:00 | 2022-08-15T20:19:59+01:00 | 4m59.244s    | 750 (601 kB in 150 RGs) | 2257 (498 kB in 452 RGs) | 2868 (576 kB in 574 RGs) | 1412 (241 kB in 283 RGs) | 1829 (236 kB in 366 RGs) |
| 01GAHF5NR01W1Q3G79C4E3SYRT | 2022-08-15T20:19:59+01:00 | 2022-08-15T20:24:59+01:00 | 4m59.245s    | 750 (600 kB in 150 RGs) | 2370 (523 kB in 474 RGs) | 2952 (593 kB in 591 RGs) | 1441 (246 kB in 289 RGs) | 1860 (240 kB in 372 RGs) |
| 01GAHFETQ0YQ9QSJ1GV36N08FG | 2022-08-15T20:24:59+01:00 | 2022-08-15T20:29:59+01:00 | 4m59.245s    | 750 (601 kB in 150 RGs) | 2513 (557 kB in 503 RGs) | 3039 (610 kB in 608 RGs) | 1444 (246 kB in 289 RGs) | 1863 (240 kB in 373 RGs) |
| 01GAHFQZP0FWXD1MT3096CKEWY | 2022-08-15T20:29:59+01:00 | 2022-08-15T20:34:59+01:00 | 4m59.244s    | 750 (600 kB in 150 RGs) | 2527 (562 kB in 506 RGs) | 2973 (597 kB in 595 RGs) | 1423 (243 kB in 285 RGs) | 1841 (238 kB in 369 RGs) |
| 01GAHG14N0ZJGPD9Q4XND1BCK1 | 2022-08-15T20:34:59+01:00 | 2022-08-15T20:39:59+01:00 | 4m59.244s    | 750 (603 kB in 150 RGs) | 2636 (584 kB in 528 RGs) | 3059 (615 kB in 612 RGs) | 1471 (251 kB in 295 RGs) | 1897 (244 kB in 380 RGs) |
| 01GAHGA9M07WP9FX53V37P1WYZ | 2022-08-15T20:39:59+01:00 | 2022-08-15T20:44:59+01:00 | 4m59.243s    | 750 (602 kB in 150 RGs) | 2573 (572 kB in 515 RGs) | 3028 (608 kB in 606 RGs) | 1504 (256 kB in 301 RGs) | 1936 (250 kB in 388 RGs) |
| 01GAHGKEK0F77ADJA4EX2G63XY | 2022-08-15T20:44:59+01:00 | 2022-08-15T20:49:59+01:00 | 4m59.243s    | 750 (598 kB in 150 RGs) | 2590 (574 kB in 518 RGs) | 2973 (596 kB in 595 RGs) | 1463 (250 kB in 293 RGs) | 1890 (245 kB in 378 RGs) |
| 01GAHGWKJ0ASEF8TQN9XFVNYG5 | 2022-08-15T20:49:59+01:00 | 2022-08-15T20:54:59+01:00 | 4m59.244s    | 750 (602 kB in 150 RGs) | 2564 (565 kB in 513 RGs) | 2968 (596 kB in 594 RGs) | 1485 (253 kB in 297 RGs) | 1917 (248 kB in 384 RGs) |
| 01GAHH5RH0VEDKMC197HGR8R6J | 2022-08-15T20:54:59+01:00 | 2022-08-15T20:59:59+01:00 | 4m59.244s    | 750 (600 kB in 150 RGs) | 2620 (583 kB in 524 RGs) | 2931 (589 kB in 587 RGs) | 1465 (250 kB in 293 RGs) | 1892 (245 kB in 379 RGs) |
| 01GAHHEXG1992RZ754ZKV7TNP9 | 2022-08-15T20:59:59+01:00 | 2022-08-15T21:04:59+01:00 | 4m59.244s    | 750 (603 kB in 150 RGs) | 2684 (598 kB in 537 RGs) | 3015 (605 kB in 603 RGs) | 1487 (254 kB in 298 RGs) | 1920 (248 kB in 384 RGs) |
| 01GAHHR2F0TY7NMEKZMZASSE5R | 2022-08-15T21:04:59+01:00 | 2022-08-15T21:09:59+01:00 | 4m59.245s    | 750 (597 kB in 150 RGs) | 2448 (540 kB in 490 RGs) | 2718 (546 kB in 544 RGs) | 1410 (240 kB in 282 RGs) | 1830 (238 kB in 366 RGs) |
| 01GAHJ17E0V3Z71570MFX12T1D | 2022-08-15T21:09:59+01:00 | 2022-08-15T21:14:59+01:00 | 4m59.244s    | 719 (584 kB in 144 RGs) | 2684 (594 kB in 537 RGs) | 2973 (597 kB in 595 RGs) | 1489 (254 kB in 298 RGs) | 1911 (247 kB in 383 RGs) |
| 01GAHJACD1SV81YP7PNGV0TP1T | 2022-08-15T21:14:39+01:00 | 2022-08-15T21:19:59+01:00 | 5m19.246s    | 752 (605 kB in 151 RGs) | 2687 (597 kB in 538 RGs) | 2888 (580 kB in 578 RGs) | 1454 (248 kB in 291 RGs) | 1875 (243 kB in 375 RGs) |
| 01GAHJKHC0ZEPN30MY9QZG7DDM | 2022-08-15T21:19:59+01:00 | 2022-08-15T21:24:59+01:00 | 4m59.244s    | 750 (600 kB in 150 RGs) | 2744 (607 kB in 549 RGs) | 2889 (580 kB in 578 RGs) | 1431 (244 kB in 287 RGs) | 1848 (239 kB in 370 RGs) |
| 01GAHJWPB0J5D2QR6B1SMAR318 | 2022-08-15T21:24:59+01:00 | 2022-08-15T21:29:59+01:00 | 4m59.243s    | 750 (603 kB in 150 RGs) | 2759 (611 kB in 552 RGs) | 2947 (592 kB in 590 RGs) | 1462 (250 kB in 293 RGs) | 1889 (245 kB in 378 RGs) |
| 01GAHK5VA1MW7P4E70NEFR4ZED | 2022-08-15T21:29:59+01:00 | 2022-08-15T21:34:59+01:00 | 4m59.244s    | 750 (602 kB in 150 RGs) | 2816 (623 kB in 564 RGs) | 2975 (597 kB in 595 RGs) | 1484 (253 kB in 297 RGs) | 1918 (249 kB in 384 RGs) |
| 01GAHKF090CW1Y6M7P1AM70HSK | 2022-08-15T21:34:59+01:00 | 2022-08-15T21:39:59+01:00 | 4m59.244s    | 731 (598 kB in 147 RGs) | 2631 (582 kB in 527 RGs) | 2809 (564 kB in 562 RGs) | 1471 (251 kB in 295 RGs) | 1901 (246 kB in 381 RGs) |
| 01GAHKR580XR0ZX4YKFNFSY46R | 2022-08-15T21:39:59+01:00 | 2022-08-15T21:44:59+01:00 | 4m59.244s    | 666 (539 kB in 134 RGs) | 2614 (579 kB in 523 RGs) | 2702 (543 kB in 541 RGs) | 1435 (245 kB in 287 RGs) | 1856 (242 kB in 372 RGs) |
| 01GAHM1A70R6XHA5HRYB9VW3WJ | 2022-08-15T21:44:59+01:00 | 2022-08-15T21:49:59+01:00 | 4m59.245s    | 750 (611 kB in 150 RGs) | 2865 (631 kB in 573 RGs) | 2935 (589 kB in 587 RGs) | 1464 (250 kB in 293 RGs) | 1895 (246 kB in 379 RGs) |
| 01GAHMAF60XRE86D1XMKTX3BF7 | 2022-08-15T21:49:59+01:00 | 2022-08-15T21:54:59+01:00 | 4m59.244s    | 750 (606 kB in 150 RGs) | 2716 (602 kB in 544 RGs) | 2779 (558 kB in 556 RGs) | 1442 (246 kB in 289 RGs) | 1869 (243 kB in 374 RGs) |
| 01GAHMKM501B6M22TRTXAZ8TBA | 2022-08-15T21:54:59+01:00 | 2022-08-15T21:59:59+01:00 | 4m59.244s    | 750 (611 kB in 150 RGs) | 3050 (672 kB in 610 RGs) | 3080 (619 kB in 616 RGs) | 1491 (255 kB in 299 RGs) | 1933 (252 kB in 387 RGs) |
| 01GAHMWS40MCDBAP1E2HTCD3SK | 2022-08-15T21:59:59+01:00 | 2022-08-15T22:04:59+01:00 | 4m59.244s    | 750 (606 kB in 150 RGs) | 2895 (645 kB in 579 RGs) | 2903 (584 kB in 581 RGs) | 1457 (249 kB in 292 RGs) | 1888 (246 kB in 378 RGs) |
| 01GAHN5Y31SMWQPQS5FEACNKFD | 2022-08-15T22:04:59+01:00 | 2022-08-15T22:09:59+01:00 | 4m59.244s    | 750 (607 kB in 150 RGs) | 2832 (627 kB in 567 RGs) | 2873 (577 kB in 575 RGs) | 1469 (250 kB in 294 RGs) | 1904 (248 kB in 381 RGs) |
| 01GAHNF320Z7G9M0Z29WASACWY | 2022-08-15T22:09:59+01:00 | 2022-08-15T22:14:59+01:00 | 4m59.244s    | 750 (604 kB in 150 RGs) | 2892 (641 kB in 579 RGs) | 2846 (572 kB in 570 RGs) | 1437 (245 kB in 288 RGs) | 1862 (243 kB in 373 RGs) |
| 01GAHNR810FA89B0J0ZA58DZZZ | 2022-08-15T22:14:59+01:00 | 2022-08-15T22:19:59+01:00 | 4m59.244s    | 750 (609 kB in 150 RGs) | 2904 (640 kB in 581 RGs) | 2898 (583 kB in 580 RGs) | 1488 (254 kB in 298 RGs) | 1922 (250 kB in 385 RGs) |
| 01GAHP1D00DFQ4PJME5QSDX5KJ | 2022-08-15T22:19:59+01:00 | 2022-08-15T22:24:59+01:00 | 4m59.244s    | 750 (606 kB in 150 RGs) | 2988 (663 kB in 598 RGs) | 2948 (593 kB in 590 RGs) | 1502 (256 kB in 301 RGs) | 1943 (254 kB in 389 RGs) |
| 01GAHPAHZ0JN44DYZKGZ0WFA3V | 2022-08-15T22:24:59+01:00 | 2022-08-15T22:29:59+01:00 | 4m59.244s    | 750 (606 kB in 150 RGs) | 2950 (658 kB in 590 RGs) | 2877 (578 kB in 576 RGs) | 1455 (248 kB in 291 RGs) | 1884 (246 kB in 377 RGs) |
| 01GAHPKPY0K8Q04N5SCEDJFWTD | 2022-08-15T22:29:59+01:00 | 2022-08-15T22:34:59+01:00 | 4m59.245s    | 750 (602 kB in 150 RGs) | 2980 (664 kB in 596 RGs) | 2908 (584 kB in 582 RGs) | 1471 (251 kB in 295 RGs) | 1904 (248 kB in 381 RGs) |
| 01GAHPWVX0FG9B33VMQSB7533H | 2022-08-15T22:34:59+01:00 | 2022-08-15T22:39:59+01:00 | 4m59.243s    | 750 (605 kB in 150 RGs) | 3056 (682 kB in 612 RGs) | 2936 (591 kB in 588 RGs) | 1479 (252 kB in 296 RGs) | 1919 (250 kB in 384 RGs) |
| 01GAHQ60W0H2KKEFPJE83WQZM4 | 2022-08-15T22:39:59+01:00 | 2022-08-15T22:44:59+01:00 | 4m59.243s    | 750 (610 kB in 150 RGs) | 3071 (683 kB in 615 RGs) | 2960 (594 kB in 592 RGs) | 1491 (255 kB in 299 RGs) | 1924 (250 kB in 385 RGs) |
| 01GAHQF5V00G9Q1SZ137FJTST3 | 2022-08-15T22:44:59+01:00 | 2022-08-15T22:49:59+01:00 | 4m59.245s    | 750 (614 kB in 150 RGs) | 3334 (741 kB in 667 RGs) | 3200 (643 kB in 640 RGs) | 1510 (257 kB in 302 RGs) | 1945 (253 kB in 389 RGs) |
| 01GAHQRAT0CET4H6RNMXFPHX8E | 2022-08-15T22:49:59+01:00 | 2022-08-15T22:54:59+01:00 | 4m59.244s    | 750 (627 kB in 150 RGs) | 3907 (881 kB in 782 RGs) | 3705 (744 kB in 741 RGs) | 1639 (280 kB in 328 RGs) | 2101 (273 kB in 421 RGs) |
| 01GAHR1FS0WVCB81EGRGTBAGZ2 | 2022-08-15T22:54:59+01:00 | 2022-08-15T22:59:59+01:00 | 4m59.245s    | 750 (609 kB in 150 RGs) | 3183 (706 kB in 637 RGs) | 3031 (610 kB in 607 RGs) | 1499 (256 kB in 300 RGs) | 1941 (253 kB in 389 RGs) |
| 01GAHRAMR03RXAEFXGVG29RWP7 | 2022-08-15T22:59:59+01:00 | 2022-08-15T23:04:59+01:00 | 4m59.245s    | 750 (611 kB in 150 RGs) | 3167 (711 kB in 634 RGs) | 2958 (594 kB in 592 RGs) | 1473 (251 kB in 295 RGs) | 1912 (249 kB in 383 RGs) |
| 01GAHRKSQ0HW11HZFVV84SKEHF | 2022-08-15T23:04:59+01:00 | 2022-08-15T23:09:59+01:00 | 4m59.244s    | 750 (613 kB in 150 RGs) | 3155 (706 kB in 631 RGs) | 2983 (599 kB in 597 RGs) | 1488 (254 kB in 298 RGs) | 1924 (250 kB in 385 RGs) |
| 01GAHRWYP001V7BKGX5MPPMBTV | 2022-08-15T23:09:59+01:00 | 2022-08-15T23:14:59+01:00 | 4m59.245s    | 750 (612 kB in 150 RGs) | 3178 (708 kB in 636 RGs) | 2983 (600 kB in 597 RGs) | 1488 (254 kB in 298 RGs) | 1918 (250 kB in 384 RGs) |
| 01GAHS63N0HKDFKE57HQ5EQAJX | 2022-08-15T23:14:59+01:00 | 2022-08-15T23:19:59+01:00 | 4m59.245s    | 750 (611 kB in 150 RGs) | 3257 (728 kB in 652 RGs) | 3069 (616 kB in 614 RGs) | 1506 (257 kB in 302 RGs) | 1947 (254 kB in 390 RGs) |
| 01GAHSF8M020F3SX0QMEYQVS7P | 2022-08-15T23:19:59+01:00 | 2022-08-15T23:24:59+01:00 | 4m59.243s    | 750 (632 kB in 150 RGs) | 4075 (908 kB in 815 RGs) | 3768 (757 kB in 754 RGs) | 1594 (272 kB in 319 RGs) | 2038 (264 kB in 408 RGs) |
| 01GAHSRDK0P541RMRZRKVCQ5KB | 2022-08-15T23:24:59+01:00 | 2022-08-15T23:29:59+01:00 | 4m59.244s    | 750 (617 kB in 150 RGs) | 3547 (797 kB in 710 RGs) | 3308 (665 kB in 662 RGs) | 1559 (266 kB in 312 RGs) | 2009 (261 kB in 402 RGs) |
| 01GAHT1JJ0830BXC8VNZKA3Y68 | 2022-08-15T23:29:59+01:00 | 2022-08-16T08:03:55+01:00 | 8h33m55.746s | 715 (590 kB in 143 RGs) | 3555 (798 kB in 711 RGs) | 3339 (670 kB in 668 RGs) | 1611 (275 kB in 323 RGs) | 2063 (267 kB in 413 RGs) |
| 01GAJQEMAPV9H6C5S8ZHYHTRE5 | 2022-08-16T08:03:56+01:00 | 2022-08-16T08:04:59+01:00 | 1m3.243s     | 160 (134 kB in 32 RGs)  | 2844 (642 kB in 569 RGs) | 2587 (520 kB in 518 RGs) | 1431 (244 kB in 287 RGs) | 1863 (243 kB in 373 RGs) |
| 01GAJQGJB0NPSJ1R409JE6N5WQ | 2022-08-16T08:05:00+01:00 | 2022-08-16T08:09:59+01:00 | 4m59.245s    | 750 (616 kB in 150 RGs) | 3591 (798 kB in 719 RGs) | 3400 (683 kB in 680 RGs) | 1576 (269 kB in 316 RGs) | 2023 (262 kB in 405 RGs) |
| 01GAJQSQA0GMY633BS607FSQJB | 2022-08-16T08:10:00+01:00 | 2022-08-16T08:14:59+01:00 | 4m59.239s    | 750 (612 kB in 150 RGs) | 3559 (795 kB in 712 RGs) | 3354 (674 kB in 671 RGs) | 1601 (273 kB in 321 RGs) | 2059 (267 kB in 412 RGs) |
| 01GAJR2W90BNABDPF1N016X061 | 2022-08-16T08:15:00+01:00 | 2022-08-16T08:19:59+01:00 | 4m59.244s    | 750 (617 kB in 150 RGs) | 3632 (812 kB in 727 RGs) | 3410 (685 kB in 682 RGs) | 1625 (277 kB in 325 RGs) | 2087 (270 kB in 418 RGs) |
| 01GAJRC180VMEM4GF4QX0VG6W0 | 2022-08-16T08:20:00+01:00 | 2022-08-16T08:24:59+01:00 | 4m59.245s    | 750 (630 kB in 150 RGs) | 4214 (933 kB in 843 RGs) | 3822 (768 kB in 765 RGs) | 1629 (278 kB in 326 RGs) | 2078 (268 kB in 416 RGs) |
| 01GAJRN671QFGC468MF4CFV9X4 | 2022-08-16T08:25:00+01:00 | 2022-08-16T08:29:59+01:00 | 4m59.245s    | 750 (625 kB in 150 RGs) | 3709 (833 kB in 742 RGs) | 3397 (682 kB in 680 RGs) | 1573 (268 kB in 315 RGs) | 2023 (262 kB in 405 RGs) |
| 01GAJRYB60SJP8FVJ67QS80GS7 | 2022-08-16T08:30:00+01:00 | 2022-08-16T08:34:59+01:00 | 4m59.244s    | 750 (629 kB in 150 RGs) | 4011 (890 kB in 803 RGs) | 3671 (738 kB in 735 RGs) | 1649 (281 kB in 330 RGs) | 2102 (272 kB in 421 RGs) |
| 01GAJS7G50GJ4E9HQT8252726K | 2022-08-16T08:35:00+01:00 | 2022-08-16T08:39:59+01:00 | 4m59.245s    | 750 (622 kB in 150 RGs) | 3675 (825 kB in 735 RGs) | 3344 (672 kB in 669 RGs) | 1566 (268 kB in 314 RGs) | 2015 (261 kB in 403 RGs) |
| 01GAJSGN40MG45HRAEYNV5QVRJ | 2022-08-16T08:40:00+01:00 | 2022-08-16T08:44:59+01:00 | 4m59.245s    | 750 (616 kB in 150 RGs) | 3659 (816 kB in 732 RGs) | 3304 (664 kB in 661 RGs) | 1559 (266 kB in 312 RGs) | 2003 (260 kB in 401 RGs) |
| 01GAJSST30JN4F7C7P1CBH4E7Q | 2022-08-16T08:45:00+01:00 | 2022-08-16T08:49:59+01:00 | 4m59.245s    | 750 (618 kB in 150 RGs) | 3717 (835 kB in 744 RGs) | 3393 (682 kB in 679 RGs) | 1591 (272 kB in 319 RGs) | 2041 (264 kB in 409 RGs) |
| 01GAJT2Z20BG79QP53MYEF3HF8 | 2022-08-16T08:50:00+01:00 | 2022-08-16T08:54:59+01:00 | 4m59.244s    | 750 (628 kB in 150 RGs) | 3732 (828 kB in 747 RGs) | 3402 (684 kB in 681 RGs) | 1598 (273 kB in 320 RGs) | 2050 (265 kB in 410 RGs) |
| 01GAJTC410CCBKK74M39KXK47W | 2022-08-16T08:55:00+01:00 | 2022-08-16T08:59:59+01:00 | 4m59.244s    | 750 (618 kB in 150 RGs) | 3739 (833 kB in 748 RGs) | 3406 (685 kB in 682 RGs) | 1614 (275 kB in 323 RGs) | 2069 (268 kB in 414 RGs) |
| 01GAJTN900D5ZW2D5X4JAF70EM | 2022-08-16T09:00:00+01:00 | 2022-08-16T09:04:59+01:00 | 4m59.238s    | 750 (620 kB in 150 RGs) | 3764 (839 kB in 753 RGs) | 3378 (679 kB in 676 RGs) | 1560 (266 kB in 312 RGs) | 2015 (262 kB in 403 RGs) |
| 01GAJTYDZ10VBAWYAE3C52SQB9 | 2022-08-16T09:05:00+01:00 | 2022-08-16T09:09:59+01:00 | 4m59.244s    | 750 (621 kB in 150 RGs) | 3722 (832 kB in 745 RGs) | 3359 (675 kB in 672 RGs) | 1599 (273 kB in 320 RGs) | 2055 (266 kB in 411 RGs) |
| 01GANF6NP40F9NEQPCV1SP38E4 | 2022-08-17T09:37:36+01:00 | 2022-08-17T09:37:45+01:00 | 9.28s        | 24 (8.2 kB in 1 RGs)    | 181 (20 kB in 1 RGs)     | 576 (25 kB in 1 RGs)     | 513 (21 kB in 1 RGs)     | 757 (46 kB in 1 RGs)     |
| 01GANQ9JZBDEZ9F07HSSXW7VK9 | 2022-08-17T11:59:01+01:00 | 2022-08-17T11:59:28+01:00 | 26.714s      | 67 (19 kB in 1 RGs)     | 200 (24 kB in 1 RGs)     | 631 (28 kB in 1 RGs)     | 546 (23 kB in 1 RGs)     | 787 (47 kB in 1 RGs)     |
| 01GANS1V8HWBTTYCFRX3W2Q0T9 | 2022-08-17T12:29:46+01:00 | 2022-08-17T12:29:59+01:00 | 13.239s      | 35 (13 kB in 1 RGs)     | 177 (19 kB in 1 RGs)     | 521 (23 kB in 1 RGs)     | 445 (19 kB in 1 RGs)     | 670 (42 kB in 1 RGs)     |
| 01GANT0TX0ZZ5YV3RR8CC38ZCV | 2022-08-17T12:46:41+01:00 | 2022-08-17T12:49:59+01:00 | 3m17.99s     | 496 (190 kB in 1 RGs)   | 1528 (172 kB in 2 RGs)   | 2507 (110 kB in 3 RGs)   | 1371 (56 kB in 2 RGs)    | 1779 (101 kB in 2 RGs)   |
| 01GANT74J0VVDXM677Z5D2K0B0 | 2022-08-17T12:49:59+01:00 | 2022-08-17T12:54:59+01:00 | 4m59.243s    | 750 (307 kB in 1 RGs)   | 1530 (199 kB in 2 RGs)   | 2427 (107 kB in 3 RGs)   | 1351 (56 kB in 2 RGs)    | 1763 (102 kB in 2 RGs)   |
| 01GANTG9H0T7D0RPQJF64JVZSH | 2022-08-17T12:54:59+01:00 | 2022-08-17T12:59:59+01:00 | 4m59.244s    | 750 (293 kB in 1 RGs)   | 1703 (221 kB in 2 RGs)   | 2561 (113 kB in 3 RGs)   | 1390 (57 kB in 2 RGs)    | 1791 (102 kB in 2 RGs)   |
| 01GANTSEG12FTHPFAGCNGWBSWB | 2022-08-17T12:59:59+01:00 | 2022-08-17T13:04:59+01:00 | 4m59.243s    | 750 (278 kB in 1 RGs)   | 1697 (224 kB in 2 RGs)   | 2523 (111 kB in 3 RGs)   | 1393 (57 kB in 2 RGs)    | 1801 (104 kB in 2 RGs)   |
| 01GANV2KF0FBW4C6HA38CRQJMP | 2022-08-17T13:04:59+01:00 | 2022-08-17T13:09:59+01:00 | 4m59.244s    | 750 (279 kB in 1 RGs)   | 1752 (230 kB in 2 RGs)   | 2555 (113 kB in 3 RGs)   | 1406 (58 kB in 2 RGs)    | 1821 (105 kB in 2 RGs)   |
| 01GANVBRE1KDB2V6SBGHXKKK51 | 2022-08-17T13:09:59+01:00 | 2022-08-17T13:14:59+01:00 | 4m59.244s    | 750 (285 kB in 1 RGs)   | 1859 (239 kB in 2 RGs)   | 2661 (118 kB in 3 RGs)   | 1427 (59 kB in 2 RGs)    | 1846 (107 kB in 2 RGs)   |
| 01GANVMXD17MK1KEHP6002DBNE | 2022-08-17T13:14:59+01:00 | 2022-08-17T13:19:59+01:00 | 4m59.244s    | 750 (278 kB in 1 RGs)   | 1866 (243 kB in 2 RGs)   | 2648 (117 kB in 3 RGs)   | 1424 (58 kB in 2 RGs)    | 1838 (106 kB in 2 RGs)   |
| 01GANVY2C13EBE4XJS169PXXH1 | 2022-08-17T13:19:59+01:00 | 2022-08-17T13:21:11+01:00 | 1m11.244s    | 180 (71 kB in 1 RGs)    | 1303 (159 kB in 2 RGs)   | 1984 (88 kB in 2 RGs)    | 1254 (52 kB in 2 RGs)    | 1639 (97 kB in 2 RGs)    |
| 01GAP118WSYADR8DN45P3MPDEB | 2022-08-17T14:49:16+01:00 | 2022-08-17T14:49:59+01:00 | 43.216s      | 110 (34 kB in 1 RGs)    | 300 (34 kB in 1 RGs)     | 698 (31 kB in 1 RGs)     | 543 (23 kB in 1 RGs)     | 782 (46 kB in 1 RGs)     |
| 01GAP12VT0TCXCEW9DRN3R7YJJ | 2022-08-17T14:50:00+01:00 | 2022-08-17T14:54:59+01:00 | 4m59.244s    | 750 (315 kB in 1 RGs)   | 4120 (425 kB in 5 RGs)   | 4475 (197 kB in 5 RGs)   | 1701 (70 kB in 2 RGs)    | 2144 (118 kB in 3 RGs)   |
| 01GAP1C0S07FRRFY7212MVRYNW | 2022-08-17T14:55:00+01:00 | 2022-08-17T14:55:27+01:00 | 26.525s      | 68 (38 kB in 1 RGs)     | 903 (102 kB in 1 RGs)    | 1444 (64 kB in 2 RGs)    | 990 (40 kB in 1 RGs)     | 1329 (80 kB in 2 RGs)    |
| 01GAP40QR2ZM6QBCWHK4YGWMKE | 2022-08-17T15:41:28+01:00 | 2022-08-17T15:44:59+01:00 | 3m30.84s     | 529 (279 kB in 1 RGs)   | 3164 (339 kB in 4 RGs)   | 3736 (164 kB in 4 RGs)   | 1565 (64 kB in 2 RGs)    | 2013 (112 kB in 3 RGs)   |
| 01GAP47JF0Q4222T6W80QH3CRE | 2022-08-17T15:44:59+01:00 | 2022-08-17T15:49:59+01:00 | 4m59.243s    | 750 (296 kB in 1 RGs)   | 1736 (218 kB in 2 RGs)   | 2437 (107 kB in 3 RGs)   | 1285 (53 kB in 2 RGs)    | 1687 (98 kB in 2 RGs)    |
| 01GAP4GQE0J0G3V2FBJQ4HCM2J | 2022-08-17T15:49:59+01:00 | 2022-08-17T15:51:33+01:00 | 1m33.244s    | 235 (91 kB in 1 RGs)    | 1290 (155 kB in 2 RGs)   | 1952 (86 kB in 2 RGs)    | 1184 (49 kB in 2 RGs)    | 1575 (93 kB in 2 RGs)    |
+----------------------------+---------------------------+---------------------------+--------------+-------------------------+--------------------------+--------------------------+--------------------------+--------------------------+

@simonswine simonswine added the kind/enhancement New feature or request label Jul 27, 2022
@simonswine simonswine force-pushed the 20220722_add-block-querier branch from 4b004b9 to 67654c0 Compare July 27, 2022 17:12
@cyriltovena
Copy link
Collaborator

Looks promising so far

@simonswine simonswine mentioned this pull request Jul 29, 2022
Merged
@simonswine simonswine force-pushed the 20220722_add-block-querier branch from 67654c0 to 386c3da Compare July 29, 2022 15:51
@simonswine simonswine force-pushed the 20220722_add-block-querier branch 2 times, most recently from eaf992b to 66aa081 Compare August 15, 2022 17:49
@simonswine simonswine force-pushed the 20220722_add-block-querier branch from aae65c4 to 6ce0288 Compare August 17, 2022 10:51
@simonswine
Add block querier to query parquet files
300dc08 
This allows to query parquet files in a local directory and/or bucket.

The current implementation will load all profiles from the matching
series finger prints in full, before checking if the timestamps matches.
@simonswine
Allow durations larger than hours
342bd25 
This replaces the standard libary duration parse with the one from
prometheus to allow time periods like days, weeks and years.
@simonswine
Add firetool CLI
96e53bb 
This allows to list local blocks
@simonswine simonswine force-pushed the 20220722_add-block-querier branch from 0da45c3 to 96e53bb Compare August 22, 2022 08:51
@simonswine simonswine changed the title 20220722 add block querier Add ability to query local blocks Aug 22, 2022
@simonswine simonswine marked this pull request as ready for review August 22, 2022 08:54
cyriltovena
cyriltovena approved these changes Aug 22, 2022
View reviewed changes
Copy link
Collaborator

@cyriltovena cyriltovena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great work. I left mostly questions and observations. I think this is a good step forward to start benchmarking different parquet format/access model.

pkg/firedb/block_querier.go Outdated Show resolved Hide resolved
pkg/firedb/block_querier.go Show resolved Hide resolved
pkg/firedb/block_querier.go Show resolved Hide resolved
pkg/firedb/block_querier.go Outdated Show resolved Hide resolved
pkg/firedb/firedb.go Show resolved Hide resolved
@simonswine simonswine force-pushed the 20220722_add-block-querier branch from b481478 to 85562ec Compare August 24, 2022 12:13
@simonswine simonswine merged commit 13e2d45 into main Aug 24, 2022
@simonswine simonswine deleted the 20220722_add-block-querier branch August 24, 2022 12:58
simonswine added a commit to simonswine/pyroscope that referenced this pull request Jun 30, 2023
@simonswine
Merge pull request grafana/phlare#127 from grafana/20220722_add-block…
461d938 
…-querier

Add ability to query local blocks
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@cyriltovena cyriltovena cyriltovena approved these changes

Assignees
No one assigned
Labels
kind/enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@simonswine @cyriltovena