forked from darold/pgbadger
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathChangeLog
2302 lines (2034 loc) · 111 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
2019-06-25 - v11.0
This release of pgBadger adds some major new features and fixes some
issues reported by users since the last four months. New features:
- Regroup cursor related query (DECLARE,CLOSE,FETCH,MOVE) into new
query type CURSOR.
- Add top bind queries that generate the more temporary files.
Require log_connection and log_disconnection be activated.
- Add --exclude-client command line option to be able to exclude log
entries for the specified client ip. Can be used multiple time.
- Allow to use time only in --begin and --end filters.
- Add -H, --html-dir option to be able to set a different path where
HTML report must be written in incremental mode. Binary files stay
on directory defined with -O, --outdir option.
- Add -E | --explode option to explode the main report into one
report per database. Global information not related to a database
are added to the postgres database report.
- Add per database report to incremental mode. In this mode there
will be a sub directory per database with dedicated incremental
reports.
- Add support to Heroku's PostgreSQL logplex format. Log can be
parsed using:
heroku logs -p postgres | pgbadger -f logplex -o heroku.html -
- When a query is > 10Kb we first limit size of all constant string
parameters to 30 characters and then the query is truncated to 10Kb.
This prevent pgbadger to waste time/hang with very long queries
when inserting bytea for example. The 10Kb limit can be controlled
with the --maxlength command line parameter.
The query is normalized or truncated to maxlength value only after
this first attempt to limit size.
This new release breaks backward compatibility with old binary or JSON
files. This also mean that incremental mode will not be able to read
old binary file. If you want to update pgBadger and keep you old reports
take care to upgrade at start of a new week otherwise weekly report will
be broken. pgBadger will print a warning and just skip the old binary
file.
There's also some bugs fixes and features enhancements.
- Add a warning about version and skip loading incompatible binary file.
- Update code formatter to pgFormatter 4.0.
- Fix pgbadger hang on Windows OS. Thanks to JMLessard for the report.
- Update tools/pgbadger_tools script to be compatible with new binary
file format in pgBadger v11.
- Add top bind queries that generate the more temporary files. This
collect is possible only if log_connection and log_disconnection
are activated in postgresql.conf. Thanks to Ildefonso Camargo for
the feature request.
- Fix auto detection of timezone. Thanks to massimosala for the fix.
- Remove some remaining graph when --nograph is used
- Force use of .txt extension when --normalized-only is used.
- Fix report of auto vacuum/analyze in logplex format. Thanks to
Konrad zichul for the report.
- Fix use of progress bar on Windows operating system. Thanks to
JMLessard for the report.
- Use a `$prefix_vars{'t_time'} to store the log time. Thanks to Luca
Ferrari for the patch.
- Update usage and documentation to remove perl command from pgbadger
invocations. Thanks to Luca Ferrari for the patch.
- Use begin and end with times without date. Thanks to Luca Ferrari
for the patch.
- Added some very minor spelling and grammar fixes to the readme file.
Thanks to ofni yratilim for the patch.
- Fix remote paths using SSH. Thanks to Luca Ferrari for the patch.
- Update regression test to works with new structure introduced with
the per database report feature.
- Fix fractional seconds in all begin and end parameters. Thanks to
Luca Ferrari for the patch.
- Fix documentation URL. Thanks to Kara Mansel for the report.
- Fix parsing of auto_explain.
Add more information about -U option that can be used multiple time.
Thanks to Douglas J Hunley for the report.
- Lot of HTML / CSS report improvements. Thanks to Pierre Giraud for
the patches.
- Update resource file.
- Add regression test for logplex format.
- Add support to Heroku's PostgreSQL logplex format. You should be able
to parse these logs as follow:
heroku logs -p postgres | pgbadger -f logplex -o heroku.html -
or if you have already saved the output to a file:
pgbadger heroku.log
The logplex format is auto-dectected like any other supported format.
pgBadger understand the following default log_line_prefix:
database = %d connection_source = %r sql_error_code = %e
or simply:
sql_error_code = %e
Let me know if there's any other default log_line_prefix. The prefix
can always be set using the -p | --prefix pgbadger option:
pgbadger --p 'base = %d source = %r sql_state = %e' heroku.log
for example.
Thanks to Anthony Sosso for the feature request.
- Fix pgbadger help on URI use.
- Fix broken wildcard use in ssh URI introduced in previous patch.
Thanks to Tobias Bussmann for the report.
- Allow URI with space in path to log file. Thanks to Tobias Bussmann
for the report.
- Fix URI samples in documentation. Thanks to Tobias Bussmann for the
patch.
- Fix t/02_basics.t to don't fail if syslog test takes more than 10s.
Thanks to Christoph Berg for the patch.
2019-02-14 - v10.3
This release of pgBadger is a maintenance release that fixes some
log format autodetection issues another pgBouncer log parsing issue
reported by users. There is also a new feature:
The -o | --outfile option can now be used multiple time to dump
output in several format in a single command. For example:
pgbadger -o out.html -o out.json /log/pgsql-11.log
will create two reports in html and json format saved in the
two corresponding files.
There's also some bugs fixes and features enhancements.
- Fix statistics reports when there a filter on database, user,
client or application is requested. Some queries was not
reported.
- Fix autodetection of pg>=10 defauilt log line prefix.
- Fix autodetection of log file with "non standard" log line prefix.
If --prefix specify %t, %m, %n and %p or %c, set format to stderr.
Thanks to Alex Danvy for the report.
- Remove extra space at end of line.
- Add minimal test to syslog parser.
- Fix a call to autodetect_format().
- Truncate statement when maxlength is used. Thanks to Thibaud
Madelaine for the patch.
- Add test for multiple output format.
- The -o | --outfile option can now be used multiple time to dump
output in several format in a single command. For example:
pgbadger -o out.txt -o out.html -o - -x json /log/pgsql-11.log
Here pgbadger will create two reports in text and html format
saved in the two corresponding file. It will also output a json
report on standard output. Thanks to Nikolay for the feature
request.
- Move detection of output format and setting of out filename into
a dedicated function set_output_extension().
- Fix another pgBouncer log parsing issue. Thanks to Douglas J.
Hunley for the report.
2018-12-27 - v10.2
This release of pgBadger is a maintenance release that fixes issues
reported by users during last three months. There is also some new
features:
* Add support to pgbouncer 1.8 Stats log format.
* Auto adjust javascript graph timezone.
There is a new command line option:
* Add --exclude-db option to compute report about everything except
the specified database.
* Add support to http or ftp remote PostgreSQL log file download.
The log file is parsed during the download using curl command
and never saved to disk. With ssh remote log parsing you can use
uri as command line argument to specify the PostgreSQL log file.
ssh://localhost/postgresql-10-main.log
http://localhost/postgresql-10-main.log.gz
ftp://localhost/postgresql-10-main.log
with http and ftp protocol you need to specify the log file format
at end of the uri:
http://localhost/postgresql-10-main.log:stderr
You can specify multiple uri for log files to be parsed. This is
useful when you have pgbouncer log file on a remote host and
PostgreSQL logs in the local host.
With ssh protocol you can use wild card too like with remote
mode, ex: ssh://localhost/postgresql-10-main.log*
Old syntax to parse remote log file using -r option is still
working but is obsolete and might be removed in future versions.
There's also some bugs fixes and features enhancements.
- Adjust end of progress bar with files with estimate size (bz2
compressed files and remote compressed files.
- Update year in copyright.
- Add information about URI notation to parse remote log files.
- Force progress to reach 100% at end of parsing of compressed
remote file.
- Extract information about PL/pgSQL function call in queries of
temporary file reports. The information is append to the details
display block.
- Fix progress bar with csv files.
- Fix reading binary file as input file instead of log file.
- Encode html output of queries into UTF8 to avoid message "Wide
character in print". Thanks to Colin 't Hart for the report.
- Add Checkpoints distance key/value for distance peak.
- Fix pgbouncer parsing and request throughput reports. Thanks
to Levente Birta for the report.
- Fix use of csvlog instead of csv for input format.
- Add support to pgbouncer 1.8 Stats log format. Thanks to Levente
Birta for the report.
- Add warning about parallel processing disabled with csvlog. Thanks
to cstdenis for the report.
- Add information in usage output about single process forcing with
csvlog format in -j and -J options. Thanks to cstdenis for the
report.
- Fix unknown line format error for multi line log while incremental
analysis over ssh. Thanks to Wooyoung Cho for the report.
- Add -k (--insecure) option to curl command to be able to download
logs from server using a self signed certificate.
- Auto adjust javascript graph timezone. Thanks to Massimino Sala
for the feature request.
- Add support to HTTP logfile download by pgBadger, for example:
/usr/bin/pgbadger http://www.mydom.com/postgresql-10.log
- Will parse the file during download using curl command.
- Fix documentation. Thanks to 0xflotus for the patch.
- Reapply fix on missing replacement of bind parameters after some
extra code cleaning. Thanks to Bernhard J. M. Grun for the report.
- Add --exclude-db option to compute report about everything except
the specified database. The option can be used multiple time.
2018-09-12 - v10.1
This release of pgBadger is a maintenance release that fixes reports
in incremental mode and multiprocess with -j option. Log parsing from
standard input was also broken. If you are using v10.0 please upgrade
now.
- Add test on pgbouncer log parser.
- Some little performances improvment.
- Fix not a valid file descriptor at pgbadger line 12314.
- Fix unwanted newline in progressbar at startup.
- Remove circleci files from the project.
- Remove dependency of bats and jq for the test suite, they are
replaced with Test::Simple and JSON::XS.
- Add more tests especially for incremental mode and input from
stdin that was broken in release 10.0.
- Sync pgbadger, pod, and README, and fix some syntax errors.
Thanks to Christoph Berg for the patch.
- Add documentation on how to install Perl module JSON::XS from
apt and yum repositories.
- Fix URI for CSS in incremental mode. Thanks to Floris van Nee
for the report.
- Fix fatal error when looking for log from STDIN. Thanks to
Jacek Szpot for the report.
- Fixes SED use for OSX builds. Thanks to Steve Newson for the
patch.
- Fix illegal division by zero in incrental mode. Thanks to
aleszeleny for the report.
- Replace SQL::Beautify with v3.1 of pgFormatter::Beautify.
2018-09-09 - v10.0
This release of pgBadger is a major release that adds some new
features and fix all issues reported by users since last release.
* Add support of pgbouncer syslog log file format.
* Add support to all auto_explain format (text, xml, json and yaml).
* Add support to %q placeholder in log_line_prefix.
* Add jsonlog format of Michael Paquier extension, with -f jsonlog
pgbadger will be able to parse the log.
* Replace the SQL formatter/beautify with v3.0 of pgFormatter.
There is some new command line option:
- Add --prettify-json command line option to prettify JSON output.
- Add --log-timezone +/-XX command line option to set the number
of hours from GMT of the timezone that must be used to adjust
date/time read from log file before beeing parsed. Note that you
might still need to adjust the graph timezone using -Z when the
client has not the same timezone.
- Add --include-time option to add the ability to choose times that
you want to see, instead of excluding all the times you do not
want to see (--exclude-time).
The pgBadger project and copyrights has been transfered from Dalibo
to the author and official maintainer of the project. Please update
your links:
- Web site: http://pgbadger.darold.net/
- Source code: https://github.com/darold/pgbadger
I want to thanks the great guys at Dalibo for all their investments
into pgBadger during these years and especially Damien Clochard and
Jean-paul argudo for their help to promote pgBadger.
- Fix checkpoint distance and estimate not reported in incremental
mode. Thanks to aleszeleny for the report.
- Fix title of pgbouncer simultaneous session report. Thansks to
Jehan Guillaume De Rorthais for the report.
- Add support of pgbouncer syslog log file format. Thanks to djester
for the feature request.
- Fix error when a remote log is empty. Thanks to Parasit Hendersson
for the report.
- Fix test with binary format. Binary file must be generated as it
is dependent of the plateform. Thanks to Michal Nowak for the
report.
- Fix case where an empty explain plan is generated.
- Fix parsing of autodetected default format with a prefix in
command line.
- Remove dependency of git command in Makefile.PL.
- Update documentation about options changes and remove of the
[%l-1] part of the mandatory prefix.
- Fix parsing of vacuum / analyze system usage for PostgreSQL 10.
Thanks to Achilleas Mantzios for the patch.
- Fix Temporary File Activity table.
- Remove dependency to git during install.
- Add --log-timezone +/-XX command line option to set the number
of hours from GMT of the timezone that must be used to adjust
date/time read from log file before beeing parsed. Using this
option make more difficult log search with a date/time because the
time will not be the same in the log. Note that you might still
need to adjust the graph timezone using -Z when the client has not
the same timezone. Thanks to xdexter for the feature request and
Julien Tachoire for the patch.
- Add support to auto_explain json output format. Thanks to dmius
for the report.
- Fix auto_explain parser and queries that was counted twice.
Thanks to zam6ak for the report.
- Fix checkpoint regex to match PostgreSQL 10 log messages. Thanks
to Edmund Horner for the patch.
- Update description of -f | --format option by adding information
about jsonlog format.
- Fix query normalisation to not duplicate with bind queries.
Normalisation of values are now tranformed into a single ? and no
more 0 for numbers, two single quote for string. Thanks to vadv
for the report.
- Fix log level count. Thanks to Jean-Christophe Arnu for the report
- Make pgbadger more compliant with B::Lint bare sub name.
- Made perlcritic happy.
- Add --prettify-json command line option to prettify JSON output.
Default output is all in single line.
- Fix Events distribution report.
- Fix bug with --prefix when log_line_prefix contain multiple %%.
Thanks to svb007 for the report.
- Add --log-timezone +/-XX command line option to set the number
of hours from GMT of the timezone that must be used to adjust
date/time read from log file before beeing parsed. Using this
option make more difficult log search with a date/time because the
time will not be the same in the log. Note that you might still
need to adjust the graph timezone using -Z when the client has not
the same timezone. Thanks to xdexter for the feature request.
- Remove INDEXES from the keyword list and add BUFFERS to this list.
- Fix normalization of query using cursors.
- Remove Dockerfile and documentation about docker run. pgBadger
comes as a single Perl script without any dependence and it can
be used on any plateform. It is a non sens to use docker to run
pgbadger, if you don't want to install anything, just copy the
file pgbadger where you want and execute it.
- Fix broken grid when no temp files activity. Thanks to Pierre
Giraud for the patch
- Add doc warning about log_in_duration_statement vs log_duration +
log_statement. Thanks to Julien Tachoire for the patch.
- Apply timezone offset to bar charts. Thanks to Julien Tachoire
for the patch.
- Delete current temp file info if we meet an error for the same PID
Thanks to Julien Tachoire for the patch.
- Consistently use app= in examples, and support appname=
Some of the usage examples used appname= in the prefix, but the
code didn't recognize that token. Use app= in all examples, and
add appname= to the prefix parser. Thanks to Christoph Berg for
the patch
- Fix wrong long name for option -J that should be --Jobs intead
of --job_per_file. Thanks to Chad Trabant for the report and
Etienne Bersac for the patch.
- Ignore blib files. Thanks to Etienne Bersac for the patch.
- Add consistency tests. Thanks to damien clochard for the patch.
- doc update : stderr is not a default for -f. Thanks to Christophe
Courtois for the patch.
- Always update pod and README. Thanks to Etienne Bersac for
the patch.
- Add some regression tests. Thanks to Etienne Bersac for the patch.
- Add editorconfig configuration. Thanks to Etienne Bersac for the
patch.
- Drop vi temp files from gitignore. Thanks to Etienne Bersac for
the patch.
- Add --include-time option to add the ability to choose times that
you want to see, instead of excluding all the times you do not
want to see. This is handy when wanting to view only one or two
days from a week's worth of logs (simplifies down from multiple
--exlucde-time options to one --include-time). Thanks to Wesley
Bowman for the patch.
- Check pod syntax. Thanks to Etienne Bersac for the patch.
- Add HACKING to document tests. Thanks to Etienne Bersac for the
patch.
- Drop obsolete --bar-graph option. Thanks to Etienne Bersac for
the patch.
- Drop misleading .perltidyrc. This file date from 2012 and
pgbadger code is far from compliant. perltidy unified diff is
10k lines. Let's drop this. Thanks to Etienne Bersac for the
patch.
- Fix use of uninitialized value in SQL formatting. Thanks to John
Krugger for the report and Jean-paul Argudo for the report.
2017-07-27 - v9.2
This release of pgBadger is a maintenance release that adds some new
features.
* Add report of checkpoint distance and estimate.
* Add support of AWS Redshift keywords to SQL code beautifier.
* Add autodetection of log format in remote mode to allow remote
parsing of pgbouncer log file together with PostgreSQL log file.
There's also some bugs fixes and features enhancements.
- Fix reports with histogram that was not showing data upper than
the last range.
- Fix parsing of journalctl without the the log line number pattern
([%l-n]). Thanks to Christian Schmitt for the report.
- Add report of checkpoint distance and estimate. Thanks to jjsantam
for the feature request.
- Append more information on what is done by script to update CSS
and javascript files, tools/updt_embedded_rsc.pl.
- Do not warn when all log files are empty and exit with code 0.
- Fix build_log_line_prefix_regex() that does not include %n as a
lookup in %regex_map. Thanks to ghosthound for the patch.
- Change error level of "FATAL: cannot use CSV" to WARNING. Thanks
to kong1man for the report.
- Fix use of uninitialized value warning. Thanks to Payal for the
report.
- Add permission denied to error normalization
- Update pgbadger to latest commit 5bdc018 of pgFormatter.
- Add support for AWS Redshift keywords. Thanks to cavanaug for the
feature request.
- Fix missing query in temporary file report when the query was
canceled. Thanks to Fabrizio de Royes Mello for the report.
- Normalize query with binded parameters, replaced with a ?.
- Sanity check to avoid end time before start time. Thanks to
Christophe Courtois for the patch.
- Fix a lot of mystyped words and do some grammatical fixes. Use
'pgBadger' where it refers to the program and not the binary file.
Also, use "official" expressions such as PgBouncer, GitHub, and
CSS. POD file was synced with README. Thanks to Euler Taveira for
the patch.
- Menu is broken when --disable-type top_cancelled_info test and
closing list must be inside disable_type test. While in it, ident
disable_lock test. Thanks to Euler Taveira for the patch.
- Fix use of uninitialized value. Thanks to johnkrugger for the
report.
- Remove test to read log file during log format auto-detection when
the file is hosted remotly. Thanks to clomdd for the report.
- Add autodetection of log format in remote mode to allow remote
parsing of pgbouncer log file together with PostgreSQL log file.
- Fix number of sessions wrongly increased after log line validation
Thanks to Achilleas Mantzios for the report.
- Minor reformatting of the pgBadger Description.
- Fix repeated info in documentation. Thanks to cscatolini for the patch.
2017-01-24 - v9.1
This release of pgBadger is a maintenance release that adds some new
features.
* Add report of error class distribution when SQLState is available
in the log_line_prefix (see %e placeholder).
* Update SQL Beautifier to pgFormatter v1.6 code.
* Improve error message normalization.
* Add --normalized-only option to generate a text file containing all
normalized queries found in a log with count.
* Allow %c (session id) to replace %p (pid) as unique session id.
* Add waiting for lock messages to event reports.
* Add --start-monday option to start calendar weeks in Monday
instead of default to Sunday.
There's also some bugs fixes and features enhancements.
- Add report of error class distribution when SQLState is available
in the log line prefix. Thanks to jacks33 for the feature request.
- Fix incremental global index on resize. Thanks to clomdd for the
report.
- Fix command tag log_line_prefix placeholder %i to allow space
character.
- Fix --exclude-line options and removing of obsolete directory
when retention is enabled and --noreport is used.
- Fix typo in "vacuum activity table". Thanks to Nicolas Gollet for
the patch.
- Fix autovacuum report. Thanks to Nicolas Gollet for the patch.
- Fix author of pgbadger's logo - Damien Cazeils and English in
comments. Thanks to Thibaut Madelaine for the patch.
- Add information about pgbouncer log format in the -f option.
Thanks to clomdd for the report.
- Add --normalized-only information in documentation.
- Fix broken report of date-time introduced in previous patch.
- Fix duration/query association when log_duration=on and
log_statement=all. Thanks to Eric Jensen for the report.
- Fix normalization of messages about advisory lock. Thanks to
Thibaut Madelaine for the report.
- Fix report of auto_explain output. Thanks to fch77700 for the
report.
- Fix unwanted log format auto detection with log entry from stdin.
Thanks to Jesus Adolfo Parra for the report.
- Add left open parentheses to the "stop" chars of regex to look
for db client in the prefix to handle the PostgreSQL client
string format that includes source port. Thanks to Jon Nelson
for the patch.
- Fix some spelling errors. Thanks to Jon Nelson for the patch.
- Allow %c (session id) to replace %p (pid) as unique session id.
Thanks to Jerryliuk for the report.
- Allow pgbadger to parse default log_line_prefix that will be
probably used in 10.0: '%m [%p] '
- Fix missing first line with interpreter call.
- Fix missing Avg values in CSV report. Thanks to Yosuke Tomita
for the report.
- Fix error message in autodetect_format() method.
- Add --start-monday option to start calendar weeks in Monday
instead of default to Sunday. Thanks to Joosep Mae for the feature
request.
- Fix --histo-average option. Thanks to Yves Martin for the report.
- Remove plural form of --ssh-option in documentation. Thanks to
mark-a-s for the report.
- Fix --exclude-time filter and rewrite code to skip unwanted line
as well code to update the progress bar. Thanks to Michael
Chesterton for the report.
- Fix support to %r placeholder in prefix instead of %h.
2016-09-02 - v9.0
This major release of pgBadger is a port to bootstrap 3 and a version
upgrade of all resources files (CSS and Javascript). There's also some
bugs fixes and features enhancements.
Backward compatibility with old incremental report might be preserved.
- Sources and licences of resources files are now on a dedicated
subdirectory. A script to update their minified version embedded
in pgbager script has been added. Thanks to Christoph Berg for
the help and feature request.
- Try to detect user/database/host from connection strings if
log_connection is enabled and log_line_prefix doesn't include
them.
Extend the regex to autodetect database name, user name, client
ip address and application name. The regex now are the following:
db => qr/(?:db|database)=([^,]*)/;
user => qr/(?:user|usr)=([^,]*)/;
client => qr/(?:client|remote|ip|host)=([^,]*)/;
appname => qr/(?:app|application)=([^,]*)/;
- Add backward compatibility with older version of pgbadger in
incremental mode by creating a subdirectory for new CSS and
Javascript files. This subdirectory is named with the major
version number of pgbadger.
- Increase the size of the pgbadger logo that appears too small
with the new font size.
- Normalize detailed information in all reports.
- Fix duplicate copy icon in locks report.
- Fix missing chart on histogram of session time. Thanks to
Guillaume Lelarge for the report.
- Add LICENSE file noting the licenses used by the resource
files. Thanks to Christoph Berg for the patch.
- Add patch to jqplot library to fix an infinite loop when trying
to download some charts. Thanks to Julien Tachoires for the help
to solve this issue.
- Script tools/updt_embedded_rsc.pl will apply the patch to resource
file resources/jquery.jqplot.js and doesn't complain if it has
already been applied.
- Remove single last comma at end of pie chart dataset. Thanks to
Julien Tachoires for the report.
- Change display of normalized error
- Remove unused or auto-generated files
- Update all resources files (js+css) and create a directory to
include source of javascript libraries used in pgbadger. There is
also a new script tools/updt_embedded_rsc.pl the can be used to
generate the minified version of those files and embedded them
into pgbadger. This script will also embedded the FontAwesome.otf
open truetype font into the fontawesome.css file.
2016-08-27 - v8.3
This is a maintenance release that fix some minor bugs. This release
also adds replication command messages statistics to the Events
reports.
- Fix auto-detection of stderr format with timestamp as epoch (%n).
- Fix histogram over multiples days to be cumulative per hour, not
an average of the number of event per day.
- Fix parsing of remote file that was failing when the file does
not exists locally. Thanks to clomdd for the report.
- Detect timezones like GMT+3 on CSV logs. Thanks to jacksonfoz
for the patch.
- Add replication command messages statistics to the Events
reports. Thanks to Michael Paquier for the feature request.
This is the last minor version of the 8.x series, next major version
will include an upgrade of boostrap and jquery library which need
some major rewrite.
2016-08-11 version 8.2
This is a maintenance release that fix some minor bug. There is also
some performances improvement up to 20% on huge files and some new
interesting features:
* Multiprocessing can be used with pgbouncer log files.
* pgBouncer and PostgreSQL log files can be used together in
incremental mode.
* With default or same prefix, stderr and syslog file can be
parsed together, csvlog format can always be used.
* Use a modal dialog window to download graphs as png images.
* Add pl/pgSQL function information to queries when available.
Here are the complete list of changes:
- Fix report of database system messages.
- Fix multi line statement concatenation after an error.
- Fix box size for report of queries generating the most
temporary files and the most waiting queries.
- Rewrite code to better handle multi-line queries.
- Fix garbage in examples of event queries with error only mode
(option -w). Thanks to Thomas Reiss for the report.
- Fix getting dataset related to query duration with the use of
auto_explain. Thanks to tom__b for the patch.
- Use a modal dialog window to download graphs as png images.
- Huge rewrite of the incremental mechanism applied to log files
to handle PostgreSQL and pgbouncer logs at the same time.
- Multiprocess can be used with pgbouncer log.
- Add code to remove remaining keyword placeholders tags.
- Fix an other possible case of truncated date in LAST_PARSED file
Thanks to brafaeloliveira for the report.
- Set default scale 1 in pretty_print_number() js function.
- Fix auto-detection of pgbouncer files that contain only stats
lines. Thanks to Glyn Astill for the patch.
- Add date to samples of queries generating most temporary files.
- Do not display warning message of empty log when quiet mode is
enable.
- Fix reading from stdin by disabling pgbouncer format detection.
Thanks to Robert Vargason for the patch.
- Fix case of duplicate normalized error message with "nonstandard
use of ...".
- Fix storage of current temporary file related request.
- Use the mnemonic rather than a signal number in kill calls.
Thanks to Komeda Shinji for the patch.
2016-04-21 version 8.1
This is a maintenance release that fix a major issue introduced with
support to pgbouncer that prevent parsing of compressed PostgreSQL
log files and adds some improvements.
Here are the complete list of changes:
- Fix one case where pid file remain after dying.
- Add requirement of log_error_verbosity = default to documentation.
- Report message "LOG: using stale statistics instead of current
ones because stats collector is not responding" in events view.
- Remove obsolete days when we are in binary mode with --noreport
- Fix wrong report of statements responsible of temporary files.
Thanks to Luan Nicolini Marcondes for the report. This patch also
exclude line with log level LOCATION to be parsed.
- Fix limit on number of sample at report generation and remove
pending LAST_PARSED.tmp file.
- Update load_stat() function and global variables to support
pgbouncer statistics. Update version to 2.0.
- Handle more kind or query types. Thanks to julien Rouhaud for
the patch.
- Fix pgbouncer log parser to handle message: FATAL: the database
system is shutting down
- Fix whitespace placed in between the E and the quote character.
Thanks to clijunky for the report.
- Fix a major issue introduced with support to pgbouncer that
prevent parsing of compressed PostgreSQL log files. Thanks to
Levente Birta for the report.
2016-02-22 version 8.0
This is a major release that adds support to pgbouncer log files.
New pgbouncer reports are:
* Request Throughput
* Bytes I/O Throughput
* Queries Average duration
* Simultaneous sessions
* Histogram of sessions times
* Sessions per database
* Sessions per user
* Sessions per host
* Established connections
* Connections per database
* Connections per user
* Connections per host
* Most used reserved pools
* Most Frequent Errors/Events
pgbouncer log files can be parsed together with PostgreSQL logs.
It also adds a two new command line options:
* --pgbouncer-only to only show pgbouncer related reports.
* --rebuild to be able to rebuild all html reports in incremental
output directory where binary data files are still available.
This release fixes a major bug introduced with journalctl code that
was prevented the use of multiprocess feature.
Here the complete list of other changes:
- Fix progress bar with pgbouncer (only events are increased).
- Sort %SYMBOLE hashtable for remove "!=" / "=" bug. Thanks to
Nicolas Gollet for the patch.
- Fix incorrect numbers on positional parameters in report Queries
generating most temporary files. Thanks to Oskar Wiksten for the
report.
- Update operators list in SQL code beautifier with last update in
pgFormatter. Thanks to Laurenz Albe for the report and the list
of missing operators.
- Cosmetic change to code and add some more debug information.
2016-01-18 version 7.3
This is a maintenance release to fix a major bug that was breaking
the incremental mode in pgBadger. It also adds some more reports and
features.
* Add --timezone=+/-HH to control the timezone used in charts. The
javascript library runs at client side so the timezone used is
the browser timezone so the displayed time in the charts can be
different from the time in the log file.
* Add /tmp/pgbadger.pid file to prevent cron jobs overlaping on
same log files.
* Add command line option --pid-dir to be able to run two pgbadger
at the same time by setting an alternate path to the pid file.
* Report information about "LOG: skipping analyze of ..." into
events reports.
* Report message "LOG: sending cancel to blocking autovacuum" into
events reports. Useful to look for queries generating autovacuum
kill on account of a lock issue.
Here the complete list of changes:
- Automatically remove obsolete pid file when there is no other
pgbadger process running (unix only)
- Update documentation about the --timezone command line option.
- Add --timezone=+/-HH to control the timezone used in charts.
Thanks to CZAirwolf for the report.
- Fix Histogram of session times when there is no data.
- Fix unclosed test file.
- Fix an other case where pgbadger.pid was not removed.
- Always display slides part on connections report even if there
is no data.
- Fix some label on sessions reports
- Add remove of pid file at normal ending.
- Fix wrong size/offset of log files that was breaking incremental
mode. Thanks a lot to CZAirwolf for the report and the help to
find the problem.
- Add command line option --pid-dir to be able to run two pgbadger
at the same time by setting an alternate path to the directory
where the pid file will be written.
- Add /tmp/pgbadger.pid file to prevent cron jobs overlaping on same
log files.
- Report information about "LOG: skipping analyze of ..." into
events reports.
- Report message "LOG: sending cancel to blocking autovacuum" into
events reports. Usefull to know which queries generate autovacuum
kill on account of a lock issue.
- Add more debug information about check log parsing decision.
2016-01-05 version 7.2
This new release fixes some issues especially in temporary files
reports and adds some features.
* Allow pgBadger to parse natively the journalctl command output
* Add new keywords from PG 9.5 for code formating
* Add support to %n log_line_prefix option for Unix epoch (PG 9.6)
There's also some new command line option:
* Adds --journalctl_cmd option to enable this functionality and
set the command. Typically:
--journalctl_cmd "journalctl -u postgresql-9.4"
to parse output of PG 9.4 log
Here is the full list of changes/fixes:
- Fix missing detailed information (date, db, etc.) in Queries
generating the largest temporary files report.
- Fix label of sessions histogram. Thanks to Guillaume Lelarge
for the patch.
- Fix to handle cancelled query that generate more than one
temporary file and more generally aggregate size on queries with
multiple (> 1GB) temporary files.
- Add "Total size" column in Temporary Files Activity table and
fix size increment when a query have multiple 1GB temporary file.
- Fix temporary file query normalization and examples.
- Fix incomplete and wrong queries associated to temporary files
when STATEMENT level log line was missing. Thanks to Mael
Rimbault for the report.
- When -w or --watch-mode is used, message "canceling statement
due to statement timeout" s now reported with other errors.
- Allow dot in dbname and user name. Thanks to David Turvey for
the patch.
- Remove use of unmaintained flotr2 javascript chart library and
use of jqflot instead.
- Fix bad formatting with anonymized values in queries.
- Display 0ms instead of 0s when qery time is under the millisecond.
Thanks to venkatabn for the report.
- Normalize cursor names. Patch from Julien Rouhaud
- Fix unregistered client host name with default pattern. Thanks to
Eric S. Lucinger Ruiz for the report.
- Remove redundant regular expressions.
- Tweaking awkward phrasing, correcting subject-verb agreements,
typos, and misspellings. Patch from Josh Kupershmid.
- Fix potential incorrect creation of subdirectory in incremental mode.
- Allow single white space after duration even if this should not appear.
- Update copyright.
2015-07-11 version 7.1
This new release fixes some issues and adds a new report:
* Distribution of sessions per application
It also adds Json operators to SQL Beautifier.
Here is the full list of changes/fixes:
- Fix unwanted seek on old parsing position when log entry is stdin.
Thanks to Olivier Schiavo for the report.
- Try to fix a potential issue in log start/end date parsing. Thanks
to gityerhubhere for the report.
- Fix broken queries with multiline in bind parameters. Thank to
Nicolas Thauvin for the report.
- Add new report Sessions per application. Thanks to Keith Fiske for
the feature request.
- Add Json Operators to SQL Beautifier. Thanks to Tom Burnett and
Hubert depesz Lubaczewski.
- Makefile.PL: changed manpage section from '1' to '1p', fixes #237.
Thanks to Cyril Bouthors for the patch.
- Update Copyright date-range and installation instructions that was
still refering to version 5. Thanks to Steve Crawford for the report.
- Fix typo in changelog
Note that new official releases must now be downloaded from GitHub and no more
from SourceForge. Download at https://github.com/dalibo/pgbadger/releases
2015-05-08 version 7.0
This major release adds some more useful reports and features.
* New report about events distribution per 5 minutes.
* New per application details (total duration and times executed) for each
query reported in Top Queries reports. The details are visible from a new
button called "App(s) involved".
* Add support to auto_explain extension. EXPLAIN plan will be added together
with top slowest queries when available in log file.
* Add a link to automatically open the explain plan on http://explain.depesz.com/
* New report on queries cumulated durations per user.
* New report about the Number of cancelled queries (graph)
* New report about Queries generating the most cancellation (N)
* New report about Queries most cancelled.
Here is the full list of changes/fixes:
- Update documentation with last reports.
- Fix number of event samples displayed in event reports.
- Add new report about events distribution per x minutes.
- Add app=%a default prefix to documentation.
- Add reports of "App(s) involved" with top queries. Thanks to Antti Koivisto
for the feature request.
- Remove newline between a ) and , in the beautifier.
- Add link to automatically open the explain plan on http://explain.depesz.com/
- Add support to auto_explain, EXPLAIN plan will be added together with top
slowest queries when available in the log file.
- Add a graph on distributed duration per user. Thanks to Korriliam for the
patch.
- Add tree new report: Number of cancelled queries (graph), Queries generating
the most cancellation (N) and Queries most cancelled lists. Thanks to Thomas
Reiss for the feature request.
- Fix case where temporary file statement must be retrieved from the previous
LOG statement and not in the following STATEMENT log entry. Thanks to Mael
Rimbault for the report.
- Add --enable-checksum to show a md5 hash of each reported queries. Thanks
to Thomas Reiss for the feature request.
2015-04-13 version 6.4
This new release fixes a major bugs in SQL beautifier which removed operator
and adds some useful improvement in anonymization of parameters values.
pgBadger will also try to parse the full csvlog when a broken CSV line is
encountered.
- Make anonymization more useful. Thanks to Hubert depesz Lubaczewski
for the patch.
- Fix previous patch for csvlog generated with a PostgreSQL version
before 9.0.
- Try continue CSV parsing after broken CSV line. Thanks to Sergey
Burladyan for the patch.
- Fix bug in SQL beautifier which removed operator. Thanks to Thomas
Reiss for the report.
- Fix loop exit, check terminate quickly and correct comments
indentation. Thanks to Sergey Burladyan for the patch
Please upgrade.
2015-03-27 version 6.3
This new release fixes some bugs and adds some new reports:
* A new per user details (total duration and times executed) for each query
reported in Top Queries reports. The details are visible from a new button
called "User(s) involved".
* Add "Average queries per session" and "Average queries duration per session"
in Sessions tab of the Global statistics.
* Add connection time histogram.
* Use bar graph for Histogram of query times and sessions times.
There's also some cool new features and options:
* Add -L | --logfile-list option to read a list of logfiles from an external
file.
* Add support to log_timezones with + and - signs for timestamp with
milliseconds (%m).
* Add --noreport option to instruct pgbadger to not build any HTML reports
in incremental mode. pgBadger will only create binary files.
* Add auto detection of client=%h or remote=%h from the log so that adding
a prefix is not needed when it respect the default of pgbadger.
* Redefine sessions duration histogram bound to be more accurate.
* Add new option -M | --no-multiline to not collect multi-line statement
and avoid storing and reporting garbage when needed.
* Add --log-duration option to force pgbadger to associate log entries
generated by both log_duration=on and log_statement=all.
The pgbadger_tools script have also been improve with new features:
* Add a new tool to pgbadger_tool to output top queries in CSV format for
follow-up analysis.
* Add --explain-time-consuming and --explain-normalized options to generate
explain statement about top time consuming and top normalized slowest
queries.
Here is the full list of changes/fixes:
- Update flotr2.min.js to latest github code.
- Add per user detail information (total duration and times executed)
for each query reported in "Time consuming queries", "Most frequent
queries" "and Normalized slowest queries". The details are visible
from a new button called "User(s) involved" near the "Examples"
button. Thanks to Guillaume Le Bihan for the patch and tsn77130 for
the feature request.
- pgbadger_tool: add tool to output top queries to CSV format, for
follow-up analysis. Thanks to briklen for the patch.
- Add geometric operators to SQL beautifier. Thanks to Rodolphe
Quiedeville for the report.
- Fix non closing session when a process crash with message:
"terminating connection because of crash of another server process".
Thanks to Mael Rimbault for the report.
- Add -L|--logfile-list command line option to read a list of logfiles
from a file. Thanks to Hubert depesz Lubaczewski for the feature
request.
- Automatically remove %q from prefix. Thanks to mbecroft for report.
- Do not store DEALLOCATE log entries anymore.
- Fix queries histogram where range was not appears in the right order.
Thanks to Grzegorz Garlewicz for the report.
- Fix min yaxis in histogram graph. Thanks to grzeg1 for the patch.
- Add --log-duration command line option to force pgbadger to associate
log entries generated by both log_duration = on and log_statement=all.
Thanks to grzeg1 for the feature request.
- Small typographical corrections. Thanks to Jefferson Queiroz Venerando
and Bill Mitchell the patches.
- Reformat usage output and add explanation of the --noreport command
line option.
- Fix documentation about minimal pattern in custom log format. Thanks
to Julien Rouhaud for the suggestion.
- Add support to log_timezones with + and - signs to timestamp with
milliseconds (%m). Thanks to jacksonfoz for the patch.
pgbadger was not recognize log files with timezones like 'GMT+3'.
- Add --noreport command line option to instruct pgbadger to not build
any reports in incremental mode. pgBadger will only create binary
files. Thanks to hubert Depesz Lubaczewski for the feature request.
- Add time consuming information in tables of Queries per database...
Thanks to Thomas for the feature request.
- Add more details about the CSV parser error. It now prints the line
number and the last parameter that generate the failure. This should
allow to see the malformed log entry.
- Change substitution markup in attempt to fix a new look-behind
assertions error. Thanks to Paolo Cavallini for the report.
- Use bar graph for Histogram of query times and sessions times.