Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log when a trace is too large to compact #2105

Merged

Conversation

scalalang2
Copy link
Contributor

@scalalang2 scalalang2 commented Feb 14, 2023

What this PR does:
Log when a trace is too large to compact.
If the total bytes of traces exceeds maximum limit, the following messages will be logged.

{
    "msg": "max size of trace exceeded",
    "tenant": "XXXXX",
    "traceId": "XXXXX",
    "discarded_span_count": 100
}

There will be some dicussions in this PR.
Please refer to following comments.

Which issue(s) this PR fixes:
Fixes #1931

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Copy link
Member

@joe-elliott joe-elliott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

two comments. looking good!

modules/compactor/compactor.go Outdated Show resolved Hide resolved
modules/compactor/compactor.go Outdated Show resolved Hide resolved
@scalalang2 scalalang2 force-pushed the scalalang2/log-when-traces-are-too-large branch 2 times, most recently from 42a31fd to 9e848ef Compare February 15, 2023 02:17
Copy link
Member

@joe-elliott joe-elliott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is looking good. two small comments.

i forgot that v2 and vParquet have different paths for counting dropped spans. thank you for tracking this down. i suppose ultimately we'd prefer both act like vParquet does, but that's too much change for this PR.

modules/compactor/compactor.go Outdated Show resolved Hide resolved
tempodb/encoding/vparquet/compactor.go Show resolved Hide resolved
@scalalang2 scalalang2 force-pushed the scalalang2/log-when-traces-are-too-large branch from 9e848ef to df93a32 Compare February 23, 2023 02:36
Copy link
Member

@joe-elliott joe-elliott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Thanks for this improvement. It looks like there's a conflict in the changelog. If you can clean that up we'll merge.

@scalalang2 scalalang2 force-pushed the scalalang2/log-when-traces-are-too-large branch from df93a32 to 3dfe897 Compare February 24, 2023 01:24
@scalalang2
Copy link
Contributor Author

I cleaned changelog, thanks joe :)

@scalalang2 scalalang2 force-pushed the scalalang2/log-when-traces-are-too-large branch from 3dfe897 to 1e0a4fe Compare February 24, 2023 01:27
@mapno mapno merged commit 24afc71 into grafana:main Feb 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Log more information when a trace is too large to compact
3 participants