Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V1.2 compaction error fix #1113

Merged
merged 4 commits into from
Nov 15, 2021
Merged

Conversation

mdisibio
Copy link
Contributor

What this PR does:
This PR patches over #1107 by logging a warning and continuing instead of failing with an error. Research concludes that somehow empty traces are being written to blocks. The compactor bookmarks have already been unintentionally skipping empties in most cases, but there are a few edge cases that are not skipped and led to this error. Therefore skipping all seems ok. We have not yet determined how the empty traces originate, or what changed between 1.1 and 1.2, but this fix should alleviate 1.2 troubles for the users encountering this bug.

Which issue(s) this PR fixes:
Fixes #1107

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Copy link
Member

@joe-elliott joe-elliott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, are we comfortable with the query path with these empty traces?

@mdisibio
Copy link
Contributor Author

To anyone who is interested in testing it out now, docker image was pushed as mdisibio2/tempo:0b566f48d. The new warning log is multiblock iterator skipping empty object...

@mdisibio mdisibio force-pushed the v1.2-compaction-error-fix branch from 0b566f4 to 3b9aad5 Compare November 15, 2021 19:29
@mdisibio mdisibio merged commit 6e99896 into grafana:main Nov 15, 2021
mdisibio added a commit that referenced this pull request Nov 15, 2021
* Skip and log empty objs in multiblockiterator, comment heavily on the predicament

* oops delete comment out changes

* text

* changelog
@mdisibio mdisibio deleted the v1.2-compaction-error-fix branch April 25, 2023 18:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Compaction sometimes fails with "failed to find a lowest object in compaction"
2 participants