Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ref(ingest): annotate transaction consumer with spans #79101

Merged
merged 1 commit into from
Oct 15, 2024

Conversation

mjq
Copy link
Member

@mjq mjq commented Oct 15, 2024

Occasionally we're seeing long ingest_consumer.process_event transactions, but the span tree doesn't reveal the bottleneck. There's a big (200ms+) gap at the start of the trace before the first child span (ingest_consumer._store_event). To track down what's happening during this time, add spans around each potentially time-consuming operation before that span.

Screenshot 2024-10-15 at 10 52 33 AM

Occasionally we're seeing long `ingest_consumer.process_event` transactions,
but the span tree doesn't reveal the bottleneck. There's a big (200ms+) gap at
the start of the trace before the first child span
(`ingest_consumer._store_event`). To track down what's happening during this
time, add spans around each potentially time-consuming operation before that
span.
@mjq mjq requested a review from lynnagara October 15, 2024 14:57
@mjq mjq requested a review from a team as a code owner October 15, 2024 14:57
@github-actions github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Oct 15, 2024
@mjq mjq requested review from 0Calories and a team October 15, 2024 15:21
Copy link

codecov bot commented Oct 15, 2024

Codecov Report

Attention: Patch coverage is 61.90476% with 8 lines in your changes missing coverage. Please review.

✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/sentry/ingest/consumer/processors.py 61.90% 4 Missing and 4 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master   #79101      +/-   ##
==========================================
- Coverage   78.30%   78.29%   -0.01%     
==========================================
  Files        7132     7132              
  Lines      313957   313963       +6     
  Branches    53079    53085       +6     
==========================================
- Hits       245833   245810      -23     
- Misses      61082    61102      +20     
- Partials     7042     7051       +9     

@mjq mjq merged commit 6328d03 into master Oct 15, 2024
50 of 51 checks passed
@mjq mjq deleted the mjq-txn-processor-spans branch October 15, 2024 15:52
cmanallen pushed a commit that referenced this pull request Oct 23, 2024
Occasionally we're seeing long `ingest_consumer.process_event`
transactions, but the span tree doesn't reveal the bottleneck. There's a
big (200ms+) gap at the start of the trace before the first child span
(`ingest_consumer._store_event`). To track down what's happening during
this time, add spans around each potentially time-consuming operation
before that span.
@github-actions github-actions bot locked and limited conversation to collaborators Oct 31, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Scope: Backend Automatically applied to PRs that change backend components
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants