Sentry no longer ingests unreal engine crash reports as of Unreal Engine 5.0

[redxdev]

Environment

SaaS (https://sentry.io/)

Version

No response

Steps to Reproduce

1. Clone https://github.com/EpicGames/UnrealEngine, checkout something recent on the 5.0 branch (example commit that currently builds on windows and displays this issue: 85fe411)
2. Configure CrashReportClient to use Sentry via modifying DataRouterUrl Engine/Programs/CrashReportClient/Config/DefaultEngine.ini
3. Build the engine/editor.
4. Launch the Unreal Editor, crash it on purpose (open or create a project, type debug crash in the console)
5. Send the crash report

Expected Result

Sentry should process and display the crash for the configured project.

Actual Result

Sentry does not show the crash as having happened.

I have confirmed that CrashReportClient is successfully sending data to Sentry's servers via Fiddler and it sends back an HTTP 200 with a GUID in the response. It never seems to be processed though.

I tracked down the issue to this specific commit: https://github.com/EpicGames/UnrealEngine/commit/a0471b76577a64e5c4dad89a38dfe7d9611a65ef

Epic changed the format of crash uploads slightly and Sentry no longer seems to be able to process them. I've confirmed that reverting the above commit results in crashes successfully appearing in Sentry, so this seems to be a matter of Sentry updating their processing to support the new format.

Thank you!

[redxdev]

It's worth noting that the specific commit that caused this is from after Epic released 5.0 EA2 - which means that the only way to confirm the issue is to build the engine yourself from source (at least until Epic puts out a more up to date preview of 5.0).

5.0 EA2 - works
https://github.com/EpicGames/UnrealEngine/commit/a0471b76577a64e5c4dad89a38dfe7d9611a65ef - doesn't work
Reverting ^ on any commit from after that - works

I can provide a prebuilt "broken" version of CrashReportClient/CrashReportClientEditor for use with a 5.0 EA2 build privately if it would help investigate this issue.

[getsentry-release]

Routing to @getsentry/owners-native for triage. ⏲️

[relaxolotl]

Thanks for the thorough report! We'll definitely take a more in-depth look at this; The direct references to specific commits help greatly in tracking down what could be going wrong on our end. We'll keep the thread updated, and I'll let you know if we need to take you up on the offer on that prebuilt version.

[relaxolotl]

Mostly a note to the team: We probably want to update https://github.com/getsentry/symbolic/blob/master/symbolic-unreal/src/container.rs#L122-L128; It looks like the offending commit stops writing the second header at the end that we're assuming is always there

[jan-auer]

@redxdev could you help me out by mailing me one of these new crash archives to [email protected]? Thanks!

[redxdev]

@jan-auer As far as I know the changes mentioned don't affect how crashes are written to disk - they only affect the protocol unreal uses to communicate with the crash processing service (unless that data gets written to a temp location I don't know about).

Would a fiddler trace (or other capture method) suffice?

[jan-auer]

Thank you, @redxdev for providing a fiddler trace. The issue was closed prematurely, it's been merged into the library and will be rolled out to Sentry.io throughout the day. I'll give an update here.

[redxdev]

Awesome. Thank you!

[jan-auer]

The fix is live on sentry.io and will be included in the next self-hosted release (likely 21.12.0).