Event JSON representation exceeds the body size limit

[jmalczak]

Hi guys

If multiple apps (mobile) write to seq using the same api key and I receive "Event JSON representation exceeds the body size limit" how can I figure out which app cause the issue? It looks like my whole seq is polluted with those messages but I can't find a way to figure out which app cause that since I was sending app id as part of the log body which is not present in the body sample included with this error event?

[nblumhardt]

Hi Jakub,

Setting an API key per log source is the intended mechanism to use on the Seq side and will be handy in the long run, if you're able to retroactively do it now.

Another option is to briefly increase the limit and try to spot some large events in a short window (say 10 mins). These messages are generated client-side - I'm guessing by Serilog in your apps? - so you'd need to reconfigure your client logger in order to change the limit anyway, so using a unique API key is probably a similar amount of work and a better solution overall.

Hope this helps,
Nick

[jmalczak]

Hey Nick

Thanks for the reply. I do use Serilog and as you wrote I would have to reconfigure client to change the limit. Regarding using an API key per log source. How do you guys advise to handle mobile clients that can log to Seq. I can imagine having a mobile app with hundreds of instances installed. In this scenario having API key per log source seems really difficult to implement.

Thanks

[nblumhardt]

datalust/serilog-sinks-seq#193

[nblumhardt]

5.2.3-dev-00260 should improve this (now on NuGet) - it'd be great to hear whether this helps, @jmalczak

[jmalczak]

@nblumhardt That's great. Thanks. I will test it and let you know if it solved the issue!

[jmalczak]

@nblumhardt Sorry for the late reply. It works great!

[nblumhardt]

Awesome, thanks for the follow-up! 👍

[malovicg]

@nblumhardt I am also facing the same message. If I understood correctly, current implementation is, when message limit is exceeded, it only displays 512 bytes (sample). I can see two problems with sample output: it is not formatted, and sometimes its short. Example:

Event JSON representation exceeds the body size limit 262144; sample: {"@t":"2023-06-29T13:28:44.4061815Z","@mt":"Saving data for {Type} failed in {DatabaseName}","@l":"Error","@x":"Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while saving the entity changes. See the inner exception for details.\r\n ---> MySqlConnector.MySqlException (0x80004005): Data too long for column 'StreetName' at row 1\r\n at MySqlConnector.Core.ResultSet.ReadResultSetHeaderAsync(IOBehavior ioBehavior) in /_/src/MySqlConnector/Core/ResultSet.cs:line 43\r\n at MySqlConnector.

Is it possible to first format the sample value? And also it would be nice to be able to control sample limit as parameter, so user can increase it (ex eventBodySampleLimitBytes). So, someone can put 1024 for their scenario.

[KodrAus]

Hi @malovicg 👋

The message for events exceeding the size limit aren't really an attempt to capture a portion of the event, just to provide a rough fingerprint you can use to help track down where they came from. Events with .NET stacktraces can get quite big, so you might need to increase your body size limit to accommodate them.

[malovicg]

Hi @KodrAus

If you look at my example, you will see that sample is not really helpful in determining where the problem is. I am building an app with multitenant hybrid architecture, and in this case I am updating data using generic methods to multiple databases. Exception does not tell me which type? Which database? It would be nice if those are resolved in actual message (sample or not). Increasing the body size limit is not really the best option to cope with this kind of things, since there is always a larger message than you anticipated. And if that happens, you would like to get a meaningful message.

In my current case, for example, I am using https://www.nuget.org/packages/EntityFrameworkCore.Exceptions.MySQL.Pomelo to get more meaningful data for EF Core MySql exceptions. And in this particular case, I am passing in batches 100 records and the size of exception message is approx 20MB!!! I really do not want to store this big message. I have done that just for curiosity, to check if message is resolved correctly, and it does say: Saving data for Supplier failed in db23 - which is very specific.

What is the alternative? Adding more logging information, like method started / finished? That is not really the best solution. This is the reason why I suggested these two improvements, if they are possible, and not overly complex to implement.

[KodrAus]

@malovicg If you're just trying to verify that those large events have some specific data in them through Seq you should be able to temporarily increase your body limit so they'll be ingested. If you're going to be receiving those large payloads regularly you could consider truncating them in your application before logging them.

[malovicg]

@KodrAus No, I am catching exceptions that I did not expect to happen. As I said, increasing limit is not very handy, since Seq then hangs a lot, due to reading large message, and obviously I do not need that large message in log. But I also cannot safely truncate the exception, since I do not know all the scenarios it could happen, and what is important and what not. All I do is:

Logger.LogError(ex, messageTemplate);

and that results in very large log. Even if I could do all these things you suggest, I just do not understand the argument against properly formatting sample message, since in many cases, even with 512 size, it can be very useful in identifying the problem.

[KodrAus]

I think we could certainly look at highlighting the ingestion log itself in the UI 👍 It's not particularly easy to scan right now.

What's important though is that the purpose of the ingestion log is not to try capture as much of an event that wasn't ingested as possible, it's just a short in-memory rolling buffer, so there's no guarantee that data will be in there, or that it will stay there. It works that way because it has to consider abuse vectors of clients flooding the system to intentionally or unintentionally DoS it.

Ultimately, if your system is generating 20MB exceptions, and you don't want to store 20MB events, then you'll need to truncate that information somehow before it arrives at Seq. If you are finding the performance of your Seq instance is particularly bad though then please do feel free to send us a diagnostic report at [email protected] and we'll gladly take a look for you 🙂 Seq is expected to handle large events with big stacktraces.