When we generate more log events than we can send

[FantasticFiasco]

@vaibhavepatel we'll continue the discussion here. Let's keep this discussion on what will happen when we generate more log events than we can send over the network.

We can create the sink in three ways. The first one is Http which creates a sink that holds the log events in memory until they are being sent over the network. We can set a limit on the number of events we hold in memory, and if we reach this limit new log events will be dropped. This configuration favours old log events.

The second one is creating the sink using DurableHttpUsingTimeRolledBuffers. This one is storing log events on disk and rotates the buffer files based on time. Each time slot has a buffer with a max size, and if that max size is reached new log events within the same slot is dropped.

The third and final one is created using DurableHttpUsingFileSizeRolledBuffers. This one is also storing log events on disk and rotates the buffer files based on file size. If the buffer overflows, log events in old buffer files are dropped, not overwritten, in favour of new.

Just wanted to clarify the behaviour of the different sink configurations.

[vaibhavepatel]

Thanks for creating this!

Overall, the behavior of all the sinks will drop the oldest logs whether it is in memory or file once the maximum limit is reached.

Now there are two use cases that I have come across when using DurableHttpUsingTimeRolledBuffers.

1. Application A that is writing lots of log files but almost all of them are small in size. Should be ~20kb or smaller and one instance/container can probably produce ~300-500 logs per second.
2. Application B that is writing few log files but majority are large in size due to the data it contains. One instance/container can probably produce ~30 MB worth of logs per second.

In both of these scenarios, based on the sink settings, the http sink can fall behind in pushing the logs to the destination based on the size and number of logs. In this case, the older logs would get dropped, causing the logs to be lost. Is there something that can be done here to improve the throughput of sending logs over the network? Maybe using multiple threads to send in parallel?

I believe you previously mentioned that the above two use cases seem to be pretty rare. It also seems unlikely for applications to generate logs faster than they are sent to the destination. And if they are, it's probably something that the app teams are okay with. However, in my case, logs are starting to be viewed as a potential source for some data analytics and being able to prevent log loss is becoming more important.

Also, I'd like to make sure that I understand the behavior for period. The period is the time waited between the Http Requests, right? So, if the period is 10 ms, then this would be the behavior:

Time | Action
0 ms | nothing
10 ms | Start Http Request 1
40 ms | Http Request 1 finished
55 ms | nothing
60 ms | Start Http Request 2
etc.

[FantasticFiasco]

Overall, the behavior of all the sinks will drop the oldest logs whether it is in memory or file once the maximum limit is reached.

No, this is not true. Only DurableHttpUsingFileSizeRolledBuffers behaves like this, the rest will drop new events until the old have been sent to the log server.

Regarding the bottleneck of sending log events and whether the situation would improve if batches of logs where send in parallell. That would depend on why we have the bottleneck. Without knowing which the least performing part of the system is, we can only make guesses.

Finally regarding the period, it's not a period between two HTTP requests, it's the period between two checks of whether new log events have been written. Let's change the period from 10 ms to 2 seconds, just to make the point. This would be the behaviour.

Time (ms)	Action
0	Nothing
1 - 1999	Assume a lot of log events are written to Serilog
2000	The sink checks whether there are any log events to send, and finds that there are
2000	HTTP request 1 started
2100	HTTP request 1 finished
2100	The sink checks whether there are any other log events to send, and finds that there are
2100	HTTP request 2 started
2200	HTTP request 2 finished
2200	The sink checks whether there are any other log events to send, and finds that there aren't
4200	The 2s period has elapsed, and it's time to check whether new log events have been written by Serilog

In conclusion, the period doesn't control the duration between HTTP requests, it controls the idle time of the sink, where log events can accumulate and being sent in batches. Longer periods means fewer but larger requests, shorter periods means more but smaller requests.