Whitespace in buffer file

[RENO-911]

Describe the bug
I've been having weird JSON parsing errors showing up in my elastic stack. When looking at the failed JSON the following was visible:

[<>\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000{\"Timestamp\":\"2021-03-11T12:05:57.4021831+00:00\",\"Level\":\"Information\",\"MessageTemplate\":\"HTTP {RequestMethod} {RequestPath} responded {StatusCode} in {Elapsed:0} ms\",\"RenderedMessage\": <more correct json>

When looking at my loggingbuffer file in Azure Kudu i've found what seems to be the issue.

I was wondering if this is a known issue, or if it could maybe stem from a mistake on my part.

configuration.WriteTo.DurableHttpUsingFileSizeRolledBuffers(config["ElasticEndpoint"], @"D:\home\LogFiles\Application\LoggingBuffer", batchFormatter: new ArrayBatchFormatter(ByteSize.MB));

Desktop (please complete the following information):

• OS: Microsoft Windows NT
• Version: 10.0.14393.0

If more context is necessary, i'd be happy to edit out sensitive info and share the whole file.

Much obliged,

[github-actions]

Hi there and welcome to this repository!

A maintainer will be with you shortly, but first and foremost I would like to thank you for taking the time to report this issue. Quality is of the highest priority for us, and we would never release anything with known defects. We aim to do our best but unfortunately you are here because you encountered something we didn't expect. Lets see if we can figure out what went wrong and provide a remedy for it.

[FantasticFiasco]

What version of the sink are you using? Could you try using the newly released 8.0.0-beta.1 and verify the behaviour in that version as well? I did some work on the buffer file reader when it comes to \r\n versus \n, and hopefully this work fixed your issue.

[FantasticFiasco]

Would it also be possible for you to share a scrubbed version of the buffer file? It would help with analysing the issue.

[RENO-911]

Thanks for the fast response, here is the log file (shortened). I'll see if the beta branch fixes this issue, ill yet you know whether it does or not.
faulty-file.txt

[FantasticFiasco]

Thanks! Can you also please check what version of this sink you are using? It would help me to isolate the issue.

[RENO-911]

Unfortunately using the latest beta version does not solve the issue for me. The version i was using before this was 7.2.0.

[FantasticFiasco]

Thanks! Good to know.

[FantasticFiasco]

I've looked into the issue and one thing that came to my mind is that buffer files are created using a BOM. I can't say that this is the reason you are experiencing the issue, but it's my only guess. If it's not I'm at a loss.

Can you please upgrade to v8.0.0-beta.2 and tell me whether the issue still persists or have been fixed?

[FantasticFiasco]

@RENO-911, have you had the opportunity to try the new beta?

[RENO-911]

I haven't had a lot of time this week, the issue might occur when i'm trying to log a dictionary type value. I haven't had the chance to check whether this was the case or not. Hopefully next week i'll be able to verify if it was, or if the new beta fixes things.

[FantasticFiasco]

Thanks! Please let me know either way. If it has something to do with a dictionary I would be interested in that as well.

[twDuke]

Hello, i might have a related problem getting this error from logstash

[2021-04-06T08:51:04,937][ERROR][logstash.codecs.json ][main][e6487ff8595eb70bddb061e16ca2a2cda7d50c56d505d32fbe362c189fb2dc6d] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('' (code 65279 / 0xfeff)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')

iirc 65279 is used as a BOM. I have tried latest 8 beta and still get it

when I use

  {
    "Name": "DurableHttpUsingTimeRolledBuffers",
    "Args": {
      "requestUri": "https://something.something"
    }
  },

edit:
hard to paste the log and make it visible, but the rogue char is appearing as the first char in the batch.
... [Source: (String)"#HERE IT APPEARS#{"events":[{"Timestamp":"2021-04-01T12:57:59.7795031+02:00","Level":"Information", ...

For fun i pulled master and instead of passing the stream as StreamContent in JsonHttpClient, i read the stream to a string and put it as a StringContent, that seeems to work.

Seems completely random when this happens for us, can happen on really trivial log entries like request logs to really big logs of complex objects.

Let me know if I can help in any way.

edit: Might have been to quick adding this comment, as it's about the buffer file, i don't see any issue with my buffer files.

edit2 : nevermind me, this happens when i run locally (on windows) our missing logs is related to something different.

[FantasticFiasco]

I think the buffer files must be the original source of the problem. I haven't previously experienced the BOM being created in the payload. If the BOM was present in the buffer files, it might have been picked up by the buffer file reader and being sent as a part of the payload. But at this point my guess is as good as yours.

Are you being able to reproduce the issue locally or in production? If yes to any of the two, would you be able to share the buffer files, or perhaps only the buffer file to which the specific log events originates from? Of course, please make sure to clean any personal or otherwise sensitive information from them that would be harmful to you or your company. Thanks!

[janichirag11]

We got similar issue, Getting a invisible special character in the log. The model binding throws an error when tried to send the data of the attached file.

When we try to read the buffer file, we found that durable sink while reading sends an additional row which is empty and just has a closing braces.

We have tempararly added a check if we get an empty closing brace, then we skip that record while sending the data. This fix is done in batchformatter.

Could you please check on what could be the cause of this issue?. Thanks in advance

LogBuffer-20211126.zip

[FantasticFiasco]

Is this occurring often in your environment? Is it reproducible in stage/non-production?

[janichirag11]

Yes, we can reporduce this in our development environment

[FantasticFiasco]

@janichirag11, are you also able to reproduce this issue with the latest pre-release?