Problem with headers' serialization

[force-net]

Describe the bug
Headers are incorrectly serialized to raw values, as result DKIM can fail and mail clients are incorrectly displays subject.

To Reproduce

Test code:

var mimeMessage = new MimeMessage();
mimeMessage.Subject = "Тестовый заголовок письма";
mimeMessage.From.Add(new MailboxAddress(Encoding.UTF8, "Раз\xa0два\xa0три\xa0четыре\xa0пять\xa0шесть\xa0семь\xa0восемь", "[email protected]"));
// replacing spaces to underscores to show problem
Console.WriteLine(mimeMessage.ToString().Replace(' ', '_'));

Result:

From:
_=?utf-8?b?0KDQsNC3wqDQtNCy0LDCoNGC0YDQuMKg0YfQtdGC0YvRgNC1wqDQv9GP0YLRjMKg0YjQtQ==?=
_=?utf-8?b?0YHRgtGMwqDRgdC10LzRjMKg0LLQvtGB0LXQvNGM?=_<[email protected]>
Date:_Fri,_30_Nov_2018_17:27:20_+0300
Subject:_
_=?utf-8?b?0KLQtdGB0YLQvtCy0YvQuSDQt9Cw0LPQvtC70L7QstC+0Log0L/QuNGB0YzQvNCw?=
Message-Id:_<9M16H8QW36U4.58648BYKPU1R3@computer>

Problems:

1. From header has empty value, with data on next string. It causes problems with DKIM checking
2. From header value length is more than 78 symbols (may be a problem, but SMTP can handle this)
3. Subject header has space value and real data on new line. It causes space in Subject after decoding in mail clients, also, DKIM check is failed.

Currently, I 'fix' problem by trimming Header.rawValue for each headers (with reflection), and all became good, but it seems, there are big problems with header serialization.

[jstedfast]

1. Bzzzzt, wrong. Check your own results. The leading _ tells a different story.
2. It's not an issue. The 78 characters is a suggestion, not a hard requirement. The only hard requirement is 998 octets for SMTP.
3. Yes, this is a bug in that there ideally should only be 1 space, but this is not causing DKIM-Signatures to fail.

None of these things are causing DKIM-Signatures to fail.

[force-net]

1 & 3. It causes DKIM check to fail with sendmail and opendkim. We've spent two days to determine issue and found that this serialization cause dkim=fail reason="signature verification failed"
It can be correct by RFC, but real software considers that signature is invalid. As result email is going to spam. It seems, it not really good situation.

Also, 3 is really bug, should I create separate issue?

[jstedfast]

No, I've got a fix already for issue 3 (just writing up some unit tests for it).

If issues 1 and 3 are really causing failures in opendkim, then opendkim is buggy. MimeKit is producing perfectly valid folded headers according to the RFCs.

I would dare you to prove otherwise.

[jstedfast]

Here's the results with the Subject header folding fix:

Date:_Fri,_30_Nov_2018_10:22:37_-0500
Subject:_=?utf-8?b?0KLQtdGB0YLQvtCy0YvQuSDQt9Cw0LPQvtC70L7QstC+0Log0L/QuNGB0YzQvNCw?=
Message-Id:_<[email protected]>

[force-net]

May be DKIM is buggy, It is really acceptable. But it is widely used, and causes problems with DKIM.
It seems, that fix in MimeKit is simplier that fix in opendkim. You need just to remove new line separator at start. 78 symbols is not hard requirement - it will not cause additional problems, but will eliminate problem with opendkim.

MimeKit is RFC compliant, I agree, but you can make it better and it will still follow RFC :)

[jstedfast]

Next time, don't say "but it seems, there are big problems with header serialization" because that just wasn't true. These were both very minor and it was really a bug in opendkim. At most you could say that MimeKit produced results that were less than perfectly ideal, but when talking about rfc2047 encoding headers... good god, perfect is unachievable.

For more reading, this is a great blog post by a Thunderbird developer: http://quetzalcoatal.blogspot.com/2013/10/why-email-is-hard-part-2.html

[jstedfast]

FWIW, these are the latest results:

From:_=?utf-8?b?0KDQsNC3wqDQtNCy0LDCoNGC0YDQuMKg0YfQtdGC0YvRgNC1wqDQv9GP0YLRjMKg0YjQtQ==?=
_=?utf-8?b?0YHRgtGMwqDRgdC10LzRjMKg0LLQvtGB0LXQvNGM?=_<[email protected]>
Date:_Fri,_30_Nov_2018_10:59:13_-0500
Subject:_=?utf-8?b?0KLQtdGB0YLQvtCy0YvQuSDQt9Cw0LPQvtC70L7QstC+0Log0L/QuNGB0YzQvNCw?=
Message-Id:_<[email protected]>

[force-net]

Thank you for fix.
In my message, I've specially added it seems, because I was not sure, but it seems... Ok, I was wrong

[jstedfast]

OpenDKIM indeed has a bug filed about this (multiple, actually).

Here's one of them: https://sourceforge.net/p/opendkim/bugs/255/

It's been filed for over a year and the developer marked it as a duplicate of an even older bug :-\

[jstedfast]

I've just released MimeKit 2.1.0 with these fixes.