Embedded images appear as attachments when TNEF is involved

[insomniac2k2]

I have a scenario where I have both tnef and regular mime messages that need attachment processing. Extracting and re-attaching attachments from regular messages works fine, but not when tnef comes into play.

Presently, I am converting the tnef message, and then processing it with existing logic:

if (_message.BodyParts.First<MimePart>().GetType().Name == "TnefPart")
{
    TnefPart tnp = new TnefPart();
    tnp = (TnefPart) _message.BodyParts.First<MimePart>();
    MimeMessage temp_message;
    temp_message = tnp.ConvertToMessage();
    _message.Body = temp_message.Body;
}

The problem seems to be when processing the message body further, any embedded messages appear and extract as attachments. Thus breaking the embedded messages for the recipient. This does not happen on regular messages with embedded images and/or attachments.

Am I doing something wrong with how I am converting. Or is this a bug?

[insomniac2k2]

I attached a tnef email for reference if needed. Due to the limitation of only being able to attach an image, I renamed the file to.jpg. To view, rename to something text readable.

[jstedfast]

I don't really understand what you are trying to do, but you've got a number of things that I find sketchy in your code.

TnefPart tnp = new TnefPart();
tnp = (TnefPart) _message.BodyParts.First<MimePart>();

Why do you create a new TnefPart here? Line 2 is fine, but line 1 makes no sense.

MimeMessage temp_message;
temp_message = tnp.ConvertToMessage();
_message.Body = temp_message.Body;

Are you aware that this code replaces the entire content of the original message with the content of the TNEF part, completely wiping out any other content & attachments that the original message may have had?

Maybe this is your intention, but in case it is not, I figured I should bring that to your attention.

The problem seems to be when processing the message body further, any embedded messages appear and extract as attachments.

Just so you're aware, there's no exact mapping from TNEF to MIME, so it comes down to interpretation. The attachments in the TNEF stream are treated as attachments because that's generally what they are. If that is not the interpretation that you'd prefer, you can clone the ConvertToMessage() logic and make whatever changes you think would make sense or you can iterate over the parts after calling ConvertToMessage() and make any adjustments that you want.

[insomniac2k2]

Thank you for your quick response. I realize how the code may seem off. The overall intention is to redirect internal, exchange based email through a gateway that scans email attachments and takes actions according to set criteria. All other framework is built. The challenge for us is converting TNEF properly without blowing up the email. We have yet to see the a TNEF based email in this scenario that has additional message body. This is why we went the route of overwriting the body entirely.

Sorry, the top part was leftovers from testing. Should have been:
TnefPart tnp = (TnefPart) _message.BodyParts.First();

I will have to look into the ConvertToMessage() logic and see what i can some up with.

[jstedfast]

The problem is that the TNEF data does not contain an AttachDisposition property and so the following code does not run:

https://github.com/jstedfast/MimeKit/blob/master/MimeKit/Tnef/TnefPart.cs#L232

The ContentDisposition is created as a side-effect of setting the attachment.FileName property when an AttachTitle property is encountered here:

https://github.com/jstedfast/MimeKit/blob/master/MimeKit/Tnef/TnefPart.cs#L306

The MimePart.FileName convenience property creates a new ContentDisposition with the default disposition of "attachment".

A "fix" for this would be to add the following code snippet before setting the filename value in various places in TnefPart.cs:

if (attachment.ContentDisposition == null)
    attachment.ContentDisposition = new ContentDisposition (ContentDisposition.Inline);

There are already a few bits of logic similar to that which just do this:

if (attachment.ContentDisposition == null)
    attachment.ContentDisposition = new ContentDisposition ();

Those would also need to be changed to pass ContentDisposition.Inline as an argument.

I'm not sure if I want MimeKit to do this by default, though, because I'm not entirely sure what other people's expectations would be. I think it could be argued either way unless I (or someone else?) can find some sort of TNEF documentation which suggests that the default disposition should be treated as "inline".

[jstedfast]

Was the problem just images being marked as "attachment" instead of "inline"? In your sample, I discovered that the image001.jpg had PR_ATTACH_FLAGS set to a value of 4.

Based on the docs I found, it suggests that these attachments should possibly get a disposition of "inline".

Docs: https://msdn.microsoft.com/en-us/library/ee160554%28v=exchg.80%29.aspx

[jstedfast]

If my understanding is correct, then the patch I just committed should solve your problem.

Let me know if I'm on the right track.

[insomniac2k2]

Thanks for looking deeper into this. Yes, the message body was coming through fine. It was just the embedded images were being taged as regular attachments and not inline as you stated.

In further troubleshooting, just by replacing the attachment label to inline did fix the message(embedded images arrived as embedded). I'm so glad that I wont have to parse the message body and do this. That would be a bit heavy.

I'll give your patch a go and see if the issue is resolved.

[jstedfast]

Have you had a chance to test the changes? I'm going to try and make a release this weekend but would prefer to know if my changes have fixed things for you before I do (I don't want to make 2 releases when I can get away with just 1 ;-)

[jstedfast]

I'm going to assume this was fixed (it seemed to be on my end).

[jstedfast]

Just released MimeKit 1.0.14 on NuGet with this fix.

[insomniac2k2]

My apologies. I've been away and very busy. I gave that fix a try, and it did in fact work. Although I did run into a seemingly brand new problem with some converted messages having their charset being changed. e.g. "..." was being changed to … . I hadn't gotten time to isolate if it was a "me thing" or a "you thing", so i neglected to respond immediately (then got bogged down shortly after). Thank you for the release. I will grab it and give it a try again.

[insomniac2k2]

Update:

I tested your build. Converted TNEF messages are preserving embedded images correctly, but I am definitely seeing charset being changed all of a sudden (I know it should be unrelated, but Im not getting it in previous mimekit versions). I am able to create the issue by putting a simple "..." in the message body. Which converts to … . I will continue to look around to see if i am causing this.

Just wanted to list it here, in case others are seeing this behavior.

[jstedfast]

At what point is the "..." being replaced with "… "? Is this happening in MimeMessage.ToString()? If so, don't use MimeMessage.ToString(), use MimeMessage.WriteTo(stream) instead.

[insomniac2k2]

I am not using ToString() for any conversion.At this point, I am checking for tnef and converting it.

            if (_message.BodyParts.First<MimePart>().GetType().Name == "TnefPart")
            {
                TnefPart tnp = (TnefPart)_message.BodyParts.First<MimePart>();
                MimeMessage temp_message;
                temp_message = tnp.ConvertToMessage();
             }
               _message.Body = temp_message.Body;

I believe that I found my problem but will need to do more testing to make sure. Our exchange server character set was "None". It seems that Mimekit may need Exchange to specify and/or encode some soft of default charset in order to interpret the tnef message correctly? UTF-8 seems to be working at first glance,

[jstedfast]

Is this in the HTML body?

[insomniac2k2]

Yes, it is HTML body. Plain text does not run into this issue. I have also found a perfect way to re-create the issue since last. It seems that if I process a tnef message and then it is received by an email server that converts it back to tnef. The message looks ok. If I then send the message back through the same process, for some reason the charset gets changed. I know that sounds like a big /facepalm, but this is what i'm working with ;)

I have also invalidated my previous assumption that I fixed the issue with changing charset on the exchange server :/

I also spent a bit more time using another third party library for converting a tnef message, and sending the message back through processing. I did not run into the same issue ("..." is not being converted to … ).

So i cannot necessarily say that it is a problem with mimekit. It could just be the way im using it? You may want to close this issue out. As it seems like embedded images are definitely being retained correctly. If you want to try to pursue my present issue w/ tnef conversion, i'd be more than happy to try to help in any way i can.

[jstedfast]

A sample tnef message would be really helpful. When you write out the message after extracting it from the tnef attachment, does the HTML body have a charset parameter? It should be utf-8.

[insomniac2k2]

I cleaned up some header info so that i could post it. I am attaching the tnef message and its converted output. I am also attaching a second run (where the break happens). tnef and converted output.

I renamed to jpg so i could attach.

[jstedfast]

Okay, so the problem is that the original character is encoded as … in the original message body.

Then the message gets forwarded and something decodes that character to be ….

This does not appear to be a MimeKit issue since it does not know how to decode HTML entities.

[insomniac2k2]

Yeah, I followed that path as well. The odd thing is that if i use a different library in this same exact scenario (first and second pass), the message comes through clean every time. I'll keep banging my head against the wall on this for a bit. Hopefully i can make some time tomorrow to take the output that i get from the other library and try to make some comparisons.

[jstedfast]

can you compare the differences between what the other tnef decoder gives you?