Changes to handle OST 2013 files

With Outlook 2013, Microsoft changed the format used for OST (and NST) files to account for larger file sizes.

My starting point was this blog entry called "OST 2013 File Format - The Missing Documentation".

1. In the file header, the value of wVer is 0x24. (084c968)
2. The number of entries in the BTPage structure is increased to 4056, requiring the following changes (2c239ac):
   • The size of the rgEntries field is increased to 4056
   • The cEnt and cEntMax fields widened to two bytes since they now have to have a higher range.
3. BBTEntry has a new field to indicate the expanded size of a block for compressed blocks. (If the expanded size and the actual size are the same, the block is not expanded). (ab166ae)
4. The block size is increased to 65512 bytes (efada4c)
5. Blocks can be zip-compressed (c483635)
6. Block trailer size has increased to 24 bytes (9b954a0)
7. HID has three new 1-bit fields which cannibalizes part of the hidBlockIndex field. (The meanings of these flags is not documented.) (336b799)
8. Blocks are aligned on 512-byte boundaries instead of 64-byte boundaries (17a13ae)

In addition to the changes listed there, I found the following differences:

1. It is possible for a folder to have a non-zero content count but not have an associated Contents Table. This was observed with the "Reminders", "To-Do Search", and "Contact Search" folders under the "Finder" folder. (3f199b8)
2. The Message Store for OST files does not include the IpmSubTreeEntryId, so the root folder needs to be found using a different mechanism - by searching for the NBTEntry which has itself as its parent. (2423d0d)
3. A new IPM type, "IPM.ApchPerson", has been introduced. This appears used for the entries in the /Root-Mailbox/IPM_SUBTREE/PersonMetadata folder. It is otherwise undocumented. (2615fed and 4de4432)
4. The message objects within /Root-Mailbox/IPM_SUBTREE/PersonMetadata contain a named property "AggregatedItemLinkIds" which reports a property type of Ox1048 (PtypMultipleGUID), but it does not conform to the description of this type in MS-OXCDATA: 2.11.1 Property Data Types, which states that it should contain "a COUNT field followed by that many PtypGuid values." It is instead a single 32-bit Integer. I have not encountered any other entities with the type PtypMultipleGUID, so I have modified the code to treat this type as a 32-bit integer and optionally logging that a PtypMultipleGUID object has been found, rather than throwing an exception. (d22d610) Note that this may indicate corruption in the underlying OST file.
5. The type of PidLidPercentComplete changes from PtypFloating64 to PtypFloating32, so I had to implement a reader for this type. (25aaab5)
6. The offsets into blocks are read as shorts (which are signed in Java), but they can be large enough that they wrap to negative values. This necessitates ensuring we mask off the upper 16 bits when converting to ints. (787a7d4)

Known Issues

1. In folder table contexts, the tag 0x0042001f (SentRepresentingNameW) contains garbage values. It has the correct value in the message Property Context. For example, in one example test file, in the Deleted Items folder, the value in the SentRepresentingNameW field was incremented for every row whose Cell Existence Bitmap indicated it was present, making it almost duplicate the row index, but the values in the heap-on-node it pointed to were often binary values and seldom the expected name values. None of the analysis I have done has pointed me in the right direction here.