EML attachment detection and inspection

The upcoming 0.9.9 version of the Profiler includes some very useful SDK additions. Among these, the addEmbeddedObject method (to add embedded objects) and a new hook notification called ‘scanning’. The scanning notification should be used for long operations and/or to add embedded objects. In this post we’ll demonstrate these new features with a little script to detect attachments in EML files.

One of the advantages of using the Profiler is that we are be able to inspect the sub-files of the attachments as well. The screenshot above shows a PNG contained in an ODT attachment. Nice, isn’t it?

But the nicest part is how little code is necessary to extend the functionality of the Profiler. These are the lines to add to the user hook configuration file:

[EML: detect attachments]
file = eml.py
scanning = detectEmlAttachments

And this is the Python code:

from Pro.Core import INVALID_STREAM_OFFSET

def detectEmlAttachmentsCb(offset, npattern, sp):
    c = sp.getObjectStream()
    # hdr range
    m = c.findFirst("\n--".encode("ascii"), 0, offset, False)
    hdrstart = 0 if m.offset == INVALID_STREAM_OFFSET else m.offset
    m = c.findFirst("\r\n\r\n".encode("ascii"), offset)
    hdrend = c.size() if m.offset == INVALID_STREAM_OFFSET else m.offset
    # make sure it's an attachment
    m = c.findFirst("Content-Disposition: attachment".encode("ascii") , hdrstart, hdrend - hdrstart)
    if m.offset == INVALID_STREAM_OFFSET:
        return 0
    # data range
    datastart = hdrend + 4
    m = c.findFirst("\r\n\r\n".encode("ascii"), datastart)
    dataend = c.size() if m.offset == INVALID_STREAM_OFFSET else m.offset
    # retrieve file name (if any)
    name = "no_name"
    m = c.findFirst('name='.encode("ascii"), hdrstart, hdrend - hdrstart)
    if m.offset != INVALID_STREAM_OFFSET:
        namestart = m.offset + 5
        namedel = "\r\n"
        if c.read(namestart, 1) == '"'.encode("ascii"):
            namedel = '"'
            namestart = namestart + 1
        m = c.findFirst(namedel.encode("ascii"), namestart)
        if m.offset != INVALID_STREAM_OFFSET:
            namesize = min(m.offset - namestart, 200)
            name = c.read(namestart, namesize).decode("utf-8")
    # add attachment
    sp.addEmbeddedObject(datastart, dataend - datastart, "?", name, "")
    return 0

def detectEmlAttachments(sp, ud):
    sp.getObjectStream().find(detectEmlAttachmentsCb, sp, "Content-Transfer-Encoding: base64".encode("ascii"))

That’s it. Of course, this is just a demonstration, to improve it we could add support for more encodings apart from ‘base64’ like ‘Quoted-Printable’ for instance.

Some email programs like Thunderbird store EML files by appending them in one single file. In fact, as you can see, the screenshot above displays the attachments of an entire Inbox database. 😉

Also notice that in the code the addEmbeddedObject method is called by specifying a base64 decode filter to load the file. We can, of course, specify multiple filters and Lua ones as well. This makes it extremely easy to load files without having to write code to decode/decrypt/decompress them. The “?” parameter leaves the Profiler to identify the format of the attachment.