News for version 1.0

The new 1.0 version of the Profiler is out with the following news:

introduced logic provider extensions
added SQLite3 support including free pages inspection
exposed internal database access to extensions
– fixed some issues when executing Python code from other threads
– made actions available in the context of the main window

The 1.0 version represents for its round number and intrinsic meaning a milestone in the development road-map. So how does the actual development stage compare to the original road-map envisioned for 1.0?

Many features we’d like to have included are not there yet. On the other hand an even bigger number of features not originally considered for this version have been added, like complete support for C/C++ types, a JavaScript debugger, an incredibly powerful Python SDK, Lua filters etc.

Talking about file formats, few important ones are still missing. For instance ELF support is yet to be added. The reason for this lies behind the original idea to add support first for Windows file types. That’s why there’s support for esoteric file types like LNK and not for ELF. This strategy has been abandoned already some time ago and as you can see in 0.9.6 we added support for Mach-O executables. Also the overall support for Android (APKs, DEX, Binary XML) is very good and that only makes the support for ELF more important. Apart from that we’re happy with the number of formats currently supported and hope to significantly increase the number next year.

Another important aspect is documentation and tutorials. While we take good care of the blog, we’re certainly guilty in this regard. Unfortunately all time spent documenting is time subtracted from creating new features. We tried to give some practical examples this year (including reversing of malware):

But even there we certainly could do more. The already existing feature set of the standard version would already need half a book to be covered, not counting explanations regarding file formats. Sooner or later an entire book will become necessary, I guess. Early adopters have the advantage of gradually following the development and easily keeping up-to-date with new features. But the term ‘early adopter’ is relative. Become one of our customers today and you’ll become an ‘early adopter’ in a year! 😉

Sorry for the sales pitch, I was saying… Yes, our product offer has increased. Few months ago we have released PE Insider, a free Portable Executable viewer for the community, based on the same code base as Profiler.

Also, we have recently announced an advanced (forensic oriented) edition of Profiler. While this does subtract some time from the standard version, it also drives development a lot and the standard version will greatly benefit from it. For instance, the newly introduced logic providers could’ve been added later were it not for the advanced version. And the benefits won’t come only as extensions to the core and internal components, but also as additional file support as we’re going to show soon.

To mark the current milestone, starting from this version we’ll change the progression of versions. Every new release will increase the minor version (rather than the last number which will be reserved for bug fixes).

We hope you will accompany us in our journey towards 2.0!

Logic Providers

The main feature of the 1.0.0 version of the Profiler is ready and thus it won’t take long for the new version to be released. This post serves as introduction to the topic of logic providers and can in no way cover all the ground.

Logic providers are a new type of extension quite similar to hooks: the callbacks are named the same. Their purpose however is different. Hooks are very powerful, but their purpose is to modify the behavior of generic scans. Logic providers, on the other hand, tell the scan engine which folders to scan, which files, etc.

Let’s take a look at a small logic provider. The ‘logicp.cfg’ entry:

[MissingSecFlags]
label = Missing security flags
descr = Perform a scan inside system and application directories searching for Portable Executables which lack certain security related flags.
file = missingsecflags.py
init = init
scanning = scanning

And the ‘missingsecflags.py’ file:

def init():
    from Pro.Core import proCoreContext
    s = proCoreContext().getSystem()
    s.addPath("C:\\Windows")
    s.addPath("C:\\Program Files") 
    s.addPath("C:\\Program Files (x86)")
    return True

def scanning(sp, ud):
    if sp.getObjectFormat() == "PE":
        obj = sp.getObject()
        # exclude .NET files
        if obj.DotNETDirectory().IsValid() == False:
            # check NX_COMPAT and DYNAMIC_BASE flags
            sp.include((obj.OptionalHeader().Num("DllCharacteristics") & 0x140) != 0x140)
            return
    sp.exclude()

Let’s now take a look at the home view in the Profiler.

Logic provider scan button

As you can see, there’s an additional scan button which belongs to the logic provider we’ve just added. The icon can be customized from the cfg file by specifying an ‘icon’ field (the path is relative to the media folder).

So let’s take a closer look at the code above. When the user clicks the scan button, the init function of our logic provider will be called.

def init():
    from Pro.Core import proCoreContext
    s = proCoreContext().getSystem()
    s.addPath("C:\\Windows")
    # ...
    return True

The init function calls getSystem (which returns a CommonSystem base class). This class can be used to initialize the scan engine. By default the system will be initialized to LocalSystem. A logic provider can even create its own system class and then set it with setSystem. As introduction it’s not useful to inspect all the methods in CommonSystem and every possible use, we leave that for future posts. In this simple case it’s not necessary to implement anything complex, we are performing a scan on the local system and so it’s enough to call the addPath method on the default class returned by getSystem.

The function then returns the True value. It can also return False and abort the scan operation. Any other value will be passed as user argument to other callbacks such as: scanning, scanned and end.

That’s a small difference between hooks and logic providers: the init function in hooks can’t abort a scan operation. Another difference is that while hooks don’t have mandatory callbacks, the init function is mandatory for logic providers, since, without it, nothing gets done. Logic providers start their own scanning operations, while hooks just attach to existing operations (even those created by logic providers).

The scanned function has the same syntax as in hooks and doesn’t require an additional explanation. The only thing worth mentioning is that hooks can be selectively called based on the file format (see formats field). This isn’t true for logic providers: their scanning/scanned callbacks will be called for every file. The logic providers API is recent and not written in stone, so it might very well be that in the future a filtering mechanism for formats will be provided for them as well.

As for the content of the scanned function, it just checks two security related flags inside of a Portable Executable and includes in the final report only those files which miss one or both flags.

Logic provider results

The scan could be made more useful to check for specific things like COM modules which miss the ASRL flag and things like that.

Also the extension doesn’t really fully benefit from the advantages brought by logic providers: it could as well be implemented as a hook, perhaps it would be even better. In this case the only advantage it provides is a shortcut in the home view for the user.

An important aspect of logic providers is that the Profiler remembers which logic providers have been used to create a report and calls their rload callback when loading that report. The rload callback exists even for hooks, but for them it’s called in any case provided the hook is enabled. It’s important to remember that the identifying name for logic providers is the value contained between brackets in the cfg file. If it’s changed, the Profiler won’t be able to identify the logic provider and print an error message in the output view.

Since this version of the Profiler also exposes its internal SQLite implementation, it’s now possible to access the internal database (main.db):

from Pro.Core import proCoreContext
db = proCoreContext().getReport().dataBase() # returns the internal SQLite handle, never to be closed!

Another useful method exposed by the Report class is retrieveFile:

c = proCoreContext().getReport().retrieveFile("C:\\somefile") # returns a NTContainer object

The retrieveFile method retrieves a file based on its name either from the project or from the disk.

Using all these features in conjunction a typical scenario for a logic provider would be:

  • The init callback is called and initializes the scan engine.
  • Information is collected either from the scanning or scanned callback, depending on the needs.
  • The end callback stores the collected data in the main database via the provided SQLite API.
  • The rload callback retrieves the collected data from the main database and creates one or more views to display it to the user.

As already mentioned this post covers only the basics and we’ll try to provide more useful examples in the future.

An analysis module for Android: announcing the Forensic Edition

We’re happy to announce the beginning of our work on a forensic oriented edition of Cerbero Profiler. This edition will contain extensions written on top of the standard edition, which are intended to help forensic analysis of supported platforms.

Let’s start with a demonstrative screenshot:

Android artifacts

(This isn’t how the final UI will look like, it just gives an idea of the sort of information which will be shown. Some columns are collapsed on purpose, because they contain real information.)

The first version aims to include support for the most used platforms. The extensions to support them will be written in Python. The reason for this technical choice is that it will enable our users to easily customize their behavior and even implement additional functionality if needed.

The technology needed to implement custom scanning logic will appear in the upcoming 1.0.0 version. It comes in the form a new type of extension named ‘logic provider’. These extensions tell the Profiler what to scan (and how) and will be displayed on the home page of the main window in the shape of additional scanning buttons:

Android artifacts

The estimated launch date is set to February and the final price is going to be 730 euros for the named license and 880 euros for the computer license. Renewal and upgrade prices have not been decided yet. Until the launch date it is possible to pre-order and obtain the discounted price of 430 euros for the named license and 580 euro for the computer license!

Our current users at the time of writing this post (those with an active support plan or pending orders) can upgrade to the advanced edition for no additional cost, just let us know! We’d like to say thanks to those users for the appreciation of our product and their loyalty.

If you’re unsure about which edition is best suited for your activities, be assured that file format support will continue to be added to the standard edition along with all other core features. The advanced edition only adds automatic tools to extract artifacts from supported platforms.

SQLite3 support and inspection of free pages

The upcoming 1.0.0 version of the Profiler introduces support for SQLite3 databases.

SQLite table

You’ll see that even viewing large tables is pleasantly fast. The SQL table control is available to the Python SDK as well: it can either be created via createView or inside a custom view with the tag sqltable.

Once the sql table view is created, it offers the following methods:

    getSQLColumns() -> NTString
    getSQLCondition() -> NTString
    getSQLTable() -> NTString
    setSQLTable(NTString const & table, NTString const & columns=NTString(), NTString const & condition=NTString()) -> bool
    setSQLTable(NTString const & table, NTString const & columns=NTString()) -> bool
    setSQLTable(NTString const & table) -> bool
    setSQLTableSelectVisible(bool b)
    setSQLite3Object(CFFObject obj)

So it’s possible to display a particular table in it or offer the possibility to the user to choose the table via setSQLTableSelectVisible.

The database can be accessed as well. The Profiler exposes its internal SQLite code in the homonymous module. It differs from the standard Python implementation and it matches the C API. For instance, to enumerate the table names in a database we can use this code:

from Pro.SQLite import *

db = obj.GetHandle() # retrieves the internal SQLite handle, never to be closed!

ret, stmt = sqlite3_prepare(db, "SELECT name FROM sqlite_master WHERE type = 'table'")
if sqlite3_step(stmt) == SQLITE_ROW:
    print(sqlite3_column_text(stmt, 0))
    sqlite3_finalize(stmt)

The handle returned by GetHandle grants only read access. In fact, to maximize speed and avoiding copy operations, the Profiler replaces the virtual file-system of the SQLite database in order for it to read directly from the CFFObject.

The exposed C API can be used to open external databases as well and will be used to access the main report database file in order to give plugins the capability to store and retrieve their own data.

Free pages inspection

When the database file contains free pages, it will be reported in the summary. Free pages usually contain deleted data and can therefore be of interest for forensic purposes.

Free pages

The image above shows a test database I’ve created. In it I created a few tables, and inserted some records containing repeated values (but keeping each record different). Then I deleted a specific record containing ‘1’s. The result is that the database now contains free pages and when inspecting them with the Profiler we can see a big part of the original data.

Keep in mind that data contained in free pages can be incomplete and is scattered. The free pages data can be retrieved programmatically as well through the method GetFreePages.

Stay tuned as there’s much more coming soon!

News for version 0.9.9

The new 0.9.9 version of the Profiler is out with the following news:

added support for docked views in the main window
added scanning and rload (report load) hook notifications
partially exposed custom views to Python
exposed addEmbeddedObject method to Python
exposed NTContainer find methods to Python
improved importing of anonymous records (C11)
– added recognition of volatile keyword in types
– moved the message box constants to the Pro.Core module
– added tools view
– added quoted-printable decoding filter
added format quota calculator extension
added experimental EML attachment detection extension

Improved importing of anonymous records

C11 supports anonymous records like the following:

struct test {
    union {
        struct {
            unsigned int a;
            unsigned int b;
        };
        struct {
            unsigned int c;
            unsigned int d;
        };
        struct {
            unsigned int e;
            unsigned int f;
        };
    };
};

Notice that not only is the union anonymous but even its substructures are. The Header Manager is now capable of correctly importing this code. As usual anonymous types will be renamed (both their type and name).

EML attachment detection and inspection

The upcoming 0.9.9 version of the Profiler includes some very useful SDK additions. Among these, the addEmbeddedObject method (to add embedded objects) and a new hook notification called ‘scanning’. The scanning notification should be used for long operations and/or to add embedded objects. In this post we’ll demonstrate these new features with a little script to detect attachments in EML files.

EML attachments

One of the advantages of using the Profiler is that we are be able to inspect the sub-files of the attachments as well. The screenshot above shows a PNG contained in an ODT attachment. Nice, isn’t it?

But the nicest part is how little code is necessary to extend the functionality of the Profiler. These are the lines to add to the user hook configuration file:

[EML: detect attachments]
file = eml.py
scanning = detectEmlAttachments

And this is the Python code:

from Pro.Core import INVALID_STREAM_OFFSET

def detectEmlAttachmentsCb(offset, npattern, sp):
    c = sp.getObjectStream()
    # hdr range
    m = c.findFirst("\n--".encode("ascii"), 0, offset, False)
    hdrstart = 0 if m.offset == INVALID_STREAM_OFFSET else m.offset
    m = c.findFirst("\r\n\r\n".encode("ascii"), offset)
    hdrend = c.size() if m.offset == INVALID_STREAM_OFFSET else m.offset
    # make sure it's an attachment
    m = c.findFirst("Content-Disposition: attachment".encode("ascii") , hdrstart, hdrend - hdrstart)
    if m.offset == INVALID_STREAM_OFFSET:
        return 0
    # data range
    datastart = hdrend + 4
    m = c.findFirst("\r\n\r\n".encode("ascii"), datastart)
    dataend = c.size() if m.offset == INVALID_STREAM_OFFSET else m.offset
    # retrieve file name (if any)
    name = "no_name"
    m = c.findFirst('name='.encode("ascii"), hdrstart, hdrend - hdrstart)
    if m.offset != INVALID_STREAM_OFFSET:
        namestart = m.offset + 5
        namedel = "\r\n"
        if c.read(namestart, 1) == '"'.encode("ascii"):
            namedel = '"'
            namestart = namestart + 1
        m = c.findFirst(namedel.encode("ascii"), namestart)
        if m.offset != INVALID_STREAM_OFFSET:
            namesize = min(m.offset - namestart, 200)
            name = c.read(namestart, namesize).decode("utf-8")
    # add attachment
    sp.addEmbeddedObject(datastart, dataend - datastart, "?", name, "")
    return 0

def detectEmlAttachments(sp, ud):
    sp.getObjectStream().find(detectEmlAttachmentsCb, sp, "Content-Transfer-Encoding: base64".encode("ascii"))

That’s it. Of course, this is just a demonstration, to improve it we could add support for more encodings apart from ‘base64’ like ‘Quoted-Printable’ for instance.

Some email programs like Thunderbird store EML files by appending them in one single file. In fact, as you can see, the screenshot above displays the attachments of an entire Inbox database. 😉

EML attachment types

Also notice that in the code the addEmbeddedObject method is called by specifying a base64 decode filter to load the file. We can, of course, specify multiple filters and Lua ones as well. This makes it extremely easy to load files without having to write code to decode/decrypt/decompress them. The “?” parameter leaves the Profiler to identify the format of the attachment.

Format quota calculator

In the upcoming 0.9.9 version of the Profiler it will be possible to create docked views even in the context of the main window. This feature combined with custom views is extremely useful if we want to create custom reports at the end of a scan.

Some time ago I needed a little script to calculate the format quotas of files in a specific directory and their sub-files: we’ll use this sample to demonstrate the new features. For example we could use it to determine what kind of files and in what percentage the System32 directory on Windows contains. Or we could use it to determine the quotas of files in a Zip archive. To make it even more useful, the script now asks the user before the scan to enter the nesting range to consider. For example the value ‘0’ means all levels (starting from 0). If we want to calculate the quotas of top level files only, we must insert ‘0-0’ (start-end). The files contained in a Zip archive can be calculated with the value ‘1-1’ and if we want to include their sub-files we must insert ‘1’.

System32 quotas

We’re probably going to include the script in the upcoming release. But in case we don’t, in order to try it out, add the following lines to the hooks configuration file:

[Format Quota Calculator]
file = quotas.py
init = typeQuotaCalcaulatorInit
end = typeQuotaCalcaulatorEnd
scanned = typeQuotaCalcaulatorScanned

And create a ‘quotas.py’ file in your ‘plugins/python’ user directory with the following content:

from os import path
import random

def generateColor():
    c = ""
    for i in range(3):
        c = c + "%0.2X" % ((random.randint(0, 200) + 300) >> 1)
    return c

def typeQuotaCalcaulatorInit():
    random.seed(0)
    # ask for nesting levels to consider
    from Pro.UI import ProInput
    ns = ProInput.askText("Format Quota Calculator (nesting level: from(-to))", "0")
    lstart = 0
    lend = -1
    if ns != None:
        ns = ns.split("-")
        if len(ns) > 0:
            lstart = int(ns[0])
        if len(ns) > 1:
            lend = int(ns[1])
    return { "lstart" : lstart, "lend" : lend, "total" : 0, "quotas" : { } }

def typeQuotaCalcaulatorEnd(ud):
    from Pro.UI import proContext, ProView
    from html import escape
    prec = "%.2f"
    mbsize = 1024 * 1024
    u = ud["total"] / 100
    # prepare content
    s = "Total size: " + (prec % (ud["total"] / mbsize)) + " MBs\n"
    ui = ""
    for k,q in ud["quotas"].items():
        ps = (prec % (q / u))
        ss = (prec % (q / mbsize))
        s = s + "\n" + k + ": " + ps + "% (" + ss + " MBs)"
        ui = ui + ""
    ui = ui + ""
    # display view
    ctx = proContext()
    v = ctx.createView(ProView.Type_Custom, "Format quotas")
    v.setup(ui)
    v.getView(1).setText(s)
    ctx.addView(v)

def typeQuotaCalcaulatorScanned(sp, ud):
    # check nesting
    nesting = sp.scanNesting()
    if ud["lstart"] > nesting or (ud["lend"] >= 0 and ud["lend"] < nesting):
        return
    c = sp.getObjectStream()
    fmt = sp.getObjectFormat()
    # if we didn't recognize the file, use extension as format identifier
    # we could also use an external signature db...
    if fmt == "":
        fmt = path.splitext(c.name())[1]
        if len(fmt) > 0:
            fmt = fmt[1:] # skip dot
    if len(fmt) == 0:
        fmt = "?"
    else:
        fmt = fmt.upper()
    # add to quotas
    size = c.size()
    ud["total"] = ud["total"] + size
    if not fmt in ud["quotas"]:
        ud["quotas"][fmt] = 0
    ud["quotas"][fmt] = ud["quotas"][fmt] + size

Remember to activate the hook from the UI before running a scan.

Of course, the view will be displayed even after an individual file scan in the workspace.

PDF quotas

In order to improve the script, we could use an external signature database for those file formats not recognized automatically.

This is a perfect example of the capabilities to extend the functionality of the Profiler. While there’s yet no estimated release date for the upcoming version, keep in tune as we hope to publish very interesting stuff soon.

Custom Views

The upcoming 0.9.9 version of the Profiler will partially expose the use of custom views. These views are used internally by the Profiler to create complex graphical UIs using short XML strings. While at the moment extensions can use PySide to create complex UIs, it’s better to avoid it if possible, since it involves an extra dependency and also because PySide might not be ported to Qt 5 in the future.

But let’s see a code snippet:

from Pro.UI import *

ctx = proContext()
v = ctx.createView(ProView.Type_Custom, "Debug Directory")
v.setup("<ui><vsplitter><table id='0'/><hex id='1'/></vsplitter></ui>")
ctx.addView(v)

These few lines will display the following view:

Empty custom view

Controls can be organized in layouts (hlayout/vlayout), splitters (hsplitter/vsplitter) and tabs (tab). These elements are called containers. Available controls are: label, pie, plot, table, tree, hex, text and media.

More controls will be available in the future and not all of the current ones can be used as it is. Some controls make sense only in combination with a callback to be notified about changes of the state of the control. The notification system will be made available to Python as well in the future, but it made sense to release a partial solution in the meantime, because many views don’t require notifications and only need a way to display information at the end of an operation.

Let’s see for example how to make use of the UI above to display information.

Custom view

This code replicates the Debug Directory UI in Portable Executables.

from Pro.UI import *

ctx = proContext()
obj = ctx.currentScanProvider().getObject()
dbgdir = obj.DebugDirectory().MakeSingle()
dbgdata = ctx.currentScanProvider().getObjectStream()
dbgdata.setRange(*obj.DebugDirectoryData(dbgdir))

v = ctx.createView(ProView.Type_Custom, "Debug Directory")
v.setup("<ui><vsplitter><table id='0'/><hex id='1'/></vsplitter></ui>")
v.getView(0).setStruct(dbgdir)
v.getView(1).setData(dbgdata)
ctx.addView(v)

Elements in a view can have attributes. We’ve only seen the id attribute used to identify the embedded controls. There are two kind of attributes: shared attributes and individual ones. Only controls have these shared attributes: width, height, min-width, max-width, fixed-width and fixed-height. If a c is prefixed to the width/height word, then the size can be expressed in characters. e.g.: fixed-cwidth=’10’. Additionally, since version 1.3, there’s also wfixed and hfixed. Both are booleans which, if true, set the fixed size policy.

Here’s a list of individual attributes for controls and containers.

  • ui
    • bgcolor (e.g. ffffff)
  • hlayout/vlayout (hl/vl)
    • margin
    • spacing
    • align (hcenter, vcenter, center, top, left, bottom, right)
  • hsplitter/vsplitter (hs/vs)
    • sizes/csizes (separated by -)
  • tab
    • index
    • titles (separated by 😉
  • label
    • bgcolor (e.g. ffffff)
    • select (bool)
    • margin
  • text
    • readonly (bool)
    • linenr (bool, show line number)
    • hline (bool, highlight current line)
    • hword (bool, highlight current word)
    • wrap (bool)
  • combo (since version 1.3)
    • edit (bool)
    • text (string, only if editable)
  • btn (since version 1.3)
    • text (string, only if editable)
  • check (since version 1.3)
    • checked (bool)
    • text (string, only if editable)
  • tline (text-line, since version 2.5)

While this post doesn’t present many usage examples, we’ll try to show additional ones in future posts.

News for version 0.9.8

Since 0.9.7 has been a massive release with lots of changes, we dedicated the new 0.9.8 version of the Profiler to improve things and fix minor bugs. Here’s the change list:

– improved support for Windows 8.1 PEs
– added language options to Header Manager
– improved anonymous types renaming logic
– improved TrueType font disassembler
– many small improvements
– fixed some minor bugs

Since some improvements are PE related, PE Insider has been updated as well.

Enjoy!

News for version 0.9.7

The new 0.9.7 version of the Profiler is out with the following news:

introduced C++ class/struct parsing with Clang
introduced headers, layouts and manual analysis in hex mode
exposed all the above to the Python SDK
added capability to turn into a portable application
– added SHA-3 hashes
– updated Qt to 4.8.5
– updated OpenSSL
– behavior change: displaying table flags now requires a double click

Enjoy!