Exposing the Core (part 1)

The main feature of the upcoming 0.9.3 version of the Profiler is the expansion of the public SDK. This basically means that a consistent subset of the internal classes will be exposed. Although it’s a subset, there’s no way to document all methods and functions. Fortunately, many of them should be quite intuitive.

Some of the most common important classes are:

  • NTContainer: this is a generic container which is used to encapsulate data such as files and memory. It’s an extremely important class, since it’s used extensively. Containers can for the time being be created through SDK functions such as: createContainerFromFile/newContainer.
  • NTBuffer/NTContainerBuffer/CFFBuffer/etc.: used to efficiently read iteratively small amounts of data from a source.
  • NTTextStream/NTTextBuffer/NTTextStringBuffer: used to output text. Indentation can be specified.
  • NTXml: used to parse XML. Fast and secure. This class is based on RapidXML.
  • CFFObject: the class from which every format class inherits (ZipObject, PEObject, etc). A very small subset of this class is exposed for now. This will change in the future.
  • CFFStruct: representation of a file format structure.
  • CFFFlags: representation of flags in a CFFStruct.

One of the new additions is that Python can now use filters as well. Do you remember the post about Widget and Views? Let’s use the same code base and change just a few lines:

from Pro import *
from PySide import QtCore, QtGui
 
class MixedWidget(QtGui.QSplitter):
    def __init__(self, parent=None):
        super(MixedWidget, self).__init__(parent)
 
        self.setWindowTitle("Mixed widget")
        self.setOrientation(QtCore.Qt.Vertical)
 
        self.model = QtGui.QDirModel()
        tree = QtGui.QTreeView()
        tree.setModel(self.model)
        self.addWidget(tree)
 
        ctx = proContext()
        self.hex = ctx.createView(ProView.Type_Hex, "")
        self.addWidget(self.hex.toWidget())
 
        tree.activated.connect(self.updateFile)
 
    def updateFile(self, idx):
        if self.model.isDir(idx) == True:
            self.hex.clear()
        else:
            # modified lines
            name = self.model.filePath(idx)
            c = createContainerFromFile(name)
            fstr = ""
            c = applyFilters(c, fstr)
            self.hex.setData(c)
            # end
 
 
ctx = proContext()
w = MixedWidget()
v = ctx.createViewFromWidget(w)
ctx.addView(v)

With just three of the modified lines we are xoring all opened files with the value 0xCC and then show the resulting data in the hex view. The Profiler provides a huge number of filters for any kind of operation and they can be chained, so we could easily compress and then encrypt a file with AES by just replacing one line in the sample above. The function applyFilters displays an optional default wait dialog to the user to interrupt the operation (if it is executing in the main thread). Please remember that the easiest way to obtain the needed filters XML string is to use the UI view and use the export command from the list (context menu->Export…).

NTBuffer generates an exception when a read operations fail. Thus, it should be used as follows:

ctx = proContext()
v = ctx.getCurrentView()
d = v.getData()
b = NTContainerBuffer(d, ENDIANNESS_LITTLE, 0)
print(str(hex(b.u8())))
try:
    b.read(10) # or b.u8(), b.u16(), etc.
except IndexError as e:
    print(str(e))

A small snippet to show how to use NTXml:

x = NTXml()
ret = x.parse("")
if ret == NTXml_ErrNone:
    n = x.findChild(None, "r")
    if n != None:
        n = x.findChild(n, "e")
        if n != None:
            a = x.findAttribute(n, "t")
            if a != None:
                print(x.value(a))

Along with the core, several of the file objects will be exposed. A text dump of a structure could be as easy as:

c = createContainerFromFile(fname)
pe = PEObject()
pe.Load(c)
out = NTTextStringBuffer()
pe.DosHeader().Dump(out) # CFFStruct::Dump
print(out.buffer)

Please notice that the code above misses several checks. We need to make sure that c is valid and Load succeds. I’ll omit these checks here to keep the code minimal.

You might say that printing out a single structure is an easy task. So let’s take a look at another cooler sample:

c = createContainerFromFile(fname)
pe = PEObject()
pe.Load(c)
out = NTTextStringBuffer()
tables = pe.MDTables("#~") # 'tables' references all .NET metadata tables
pe.DisassembleMSIL(out, 0x06000001) # .NET token (MethodDef | index)
print(out.buffer)

These few lines output an entire .NET method such as:

private static void Main(string [] args)
{
 locals: int local_0,
         int local_1

 ldc_i4_2
 stloc_0 // int local_0
 ldloc_0 // int local_0
 stloc_1 // int local_1
 ldloc_1 // int local_1
 ldc_i4_1
 sub
 switch
  goto loc_22
  goto loc_60
 br_s loc_71
loc_22:
 try
 {
  ldstr "h"
  call System.Console::WriteLine(string) // returns void
  leave_s loc_81
 }
 catch (System.ArgumentNullException)
 {
  pop
  ldstr "null"
  call System.Console::WriteLine(string) // returns void
  leave_s loc_81
 }
 catch (System.ArgumentException)
 {
  pop
  ldstr "error"
  call System.Console::WriteLine(string) // returns void
  leave_s loc_81
 }
loc_60:
 ldstr "k"
 call System.Console::WriteLine(string) // returns void
 ret
loc_71:
 ldstr "c"
 call System.Console::WriteLine(string) // returns void
loc_81:
 ret
}

Nice, isn’t it? Remember we can change the indentation programmatically.

Of course, it will also be possible to get the object currently being analyzed and similar stuff. But we’ll see how to do that in another post.

If you’re wondering why the case convention for methods is not always the same, the reason is simple. CFFObject/CFFStruct/etc are based on older code which followed the Win32-like convention. Consequently all derived classes like PEObject follow this convention. All other classes use the camel-case convention.

News for version 0.9.2

The new version of the Profiler is out with the following news:

removed virtual memory constraint: large files are now supported
added decompression bomb detection
added media preview for image files
added preview for several PE resources
added text preview for Office Word Documents
added format selection to open file dialog
display format choose dialog when more than one format has been detected
added XFA interactive forms detection inside PDFs
added from/to hex and base64 filters
automatically detect files in Zip archives missing a Central Directory
increased PySide integration
– fixed Office VBA extraction bug
– fixed bug in PDF V4 and V5 Revision encryption

Format detection & selection

To better help with the identification of files which can be interpreted as different formats, the individual file dialog features now some additions.

As you can see the identified formats for the currently selected file are listed (it’s a simple GIF file with a PDF appended at the end). The dialog gives the user also the ability to manually choose the format to use for loading the file. While all this could be achieved even before, it wasn’t as handy as it is now.

However, it wouldn’t make sense to display the file selection dialog when the user uses the shell integration or drops a file to open it. So, instead the Profiler displays a choice dialog for the format in case multiple formats are detected.

Conversion filters

Some new filters are available: from/to hex/base64.

While the actions in the Profiler already feautured a mechanism to do these conversions, having them as filters is extremely useful, because it allows to use them to load embedded files or to convert large portions of data.

Damaged Zip archives

While it has always been possible to manually extract through filters data or partial data from damaged Zip files (e.g. those missing a Central Directory), now the embedded data is automatically analyzed and ready for inspection. This means that even when a Zip archive is truncated and some compressed files are truncated as well, they will nonetheless be automatically detected and be available for inspection by the user.

As you can see many improvements have been introduced. The most important of them is of course the removal of the virtual memory constraints as it represents an important step in the roadmap of the Profiler. Stay tuned as the next version will be important as well!

XFA Interactive Form Inspection

The upcoming 0.9.2 version of the Profiler introduces detection of Acro/XFA interactive forms inside PDFs. This technology has been abused numerous times (some recent cases come to mind), so it is now being reported as a potential threat.

The video below shows the inspection of a XFA Interactive Form and how to load a base64-encoded GIF image embedded in it.

Stay tuned!

Previews

The upcoming version 0.9.2 of the Profiler adds previews for various things: images (all supported formats), several Portable Executable resources and Office Word Documents (text-only).

PE resources preview

Since media elements are rendered through third-party code, the Profiler displays a warning box before actually rendering a media element.

Preview warning

The ‘Allow all’ button allows media elements for the current session only. If the Profiler is running in a safe environment (like a VM), the user can decide to permanently disable the warning box and allow all media elements.

Preview settings

Last but not least, text-only preview of Office Word Documents has been introduced. This allows users to safely inspect the text content of a document without processing the file with an official viewer which could be the target of exploits.

Office document preview

While there are already enough new features to release, some smaller additions will be squeezed into 0.9.2 during the next days. Stay tuned!

Zip bomb revisited

The upcoming 0.9.2 release of the Profiler removes the virtual memory constraint, meaning that it is now able to open and process files of any size (the hex editor can edit large files as well). This feature has actually been in the TODO list from day 1 and I’ll write about the internals of it in some other post, in order to better demonstrate the capabilities gained by these changes to the core.

Also, because of the increased functionality, it made sense to add detection for Zip (decompression) bombs. Almost a year ago we’ve talked about Zip bombs, but it was limited to the safe exploration of such files. Let’s start with the new ‘Limits’ page in the setup.

Limits

  • What had been once the maximum size of a file is now the size of virtual memory the memory pool is allowed to use. Of course, the more virtual memory is granted, the faster it becomes to analyze large files. But it’s only a matter of speed, you can choose to give to the pool the bare minimum, it’ll work just as well.
  • The nesting option shouldn’t need any explanation since it hasn’t changed. But just for completeness: it specifies the maximum level of scanning into a root object. If more levels are available, it is signaled. For instance, level 0 specifies that children objects should not be scanned automatically (though they might still be opened manually by the user).
  • The maximum file size can be used to discard files larger than the specified size during batch scan operations. The default is 0, which stands for infinite.
  • The decompression bomb threshold is the limit we’re interested in for this post. It represents a cumulative size which can’t be exceeded. In other words if an archive contains 1 file of 100+ GBs (let’s use this number for the purpose of this example) it’s the same as whether one sub-archive contains 100 files of 1 GB each (plus a single byte in excess). If the threshold is exceeded, it will be reported as a threat. While 100 GBs is the default, you can speed up scanning by specifying a lower limit.
  • The maximum number of children files is cumulative as well. This constraint depends on virtual memory limitations (as all children are shown in a tree). 100.000 (the default) is a safe choice. If more children than the imposed limit are present, it is signaled.

So let’s again take the famous 42.zip as an example of Zip bomb and let’s scan it. We’ll get this in the summary:

Decompression bomb threat

Please note that the threat may not be reported in the summary of the root object itself, but in one of its children objects (once the bomb threshold has indeed been exceeded). But since we know there’s a threat (as reported by the risk factor) we can just jump to it by pressing F2 in the hierarchy view.

Widgets and Views

The last release of the Profiler featured some significant improvements. So while it also included initial PySide support, there wasn’t much time to make it really nice. One of the missing things was the ability to mix internal Profiler views (such as the hex editor) with PySide widgets. With the upcoming 0.9.2 release it will be possible to create a view and obtain a PySide widget with just one method:

widget = view.toWidget()

This way one can make use of advanced internal views of the Profiler and combine them with other custom controls. Let’s see a practical example.

Mixed widget

The widget in the screenshot combines a QTreeView with a directory model and a hex view. When a file is activated in the tree, it is opened by the hex editor. To try it out, just press Ctrl+Alt+R and enter the following code:

from Pro import *
from PySide import QtCore, QtGui

class MixedWidget(QtGui.QSplitter):
    def __init__(self, parent=None):
        super(MixedWidget, self).__init__(parent)

        self.setWindowTitle("Mixed widget")
        self.setOrientation(QtCore.Qt.Vertical)

        self.model = QtGui.QDirModel()
        tree = QtGui.QTreeView()
        tree.setModel(self.model)
        self.addWidget(tree)

        ctx = proContext()
        self.hex = ctx.createView(ProView.Type_Hex, "")
        self.addWidget(self.hex.toWidget())

        tree.activated.connect(self.updateFile)

    def updateFile(self, idx):
        if self.model.isDir(idx) == True:
            self.hex.clear()
        else:
            name = self.model.filePath(idx)
            self.hex.setFileName(name)


ctx = proContext()
w = MixedWidget()
v = ctx.createViewFromWidget(w)
ctx.addView(v)

Amazingly little code snippet, right? Please note that the ProHexView setFileName method is also a new addition to the SDK.

News for version 0.9.1

The new version of the Profiler is out with the following news:

added capability of opening multiple analysis views
added capability of switching root object in the workspace
added navigation in analysis views
added bookmarks
added PySide integration
– added user application data folder support
– added history for the Python command line and script dialog
– added save option to the keys input dialog
– improved notes: the toolbar now signals their presence
– updated Qt to 4.8.4

Also a new Demo version has been released, which as usual can be found on the product page.

UI Improvements & Bookmarks

The upcoming 0.9.1 version of the Profiler features some important UI improvements and the introduction of bookmarks. Among the UI improvements there’s:

  • the ability to switch root in the workspace
  • multiple analysis views displaying data of different roots
  • navigation

In this case a video is probably more worth than a thousand words.

These new features lay down the groundwork for some more interesting capabilities which will be added soon. Stay tuned! 🙂

PySide support

This is really a small addition which took just a couple of hours of work, but since it can come very handy, it’s worth dedicating a post to it. The upcoming 0.9.1 version of the Profiler adds explicit support for PySide. Thus, it will be possible to create Qt widgets and add them to the workspace.

Installing PySide

First of all, let’s install PySide. There are 3 ways to do this.

1) Install it from the qt-project page.

Make sure you select the package matching the current Python version used by the Profiler.

2) Install the package we compiled for you. It’s vanilla, directly from the original sources, but it has the advantage that it is guaranteed to work. In fact, at the time of writing the official package contains a bug (missing shiboken Python module) and so the first one is not really an option until it is not fixed.

Download
SHA1: 2024348E79890A167BB231098A6A16FC8BB02C9E

3) You can compile PySide yourself following the instructions at qt-project. At the end, use the installer created inside ‘c:\pyside-setup\dist’.

A code sample

Using it is even easier than the setup process. Basically ProContext has a new method called createViewFromWidget which takes as a parameter a widget created by PySide and returns a ProView which in turn can be added to the workspace.

Adding a widget to the workspace only takes the following line:

ctx.addView(ctx.createViewFromWidget(widget))

Therefore using an existing widget and adding it to the workspace is very easy. Let’s see a real-world widget like an official PySide sample: PySide/examples/effects/lighting.py. It’s sufficient to remove:

if __name__ == '__main__':

    import sys

    app = QtGui.QApplication(sys.argv)

    lighting = Lighting()
    lighting.setWindowTitle("Lighting and Shadows")
    lighting.resize(640, 480)
    lighting.show()

    sys.exit(app.exec_())

And add:

lighting = Lighting()
lighting.setWindowTitle("PySide widget")

ctx = proContext()
ctx.addView(ctx.createViewFromWidget(lighting))

Now we can add an action to execute the code or just insert it in the custom script box (Ctrl+Alt+R) and the view will be shown like this:

To set a custom icon for the view use setWindowIcon.

As usual stay tuned as the upcoming version is going to include some major additions and significant changes.

News for version 0.9.0

We’d like to wish everybody merry Christmas and a happy new year!

We’re just in time to place under the Christmas tree a new version of the Profiler with the following news:

added Java Class support including byte code disassembler and layout ranges
added .NET support including byte code disassembler and layout ranges
added DEX support including byte code disassembler and layout ranges
added dedicated view to display data like raw PDF objects
added PE MUI resources validation
– added Adler32 to filters
– updated jsbeautifier
separated malicious threats from intrinsic ones in the report view
– fixed update with unprivileged user account on Windows
– fixed several bugs

The main addition in this new release as seen previously is the managed trio Java, DEX and .NET.

Dedicated view to display raw data

Previously PDFs had 3 views to display objects: one for the dictionary, one for the decoded stream and one for the decoded stream shown as text. Now there’s also a ‘raw data’ view to show the object unmodified just as it is in the file.

Highlighted with different colors you can see the dictionary/value and the stream part. The same applies to child objects which are highlighted in the stream of their parent.

PE MUI resources validation

Following a short post about MUI resources, validation for them has been added. Also some bug fixes related to resource validation.

Separated malicious threats from intrinsic ones

In order to better separate intrinsic risk factors from malicious threats, they are now shown separately in the report view.

Have some nice holidays and stay tuned as we’ll try to add even more features the upcoming year.

Christmas hat image from freevector.com