News for version 0.9.0

We’d like to wish everybody merry Christmas and a happy new year!

We’re just in time to place under the Christmas tree a new version of the Profiler with the following news:

added Java Class support including byte code disassembler and layout ranges
added .NET support including byte code disassembler and layout ranges
added DEX support including byte code disassembler and layout ranges
added dedicated view to display data like raw PDF objects
added PE MUI resources validation
– added Adler32 to filters
– updated jsbeautifier
separated malicious threats from intrinsic ones in the report view
– fixed update with unprivileged user account on Windows
– fixed several bugs

The main addition in this new release as seen previously is the managed trio Java, DEX and .NET.

Dedicated view to display raw data

Previously PDFs had 3 views to display objects: one for the dictionary, one for the decoded stream and one for the decoded stream shown as text. Now there’s also a ‘raw data’ view to show the object unmodified just as it is in the file.

Highlighted with different colors you can see the dictionary/value and the stream part. The same applies to child objects which are highlighted in the stream of their parent.

PE MUI resources validation

Following a short post about MUI resources, validation for them has been added. Also some bug fixes related to resource validation.

Separated malicious threats from intrinsic ones

In order to better separate intrinsic risk factors from malicious threats, they are now shown separately in the report view.

Have some nice holidays and stay tuned as we’ll try to add even more features the upcoming year.

Christmas hat image from freevector.com

.NET support

Although there haven’t been customer requests for this, the upcoming 0.9.0 version of the Profiler adds support for .NET, which includes format, layout ranges and an MSIL disassembler.

As usual, let’s begin with the format itself. Since some users probably have used CFF Explorer to inspect the .NET format in the past, I have kept the same format view in the Profiler as well.

So how is it better than CFF Explorer? First of all, it’s very fast. There’s absolutely no wait time in opening a large metadata set as the following.

And then one handy feature which was not available in CFF Explorer, is the capability to display a second set of metadata. For instance, .NET native images (meaning those files in the assembly cache created with ngen.exe) contain two sets of metadata. The Profiler lets you inspect both sets.

Layout ranges cover all PE parts, so it was normal to add them for .NET as well. These are the ranges available for .NET:

Let’s see them in the hex editor.

If you’re asking yourself why it’s all gray, it’s because more ranges are blended together, meaning that all .NET metadata is contained in the .text section of the PE and that section is marked as executable even if the assembly contains only MSIL code. While it doesn’t make much sense to mark as executable a region of data containing strings and tables, my best guess is that old versions of Windows had no in-built support for .NET in the loader, which is why assemblies contain a single ‘mscoree.dll’ import descriptor and the entry point is just a jmp to the only IAT thunk (_CorExeMain for executables, _CorDllMain for dlls). That API then loads the .NET framework if necessary. Since that single native jmp instruction requires a section in the PE marked as executable and because the granularity of sections is the same as virtual memory pages, it was probably considered a waste to use an entire memory page just for a jmp instruction and so the result is that everything is contained into a single executable section. This could probably be changed as new versions of the framework do not even run on older systems.

However, for our inspection purposes it suffice to get rid of the Code range by pressing Ctrl+Alt+F.

We can now inspect .NET layout ranges without the conflict. One nice aspect about it is that the code of IL methods is easy to distinguish between its header and extra sections.

Included is also an IL disassembler. Its purpose is to let our customers quickly browse the contents of an assembly. As such readability was a priority and the output has been grouped for classes: I have always found it cumbersome in ILDasm to open every single method to inspect an assembly.

Here’s an output example:

  private static void Main(string [] args)
  {
    locals: int local_0,
            int local_1

    ldc_i4_2
    stloc_0 // int local_0
    ldloc_0 // int local_0
    stloc_1 // int local_1
    ldloc_1 // int local_1
    ldc_i4_1
    sub
    switch
      goto loc_22
      goto loc_60
    br_s loc_71
loc_22:
    try
    {
      ldstr "h"
      call System.Console::WriteLine(string) // returns void
      leave_s loc_81
    }
    catch (System.ArgumentNullException)
    {
      pop
      ldstr "null"
      call System.Console::WriteLine(string) // returns void
      leave_s loc_81
    }
    catch (System.ArgumentException)
    {
      pop
      ldstr "error"
      call System.Console::WriteLine(string) // returns void
      leave_s loc_81
    }
loc_60:
    ldstr "k"
    call System.Console::WriteLine(string) // returns void
    ret
loc_71:
    ldstr "c"
    call System.Console::WriteLine(string) // returns void
loc_81:
    ret
  }

And the original code:

        static void Main(string[] args)
        {
            int a = 2;
            switch (a)
            {
                case 1:
                    try
                    {
                        System.Console.WriteLine("h");
                    }
                    catch (ArgumentNullException)
                    {
                        System.Console.WriteLine("null");
                    }
                    catch (ArgumentException)
                    {
                        System.Console.WriteLine("error");
                    }
                    break;
                case 2:
                    System.Console.WriteLine("k");
                    break;
                default:
                    System.Console.WriteLine("c");
                    break;
            }
        }

There’s one last thing worth mentioning. .NET manifest resources are displayed as sub-files.

However, the parsing of ‘.resources’ files was still too partial and thus won’t be included in 0.9.0.

From the last two posts you may have guessed the topic of the upcoming release. So stay tuned as there’s yet more to come.

Java Class support

The upcoming 0.9.0 version of the Profiler adds support for Java class files. This will be especially useful for malware analysis. The support includes: disassembler, format views and layout ranges. Let’s see some screeshots of the format itself first.

Here’s a view of the constants:

Note: Utf8 strings are highlighted in orange just to distinguish them.

Here’s a view of the methods with their attributes:

And here’s the layout ranges view:

Again strings are in orange, while the actual code of a method is in a slightly lighter green than the method itself.

Since the format of class files is contiguous, it’s extremely easy to use layout ranges to create a new custom class file using the hex editor.

And finally, the disassembler:

The output shown in the screenshot above:

// SourceFile: HelloWorld.java

super class HelloWorld extends java.lang.Object
{

  static float f1;
  public static java.lang.String hello;

  HelloWorld()
  {
    // max_stack = 1  max_locals = 1
// line 1
    aload_0
    invokespecial java.lang.Object.() // returns void
    return
  }

  public static void main(java.lang.String[])
  {
    // max_stack = 7  max_locals = 6
// line 7
    ldc2_w 454.546
    dstore_1
// line 8
    ldc2_w 552441554577111995
    lstore_3
// line 9
    getstatic java.lang.System.out // java.io.PrintStream
    ldc "The value of i is: %f and %d"
    iconst_2
    anewarray java.lang.Object
    dup
    iconst_0
    getstatic HelloWorld.f1 // float
    invokestatic java.lang.Float.valueOf(float) // returns java.lang.Float
    aastore
    dup
    iconst_1
    lload_3
    invokestatic java.lang.Long.valueOf(long) // returns java.lang.Long
    aastore
    invokevirtual java.io.PrintStream.format(java.lang.Stringjava.lang.Object[]) // returns java.io.PrintStream
    pop
// line 10
    getstatic java.lang.System.out // java.io.PrintStream
    getstatic HelloWorld.hello // java.lang.String
    invokevirtual java.io.PrintStream.println(java.lang.String) // returns void
// line 13
    try
    {
      getstatic java.lang.System.out // java.io.PrintStream
      ldc "test2"
      invokevirtual java.io.PrintStream.println(java.lang.String) // returns void
    }
    catch (java.lang.Exception)  goto loc_57
// line 18
    goto loc_67
// line 15
loc_57:
    astore 5
// line 17
    getstatic java.lang.System.out // java.io.PrintStream
    ldc "error"
    invokevirtual java.io.PrintStream.println(java.lang.String) // returns void
// line 19
loc_67:
    return
  }

  static void ()
  {
    // max_stack = 1  max_locals = 0
// line 3
    ldc 43
    putstatic HelloWorld.f1 // float
// line 4
    ldc "Hello world!"
    putstatic HelloWorld.hello // java.lang.String
    return
  }

}

And the original source file:

class HelloWorld
{
  static float f1 = 43;
  public static String hello = "Hello world!";
  public static void main(String[] args)
  {
    double d1 = 454.545774;
    long l1 = 552441554577111995L;
    System.out.format("The value of i is: %f and %d", f1, l1);
    System.out.println(hello);
    try
    {
      System.out.println("test2");
    }
    catch (Exception e)
    {
      System.out.println("error");
    }
  }
}

There’s yet more to come. Stay tuned. 🙂

News for version 0.8.9

The new version is out. 🙂 Here’s a recap of the latest improvements:

increased python integration and exposed more parts of the SDK
– added SDK documentation to the docs directory
– added Python command line
– added global and individual file notes
improved filters and added range parameters
– introduced fullscreen modality in workspace (F11)

This has been mostly a transition release and what took most of the time were structural changes in how the SDK is exposed to Python so that future releases will benefit from it. The main advantage for the user may be the addition of range parameters in the filters and the introduction of file notes, which is a small feature, but very useful in the context of analysis.

The next release will bring some new file formats and interesting improvements.

Damaged Zip archive (video)

In this video we can see how to inspect a damaged Zip archive using the Profiler in a real-world scenario. Although soon the automatic recovery of damaged Zip archives will be available and it will be possible to perform this sort of task programmatically, it’s still useful to see how to do this kind of thing manually.

Filters with range parameters

The upcoming 0.8.9 release improves filters and introduces range parameters. If you don’t know what filters are you can take a look at the original introductory post.

What is now possible to do is to specify an optional range for the filter to be applied to. This is extremely useful, since many times we need to modify only a portion of the input data. Take, for instance, a file in which a portion of data is encrypted (or compressed) and we want to keep the outer parts not affected by the filter. This would be the layout of the input data:

Data layout

To handle the outer parts (before and after) we have an extra parameter called ‘trim’. This parameter can be set to one of the following values:

  • no: the outer parts are not trimmed and kept in the output data
  • left: the left part is trimmed while the right one is kept
  • right: the right part is trimmed while the left one is kept
  • both: both outer parts are trimmed and the output data will contain only the output of the filter

Let’s try to understand this better with a practical example. Let’s consider a compressed SWF file. The Profiler handles the decompression automatically, but just for the sake of this post we’ll decompress it manually. In case you don’t know, the format of a compressed SWF is as following:

3 bytes: CWS signature (would be FSW uncompressed files)
4 bytes: uncompressed file size
1 byte: version

To dig deeper into the file format you may want to take a look at a new web page we are preparing at fileanalysis.net. But for the sake of our example you need only to know the initial fields of the header. To decompress a SWF we need to decompress the data following the header and then to fix the first signature byte and we’ll do this manually and graphically with filters.

Let’s start by selecting the compressed input data and then by trying to add the unpack/zlib filter.

ZLib filter

As you can see we are prompted with a dialog asking us if we want to use the range defined by the selection of the hex editor on the left (input data). We accept and specify that we want to keep the data on the left (although in this case we could have just disabled the trimming completely, let’s just pretend there’s some data on the right we want to discard).

Now we need to fix the signature byte. To do this we use the misc/basic filter (set operation, offset 0, size 1 and no trimming).

By running the preview we’ll have the decompressed SWF.

The exported filters will look like this:


    
    

While this was a very simple case, much more complicated cases can be handled. Also remember that filters can be used to specify how to load embedded files, which means that, for instance, it’s easy to decrypt a file contained into another file and inspect it without ever having to save the decrypted data to disk.

I hope this post offers some understanding of an advanced use of filters.

Python SDK improvements

The upcoming 0.8.9 release of the Profiler improves integration with Python and the SDK exposes new functionality. Moreover, it lays down the groundwork needed to expand the SDK in the next releases. Documentation of the SDK has been included in the docs directory and also a Python command line has been added to the workspace.

To offer a glimpse of the capabilities here is a small code snippet to create a custom plot which could be used, for instance, to display entropy.

import random

ctx = proContext()
v = ctx.createView(ProView.Type_Plot, "Test") 
v.setPlotTitle("Random")
v.setAxisScale(ProPlotView.xBottom, 0, 100)
v.setAxisScale(ProPlotView.yLeft, 0, 50)
x = NTDoubleVector()
y = NTDoubleVector()
i = 0
while i < 100:
    x.append(i)
    y.append(random.randrange(50))
    i = i + 1
v.addCurve(x, y)
ctx.addView(v)

And the screenshot of the created plot.

Custom plot

While this doesn't sound too exciting at first, the on-going SDK expansion will allow to do some very interesting things. Stay tuned as some more technical posts are coming.

News for version 0.8.8

While we talked about some of the news of this version, there are some more which are worth mentioning.

introduced new multi-file report and project technology with compression and encryption
introduced new UI for workspace mode
added Windows Lnk support
– added file extensions scan option
– added directory scan to command line
added PNG CRC validation
added new filters: misc/replace and dev/array
– several UI improvements
– hex editor improvements
– increased memory limit

New workspace UI

The workspace features a new dock-based UI. It is easy to get used to the new UI as it is completely intuitive.

Workspace UI

Now our users can completely costumize their analysis workspace.

PNG CRC validation

This feature was requested by one of our customers and it can come handy during forensic analysis. The CRC of each PNG chunk is verified and those which don’t match are signaled in the format view (highlighted in red).

PNG CRC

New filters: misc/replace and dev/array

Two new filters have been added. misc/replace is self explanatory: it replaces bytes and strings. While dev/array is a small addition which can come handy all those times we need to copy some bytes to an array in a programming language. We can specify the language, the radix and the number of columns and we get something like:

unsigned char data[64] =
{
    0x4D, 0x5A, 0x90, 0x00, 0x03, 0x00, 0x00, 0x00,
    0x04, 0x00, 0x00, 0x00, 0xFF, 0xFF, 0x00, 0x00,
    0xB8, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
    0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
    0x00, 0x00, 0x00, 0x00, 0xF8, 0x00, 0x00, 0x00
};

We hope you enjoy this new version.

Windows Link Support

The next Profiler update, along with several new features, will also include preliminary support for the Windows Link files (also known as Shell Links). This format has been introduced with Windows 95 and is frequently taken into consideration in forensic analysis procedures to discover usage information about files and folders.

In the image below, a link file that opens the Cerbero homepage using Internet Explorer is being shown.

Windows Link Format

Next generation reports

The upcoming 0.8.8 version of the Profiler features a new way of generating reports. In particular it will be possible to scan multiple files and save a single report for all of them. Moreover, the report can contain the original files (either plain or compressed and/or encrypted).

But let’s proceed in order. While it is certainly easy to use the new features, reading this article will help you understanding of the under the hood details.

Let’s begin with a custom scan of the SysWOW64 directory.

When the scan has finished, we get the list of scanned files.

At this point we can click on “Save report”.

In the options above I decided to include all files in the saved project and use the password “test” to encrypt them. This comes handy if we want to move our analysis to another system or send it to a colleague.

What we also can do is to save the unpacked database files instead of the project.

This will create a directory with the extension “cprodb” containing two files. A main.db and an objects.sqlite file. Both are SQLite databases: main.db contains all kinds of information while objects.sqlite is reserved to store data about scanned files. Plugins will have access to main.db to store their own data (to be documented in some future post), keeping in mind that tables which begin their name with “pro” are reserved for internal use.

When a report is saved, not as a project, but in its unpacked state, it can be saved back to a project by clicking “Save project”. If no project is associated with the report, the user will be asked to select an existing project. Unpacked reports are modified directly and don’t need to be saved.

Most of the work for this feature went into testing database technologies suitable to contain a large number of elements and to stress-test them. In the end I decided to implement an abstract interface so that in the future other kind of databases can be added and the user will be in the position to decide whether he prefers to use SQLite or some other database to store information of the scanned files.

However, even by using the current SQLite implementation there should be no problem in generating reports with tens of millions of scanned files. I personally did tests up to 100 million of entries.

Here’s a screenshot with a saved report of an entire Virtual Machine: the project includes all of the files.

The new reporting technology is going to enable other important features (soon to be documented). Please be patient, because it will still take some more weeks to deliver the 0.8.8 version, which will include other significant improvements and features.