Tag: SDK

Internal Project Files

The upcoming 5.6 version of Cerbero Suite introduces a new major core feature, namely the capability to generate files which do not exist on disk and store them in the analysis report.

While this feature doesn’t seem so important, it has countless real-world applications. For example, an unpacker may unpack a file during the scanning process and store the resulting file as an internal file. When the unpacked file is requested, the operation bypasses the unpacker and directly accesses the internal file.

In the following example a dummy internal file is generated for a scanned file and adds it as an embedded object to the generated report.

from Pro.Core import *

def scanning(sp, ud):
    # skip if it's a nested scan: avoid recursion
    if sp.isNestedScan():
        return
    # a global report is needed to store internal files
    r = sp.getGlobalReport()
    if not r:
        return
    # generate an internal file id
    uid = r.newInternalFileUID()
    if not uid:
        return
    # retrieve the path on disk for the internal file
    path = r.newInternalFilePath(uid)
    # generate the content of the internal file
    with open(path, "w") as f:
        f.write("hello " * 5)
    # save the internal file
    r.saveInternalFile(uid, "TEST FILE")
    # add the internal file as embedded object
    sp.addInternalFile(uid, "", "Test")

The lines in the ‘hooks.cfg’ configuration file:

[IntFileTest_1]
label = Internal file test
file = intfile_hook.py
scanning = scanning
enable = yes

What follows is a screenshot of the result of this operation.

Internal files can be referenced as embedded objects as well as root objects. When referencing an internal file from a root entry in the report it is enough to set the file name of the entry as following:

REPORT_INT_ROOT_PREFIX + uid

This means that not only embedded objects, but also root objects can reference internal files which may be temporary if the project is not saved by the user.

We’ll soon use internal files to create new and also expand existing packages for Cerbero Suite.

PDB support (including export of types)

The main feature of the upcoming 2.4 version of Profiler is the initial support for the PDB format. Our code doesn’t rely on the Microsoft DIA SDK and thus works also on OS X and Linux.

Since the PDB format is undocumented, this task would’ve been extremely difficult without the fantastic work on PDBs of the never too much revered Sven B. Schreiber.

Let’s open a PDB file.

As you can see the streams in the PDB can be explored. The TPI stream (the one describing types) offers further inspection.

All the types contained in the PDB can be exported to a Profiler header by pressing Ctrl+R and executing the ‘Dump types to header’ action.

Now the types can be used from both the hex editor and the Python SDK.

We can explore the dumped header by using, as usual, the Header Manager tool.

The type showed above in the hex editor is simple. So let’s look what a more complex PDB type may look like.

The PDB code is also exposed to the SDK. This is a small snippet of code, which dumps all the types to a text buffer and then displays them in a text view.

from Pro.Core import *
from Pro.UI import *
from Pro.PDB import *

def showPDBTypes():
    ctx = proContext()
    out = proTextStream()
    out.setIndentSize(4)

    obj = ctx.currentScanProvider().getObject()
    tpi = obj.GetStreamObject(PDB_STREAM_ID_TPI)
    tpihdr = obj.TPIHeader(tpi)
    tiMin = tpihdr.Num("tiMin")
    tiMax = tpihdr.Num("tiMax")
    tctx = obj.CreateTypeContext(tpi)
    for ti in range(tiMin, tiMax):
        tctx.DumpType(out, ti)

    view = ctx.createView(ProView.Type_Text, "PDB Test")
    view.setLanguage("XML")
    view.setText(out.buffer)
    ctx.addView(view)

showPDBTypes()

In order to dump all types to a single header, you can use the DumpAllToHeader method.

Python SDK improvements

The upcoming 0.8.9 release of the Profiler improves integration with Python and the SDK exposes new functionality. Moreover, it lays down the groundwork needed to expand the SDK in the next releases. Documentation of the SDK has been included in the docs directory and also a Python command line has been added to the workspace.

To offer a glimpse of the capabilities here is a small code snippet to create a custom plot which could be used, for instance, to display entropy.

import random

ctx = proContext()
v = ctx.createView(ProView.Type_Plot, "Test") 
v.setPlotTitle("Random")
v.setAxisScale(ProPlotView.xBottom, 0, 100)
v.setAxisScale(ProPlotView.yLeft, 0, 50)
x = NTDoubleVector()
y = NTDoubleVector()
i = 0
while i < 100:
    x.append(i)
    y.append(random.randrange(50))
    i = i + 1
v.addCurve(x, y)
ctx.addView(v)

And the screenshot of the created plot.

Custom plot

While this doesn't sound too exciting at first, the on-going SDK expansion will allow to do some very interesting things. Stay tuned as some more technical posts are coming.

Parameters & Settings

In the upcoming release of the Profiler (0.7.4) actions and scripts have a way to ask the user for parameters and settings: a new set of APIs featuring a property editor dialog.

Properties

Here’s the complete code of the sample followed by explanations.

import ProUI

xml = """


  <section label="General">
    <property id="0" label="Name" type="edit" value="object" />
    <property id="1" label="Size" type="static" value="(0,0)">
        <property id="2" label="Width" type="integer" value="0" signed="false" radix="10" align="0" maxbits="16" />
        <property id="3" label="Height" type="integer" value="0" signed="false" radix="10" align="0" maxbits="16" />
    </property>
  </section>
  
  <section label="Options">
    <property id="4" label="Word wrap" type="check" value="false" />
    <property id="5" label="Syntax" type="combo" value="1">
      <list>
        <i>JavaScript</i>
        <i>C++</i>
        <i>Pascal</i>
      </list>
    </property>
  </section>
  
  <section label="Files">
    <property id="6" label="Open file" type="open-file" value="C:\\test.txt" />
    <property id="7" label="Save file" type="save-file" value="C:\\test2.txt" />
    <property id="8" label="Select directory" type="open-directory" value="C:\\temp" />
  </section>
  
  <section label="Content">
    <property id="9" label="Text" type="text">
        This \tis a\nsample text
    </property>
  </section>
  
  <section id="20" label="Numbers">
    <property id="10" label="Base address" type="integer" value="401000" signed="false" radix="16" align="8" />
    <property id="11" label="Char" type="integer" value="127" signed="true" radix="10" align="0" maxbits="8" />
  </section>
  
"""

def UpdateSize(pe):
    sz = "(" + str(pe.getValue(2)) + "," + str(pe.getValue(3)) + ")"
    pe.setValue(1, sz)

def ParamsCallback(pe, id, userdata):
    print("changed: " + str(id) + " value: " + str(pe.getValue(id)))
    if id == ProUI.ProPropertyEditor.Init:
        UpdateSize(pe)
    elif id == 2 or id == 3:
        UpdateSize(pe)
        pe.setValue(0, "test2")
        pe.setValue(5, 2)
        pe.setValue(10, 0x2000)
    elif id == 4:
        b = pe.isVisible(20) == False
        pe.setVisible(20, b)
        if b == False:
            pe.setErrors([6, 7])
        else:
            pe.clearErrors()
    return True

def TestAction():
    context = ProUI.getContext()
    params = context.askParams(xml, "Test", ParamsCallback, None)
    print(params)
    return 0

Let’s start with the action code.

def TestAction():
    context = ProUI.getContext()
    params = context.askParams(xml, "Test", ParamsCallback, None)
    print(params)
    return 0

This code shows a property editor dialog specifying an XML string to create the dialog, a settings key (optional) and a callback (optional). The return value is a dictionary with the values of the properties with their id as key or None when the dialog is rejected.

{0: 'test', 1: '(4,0)', 2: 4, 3: 0, 4: False, 5: 1, 6: 'C:\\test.txt', 7: 'C:\\test2.txt', 8: 'C:\\temp', 9: 'This \tis a\nsample text', 10: 8192, 11: 127}

I’ll talk later about what the settings key means. Let’s first understand the XML syntax.

The XML root tells the function to create a property editor with the (optional) title “Settings”.

<section label="General">

A section is created. Properties do not need a section as parent, but it might be visually more appealing to specify one. Child nodes of sections are properties. Properties can have other properties as child nodes, but not sections.

<property id="0" label="Name" type="edit" value="object" />

The first property being created is a single line edit field with the id of 0 and a value of “object”. id and type attributes are mandatory for properties. Sections may optionally specify an id as we’ll see later.

    <property id="1" label="Size" type="static" value="(0,0)">
        <property id="2" label="Width" type="integer" value="0" signed="false" radix="10" align="0" maxbits="16" />
        <property id="3" label="Height" type="integer" value="0" signed="false" radix="10" align="0" maxbits="16" />
    </property>

Here we have one static property with two integer child properties. A static property is a non-editable text which can only be set programmatically.

An integer property can specify various things, although the only mandatory attributes remain id and type. Most of the attributes are self-explanatory. align specifies the 0s which may prefix the number to obtain the desired alignment. For example, the number 1 with an alignment of 4 will be displayed as 0001. maxbits specifies the maximum number of bits the integer can measure (at the time it defaults to 10000).

  <section label="Options">
    <property id="4" label="Word wrap" type="check" value="false" />

Inside a new section a check property is specified.

    <property id="5" label="Syntax" type="combo" value="1">
      <list>
        <i>JavaScript</i>
        <i>C++</i>
        <i>Pascal</i>
      </list>
    </property>

Following there’s a combo property. The list of the combo is specified as the child node list and the default index is specified as the value attribute.

  <section label="Files">
    <property id="6" label="Open file" type="open-file" value="C:\test.txt" />
    <property id="7" label="Save file" type="save-file" value="C:\test2.txt" />
    <property id="8" label="Select directory" type="open-directory" value="C:\temp" />
  </section>

These three properties are related to file operations. When the user activates one, he will be able to open a file dialog to perform the requested operation.

  <section label="Content">
    <property id="9" label="Text" type="text">
        This \tis a\nsample text
    </property>
  </section>

The text property specifies a multi-line text field. When the user activates this property, a multi-line text input dialog is displayed in order to change the value.

  <section id="20" label="Numbers">
    <property id="10" label="Base address" type="integer" value="401000" signed="false" radix="16" align="8" />
    <property id="11" label="Char" type="integer" value="127" signed="true" radix="10" align="0" maxbits="8" />
  </section>

The properties in this section are not any different than those seen before, but it should be noted that in this case the section has an id attribute. Giving an id to a section makes it possible to set the visibility or the enabled/disabled state of the section and its children.

Let’s analyze the callback.

def UpdateSize(pe):
    sz = "(" + str(pe.getValue(2)) + "," + str(pe.getValue(3)) + ")"
    pe.setValue(1, sz)

def ParamsCallback(pe, id, userdata):
    print("changed: " + str(id) + " value: " + str(pe.getValue(id)))
    if id == ProUI.ProPropertyEditor.Init:
        UpdateSize(pe)
    elif id == 2 or id == 3:
        UpdateSize(pe)
        pe.setValue(0, "test2")
        pe.setValue(5, 2)
        pe.setValue(10, 0x2000)
    elif id == 4:
        b = pe.isVisible(20) == False
        pe.setVisible(20, b)
        if b == False:
            pe.setErrors([6, 7])
        else:
            pe.clearErrors()
    return True

ParamsCallback has three arguments. pe is the ProPropertyEditor class instance. id is the property being modified or the notification code (Init, Accept). userdata is the custom data specified in the askParams method, which in this case is None.

The UpdateSize function updates the value of the static property when one of its children has been changed. Other fields are changed for the purpose of demonstration.

        pe.setVisible(20, b)

This line sets the visibility of the last section.

        if b == False:
            pe.setErrors([6, 7])
        else:
            pe.clearErrors()

The setErrors method allows to highlight properties in red. The idea is that a callback might perform some checks when being notified with the ProPropertyEditor.Accept code, highlight properties which are not accepted and return False to ask the user to enter correct values. Calling clearErrors or setErrors with an empty list will achieve the same result.

Here’s a screenshot with two highlighted properties and the last section hidden.

Properties 2

Let’s go back to askParams method. I haven’t yet explained the settings key (“Test”). This is an optional argument: it specifies if and where the values of the properties should be stored in case the dialog is accepted. The specified key name should be similar or equal to the name of the action to avoid conflicts. If the property dialog changes and the old settings must be discarded, it can be achieved by specifying a version at the end of the key name: “Test#1”. When the version number is omitted, it defaults to 0.

It should be noted that static and multi-line text properties are not saved automatically. The latter to avoid too large values being stored. However, it is still possible to save and restore these values through the Init and Accept notification codes by using the settings API. 

# restore
ProUI.ProSettings.getValue("Test/mytext")
# save
ProUI.ProSettings.setValue("Test/mytext", text)

Key names starting with “_” are reserved and shouldn’t be used.

Finally, let’s see how actions can now optionally specify a configuration function.

Configure action

[TestAction]
label = Test
file = testfile.py
context = any
config = TestConfig

config specifies the name of a function to be called inside of file.

This new set of APIs opens the door to many interesting customizations for actions, scripts and other components, and we will soon show you some of them. 🙂

Python 3 SDK: actions & custom scripts

The new version 0.7.3 of Cerbero Suite features a powerful Python 3 SDK, which enables to run custom scripts and actions. Let’s first take a look at a simple script. Just press Ctrl+R (or “Execute action…” in the context menu of a view) and go to “Custom”:

Simple script

As it is easy to guess, this basic script shows a message box. Message boxes can be used to notify things to the user or to ask him a question. Most of the time they won’t be necessary and the standard output can be used instead. All the output produced by Python will be visible in the output console. In fact, the console will become visible when something is printed to it (this behavior can be changed from the options).

Output console

The SDK can be used to retrieve data from views, set their data, create new views and so on. But before looking at a more advanced script, let’s talk about a new feature of Cerbero Suite: actions. For the purpose of demonstration let’s take a malware with obfuscated JavaScript.

Obfuscated JS

And now let’s again press Ctrl+R in the context of the obfuscated JavaScript.

Actions

By activating the “Beautify JavaScript” action we will get a beautified version (jsbeautifier.org) of the previously obfuscated JavaScript.

Beautified JS

Python actions are defined in the config/actions.cfg file.

[JSBeautify]
category = JavaScript
label = Beautify JavaScript
file = javascript.py
context = text

The section name (JSBeautify) specifies the id of the action and is also the name of the function to be called in file. The file field supports absolute paths as well, otherwise the script will be loaded from plugins/python. The category and label specify in which category inside the execute action dialog the action should be grouped and its description. When the category field is omitted, it will default to “Other”.

The context field is very important as it specifies when the action should be available for use. In this specific case, the action can be used in any text view. An action can also be available in more than one context.

; available both in text and hex views
context = text|hex

; available in text and hex views only when text or data is selected
context = text|hex|sel

; always available even when not in a view
context = any

Now let’s see how to create an action which decodes some selected text from base64 and shows the decoded bytes in a new hex view. First it is necessary to define the action.

[Base64Decode]
category = Samples
label = Base64 decoder
file = samples.py
context = text|sel

And here’s the Python code.

from Pro.UI import *

def Base64Decode():
    context = proContext()
    view = context.getCurrentView()
    if view.isValid() and view.hasSelection():
        text = view.getSelectedText()
        decview = context.createView(ProView.Type_Hex, "Base64 decoded data")
        import base64
        decview.setBytes(base64.b64decode(text.encode("utf-8")))
        context.addView(decview)
    return 0

Let’s see it in action with a PGP public key.

PGP Public Key

And the decoded data.

PGP decoded key

Although the SDK is brand new, you will see very soon some new useful actions implemented. 🙂