Remote Containers

In our previous post we introduced multi-processing as implemented in the upcoming Cerbero Suite 5.2 and Cerbero Engine 2.2. In this post we’re going to talk about remote containers, which are an additional functionality of our multi-processing technology.

Containers (NTContainer) are a way to encapsulate any kind of raw data (e.g.: memory, files) and are used ubiquitously. There might be occasions in which a manager wants to share a container with a worker.

The API to accomplish this is very simple: all the manager has to do is to share the container using shareContainer() and the worker can access the container using getSharedContainer().

In the following example a 10 mega-bytes container is created with a signature appended at the end. A local and remote search is performed to find the signature.

from Pro.Core import NTContainer, MB_SIZE, NTTime
from Pro.MP import *
import time

remote_code = r"""
from Pro.Core import NTTime
from Pro.MP import *

def main():
    c = proWorkerObject().getSharedContainer('NAME')
    # remote search
    magic = b'\xAA\xBB\xCC\xDD'
    t = NTTime()
    t.start()
    match = c.findFirst(magic)
    print('remote search (ms): ' + str(t.elapsed()))

main()
"""

def main():
    magic = b"\xAA\xBB\xCC\xDD"
    buf = b"\xFF" * 10 * MB_SIZE + magic
    c = NTContainer()
    c.setData(buf)
    
    # local search
    t = NTTime()
    t.start()
    match = c.findFirst(magic)
    print("local search (ms):", t.elapsed())

    m = ProManager()
    m.setOptions(ProMPOpt_AtomicOutput)
    
    m.shareContainer("NAME", c)
    
    worker_id = m.startWorker()

    m.evalPythonCode(worker_id, remote_code)
    
    while m.isBusy():
        m.processMessages()
        time.sleep(0.1)
    
main()

The output is:

local search (ms): 17
remote search (ms): 351

The reason for the time difference is, of course, that accessing the remote data is comparatively slower. This factor needs to be taken into consideration when working with remote containers.

Yet another limitation regarding remote containers is that they are read-only. This is for security reasons, as it wouldn’t be safe to allow other processes to change the original container.

In the next example the code asks the user to choose a Windows executable (PE), opens it and shares the container. The import table of the PE is then parsed from the worker process.

from Pro.Core import *
from Pro.UI import *
from Pro.MP import *
import time

remote_code = r'''
from Pro.Core import *
from Pro.MP import *
from Pro.PE import *

def main():
    c = proWorkerObject().getSharedContainer("PE")
    # print imported modules
    obj = PEObject()
    if not obj.Load(c):
        print("error: couldn't load file")
        return
    imp = obj.ImportDirectory()
   
    it = CFFStructIt(imp)
    while it.hasNext():
        cur = it.next()
        name_rva = cur.Uns("Name")
        name_offs = obj.RvaToOffset(name_rva)
        if name_offs != INVALID_STREAM_OFFSET:
            name = obj.ReadUInt8String(name_offs, 1000)[0]
            name = name.decode("utf-8", errors="ignore")
            print("imported module: " + name)
            

main()
'''

def main():
    fname = proContext().getOpenFileName("Select Windows executable...", str(), "Executable files (*.exe)")
    if not fname:
        return

    c = createContainerFromFile(fname)
    if c.isNull():
        return

    m = ProManager()
    m.setOptions(ProMPOpt_AtomicOutput)
    
    m.shareContainer("PE", c)
    
    worker_id = m.startWorker()

    m.evalPythonCode(worker_id, remote_code)
    
    while m.isBusy():
        m.processMessages()
        time.sleep(0.1)
    
main()

An example of output is:

imported module: KERNEL32.dll
imported module: SHLWAPI.dll

In the following example the shared container is shown in a hex view from the worker process.

from Pro.Core import *
from Pro.UI import *
from Pro.MP import *
import time

remote_code = r'''
from Pro.Core import *
from Pro.UI import *
from Pro.MP import *

def main():
    c = proWorkerObject().getSharedContainer("DATA")
    ctx = proContext()
    hv = ctx.createView(ProView.Type_Hex, "Remote Container Data")
    hv.setData(c)
    dlg = ctx.createDialog(hv)
    dlg.show()

main()
'''

def main():
    fname = proContext().getOpenFileName("Select a file...")
    if not fname:
        return

    c = createContainerFromFile(fname)
    if c.isNull():
        return

    m = ProManager()
    m.setOptions(ProMPOpt_AtomicOutput)
    
    m.shareContainer("DATA", c)
    
    worker_id = m.startWorker()

    m.evalPythonCode(worker_id, remote_code)
    
    while m.isBusy():
        m.processMessages()
        time.sleep(0.1)
    
main()

This is a screenshot from running the last example.

Sharing containers with workers is a very inexpensive operation in terms of resources. Therefore, sharing many containers is not an issue.