[go: up one dir, main page]

Draft: Add function serialization via dill

Description

This adds the ability to serialize python function objects in HEROS. It does so by employing the dill module which itself only relies on python>=3.8 and setuptools >=4 (so not too much dependency bloating).

Being able to send functions back and forth comes a lot of new abilities and capabilities. It allows the remote side to send me a function to run on my machine. Or, even more relevant, it enables me now to instruct a remote side to run code that I define locally. When using it right, I can even access the remote object from within my locally defined code.

Let's make a few examples. For all of them let's define the following class as a LocalHERO

class TestObject(LocalHERO):

    foovar: str = ""
    testme: int = 0

    def hello(self) -> str:
        self.testme += 1
        return "world"

    def get_power_func(self, power):
        def inner(x):
            return x**power
        return inner

    def run_func(self, func, args, kwargs):
        return func.__get__(self)(*args, **kwargs)

With this HERO running, we can go to another machine, run obj = RemoteHERO("my_name"), and do the following things:

EXAMPLE 1 (receive functions with closure)

In [0]: f = obj.get_power_func(3)

In [1]: f
Out[1]: <function __main__.TestObject.get_power_func.<locals>.inner(x)>

In [2]: f(3)
Out[2]: 27

This means the function was transferred from the RemoteHERO to the local interpreter and then run by the local interpreter. It is important to note, that also the closure of the function is transferred. This allows the definition of the parameter power to be properly handled.

EXAMPLE 2 (send functions and accessing the remote object)

In [0]: def greeter(self, name):
            print(f"hello {name}, this is {self.foovar}")
            a = self.hello()
            return a

In [1]: obj.foovar = "RemoteHERO"

In [2]: obj.run_func(greeter, ["HERO"], {})
Out[2]: 'world'

On the console of the remote side you will see the print hello HERO, this is RemoteHERO. This shows that we executed the locally defined function greeter on the remote side as a method attached to the remote object, i.e. we can access the object as self. Of course this access is enabled by using the __get__ method in the run_func of the LocalHERO.

Possible applications

  • Use the processors in herostools to run a logic defined locally. This is very similar to what the mapping in a python multiprocessing.Pool does. It also has to serialize the function to be run and sends it into the newly created processes. I think we can now do a very similar thing and transparently get network-distributed compute network via HEROS
  • This could be the corner stone interact with multiple realtime control systems.
  • ... the only limit is your imagination

Note: This implementation is python only and can most likely not be implemented for other languages in the future. In general such a functionality cannot be realized with compiled languages. So maybe that is not too bad to be python specific here (sorry, typescript...)

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Checklist:

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation

Merge request reports

Loading