Example: Run System

Showcases the API Usage of the run system proFit v0.5.dev

import numpy as np
import matplotlib.pyplot as plt

Setup

The Worker

The simulation we want to run is wrapped in a Worker. We need to name the output variables, the input variables can be inferred from the simulations’s argument list.

As an alternative, the CommandWorker can be configured to run any command/executable. It will prepare a directory for each simulation, filling the input values according to a template and read the output files to transmit them back.

Note: for the LocalRunner and SlurmRunner it is only possible to use a custom Worker which is importable (e.g. defined in a separate file which is specified in the configuration). Custom Workers defined in a script or notebook can only be used with the ForkRunner for now.

from profit.run import Worker


@Worker.wrap("simulation")
def simulation(u, v) -> "f":
    return u * np.cos(v)

The Interface

To manage the flow of data between the Workers and the Runner, we choose and configure an Interface.

Memorymap Interface (memmap)	ZeroMQ Interface (zeromq)
single memory mapped file	messages via a protocol (tcp)
only local, but fast	local and distributed (HPC)

from profit.run import RunnerInterface

interface = RunnerInterface["memmap"](
    size=10,
    input_config={"u": {"dtype": float}, "v": {"dtype": float}},
    output_config={"f": {"dtype": float, "size": (1, 1)}},
)

The Runner

The Runner is the central components of the run system which brings everything together. The Runner is also responsible for starting and distributing the individual Workers.

Fork Runner (fork)	Local Runner (local)	Slurm Runner (slurm)
forking / Process	via the shell / subprocess	via the Slurm scheduler
fastest, supports temporary Workers		submits Slurm jobs (HPC)

from profit.run import Runner

runner = Runner["fork"](
    interface=interface,
    worker="simulation",  # don't require the Worker, just it's label or config dictionary
)

Running

runner.next_run_id = 0  # reset Runner
runner.spawn({"u": 1.2, "v": 2})

U = np.random.random(9)
V = np.linspace(0, 2, 9)
parameters = [{"u": u, "v": v} for u, v in zip(U, V)]
runner.spawn_array(parameters, progress=True, wait=True)

submitted: 100%|██████████| 9/9 [00:00<00:00, 19.82it/s]
finished : 100%|██████████| 10/10 [00:00<00:00, 98.31it/s]

Results

import pandas as pd

pd.DataFrame(runner.input_data).join(pd.DataFrame(runner.output_data))

	u	v	f
0	1.200000	2.00	-0.499376
1	0.382964	0.00	0.382964
2	0.887557	0.25	0.859965
3	0.176309	0.50	0.154726
4	0.846768	0.75	0.619570
5	0.944341	1.00	0.510230
6	0.483903	1.25	0.152585
7	0.832679	1.50	0.058901
8	0.320214	1.75	-0.057077
9	0.993055	2.00	-0.413257

fig, ax = plt.subplots(tight_layout=True)
sc = ax.scatter(
    runner.input_data["u"], runner.input_data["v"], c=runner.output_data["f"]
)
plt.colorbar(sc, ax=ax, label="f")
ax.set(
    xlabel="u",
    ylabel="v",
)
None

Clean Runner & Interface

runner.clean()