Remote execution

The Remote runner executes the steps needing the instruments on the machine which has access to them, while the notebook runs elsewhere. It follows the multi-tier architecture of the interactive usage of Qibocal, and the conventions of Qibocamp.

from qibocal_widgets import Executor
from qibocal_widgets.marimo import Marimo
from qibocal_widgets.remote import Remote

executor = Executor(
    None, targets=[0, 1], runner=Remote("lab"), frontends=[Marimo()]
)
completed = executor(bound)  # acquired remotely, fitted and reported locally
completed

Tiers

What happens depends on the connection options, not on any detection:

Options	Steps needing the instruments
none	run in the current process (already on the compute node): nothing dispatched
queue and/or slurm	one blocking srun job per step, from the current machine (access node)
ssh	the same, through SSH (laptop to access node); without queue, steps run on the SSH destination itself

Only the acquisition needs the instruments. Fit and report always run locally, from the acquired data. Updates only modify the platform parameters:

• with a local platform (e.g. Executor.create(platform="chip", ...) on the access node), the update is applied to it, and its state is sent along with every following acquisition;

• without (e.g. Executor(None, ...) on a laptop), the update runs remotely, on the access node and outside of the queue, and the runner keeps the resulting state for the following acquisitions.

Each job recreates the platform from its name and applies the current state, so jobs are independent, as SLURM requires.

Since the passthrough tier changes nothing, the same notebook works everywhere: select the connection with the QIBOCAL_WIDGETS_CONNECTION environment variable and use Remote().

Connections

Connection options can be given directly, or stored by name:

from qibocal_widgets.remote import Connection, Remote, save_connection

save_connection(
    "lab",
    Connection(
        ssh="-J bastion me@login",  # ssh arguments, ending with the destination
        queue="qpu",  # SLURM partition
        slurm="--time=30",  # further srun options
        platform="chip",  # resolved on the remote machine
        platforms_dir="~/platforms",  # where, on the remote machine
        requirements="qibocal @ git+https://github.com/qiboteam/qibocal.git@protodev",
    ),
)

Remote("lab")  # the stored connection
Remote("lab", queue="other")  # with overrides
Remote(queue="qpu", platform="chip")  # without any file

Named connections live in remotes.json, in the configuration directory shared with Qibocamp ($QIBOCAMP_CONFIG_DIR, else $XDG_CONFIG_HOME/qibocamp, else ~/.config/qibocamp):

{
  "version": 1,
  "connections": {
    "lab": {
      "ssh": "-J bastion me@login",
      "queue": "qpu",
      "platform": "chip",
      "platforms_dir": "~/platforms"
    }
  }
}

The platforms are found in platforms_dir on the remote machine, which replaces its QIBOLAB_PLATFORMS for every remote step (~ is expanded there). Without it, the remote environment must set the variable itself. The field has the name and meaning of the same setting in Qibocamp’s agent configurations.

The fields mirror the notebook connections of Qibocamp, so that its launcher can manage them as well. SSH host aliases and jump hosts from ~/.ssh/config are honored; connections never prompt for passwords (BatchMode), so set up key-based authentication.

Environments and protocols

Remote steps run in a Python environment on the remote machine, under ~/.cache/qibocamp/envs, the same location used by Qibocamp. A named connection has one of its own by default, named after it under _remote/ (lab uses ~/.cache/qibocamp/envs/_remote/lab), shared with no other connection nor notebook session unless venv names it on purpose; a connection given inline uses qibocal. Installing it is a separate operation, never performed by the steps:

remote = Remote("lab")
remote.install()  # once, and whenever the requirements change
remote.install(rebuild=True)  # from scratch

Installation uses uv when available (venv and pip otherwise), runs on the access node (where network access usually is) unless rebuild_on_compute is set, and is skipped when the requirements did not change. Qibocal is always required, and can be pinned in the requirements.

Everything travels as JSON and NPZ, nothing is pickled: protocols are sent by reference (module:attribute), parameters as JSON, platform state and data in Qibocal’s own formats. Built-in protocols work out of the box. Further protocols must be defined at the top level of an importable module, whose package is listed in the requirements of the connection; protocols defined in a notebook cell cannot be executed remotely.

Failures raise RemoteError, carrying the remote traceback or error output.

Background execution

executor(bound) blocks until the job has gone through the queue. executor.submit(bound) returns at once, with a Pending execution, so that the notebook stays interactive while the job waits: the platform explorer and the queue monitor keep working.

pending = executor.submit(protocols.ramsey(delay=(10, 1000, 100)), nshots=500)

pending()  # the result, or None while running: never blocks
pending.result()  # waits for the result
await pending  # with top-level await

Several submissions are queued together. Their results are recorded, fitted, reported, and update the platform in the order they start (submission, or confirmation, see below), whatever the order their jobs complete in; each acquisition uses the platform state at the time it starts, so acquisitions submitted together do not see each other’s updates.

In Marimo, background executions integrate with the reactive runtime, with nothing to set up: they run in a mo.Thread, and a pending execution is reactive state, so that every cell referencing it runs again when it completes. Displayed as the output of the cell submitting it, it shows a spinner, replaced by the result when ready. Downstream cells wait for it with mo.stop, without blocking anything:

# one cell
pending = executor.submit(protocols.ramsey(delay=(10, 1000, 100)), nshots=500)
pending  # a spinner, then the result

# another cell: runs again when the execution completes
ramsey = pending()
mo.stop(ramsey is None, mo.md("*Waiting for the acquisition…*"))
ramsey.results

It is a mo.state getter like any other, for any other use of Marimo’s reactivity. (await pending also works, but holds the notebook until the result arrives.)

In Jupyter, the pending execution’s display updates by itself when the result arrives, and pending.result() waits for it where needed; dependent cells do not run again by themselves, as usual in Jupyter.

Progress. Qibolab logs an estimate of the duration of every execution on the platform, and drivers may refine it just before starting (Qblox does, accounting for its overheads). The estimates are captured where the acquisition runs, locally or by the remote worker, which streams them back while the job runs: the displayed spinner becomes a progress bar, filling in the estimated time. Each new estimate starts the bar again, since the acquisition starts after it, showing the first one for comparison. Drivers estimating each batch (Qblox) also count them: batches done: 2/?, the total being unknown in advance; the count starts again with each execution on the platform. The estimates received are in pending.estimates. Capturing them changes nothing in the logs: they are only emitted where they already were.

Avoiding repeated acquisitions. Notebooks run cells again, unintentionally too (in Marimo, whenever an upstream cell changes). With reuse=True, submit returns an identical execution instead of acquiring again: one still running, or one recorded in the session, loaded from disk. Executions are identical with the same protocol, parameters, overrides, and targets, whatever the platform state: the updates of other executions do not cause acquiring again (submit without reuse for that). Nothing is cached besides the session itself, which records a small key next to each execution; reusing recorded executions works across kernel restarts (attaching the session), for protocols registered in Qibocal, which it can load back. A new executor starts a new session, unless attached to an existing one: in Marimo, running again the cell defining the executor leaves nothing to reuse.

Acquiring on request. Notebooks run cells for many reasons: in Marimo, cells submitting executions reference the executor, so they run whenever the executor’s cell runs, including the first time, because a single experiment needs it. With confirm=True, nothing starts by itself: each execution is ready, displayed as a Run button (e.g. “▶ Run ramsey”, its overrides in the tooltip), and starts when clicked.

executor = Executor(None, runner=Remote("lab"), frontends=[Marimo()],
                    confirm=True, reuse=True)

pending = executor.submit(protocols.ramsey(delay=(10, 1000, 100)), nshots=500)
pending  # a Run button, then the spinner, then the result

A notebook full of experiments then opens with their buttons, and the results of those already recorded in the session (reused executions need no confirmation). Both are executor defaults, overridden by submit’s arguments of the same names; both are off by default.

• pending.start() confirms from code, as the button does.

• Downstream cells see pending() as None until the execution completes, as while running. Waiting for an execution that is not started (pending.result(), await pending) raises NotStartedError instead: it would block the confirmation itself.

• Executions are processed in the order they start, not in the order they were submitted: one waiting for confirmation does not hold the others.

• Where no frontend can show the button (e.g. a script, or a notebook run as an app without a browser), executions start at once, so that the same notebook runs everywhere.

Running again. A completed execution shows a Run again button (pending.rerun() from code), whatever reuse: the protocol is acquired again, and recorded anew in the session, where reuse then finds the latest. The same handle follows the new execution: cells referencing it see it running again, then the new result. Each protocol runs again on its own: the protocols depending on its update, submitted elsewhere, are not repeated, and with reuse they keep their recorded results until run again as well.

Alternatively, in Marimo only, set the runtime to mark dependent cells as stale instead of running them (on_cell_change = "lazy", in the notebook settings or under [tool.marimo.runtime] in pyproject.toml).

The job is held by a background thread and an open SSH connection: the kernel and the connection must stay alive until it completes. Detached jobs, surviving disconnections and restarts, are planned (#8).

Queue

While acquisitions wait in the queue, the queue itself can be followed from the notebook, through the same connection:

remote = executor.runner
remote.queue()  # asked once: jobs, node groups, summary, remote user
remote.monitor()  # a refreshable view, for Jupyter and Marimo

By default, the whole partition the acquisitions are submitted to is shown, so that the jobs ahead are visible. The jobs of the remote user are highlighted, and those submitted by the runner are tagged: its srun jobs are named qibocal-widgets-<action>, unless the slurm options set a name. The monitor refreshes periodically while displayed (every 30 seconds by default, interval=0 to refresh only on request), and can show only the user’s jobs. Every refresh asks the controller, so keep the interval reasonable.

The query (squeue, sinfo and whoami in a single round trip) and its fields are the ones of Qibocamp, whose launcher shows the same queue.