Process orchestration for taktora — architecture (arc42)¶
Architecture documentation for a process-orchestration layer above taktora, structured per the arc42 template (12 sections) and encoded with sphinx-needs using the useblocks “x-as-code” conventions (https://x-as-code.useblocks.com/how-to-guides/arc42/index.html).
taktora deliberately ships no supervisor, launcher, or process manifest: restart policy is “the host’s responsibility, matching taktora-executor’s existing posture” (see the connector cross-cutting requirements). This chapter specifies how an integrator wires one or more taktora processes into a supervised deployment — who launches them, in what order, how liveness is watched, how faults are recovered, and how cross-process integrity and fail-safety are preserved.
Note
Provenance / status. This is a design proposal authored at
open (drafted, not yet reviewed). Every req here is defined
inline for a self-contained handoff; when the concept is accepted the
requirements should migrate to requirements/orchestration/ and the
building blocks should reference concrete crate IDs. Nothing in this
chapter is implemented in the workspace today except the taktora-side
contract surface of taktora orchestration contr... (BB_0200), which already exists.
Top-level feature¶
An orchestration layer that launches, sequences, supervises, and recovers one or more taktora-executor OS processes as a coordinated deployment, using taktora’s existing lifecycle/health contract as the sole coupling surface. This is an integration-layer feature. It sits above the runtime heart (PLC runtime heart on iceoryx2 (FEAT_0010)) and the connector framework (Connector framework (FEAT_0030)); it is explicitly not part of the executor core, which stays a single-process reactor. Two realisations are in scope: reuse of an existing init/supervisor (systemd) and a bespoke taktora-native supervisor (“conductor”). |
1. Introduction and goals¶
The reason-to-exist is a gap, not a feature request against the
executor. A taktora process — one Executor (one WaitSet
dispatch thread) plus its connectors and one iceoryx2 Node, pinned to
a single IntegrityLevel — is a well-behaved orchestratee: it exits
cleanly on signal (Signal-driven shutdown (REQ_0050)), stops on a Stoppable handle
(Programmatic shutdown wakeup (REQ_0051)), gates its own cold start behind an admission check
(Cold-start integrity-verifi... (TSR_0011)), emits a bounded-period heartbeat (Heartbeat for Element B mon... (TSR_0010)),
and drives its outputs safe on a fatal fault without any taktora code
running afterwards (Abort on framework-invarian... (ADR_0065)). What is missing is the process
above it that turns N such binaries into a deployment.
The quality goals the orchestration layer is optimised for:
Processes shall start in a declared dependency order, and a dependent
shall not be considered started until its provider is ready (not
merely spawned). This closes the reader-before-writer race that the
|
Loss of a process’s liveness shall be detected within FTTI/2 (≤ 50 ms for the automotive target) via the heartbeat of Heartbeat for Element B mon... (TSR_0010), and the configured recovery action (restart of the process or its run group) shall be applied deterministically. |
The safe-state path shall not depend on the orchestrator, the crashed process, or any taktora code running after the fault. Outputs reach a safe state through the fieldbus watchdog (Abort on framework-invarian... (ADR_0065), Output-slave watchdog enabl... (AOU_0016)) regardless of what the supervisor does next. |
Where a hardened init/supervisor already provides ordering, restart, and watchdog semantics (systemd on Linux), the deployment shall reuse it rather than reimplement it. A bespoke supervisor is justified only by requirements the reused tool cannot meet (cross-process system states, safety-argument coupling). |
Startup, restart, and state transitions shall not weaken the spatial-isolation invariant of Cross-process hosting mode (TSR_0009) / Integrity-level declaration... (TSR_0003): safety-critical and QM-grade code stay in distinct OS processes communicating only over iceoryx2 single-writer channels (Single-publisher iceoryx2 t... (TSR_0007), Integrator unsafe-Rust disc... (AOU_0008)). |
2. Constraints¶
The orchestrator shall treat a taktora process as an opaque managed unit. It shall not reach inside a process to schedule items; the executor owns in-process scheduling. There is no in-process supervisor and none shall be introduced. |
Cross-process data flow shall be exclusively iceoryx2 SHM channels (Integrator unsafe-Rust disc... (AOU_0008)). The orchestrator’s own control plane (launch, health, commands) may use any transport, but shall not become a second data path between managed processes. |
Unlike a full execution-management daemon, taktora does not checksum or
verify signatures of the binaries it runs. If provenance is required,
the launcher (Launcher (BB_0202)) shall verify before |
The production realisation targets Linux (the platform of the
|
Consistent with taktora’s existing posture, the decision to restart a crashed process lives in the orchestration layer, never in the executor. The executor’s contribution is to fail fast and observably (Abort on framework-invarian... (ADR_0065)), not to self-heal. |
3. Context and scope¶
The orchestrator sits between the platform init/operator and a set of taktora processes, coupling to each process only through taktora’s lifecycle/health contract (taktora orchestration contr... (BB_0200)). A diverse, independent Element B monitor (per the SEooC decomposition, ASIL Decomposition) is a peer process the orchestrator launches but does not implement.
flowchart TB
OP["Platform init / operator<br/>(systemd, or boot script)"]
ORCH["Orchestrator<br/>(systemd units, or taktora-conductor)"]
subgraph MANAGED["Managed taktora processes"]
direction LR
P1["SC executor process<br/>(heartbeat, admission, HealthEvent)"]
P2["Gateway process<br/>(ConnectorGateway + stack)"]
P3["QM executor process<br/>(telemetry / non-safety)"]
end
MON["Element B monitor<br/>(diverse, independent process)"]
IOX[("iceoryx2 SHM<br/>data plane")]
BUS[("fieldbus + drives<br/>SM-watchdog → safe state")]
OP --> ORCH
ORCH -->|"launch · order · restart · state"| MANAGED
P1 -->|"heartbeat ≤ FTTI/2"| MON
MANAGED <--> IOX
P2 -->|"PDO"| BUS
In scope. Launch and dependency-ordered bring-up; readiness handshake; iceoryx2 service/SHM provisioning; liveness watch and restart; cross-process system-state transitions; reverse-order shutdown; optional launch-time executable authentication. Out of scope. In-process scheduling (owned by the executor); hard-real-time guarantees; the safe-state mechanism itself (owned by the fieldbus watchdog, Abort on framework-invarian... (ADR_0065)); the content of the Element B monitor’s diverse check (an integrator Assumption-of-Use); QNX realisation. |
4. Solution strategy¶
The strategy is two-track, chosen per deployment by whether an existing supervisor can meet the requirements.
Track A — reuse systemd (default, Linux). A taktora process is already
Type=notify/watchdog/restart-shaped, so ordinary systemd units express ordering, readiness, restart, and a watchdog fed by the heartbeat. Zero bespoke code. Chosen unless a requirement below forces Track B.Track B — taktora-conductor (bespoke supervisor). A small supervisor binary that reads a manifest in a declarative service / run-group / system-state / dependency / handshake) and drives the same taktora contract surface. Chosen when the deployment needs cross-process system states or a safety-argument coupling systemd does not express.
Context. Ordering, restart with backoff, and a liveness watchdog
are solved problems in hardened init systems. taktora already meets
the systemd notify/watchdog contract: Decision. On Linux, the default realisation is systemd units. A
taktora process raises Alternatives considered.
Consequences. systemd gives ordering, restart, and watchdog for free but expresses no safety argument and only crude “targets” for states; deployments needing cross-process system states or the SEooC coupling escalate to Track B. |
Context. Where Track A is insufficient, the missing concepts — grouping processes controlled together, sets of groups that run in a given vehicle/machine mode, inter-process start dependencies — are exactly the classic process-supervisor model (Run Group, System State, dependency, handshake). taktora’s own mode/state-machine feature (Mode / state-machine framework (FEAT_0019)) is in-process only and does not span processes. Decision. Alternatives considered.
Consequences. Integrators familiar with PLC/init supervisors map their mental model directly. The conductor stays a thin control plane; the data plane is untouched (iceoryx2 shared memory is t... (CON_0201)). |
Context. “Process spawned” is not “process ready”. A dependent started against a not-yet-subscribed provider silently loses the first publications (the example’s late-joiner hazard). A robust handshake needs an in-band readiness signal. Decision. The readiness predicate for a service is its first heartbeat tick or first ``HealthEvent::Up`` on the health channel — emitted only after the admission gate (Cold-start integrity-verifi... (TSR_0011)) has admitted the item set. The orchestrator blocks a dependent’s launch on the provider’s readiness (bounded by a per-service handshake timeout); timeout is a start fault, handled by the run group’s policy. Alternatives considered.
Consequences. Readiness reuses the exact channel the watchdog uses (Heartbeat for Element B mon... (TSR_0010)), so no new taktora surface is required for Track A or Track B. |
Context. The SEooC concept claims ASIL D by decomposition
( Decision. The orchestrator launches the Element B monitor as a separate process and wires the heartbeat channel to it, but does not implement or subsume it. The safe-state reaction remains the fieldbus watchdog (Abort on framework-invarian... (ADR_0065), Output-slave watchdog enabl... (AOU_0016)), so it holds even if the orchestrator itself dies (Fail-safety independent of ... (QG_0202)). A conductor that also acted as the monitor would defeat the independence argument. Alternatives considered.
Consequences. The safety story is a deployment property (three+ processes: control, monitor, fieldbus-enforced safe state), not a supervisor feature. |
5. Building block view¶
The orchestration layer decomposes into a taktora-side contract surface (already present) and a supervisor-side set of blocks (Track B; Track A maps them onto systemd primitives instead).
The set of existing taktora hooks an orchestrator couples to. This is the only taktora-side surface; it needs no new code.
|
The declarative model parsed by [[service]] # one managed process
id = "nc_hotpath"
exe = "/opt/app/nc_hotpath"
integrity = "SafetyCritical"
sched = { policy = "FIFO", prio = 80, affinity = [3] } # REQ_0041 / REQ_0040
heartbeat_timeout_ms = 20 # readiness + watchdog bound (TSR_0010)
restart = "on-failure" # CON_0204
verify_signature = true # optional; CON_0202
[[service]]
id = "ecat_gateway"
exe = "/opt/app/ecat_gateway"
integrity = "SafetyCritical"
restart = "on-failure"
[[run_group]] # controlled together
id = "motion"
members = ["ecat_gateway", "nc_hotpath"]
deps = [{ from = "nc_hotpath", needs = "ecat_gateway", state = "Up" }]
[[system_state]] # a machine/vehicle mode
id = "drive"
run_groups = ["motion", "telemetry"]
|
Spawns each service via |
Pre-creates the iceoryx2 services a run group needs with pinned QoS
(single-writer for safety-critical channels, Single-publisher iceoryx2 t... (TSR_0007);
|
Subscribes to every service’s heartbeat + |
The Track-A alternative to Health aggregator and resta... (BB_0204): a thin adapter that raises
# nc-hotpath.service
[Service]
Type=notify
ExecStart=/opt/app/nc_hotpath
WatchdogSec=100ms
Restart=on-failure
CPUAffinity=3
Requires=ecat-gateway.service
After=ecat-gateway.service
|
Requirements realised¶
The orchestrator shall start services in an order consistent with the declared dependency graph, and shall reject a manifest whose dependency graph contains a cycle. |
A service with unmet dependencies shall not be launched until each
provider signals readiness (first heartbeat or |
The orchestrator shall ensure the iceoryx2 services a run group requires exist with their declared QoS before any consumer of those services is launched. |
The orchestrator shall detect service liveness loss within FTTI/2 and apply the declared restart policy for the affected service or run group. |
On a system-state change request, the orchestrator shall start the run groups of the target state that are not running and stop those not associated with it, leaving shared run groups already running untouched. |
The orchestrator shall stop a run group’s services in the reverse of
their start order, delivering SIGTERM (or |
For a safety deployment, the orchestrator shall launch the integrator’s diverse monitor as a distinct process wired to the safety-critical services’ heartbeat channel, and shall not itself perform the diverse safety check. |
Where configured, the launcher shall verify a binary’s integrity (checksum + signature) before executing it, refusing to start the service on failure. |
6. Runtime view¶
Scenario R1 — dependency-ordered cold start (motion run group).
sequenceDiagram
participant C as Orchestrator
participant S as SHM provisioner
participant GW as ecat_gateway
participant NC as nc_hotpath
participant MON as Element B monitor
C->>S: provision iceoryx2 services (QoS pinned)
C->>GW: launch
GW-->>C: HealthEvent::Up (ready)
Note over C: dep {nc_hotpath needs ecat_gateway=Up} satisfied
C->>NC: launch
NC->>NC: admission_check → RUNNING
NC-->>C: first heartbeat (ready)
C->>MON: launch, wire NC heartbeat
Note over C,MON: run group "motion" is up
Scenario R2 — heartbeat miss → restart. The aggregator
(Health aggregator and resta... (BB_0204)) sees no nc_hotpath heartbeat within
heartbeat_timeout_ms; it applies restart = on-failure: stop the
run group in reverse order (Reverse-order shutdown (REQ_1205)), then re-run R1 for the
group. Meanwhile the SM-watchdog has already held/safed outputs — the
restart is not on the safety path.
Scenario R3 — crash → fail-safe → restart. A framework-invariant
violation in nc_hotpath triggers the fatal handler and
process::abort (Abort on framework-invarian... (ADR_0065)); no destructors run, the master
stops emitting PDO frames, each output slave’s SM-watchdog expires within
≤ FTTI/2 and applies safe-state values. The child’s abort exit is
observed by the launcher (Launcher (BB_0202)); the restart engine
(Health aggregator and resta... (BB_0204)) recovers per policy. Safe-state reached with zero
dependency on the orchestrator (:need:`QG_0202`).
Scenario R4 — system-state transition drive → parking. The
orchestrator diffs the target state’s run groups against the running set
(Cross-process system-state ... (REQ_1204)): starts parking-only groups, stops
drive-only groups in reverse order, leaves groups common to both
running.
Scenario R5 — graceful shutdown. SIGTERM to the orchestrator ⇒ stop
every run group in reverse start order (Reverse-order shutdown (REQ_1205)); each taktora
process returns cleanly from run() (Signal-driven shutdown (REQ_0050)); the SHM
provisioner reaps services last.
7. Deployment view¶
Each taktora process is a |
A
flowchart TB
CON["taktora-conductor"]
subgraph SCP["SC process — nc_hotpath (FIFO 80, core 3)"]
NC["Executor: cyclic NC @1 ms (motion + cia402)"]
end
subgraph GWP["gateway process — ecat_gateway"]
EC["ConnectorGateway + ethercrab"]
end
subgraph QMP["QM process — telemetry"]
TE["Executor: NDJSON export (off-RT)"]
end
MON["Element B monitor (separate process)"]
IOX[("iceoryx2 SHM")]
BUS[("EtherCAT + CiA 402 drives")]
CON -->|"provision · launch · order · restart"| SCP & GWP & QMP & MON
NC <--> IOX <--> EC
NC -->|"AxisStatus"| IOX --> TE
NC -->|"heartbeat ≤ FTTI/2"| MON
EC -->|"PDO"| BUS
|
Mapping of orchestration concerns to each track:
Concern |
Track A (systemd) |
Track B (conductor) |
|---|---|---|
Ordering |
|
dependency graph (Dependency-ordered startup (REQ_1200)) |
Readiness |
|
first heartbeat / Up (Readiness handshake before ... (REQ_1201)) |
Liveness watchdog |
|
health aggregator (Health aggregator and resta... (BB_0204)) |
Restart |
|
restart engine (Health aggregation and reco... (REQ_1203)) |
System states |
systemd targets |
|
SHM provisioning |
|
SHM provisioner (SHM provisioner (BB_0203)) |
Element B monitor |
separate unit |
separate service (Independent Element B monit... (REQ_1206)) |
8. Crosscutting concepts¶
Integrity preservation across restart. A restarted service is
re-launched with the same IntegrityLevel pin and the same
single-writer channel capabilities (Cross-process hosting mode (TSR_0009), Single-publisher iceoryx2 t... (TSR_0007));
the admission gate (Cold-start integrity-verifi... (TSR_0011)) re-verifies the isolation context
on every cold start, so a restart cannot silently downgrade isolation
(Cross-process integrity pre... (QG_0204)).
Shared-memory lifecycle. iceoryx2 services are persistent
/dev/shm resources; the first opener creates, later openers attach,
and open_or_create reaps stale services. The provisioner
(SHM provisioner (BB_0203)) makes creation explicit and ordered so QoS is pinned
by the owner, not raced by whichever consumer starts first.
Time and watchdog budget. All liveness bounds are expressed against FTTI/2 (≤ 50 ms automotive): heartbeat period (Heartbeat for Element B mon... (TSR_0010)), health transition latency (Bounded health-event latency (TSR_0006)), and the SM-watchdog timeout (Output-slave watchdog enabl... (AOU_0016)). The orchestrator’s detection timeout must be ≥ the heartbeat period and ≤ FTTI/2.
Observability. The orchestrator consumes the same health/heartbeat channels used for supervision; per-process logs flow through the existing logging facade / DLT backend. No separate orchestration telemetry plane is introduced (iceoryx2 shared memory is t... (CON_0201)).
Security. taktora authenticates nothing (taktora performs no executa... (CON_0202)); if the threat model requires it, Optional launch-time execut... (REQ_1207) moves signature verification into the launcher — the one place a process boundary is crossed.
QNX parallel. On QNX the same model maps onto the platform’s own launch/security-policy tooling; the manifest and readiness/handshake semantics are portable, the launcher and scheduling primitives are not.
9. Architecture decisions¶
ID |
Title |
Status |
Refines |
|---|---|---|---|
Reuse an existing init/supervisor where it suffices |
open |
||
Model the bespoke supervisor on a declarative service model |
open |
||
Readiness is first heartbeat / HealthEvent::Up, not spawn |
open |
||
Safety supervision needs a diverse Element B, not the conductor |
open |
10. Quality requirements¶
Quality scenarios refining the goals of §1 into measurable acceptance criteria:
Goal |
Scenario |
Measure |
|---|---|---|
A dependent launched before its provider is ready |
0 lost first-cycle publications across N cold starts |
|
A service stops emitting heartbeats |
fault raised ≤ FTTI/2 (≤ 50 ms) |
|
Orchestrator killed at the moment a service crashes |
outputs still reach safe state via SM-watchdog |
|
Deployment without cross-process states |
realised with systemd units only, no bespoke code |
|
Service restarted |
re-admitted with identical integrity + channel caps |
11. Risks and technical debt¶
A crashed process may leave a |
Heartbeat-as-readiness can time out if a provider is CPU-starved at start. Mitigation: separate (longer) handshake timeout from the (tighter) steady-state watchdog timeout; pin/prioritise SC services (SCHED_FIFO priority on Linux (REQ_0041)). |
taktora runs unverified binaries by default. Until Optional launch-time execut... (REQ_1207) is implemented, provenance depends entirely on filesystem/OS controls. |
Track A assumes systemd; QNX and minimal-init targets force Track B or a platform-native supervisor, widening the bespoke surface. |
An aggressive detection timeout under transient load can restart a
healthy service repeatedly. Mitigation: bounded backoff; detection
timeout ≥ heartbeat period; correlate with |
Separate-process deployments require the peer’s SHM pool to be sized ahead of time; under-sizing surfaces only at runtime. Mitigation: derive sizes from the manifest’s channel specs. |
If future work lets the conductor take safety-relevant reactions, it risks becoming a common-cause dependency with the control plane. Mitigation: keep safety reactions in the fieldbus watchdog and the diverse Element B monitor only. |
12. Glossary¶
The process (systemd, or |
|
One managed taktora OS process, the unit of orchestration (The OS process is the unit ... (CON_0200)). |
A set of services controlled together — started in dependency order, stopped in reverse. |
A set of run groups that should run together in a given machine/vehicle mode. |
The gate a provider must pass before dependents launch: its first
heartbeat or |
The diverse, independent monitor process supplied by the integrator to close the ASIL-D-by-decomposition independence argument (ASIL Decomposition). |
Fault-Tolerant Time Interval. All liveness and watchdog bounds here are expressed against FTTI/2 (≤ 50 ms automotive). |
A taktora process viewed as a managed unit — designed to be launched, watched, and recovered by an orchestrator, exposing only the contract surface of taktora orchestration contr... (BB_0200). |