ROS 2 Warehouse Cell | Nate Biddle

Engineering brief

Testing the failure modes behind a moving fleet.

The visible Gazebo cell sits on top of a deterministic Python plant, a public ROS 2 contract, Fast DDS communication, and an optional SQLite checkpoint and event outbox. The suite treats a timeout as an uncertain outcome, duplicate delivery as a consumer-design problem, and concurrent order handling as an invariant problem rather than a happy-path API check.

Runtime: Ubuntu 22.04, ROS 2 Humble, Gazebo Sim 8, Python 3.10, colcon
Warehouse Model: 4 AGVs, 24 cases/SKUs, 2 picker cells, racks, lanes, chargers, conveyors, pallets
Distributed Surface: Topics, services, action, QoS, Fast DDS, diagnostics, SQLite checkpoint and outbox
Qualification Record: 104 executed checks passed; opt-in Kubernetes Pod restart remained host-blocked

Recorded qualification run

Gazebo and pytest observing one system.

The recording pairs the live cell with the 105-case qualification collection. Tests drive ROS state, services, actions, faults, and process boundaries while Gazebo mirrors authoritative domain poses. The visual scene makes behavior inspectable; ROS state, events, responses, durable rows, timing, and process evidence decide the result.

System under test

A distributed control surface with explicit ownership.

01 / Stimulus

SQA Clients

rclpy clients submit orders, actions, faults, stop commands, concurrent work, and bounded deadlines.

02 / Transport

ROS 2 + Fast DDS

Topics, services, an action, QoS profiles, callback groups, asynchronous futures, and isolated domains.

03 / Authority

Controller + Plant

A four-thread executor adapts the deterministic engine for dispatch, A* routing, inventory, faults, and pallet flow.

04 / Recovery + Projection

State, Events, SQLite

Checkpoint/outbox state feeds snapshots, live events, metrics, diagnostics, Gazebo, and the health gateway.

Source of truth: the ROS-adapted domain engine, not rendered pixels.

Persistence boundary: one opt-in SQLite writer, not a replicated database.

Isolation: deterministic seeds, unique ROS domains, Gazebo partitions, and owned process groups.

Qualification strategy

Test design follows failure boundaries, not feature checklists.

104passing checks

75unit + tooling

14integration

8process + system

7end-to-end

WH-NETEM-001

Real DDS Network Impairment

Fast DDS is forced onto UDPv4 so shared memory cannot bypass Linux tc netem. A 180 ms one-way delay measured 0.374 seconds round trip; a 300 ms delay created a committed-but-unknown outcome; 100% packet loss isolated the request until the link healed.

WH-DIST / WH-NET

Retries Without Duplicate Work

Stable request IDs and content fingerprints distinguish first-time acceptance, an equivalent replay, and conflicting reuse. Tests abandon or delay successful responses, reconcile authoritative state, retry, and require one reservation and one order effect.

WH-OUTBOX-CRASH-001

At-Least-Once Event Handling

A failpoint kills the controller after DDS publication but before the SQLite outbox row is marked. The same event ID arrives twice after restore, while the consumer's atomic claim applies it once and the warehouse records one durable completion.

WH-TXN / WH-ROUTE

Concurrency and Atomic Reservation

Two ROS clients are released together against scarce inventory. One wins, one receives a clean rejection, and no partial reservation or lost update survives. Surge scenarios also require bounded completion without collision, deadlock, or starvation of useful work.

WH-PAUSE / WH-RESTART

Process Pause and Recovery

SIGSTOP freezes a live controller until independent diagnostics report stale state, then coherent progress resumes. A separate mid-order SIGKILL restores the same run, revision, ownership, and work from SQLite and completes once.

WH-STREAM / WH-QOS / WH-GZ

Snapshots, Streams, and Convergence

Retained state serves late joiners while volatile events preserve live transition context through event, correlation, causation, run, and revision IDs. Separate checks reconcile event chains to state and verify stopped AGV poses converge into Gazebo.

Fleet scalability

More vehicles exposed the actual bottleneck.

The same deterministic surge workload -- three orders, 12 cases, and two picker docks -- ran five repeatability executions at each fleet size. Moving from one AGV to two reduced logical cycle time by 46.559%. Moving from two to four improved it by only 0.431%, showing that picker and dock capacity, not transport, had become the limiting resource.

These are deterministic model characterizations, not independent statistical samples or a physical warehouse capacity claim.

1 AGV217.25 s cycle

3.314 items/min

2 AGVs116.10 s cycle

6.202 items/min

4 AGVs115.60 s cycle

6.228 items/min

Deployment, observability, and evidence

Qualification continues beyond the test assertion.

Kubernetes POC

A non-root ROS image, single-replica Recreate Deployment, ClusterIP Service, PVC-backed SQLite state, and same-Pod health gateway define a bounded controller-recovery topology without Gazebo.

Pod Replacement Oracle

The opt-in kind workflow records Pod UID, run ID, revision, and an active durable order; deletes the Pod; then requires a new UID, restored readiness, non-regressing revision, and retained work.

Visible Prerequisite Failure

Static manifest and health-gateway checks executed successfully. The Pod-restart run could not create a cluster on this host because kind/kubectl and Docker access were unavailable, so the tooling emitted a JIRA-ready prerequisite report instead of implying runtime success.

Native Stack Evidence

An automated GDB scenario captures thread inventory, loaded libraries, locals where available, and thread apply all bt full. The retained 744-line stack traces rcl_wait through rclpy and CPython alongside RMW/Fast DDS worker threads.

Operational Signals

Versioned metrics, diagnostics, structured events, ROS/Gazebo logs, health/readiness endpoints, process backtraces, core-dump guidance, pdb, faulthandler, bags, and tracing support triage from application state down to native waits.

CI and Defect Evidence

Colcon-aware runners produce JUnit XML, machine-readable JSON, reviewer-friendly Markdown, raw evidence attachments, and bounded JIRA drafts with environment, reproduction command, expected result, actual result, and failure detail.

Reviewable evidence

The claims are backed by retained docs and run artifacts.

The qualification report is the quickest read. The network JSON and full GDB stack provide raw scenario evidence; the master guide, test plan, architecture, and reporting guide show how the suite was designed.

Final Qualification Report 104 passes, timings, scenarios, environment, reproduction Netem Measurements Latency, timeout, partition, transport, and one-effect evidence Full GDB Stack 744-line ROS, rclpy, CPython, RMW, and Fast DDS thread capture Kubernetes Prerequisite Report Explicit host limitation and JIRA-ready failure evidence Master SQA Guide System model, distributed semantics, test method, operations Test Plan Scenario catalogue, invariants, oracles, entry and exit criteria Architecture ROS contract, authority, consistency, delivery, process boundaries Reporting Workflow JUnit conversion, artifact retention, triage, and JIRA workflow Debugging Guide GDB, core dumps, Python, ROS graph, QoS, Gazebo, and tracing JSON Summary Machine-readable test results, properties, and measurements JUnit XML CI-ready per-test results and scenario properties

Scope discipline

Strong evidence, stated at the boundary actually tested.

The DDS impairment result is real same-host Fast DDS UDPv4 traffic through a disposable Linux network namespace, not a claim about a production multi-host network.
The scaling result characterizes a deterministic warehouse model and identifies a modeled picker/dock bottleneck; it is not physical fleet sizing or an SLO.
The Kubernetes topology, manifest, probe behavior, and smoke workflow are implemented, while live Pod replacement remains unvalidated until the required local runtime is available.
The cell is a robotics software and SQA fixture, not certified machinery control, safety-rated emergency-stop hardware, localization, or manipulation validation.