# Warehouse Cell Architecture ## System at a glance The POC is a deterministic warehouse-domain simulator wrapped by ROS 2 and mirrored into a populated Gazebo world. ROS state is authoritative; Gazebo provides spatial visualization. Tests drive and observe the same public ROS contract used by the demo. ```text warehouse_interfaces messages / services / action | +-----------------------+-----------------------+ | | v v warehouse_core SQA test client cell_controller pytest + rclpy | +--> /warehouse/state ------+------------------> assertions +--> /warehouse/events -----+------------------> timeline | | | v | warehouse_gazebo/state_visualizer | | | v | Gazebo Transport /world/warehouse/set_pose | | +--------------------> Gazebo Sim 8 | v visible warehouse cell warehouse_gazebo/warehouse.launch.py starts/configures the assembled process set. ``` The domain engine is kept free of ROS and Gazebo imports. This gives the project a fast, deterministic unit-test seam while preserving black-box tests around the deployed ROS processes. ## Package inventory ### `warehouse_interfaces` The shared interface contract contains: | Kind | Type | Purpose | |---|---|---| | Message | `CellState` | Versioned aggregate snapshot with process-run/revision, scenario/time, emergency-stop state, entities, and throughput. | | Message | `VehicleState` | AGV pose, motion, load, battery, target, and fault. | | Message | `ItemState` | Case identity/SKU, pose, location, order, pallet layer. | | Message | `PickerState` | Picker activity, queue, pallet, active item, and fault. | | Message | `OrderState` | Requested quantities and order progress. | | Message | `WarehouseEvent` | Versioned, uniquely identified, sequenced/correlated/caused, timestamped operational event. | | Message | `TaskCommand` | Generic correlated command representation for extensibility/observation. | | Service | `SubmitOrder` | Submit SKU/quantity lines with a business idempotency key and disposition. | | Service | `ConfigureNetworkFault` | Configure the SQA-only application service relay's deterministic pass/delay/drop/partition mode. | | Service | `ResetCell` | Reset to a named deterministic scenario and seed. | | Service | `InjectFault` | Apply a supported timed or persistent fault. | | Service | `ClearFault` | Clear a target's active fault. | | Service | `SetEmergencyStop` | Assert or reset the simulated cell-wide stop. | | Action | `FulfillOrder` | Long-running idempotent order request with progress feedback, terminal result, and disposition. | The `.msg`, `.srv`, and `.action` files are the source of truth for fields and units: ```bash ros2 interface package warehouse_interfaces ros2 interface show warehouse_interfaces/msg/CellState ros2 interface show warehouse_interfaces/action/FulfillOrder ``` ### `warehouse_core` The pure Python domain layer contains: - `models.py`: poses, cases, AGVs, pickers, orders, jobs, events, and snapshots; - `engine.py`: deterministic ticking, dispatch, travel, pick/pallet flow, faults, emergency stop, and invariant checking; - `routing.py`: grid-based A* routing and route validation; - `pallet.py`: deterministic layered/interlocked pallet placement logic; - `state_store.py`: optional single-controller SQLite checkpoint and at-least-once event outbox; - `event_dedup.py`: live-consumer atomic `event_id` claim/apply-once helper for at-least-once streams; - `network_fault_relay.py`: SQA-only `SubmitOrder` application-boundary delay/drop/partition proxy; - `metrics.py`: versioned JSON monitoring schema, renderer, and tolerant parser; - `scenarios.py` and `config/scenarios.yaml`: populated scenario definitions. Installed ROS executables are: - `cell_controller`: adapts the domain engine to ROS topics, services, action, parameters, and timers; - `health_monitor`: observes state/events and reports liveness/invariant health; - `health_gateway`: exposes latest-state freshness and a read-only state/metrics projection over HTTP for container probes; - `network_fault_relay`: forwards test order calls through a deterministically configurable application boundary; - `scenario_client`: submits or demonstrates named scenarios from the command line. The important design boundary is: ```text pure domain methods <-> cell_controller serialization/callback adapter <-> ROS graph ``` Unit tests belong on the left; interface and functional tests exercise the right. ### `warehouse_gazebo` This package installs: - `worlds/warehouse.sdf`, a warehouse world for Gazebo Sim 8; - reusable models for AGVs, cases, pickers, pick cells, pallets, racks, and charging/staging assets; - `state_visualizer`, which maps ROS state entity IDs to Gazebo entities; - `warehouse.launch.py`, the unified launch that starts `cell_controller`, `health_monitor`, Gazebo, and the visual adapter by default. The visualizer reads `/warehouse/state` and calls `/world/warehouse/set_pose`. It must not feed a second competing domain state back into `cell_controller`. ### Unified bringup There is no separate `warehouse_bringup` package in this POC. `warehouse_gazebo/warehouse.launch.py` is intentionally the sole assembled launch so demo and test topologies cannot drift. It starts the controller, monitor, Gazebo server/client, and visualizer; propagates scenario, seed, visual/headless, controller-rate, recording, node/Gazebo GDB, debug, and optional persistence choices; and shuts the graph down when Gazebo exits. `state_db_path` is empty and `controller_respawn` is false by default; enable respawn only with an intentional persistence test and isolated database path. Discover the installed launch surface rather than relying on memory: ```bash ros2 launch warehouse_gazebo warehouse.launch.py --show-args ``` Operator recovery exercise: ```bash ./scripts/run_demo.sh --state-db /tmp/warehouse-recovery.sqlite3 \ --respawn-controller # Explicitly discard the file's prior checkpoint before the next run. ./scripts/run_demo.sh --state-db /tmp/warehouse-recovery.sqlite3 \ --fresh-state --respawn-controller ``` Use one database path per isolated test/run. `--respawn-controller` requires a state database; the ordinary command without `--state-db` remains in-memory. ## Public ROS contract The black-box test client uses these stable names: | Name | Kind/type | Semantics | |---|---|---| | `/warehouse/state` | `warehouse_interfaces/msg/CellState` topic | Periodic aggregate state used by UI, tests, and visualizer. | | `/warehouse/events` | `warehouse_interfaces/msg/WarehouseEvent` topic | Operational transition/fault/safety events. | | `/warehouse/reset` | `warehouse_interfaces/srv/ResetCell` service | Reset state and deterministic seed. | | `/warehouse/submit_order` | `warehouse_interfaces/srv/SubmitOrder` service | Validate an order and return `ACCEPTED`, idempotent `REPLAYED`, `CONFLICT`, or `REJECTED` using `request_id`. | | `/warehouse/inject_fault` | `warehouse_interfaces/srv/InjectFault` service | Inject a supported fault. | | `/warehouse/clear_fault` | `warehouse_interfaces/srv/ClearFault` service | Clear a target fault. | | `/warehouse/emergency_stop` | `warehouse_interfaces/srv/SetEmergencyStop` service | Control the simulated stop state. | | `/warehouse/fulfill_order` | `warehouse_interfaces/action/FulfillOrder` action | Long-running idempotent order workflow with `request_id`, progress feedback, terminal result, and disposition. | | `/warehouse/commands` | `warehouse_interfaces/msg/TaskCommand` topic | Asynchronous submit/fault/stop command ingress; `command_id` is the order-submit idempotency key. | | `/warehouse/metrics` | `std_msgs/msg/String` topic | Periodic JSON operational summary. | | `/diagnostics` | `diagnostic_msgs/msg/DiagnosticArray` topic | Health, stale-state, battery, fault, stop, and separation diagnostics. | | `/warehouse_fault_proxy/submit_order` | `warehouse_interfaces/srv/SubmitOrder` service | SQA-only proxy ingress used to place deterministic faults before/after the real order service call. | | `/warehouse_fault_proxy/configure` | `warehouse_interfaces/srv/ConfigureNetworkFault` service | Select `PASS`, `DELAY_BEFORE_FORWARD`, `DELAY_AFTER_COMMIT`, `DROP_BEFORE_FORWARD`, or `PARTITION`. | | `/world/warehouse/set_pose` | Native Gazebo Transport service | Update a named visual entity pose; not a ROS graph service in this POC. | Verify names and QoS from a running overlay: ```bash ros2 topic list -t ros2 topic info /warehouse/state --verbose ros2 service list -t | rg '^/warehouse/' ros2 action list -t gz service -l | rg '^/world/warehouse/set_pose$' ``` The `/warehouse_fault_proxy/*` endpoints exist only when the test relay is started. They are a service-level fault-injection seam, not the production order endpoint and not a packet/DDS/network emulator. ## Consistency, delivery, and recovery boundaries ### Authoritative state versus projections | Surface | Role | Retention/recovery boundary | |---|---|---| | `WarehouseEngine` in `cell_controller` | Sole authoritative business state for the running process. | In memory by default; optionally checkpointed to one local SQLite file. | | `/warehouse/state` | Periodic full snapshot projection with schema version, controller `run_id`, and within-run revision. | Controller publisher is reliable/transient-local/keep-last 1, so a compatible late subscriber can receive the latest retained sample while that publisher lives. It is not persistent storage. | | `/warehouse/events` | Live transition/fault/safety projection with event ID, within-run sequence, correlation, and causation metadata. | The ROS topic is reliable/volatile/keep-last 100 and is not a durable broker/audit log. Optional SQLite retains unpublished outbox rows for restart publication. | | `/warehouse/metrics` | Periodic schema-versioned JSON observability projection. | Derived data for monitoring; never an order or inventory source of truth. | | `/diagnostics` | Health projection from the monitor. | Useful current diagnosis, not a complete incident history. | | Gazebo entities | Spatial visualization projection. | Mirrors ROS domain poses; a visually updated case is not proof of a committed warehouse transition. | Tests reconcile projections against the engine's public snapshot rather than electing whichever message arrived last as authoritative. Snapshot and event streams can be observed at different times: the controller mutates the engine and queues events inside a callback, then a timer publishes the next snapshot and drains queued events. In default in-memory mode, a process exit between those steps can erase the pending evidence. With SQLite enabled, mutation checkpoints and pending outbox rows narrow that window, but publish-before-mark may redeliver an event and therefore requires `event_id` deduplication. `WH-OUTBOX-CRASH-001` proves the concrete window: generation 1 publishes one `ORDER_ACCEPTED` and hard-exits before marking its row; generation 2 publishes the same `event_id` again. A live deduplicator sees two deliveries but applies one effect. ### Request outcome model For a ROS service or action, transport completion and business completion are separate facts: ```text request dispatched | +--> explicit rejection -----------------> known no-commit path | +--> accepted/mutated --> response seen -> known outcome for this process epoch | response lost -> uncertain client outcome | +--> server pause/exit ------------------> timeout or broken future; inspect state ``` A timeout does not prove that the request failed before mutation. `/warehouse/submit_order` and `/warehouse/fulfill_order` accept a stable `request_id`: first application returns `ACCEPTED`, an equivalent retry returns the existing order as `REPLAYED`, and changed content under the same key returns `CONFLICT` without mutation. Asynchronous order commands use `TaskCommand.command_id` the same way. The ledger survives only the current process unless `state_db_path` enables checkpointing. Unit tests cover replay/conflict; live `WH-DIST-001` abandons an unobserved first response at the application node, independently observes commit, then proves replay from a new client. `WH-NET-001..003` place deterministic delay/drop/partition behavior in a test-only ROS service relay. In particular, `DELAY_AFTER_COMMIT` lets the authoritative call finish but withholds its result past a real client deadline; reconciliation then finds one order and same-key retry returns `REPLAYED`. Pre-forward drop/partition modes guarantee zero authority calls until a new retry after `PASS`. These are application-boundary behaviors, not packet loss, DDS retransmission/discovery, kernel traffic shaping, or asymmetric-network proof. `WH-NETEM-001` is the distinct real-packet seam. The complete client/controller graph runs in a disposable unprivileged Linux user/network namespace, Fast DDS builtin transport is forced to UDPv4 so shared memory cannot bypass the test, and `tc netem` changes the namespace's loopback qdisc. It exercises 180/300 ms one-way delay and a bounded 100%-packet-loss partition, then reconciles state and same-key retries after healing. Because DDS is reliable, either the retransmitted original or explicit recovery retry may win; `ACCEPTED` versus `REPLAYED` is nondeterministic at that race, while one order effect is mandatory. The evidence is real kernel/DDS loopback behavior, but remains same-host, symmetric, short-duration, and does not qualify a physical or multi-host network. ### Transaction and concurrency boundary The live controller uses a four-thread `MultiThreadedExecutor`, one reentrant callback group, and an `RLock` around engine access. Order validation, idempotency lookup/fingerprint comparison, and reservation are one process-local critical section: all expected validation occurs before order/case mutation, so an expected rejection/conflict leaves inventory unchanged. This prevents the modeled lost-update race when two service callbacks compete for one case, and `WH-TXN-001` exercises that public boundary. This critical section is not a distributed transaction. With SQLite enabled, the controller checkpoints a mutation before its service/action response escapes. At the timer boundary, it writes the new checkpoint and drained events to the local outbox in one SQLite transaction, publishes pending rows in sequence order, then marks them published. A crash after publish but before marking can redeliver an event; consumers use `event_id` to deduplicate. The provided deduplicator is process-local memory, so a consumer that must survive its own crash must persist the event claim in the same transaction as its business effect. DDS publication, service response transport, and the SQLite commit cannot be made one atomic operation, and there is no general rollback for an unexpected exception after in-memory mutation. ### Restart, failover, and disaster recovery The ordinary demo leaves `state_db_path` empty and therefore remains in-memory. An opt-in persistence/recovery exercise is available: the local SQLite checkpoint stores logical state, in-flight routes/jobs, processed idempotency requests, event history/pending events, run/revision, and configuration; `restore_from_checkpoint:=true` reloads it; `controller_respawn:=true` can restart the process when used deliberately with a database path. `WH-RESTART-UNIT-001` proves checkpoint reconstruction and once-only completion below the live process boundary. Live `WH-RESTART-001` abruptly kills controller generation 1 mid-order, starts generation 2 against the same temporary database, and proves the same run/revision/order resumes to one observed and durable `ORDER_COMPLETE`. The separate `WH-OUTBOX-CRASH-001` case targets the duplicate-after-publish window. These tests do not prove launch-driven respawn specifically, client/action reconnection, corrupt-database behavior, host loss, or performance under checkpoint I/O. There is still no standby replica, leader election, replicated store, backup/restore procedure, declared RPO/RTO, or split-brain protection. A retained DDS sample is not storage, deterministic `/warehouse/reset` is not disaster recovery, and respawning one controller from local SQLite is recovery—not failover. ## State and ownership ### Authoritative ownership - `WarehouseEngine` owns business state: inventory, assignments, case ownership, order progress, pallet contents, faults, and simulated domain position. - `cell_controller` owns the ROS representation and callback ordering around that engine. - Gazebo owns the rendered/physics world, but its entities mirror the ROS positions for this POC. - The test harness owns scenario correlation, observations, deadlines, assertions, and evidence. ### Case location lifecycle ```text storage -> dispatch_queue / RESERVED -> vehicle / IN_TRANSIT -> picker queue / AT_PICKER -> pallet / PALLETIZED ``` Exception paths must retain one location/owner: a fault does not clone or lose a case. Order completion requires every reserved case to reach the pallet exactly once. ### Timing The engine advances deterministically from a ROS wall timer. At the default `tick_hz=20` and `simulation_speed=3.0`, each callback advances the logical plant by `3/20` seconds. `CellState.simulation_time_sec` is the domain clock and resets with the scenario. The POC deliberately does not bridge Gazebo `/clock` into ROS, and the visualizer runs with `use_sim_time=false`; Gazebo's `paused` argument therefore does not pause the logical controller. Harness timeouts use Python's monotonic wall clock. This split is a documented POC simplification, not a model for physics-coupled control. The engine supports stable fleet prefixes of 1, 2, or 4 AGVs. `WH-SCALE-001` keeps the three-order/12-case/two-picker surge workload fixed across those configurations. Its median logical cycles are 217.25, 116.10, and 115.60 seconds respectively: a 46.559% reduction from one to two and only 0.431% from two to four. In this model the bottleneck shifts from transport toward the fixed picker/dock capacity; adding modeled vehicles cannot create linear throughput once downstream service is saturated. This is deterministic domain characterization, not physical fleet sizing or live ROS/Gazebo performance. ## Deployment and process boundaries The default POC favors separate processes because they are observable and individually debuggable. A launch file starting several nodes is orchestration, not ROS node composition. If native composable nodes are added later, test both the deployed container and an isolated debug topology because crash, intra-process transport, and GDB boundaries change. Typical process set: ```text gz sim server gz sim GUI client (omitted in headless mode) cell_controller health_monitor state_visualizer scenario_client (demo only; exits after submission/observation) pytest SQA probe (test runs only) ``` The separate container exercise deliberately deploys only the control/recovery slice: ```text Deployment warehouse-cell (replicas=1, strategy=Recreate) Pod (shared loopback/DDS namespace) +-- controller container | +-- cell_controller | +-- /var/lib/warehouse/state.sqlite3 --> PVC +-- health-gateway sidecar +-- /warehouse/state subscriber +-- HTTP :8080 /healthz process liveness /readyz recent ROS state required /state read-only state projection /metrics Prometheus text projection | +--> ClusterIP Service warehouse-cell:8080 ``` `deploy/docker/Dockerfile` is a non-root ROS 2 Humble controller image and excludes Gazebo. `deploy/k8s/warehouse-cell.yaml` uses a 256 MiB `ReadWriteOnce` PVC backed by a local `hostPath` PV, plus read-only root filesystems and an `emptyDir` for `/tmp`. `ROS_LOCALHOST_ONLY=1` is intentional because the two containers share one Pod network namespace. The HTTP Service is an orchestration observation boundary; it neither carries order commands nor brokers DDS. `/healthz` proves only that the gateway HTTP process is live; `/readyz` requires a fresh ROS state and therefore reflects controller-to-gateway progress. The controller container has no dedicated liveness probe, so a hung controller makes the Pod unready but is not necessarily restarted by this manifest. A controller process exit, whole-Pod deletion, and a non-progressing process are distinct failure cases. One replica and `Recreate` avoid an intended overlapping rollout against a single SQLite writer. They do not implement leader election or split-brain fencing. The local `hostPath` survives a Pod replacement on the same disposable kind node but is not portable, replicated, backed up, or qualified for multi-node rescheduling. `WH-K8S-STATIC-001..002` cover the checked-in topology statically, and `WH-K8S-GATEWAY-001` executes the HTTP freshness projection without a cluster. `test/deployment/test_kubernetes_restart.py` wraps `scripts/k8s_smoke.sh` for opt-in runtime `WH-K8S-001` (explicit whole-Pod deletion, new Pod UID, readiness, same run, non-regressing revision, retained order). It was skipped by design in the final non-opted-in full run; an explicit attempt fails its prerequisite gate before cluster creation because `kind`/`kubectl` are absent, and Docker socket access is also unavailable on this host. ## Failure domains | Failure domain | Symptom | First checks | |---|---|---| | Domain logic | Illegal transition, wrong count/route, invariant event | Reproduce in unit test with same scenario/seed. | | ROS adapter | Service hangs, stale state, conversion error | Node log, graph, QoS, executor/callback stacks. | | DDS/discovery | No endpoints or intermittent connection | `ROS_DOMAIN_ID`, RMW, topic info, discovery readiness. | | SQA network relay | Proxy timeout/drop disposition or unexpected authoritative mutation | Configured relay mode, pre/post-forward boundary, client monotonic deadline, authoritative order/event counts. | | Linux/DDS packet path | Latency bound missed, response timeout, no state, recovery race | Namespace capability, forced UDPv4/no shared memory, active qdisc, monotonic timing, post-heal state/idempotency. | | Gazebo server | World/service unavailable or server exits | Gazebo log, world parse, Transport service list, resource path, GPU/physics error. | | Visualizer/Transport adapter | Correct ROS state but stale/wrong entity | Entity names, `/set_pose`, coordinate mapping, Gazebo bindings. | | GUI/rendering | Server/test healthy but client blank/crashes | Display/Wayland/X11, GPU/driver, GUI log; compare headless run. | | Container/Pod | Gateway live but not ready, Pod replacement, PVC mount/write failure | `/healthz` versus `/readyz`, Pod events/logs/UID, PVC/PV binding, state file permissions, pre/post run/revision. | | Test harness | False timeout, leaked context/process | Wall-vs-sim time, fixture teardown, last-state diagnostic. | ## Encoding and contract evolution The ROS IDL and JSON metrics projection are separate schemas: - `.msg`, `.srv`, and `.action` files define generated ROS types. Compatibility includes field types/order, array relationships, units, state vocabulary, QoS, and supported old/new build combinations. - `/warehouse/metrics` carries JSON in `std_msgs/msg/String`. Version 1 requires `schema_version`, `scenario`, `simulation_time_sec`, `completed_orders`, `palletized_items`, `throughput_items_per_min`, and `emergency_stop`. The current producer also emits `run_id`, `revision`, and `queue_depth`; the version-1 parser supplies safe defaults for those three fields when reading an older document. An additive JSON field can be compatible when consumers validate their required subset and ignore unknown fields. Removal, rename, type/unit change, and malformed JSON require explicit negative tests. ROS IDL evolution must be proven with producers and consumers generated from the versions actually deployed; “added one field” is not by itself a compatibility guarantee. Recorded bags and fixtures are versioned consumers too. The POC has a metrics schema version and schema metadata on aggregate state/events, but it has no interface-version handshake or mixed-version launch topology. The umbrella `WH-SCHEMA-001` scenario protects version-1 JSON round-trip, older-document defaults, additive fields, and actionable malformed/type errors; broader ROS interface compatibility remains a deliberate future qualification activity. ## Design constraints and extension points - Preserve deterministic domain functions when adding realism; put nondeterministic I/O behind adapters. - Version interface semantics deliberately. Adding a field is not enough—document units, optionality, QoS, and state meaning. - Keep IDs stable and globally unambiguous within a run. - Add coordinate-frame names before introducing multiple frames or real localization; the current aggregate poses are simplified. - For physics-authoritative movement, create one explicit command/feedback loop and retire pose mirroring as the authority. - For production-like safety, create a separate safety architecture; do not extend the simulated emergency-stop service and call it safety-rated. - A future production decomposition may introduce a separate bringup package; if it does, keep one authoritative launch topology and add launch-contract tests before migration. The full component, test, ROS, and safety discussion is in [MASTER_SQA_GUIDE.md](MASTER_SQA_GUIDE.md).