RMW Backends: Zenoh, XRCE-DDS, Cyclone DDS

Capability truth (services / events / QoS per backend) is the generated Per-RMW Feature Matrix; version pins live on Support Status. This page is the architectural narrative.

nano-ros supports three RMW (ROS Middleware) backends for connecting embedded devices to a ROS 2 network. Each backend targets different deployment scenarios and resource constraints. Each Node picks its backend at build time; one binary can link multiple backends and bridge between them — see Cross-backend Bridges for the multi-RMW pattern.

Zenoh (rmw-zenoh)

The Zenoh backend uses zenoh-pico, a lightweight C client for the Zenoh protocol. The MCU participates directly in the Zenoh network – there is no protocol translation layer.

How it works:

1. The MCU runs zenoh-pico in client mode, connecting to a zenoh router (rmw_zenohd, ROS 2’s own) over TCP, UDP, or TLS.
2. zenoh-pico creates publishers and subscribers directly on the Zenoh network.
3. ROS 2 nodes running rmw_zenoh_cpp connect to the same router, enabling transparent interop.
4. zenoh itself defines a peer mode in which two zenoh-pico devices talk directly with no router. nano-ros does not ship it. The shim is compiled without zenoh-pico’s multicast transport and scouting (nros_zpico_build::MULTICAST_TRANSPORT = false) because those are three more code paths in a library whose job here is fitting on an MCU, and every nano-ros deployment reaches its peers through a router or an agent. NROS_SESSION_MODE=peer is therefore refused at session open with a message saying so — see issue 0682. Enabling it means flipping that constant, rebuilding the shim, and accepting the footprint.

Key characteristics:

• Client/router topology; zenoh’s router-free peer mode is compiled out (see above)
• The router forwards messages without interpreting them – it is not a protocol translator
• If the router crashes, peers in client mode lose routing but the MCU continues running
• Full ROS 2 graph discovery via liveliness tokens
• Transport options: TCP, UDP, TLS (via zpico-smoltcp or platform sockets)

XRCE-DDS (rmw-xrce)

The XRCE-DDS backend uses Micro-XRCE-DDS-Client, the same client library used by micro-ROS. It follows an agent-based model where a lightweight client on the MCU delegates entity creation to an agent process.

How it works:

1. The MCU runs the XRCE-DDS client, connecting to an Agent process over UDP or serial (UART).
2. The client sends requests to the Agent: “create a publisher on topic X with type Y.”
3. The Agent creates full DDS entities on behalf of the client and bridges data between the XRCE protocol and the DDS data space.
4. ROS 2 nodes using any DDS-based RMW (FastDDS, Cyclone DDS) communicate through the Agent.

Key characteristics:

• Agent is mandatory – the MCU cannot participate in the network without it
• If the Agent crashes, the MCU loses all connectivity
• Fully static memory allocation on the MCU (no heap required)
• Client-side discovery is not supported; the Agent handles it
• Transport options: UDP, serial (HDLC framing)

Cyclone DDS (rmw-cyclonedds)

Maturity status. Cyclone DDS supports pub/sub + service round-trips on the native (host) build (full service implementation in src/service.cpp); status events (liveliness, deadline-miss, etc.) are NOT wired to Cyclone listeners yet, and the service-availability probe is deferred (service_server_available returns NROS_RMW_RET_UNSUPPORTED — poll-with-retry instead of waiting on it). Wire-level interop with stock rmw_cyclonedds_cpp (Humble) works for topics and services. Full known-limitations list: docs/reference/cyclonedds-known-limitations.md.

The Cyclone DDS backend uses Eclipse Cyclone DDS, the same DDS implementation that ROS 2 ships with via rmw_cyclonedds_cpp. Built as a standalone C++ library at packages/rmw/cyclonedds/nros-rmw-cyclonedds/ that registers itself with the runtime through the C ABI vtable in nros-rmw-cffi.

How it works:

1. The application (or platform) calls nros_rmw_cyclonedds_register() once before nros::init() — this is automatic when the consumer’s CMake build sets -DNANO_ROS_RMW=cyclonedds.
2. The runtime stores the vtable pointer; subsequent nros::init() calls dispatch through it for session, publisher, subscriber, and service operations.
3. dds_create_domain / dds_create_participant / dds_create_topic / dds_create_writer / dds_create_reader are invoked under the hood; Cyclone owns its own RX threads.
4. ROS 2 nodes using stock rmw_cyclonedds_cpp interoperate directly — same wire protocol, same discovery, no key rewriting (unlike rmw_zenoh’s <domain>/<topic>/<type>/... scheme).

Key characteristics:

• Pure-C++ backend (not a Cargo crate) — Autoware contributors can read and extend the wrapper using the same patterns as actuation_module/include/common/dds/.
• ROS 2 Tier-1 wire compat — pinned to Cyclone DDS tag 0.10.5 to match ros-humble-cyclonedds 0.10.5 + ros-humble-rmw-cyclonedds-cpp 1.3.4.
• Static ddsi_config via dds_create_domain_with_rawconfig skips the XML parser; embedded-friendly.
• Discovery via SPDP multicast or unicast peer list (mirrors Cyclone’s standard config knobs).
• Heap required (Cyclone uses malloc); BUILD_SHARED_LIBS=ON produces libddsc.so for POSIX, static link for embedded.
• Services wired (create/recv/reply — service.cpp); the availability probe (service_server_available) and actions are still unsupported.
• No status events yet — register_subscription_event / register_publisher_event / assert_publisher_liveliness slots are not wired to Cyclone listeners yet.

Build:

Contributors: the in-tree Cyclone DDS build/test lanes are in Per-Platform Contributor Lanes.

Each example picks its RMW via -DNANO_ROS_RMW=cyclonedds at configure time; the root CMakeLists.txt add_subdirectory’s packages/rmw/cyclonedds/nros-rmw-cyclonedds/ and links the resulting target into NanoRos::NanoRos. No build/install/ prefix, no find_package(NrosRmwCyclonedds) deleted both.

Availability follows from config — the build provisions Cyclone. Selecting cyclonedds (-DNANO_ROS_RMW=cyclonedds / --features rmw-cyclonedds) is your whole responsibility; nros_provide_cyclonedds() resolves the library: a prebuilt install you point at (-DCMAKE_PREFIX_PATH=<install> / -DCycloneDDS_DIR=), else self-provision from source — by default the pinned third-party/dds/cyclonedds submodule, or your own checkout via -DCYCLONEDDS_SOURCE_DIR=<path>. So a bare cmake/cargo build needs no just cyclonedds pre-step (that is a contributor-only recipe); freertos / threadx-rv64 / native examples build Cyclone on demand (sccache-accelerated), gated on the relevant cross toolchain. On a tier without the toolchain, the embedded-Cyclone tests are filtered out of test-all (skipped, not failed). idlc is a host tool, found on PATH (e.g. a ROS 2 install) or via -DIDLC_EXECUTABLE=. See docs/development/sdk-tiers.md § “CycloneDDS — self-provisioned in CMake”.

Known limitations. Cyclone DDS currently has a 2× CDR roundtrip per message, deferred status-event wiring, pending service request-id correlation, and incomplete Cortex-A/R Zephyr board support. See docs/reference/cyclonedds-known-limitations.md for the full list. ARMv8-R toolchain prep (Cortex-A 64-bit FVP, Cortex-R52 hardware) is in docs/reference/zephyr-armv8r-setup.md.

Comparison

¹ Footprint figures are order-of-magnitude planning numbers, not measurements — they vary with features, message set, and opt level. Size your own image with arm-none-eabi-size on your build; the RAM knobs are enumerated in the Static Pool Inventory.

Multi-backend binaries (bridges)

A single nano-ros binary can link more than one RMW backend and forward traffic between them. The bridge pattern is useful for:

• Translating between protocols — a gateway node running zenoh ingress + DDS egress lets MCU fleets on zenoh-pico talk to an Autoware stack on Cyclone DDS without a separate translator.
• Hard-real-time + best-effort split — a high-priority Node on Cyclone DDS for control loops, a low-priority Node on Zenoh for telemetry, both in one process.
• Bringing up an XRCE Agent-free fleet — bridge XRCE devices to a Zenoh network so they look like first-class participants to stock ROS 2.

The pattern uses Executor::open_with_rmw("<name>", ...) to pin the primary session and node_builder("name").rmw("<other>").build() to open additional sessions on other backends. Both backends must be in the binary’s link line (Cargo manifest deps + register() call each).

A worked example lives at examples/bridges/native-rust-zenoh-to-dds/; the time-triggered variant under examples/bridges/tt-zenoh-to-xrce/ shows the same pattern under an ARINC-653-style cyclic schedule.

Full walkthrough: Cross-backend Bridges.

RMW Selection

RMW selection is a declared, language-agnostic, per-deploy value — you set it once in system.toml ([system].rmw, optionally overridden per image by [image.<id>].rmw) or via a CLI/build flag, and the toolchain lowers that declaration to each language’s native build mechanism: a Rust cargo rmw-<x> feature, or a CMake -DNANO_ROS_RMW cache var. The cargo feature and the CMake var below are the lowering targets the build uses — not the user-facing knob. A binary links the common rmw-cffi runtime plus exactly one backend; the linked backend self-registers through the nros_rmw_vtable_t C ABI, and the registry walker resolves it at Executor::open. See RFC-0031 for the full selection-and-lowering model and precedence rules.

Cargo.toml (Rust)

Lowered form of the declared RMW. The selection point is the board crate (RFC-0031’s phase-248 C5b amendment): an app declares one rmw-<x> feature that activates the backend dep and forwards to the board crate’s matching feature. Codegen no longer emits nros/rmw-<x> or nros/platform-<y> — the board brings both.

[features]
default = ["rmw-zenoh"]
# Each `rmw-*` activates the app-owned optional backend dep and forwards to
# the board crate, which is what actually selects the backend.
rmw-zenoh = ["dep:nros-rmw-zenoh", "nros-board-linux/rmw-zenoh"]
rmw-cyclonedds = [
    "nros/rmw-cyclonedds",              # the marker feature; see below
    "dep:nros-rmw-cyclonedds-sys",
    "nros-board-linux/rmw-cyclonedds",
]

In a WORKSPACE you write none of this: [image.<id>].rmw in the bringup’s system.toml is the declaration, and the Entry pkg’s generated manifest carries the lowered form. The block above is the standalone copy-out shape, where there is no bringup to declare it in.

Cyclone DDS backend — Rust runtime sees the generic rmw-cffi C-ABI

vtable; actual Cyclone wiring lives C++-side under

packages/rmw/cyclonedds/nros-rmw-cyclonedds/ and is selected at CMake

configure time via -DNANO_ROS_RMW=cyclonedds. The Rust

manifest only carries the rmw-cffi feature; no Rust shim dep.

[dependencies] nros = { path = “<…>/packages/api/nros”, default-features = false, features = [“std”, “rmw-cffi”, “platform-posix”] }

Each example also calls `<backend>::register()` from `main()` before
`Executor::open` — this drags the rlib's CGU into the binary so the
linkme distributed-slice walker finds the backend. C/C++ builds rely
on the CMake-emitted strong stub from `nano_ros_link_rmw(... RMW <x>)`
instead.

For C++ consumers, the declared RMW lowers to the CMake cache var:

```bash
cmake -S . -B build -DNANO_ROS_RMW=cyclonedds  # zenoh / xrce / cyclonedds

Kconfig (Zephyr)

# Zenoh backend
CONFIG_NROS_RMW_ZENOH=y

# XRCE-DDS backend
CONFIG_NROS_RMW_XRCE=y

# Cyclone DDS backend (Cortex-A/R Zephyr targets)
CONFIG_NROS_RMW_CYCLONEDDS=y

Enabling more than one simultaneously produces a compile_error!().

Cyclone DDS — per-platform configuration profile

The Cyclone DDS backend speaks raw RTPS over UDP multicast and unicast. The host networking stack on every RTOS needs IGMP enabled, adequate net- buffer pool sizing for the SPDP / SEDP discovery burst, and a way to join a multicast group. Concrete deltas per platform:

Zephyr (platform-zephyr)

# IGMP for SPDP multicast discovery (239.255.0.1:7400+).
CONFIG_NET_IPV4_IGMP=y

# Multicast group slots — RTPS uses up to 4 builtin groups per
# participant (SPDP + SEDP pubs / subs / topics).
CONFIG_NET_IF_MCAST_IPV4_ADDR_COUNT=4

# Net-buffer pools — defaults of 14 / 36 are too small for the SEDP
# burst when more than two service / action entities exist on each
# participant. Symptom of undersizing: runtime starvation as the
# discovery exchange backs up. Cortex_a9 has 512 MB SRAM; the cost
# is trivial.
CONFIG_NET_PKT_RX_COUNT=256
CONFIG_NET_PKT_TX_COUNT=128
CONFIG_NET_BUF_RX_COUNT=512
CONFIG_NET_BUF_TX_COUNT=256
CONFIG_NET_BUF_DATA_SIZE=512

# Heap — Cyclone DDS (malloc) + heapless mailboxes.
CONFIG_COMMON_LIBC_MALLOC_ARENA_SIZE=4194304
CONFIG_HEAP_MEM_POOL_SIZE=524288

On qemu_cortex_a9, additionally use a board overlay to bump the GEM driver’s RX / TX descriptor ring (default 32) so a brief drainer pause doesn’t immediately spill:

&gem0 {
    promiscuous-mode;
    rx-buffer-descriptors = <128>;
    tx-buffer-descriptors = <64>;
};

native_sim is not yet supported — Zephyr’s NSOS driver doesn’t forward IP_ADD_MEMBERSHIP to the host kernel (no IPPROTO_IP case in nsos_adapt_setsockopt). Use qemu_cortex_a9 for DDS-on-Zephyr testing instead.

FreeRTOS + lwIP (platform-freertos)

// FreeRTOSConfig.h / lwipopts.h
#define LWIP_IGMP            1   // SPDP multicast
#define LWIP_SO_RCVTIMEO     1   // Cyclone DDS recv timeouts
#define LWIP_BROADCAST       1
#define IP_REASSEMBLY        1   // RTPS DATA_FRAG fragments
#define MEMP_NUM_NETBUF      32  // discovery burst headroom

NuttX (platform-nuttx)

CONFIG_NET_IGMP=y
CONFIG_NET_BROADCAST=y
CONFIG_NET_UDP_NRECVS=4
CONFIG_NET_RECV_TIMEO=y

ThreadX + NetX Duo (platform-threadx)

tx_user.h / nx_user.h:

#define NX_ENABLE_IGMPV2          // IGMP v2 for RTPS multicast
// NetX BSD layer init must call `bsd_initialize` early — required
// for `setsockopt(SO_RCVTIMEO)` support.

Bare-metal smoltcp (platform-mps2-an385, platform-stm32f4,

platform-esp32-qemu)

#![allow(unused)]
fn main() {
// Bridge config in the board crate.
let mut config = smoltcp::iface::Config::new(...);
config.multicast_groups = vec![Ipv4Address::new(239, 255, 0, 1)];
// MulticastConfig::Strict in smoltcp 0.x; the bridge must expose
// a `join_multicast_group` API.
}

POSIX (platform-posix)

No configuration needed — the kernel does IGMP and setsockopt support natively. Just ensure the loopback interface is up (default).

When to Use Which

Choose Zenoh when:

• You need ROS 2 interop with rmw_zenoh_cpp (the recommended ROS 2 middleware for Jazzy+)
• You want peer-to-peer communication without any bridge process
• Your MCU has at least ~16 KB of heap and ~100 KB of flash
• You are using TCP or UDP networking
• You want simpler deployment (the router is a single process started with ros2 run rmw_zenoh_cpp rmw_zenohd)

Choose XRCE-DDS when:

• Your MCU has very limited RAM (under 8 KB available for middleware)
• You need serial (UART) transport – useful for MCUs without networking hardware
• You are integrating with an existing micro-ROS or DDS deployment
• You want zero heap allocation on the MCU side

Choose Cyclone DDS when:

• You want direct, brokerless ROS 2 interop with no router and no Agent
• Your MCU has at least ~32 KB RAM (heap for Cyclone + RTPS state) and a network stack with IGMP
• You’re integrating with a DDS-based ROS 2 deployment running stock rmw_cyclonedds_cpp and want full RTPS wire-compat without protocol translation
• Your tolerance for failure is “samples from peer X stop arriving” rather than “the whole router goes down”
• Note: Cyclone DDS is pub/sub-only today (no services / actions yet)

Any of the three works well for:

• Standard pub/sub and service patterns
• Integration with ROS 2 desktop nodes
• QEMU-based development and testing