Synthetic Data & Teleoperation

Generate synthetic data with Isaac Sim Replicator as tracked runs, drive the robot through upstream ROS teleop stacks, and record demonstrations that land as first-class datasets — so the chain record → inspect → train → evaluate needs no glue code. Caasi adds no SDG engine and no teleoperation of its own: it resolves what is installed and launches it.

Synthetic data — caasi dataset generate

The design did not change: Caasi implements no SDG engine. Generation reuses the experiment runner (backend: sim) after checking that the Replicator extension resolves from the sdg catalog domain, and the run writes into CAASI_DATASET_DIR. Inspection and validation are filesystem-only summaries of the synthetic tree — one rgb/ frame counts as one produced episode and no image is ever decoded.

Capabilities: where synth status went

caasi config catalog sdg [--json]
caasi doctor --component data [--verbose] [--json]
caasi dataset list [--limit|-n N] [--json]

The status verb split across the commands that already owned its two halves. The sdg catalog domain still describes the same two capabilities — replicator (exts/omni.replicator.core, core) and writer_kit (exts/omni.replicator.writer.kit) — both detected as paths under the Isaac Sim installation. caasi config catalog sdg prints that definition, and every field stays overridable with caasi config set catalog.sdg.<capability>.<field> <value>. The datasets a synthetic run produced — including one still generating — are listed by caasi dataset list, with each one's status and source read from its metadata.json.

Whether the capability is installed at all is gated by doctor's data component, which also covers the bag-storage plugins and the download CLIs. On a machine without Isaac Sim this is the real output:

shellcaasi doctor --component data
Environment Diagnostics

Data & Recording
  ✗ Replicator — not found (exts/omni.replicator.core)
      ↳ Install it, or point Caasi at the renamed upstream with `caasi config set
catalog.data.replicator.paths [...]`.
  • MCAP bag storage — not found (rosbag2_storage_mcap)
  • SQLite3 bag storage — not found (rosbag2_storage_default_plugins)
  • Hugging Face CLI — not found (hf, huggingface-cli)
  • NGC CLI — not found (ngc)
✘ 1 issue(s) found, 0 warning(s). Run with --verbose for hints.
exit code: 1

Generating: caasi dataset generate

caasi dataset generate CONFIG_PATH [--episodes N] [--workers N] [--record-images]
            [--record-depth] [--record-lidar] [--name NAME] [--dry-run] [-- SCRIPT_ARGS…]

The canonical synthetic-data launcher, and the command the old synth generate became. Its full option table lives with dataset generate; what follows is the part specific to synthetic data. CONFIG_PATH is the same experiment YAML sim run consumes — backend: sim plus a script that drives Replicator — --episodes N still becomes the script arg --episodes N and is recorded in metadata.json, --dry-run still prints the command without starting it, and anything Caasi doesn't recognize is appended verbatim to the script's argv.

What happens on launch:

1. The Replicator gate runs first: sdg.replicator must resolve, even for --dry-run. When it doesn't, the real failure is Error: Isaac Sim Replicator was not found (exts/omni.replicator.core). Install Isaac Sim, or point Caasi at it with `caasi config set catalog.sdg.replicator.paths [...]`. (exit 1).
2. The dataset directory is created as <name-slug>-<YYYYmmdd-HHMMSS> in the project's datasets/ or paths.datasets (default ~/.caasi/datasets), with --name supplying the slug — the same rules as dataset generate.
3. The script receives --dataset-dir <dir> and the environment variable CAASI_DATASET_DIR (plus --headless when the experiment declares it) — the generator writes the synthetic tree there.
4. A detached run (kind: dataset, the experiment's backend) is started and metadata.json is written.

The synthetic dataset tree:

sdg-dome-20260907-101500/
├── metadata.json
├── rgb/              # one frame = one produced episode
├── depth/
├── segmentation/
├── bounding_boxes/
└── metadata/

and its metadata.json:

{
  "name": "sdg-dome",
  "created": "2026-09-07T10:15:00+08:00",
  "generator": "replicator",
  "replicator": "/opt/isaac-sim-6.0/exts/omni.replicator.core",
  "experiment": "/home/you/demo/experiments/sdg_dome.yaml",
  "backend": "sim",
  "episodes": 200,
  "status": "generating",
  "run_id": "20260907-101500-sdg-dome"
}

shellcaasi dataset generate experiments/sdg_dome.yaml --episodes 200 --dry-run
Dry run — nothing was started:
  command: /opt/isaac-sim-6.0/python.sh /home/you/demo/scripts/sdg_collect.py --episodes 200 --headless --dataset-dir <dir>
  cwd:     /home/you/demo/experiments
caasi dataset generate experiments/sdg_dome.yaml --episodes 200
Dataset generation started: /home/you/.caasi/datasets/sdg-dome-20260907-101500
  Follow it with: caasi logs 20260907-101500-sdg-dome -f

Inspecting: caasi dataset inspect

caasi dataset inspect QUERY [--json]

The synthetic-data half of dataset inspect; QUERY is the usual dataset argument — a path, latest/last/newest, or a unique name/prefix, with unknown or ambiguous → Error: No dataset matching '<query>'. (exit 1). Inspect counts files, never decodes images:

• episodes produced = number of files in rgb/;
• per-directory file counts (recursive) for rgb, depth, segmentation, bounding_boxes and metadata;
• total file count and byte size of the whole dataset;
• elapsed seconds since the metadata's created timestamp;
• the linked run's id and live status.

shellcaasi dataset inspect latest
sdg-dome /home/you/.caasi/datasets/sdg-dome-20260907-101500
  generator: replicator
  episodes: 200 produced, 200 declared
  size: 1.9 GB
  elapsed: 3421.5s
  Run: 20260907-101500-sdg-dome (succeeded)
directory       files
rgb               200
depth             200
segmentation      200
bounding_boxes    200
metadata            3
caasi dataset inspect latest --json | jq '{episodes_produced, episodes_declared}'
{ "episodes_produced": 200, "episodes_declared": 200 }

JSON payload: {"path", "name", "generator", "episodes_declared", "episodes_produced", "counts", "files", "bytes", "elapsed", "run"}. run is null when unlinked or {"id", "status": "unknown"} if the run was deleted; elapsed is null when created is unparseable.

Validating: caasi dataset validate

caasi dataset validate QUERY [--json]

caasi dataset validate starts from the structural checks (metadata present, valid JSON object, name set, declared episode count vs episodes/) and, for a synthetic dataset, adds two SDG rules with these exact issue strings:

• when metadata declares an integer episodes and rgb/ exists: metadata declares 200 episode(s) but rgb/ holds 197;
• for each of depth/, segmentation/, bounding_boxes/ that exists: segmentation/ holds 0 file(s) but rgb/ holds 197 — each must match the rgb/ count.

shellcaasi dataset validate latest
2 issue(s) found:
  • metadata declares 200 episode(s) but rgb/ holds 197
  • segmentation/ holds 0 file(s) but rgb/ holds 197
exit code: 1
caasi dataset validate latest --json
{
  "path": "/home/you/.caasi/datasets/sdg-dome-20260907-101500",
  "valid": false,
  "counts": { "rgb": 197, "depth": 197, "segmentation": 0, "bounding_boxes": 197, "metadata": 3 },
  "issues": ["metadata declares 200 episode(s) but rgb/ holds 197",
             "segmentation/ holds 0 file(s) but rgb/ holds 197"]
}
exit code: 1

A consistent dataset prints Dataset at <path> is consistent. and exits 0; any issue exits 1 — in both human and JSON mode.

caasi teleop

Drive a robot, record demonstrations, replay recorded bags. Every teleop command delegates: each backend is an upstream ROS package (or an upstream script) that Caasi only resolves and launches — ros2 launch for the keyboard/joystick stacks, the resolved Isaac Sim XR script, and ros2 bag record/play for demonstrations. Caasi implements no teleoperation itself.

The group exposes exactly four verbs — start, record, stop, replay; there is no teleop status. What is installed lives in the teleop catalog domain:

shellcaasi config catalog teleop
Teleoperation teleop · ros
capability  upstream targets       launch file                        caasi group
keyboard    teleop_twist_keyboard  teleop_twist_keyboard              —
                                   teleop-launch.py
joy         teleop_twist_joy, joy  teleop_twist_joy                   —
                                   teleop-launch.py
xr          isaac-sim.xr.vr.sh     —                                  —
record      rosbag2                —                                  —

Every field is overridable: `caasi config set catalog.<domain>.<capability>.<field> <value>`

caasi teleop start

caasi teleop start [CONFIG.yaml] [ARGS…] [--backend|-b CAPABILITY] [--device PATH]
            [--name NAME] [--dry-run]

Two modes, decided by the first positional argument:

• Stack mode (no YAML): the backend is picked --backend → the catalog.teleop.default setting → the first installed of keyboard, joy, xr (in that order). Launch-file stacks run through ros2 launch <package> <launch-file> and --device becomes the launch argument device:=<path>; a resolved script such as Isaac Sim's isaac-sim.xr.vr.sh runs directly, with --device as its first positional argument.
• Experiment mode: when the first argument ends in .yaml/.yml, that experiment YAML launches the robot's sim through the normal experiment runner.

Everything unrecognized is passed through to the stack or the experiment. The run is tracked with kind: teleop (backend ros for launch-file stacks, otherwise the resolved tool); --name overrides the run name, which defaults to the capability key or the experiment name.

shellcaasi teleop start -b joy --device /dev/input/js0 --dry-run
Dry run — nothing was started:
  command: /opt/ros/jazzy/bin/ros2 launch teleop_twist_joy teleop-launch.py device:=/dev/input/js0
caasi teleop start
Teleop started: 20260907-210145-keyboard
  Follow it with: caasi logs 20260907-210145-keyboard -f

On a machine where nothing is installed, the real failures are:

shellcaasi teleop start -b keyboard --dry-run
Error: 'keyboard' is not available (teleop_twist_keyboard). Install it, or point Caasi at the renamed upstream with `caasi config set catalog.teleop.keyboard.packages [...]`.
exit code: 1
caasi teleop start --dry-run
Error: No teleop capability is installed. Install it, or point Caasi at the renamed upstream with `caasi config set catalog.teleop.<capability>.<field> [...]`.
exit code: 1

caasi teleop record

caasi teleop record [--topics|-t TOPIC]… [--name NAME] [--dry-run] [-- EXTRA…]

ros2 bag record into a dataset directory. -t is repeatable; with no topics everything is recorded (-a). Unrecognized arguments are appended to the ros2 bag record command. The ros2 CLI is required — otherwise Error: ros2 CLI not found; source a ROS 2 distro first. (exit 1); when the current shell has not sourced a distro, the run's environment sources the detected one.

The destination is <name-slug>-<YYYYmmdd-HHMMSS> under the usual datasets base (--name defaults to teleop), and metadata.json is written at launch — the bridge that makes a recording a first-class dataset for dataset inspect:

{
  "name": "pick-place",
  "created": "2026-09-07T21:05:12+08:00",
  "kind": "teleop",
  "topics": ["/cmd_vel", "/joint_states"],
  "status": "recording",
  "run_id": "20260907-210512-pick-place"
}

The run is tracked with kind: bag, backend: ros2 and extra: {topics, dataset}.

shellcaasi teleop record --dry-run
Dry run — nothing was started:
  command: /opt/ros/jazzy/bin/ros2 bag record -a -o /home/you/.caasi/datasets/teleop-20260907-204831
  dest:    /home/you/.caasi/datasets/teleop-20260907-204831
caasi teleop record -t /cmd_vel -t /joint_states --name pick-place
Recording demonstrations: /home/you/.caasi/datasets/pick-place-20260907-210512
  Follow it with: caasi logs 20260907-210512-pick-place -f

caasi teleop stop

caasi teleop stop

Stops every running/paused run of kind teleop or bag, newest first — one line per run; a no-op when nothing is active. This is how a recording is closed out.

shellcaasi teleop stop
Stopped 20260907-210145-keyboard (teleop).
Stopped 20260907-210512-pick-place (bag).
caasi teleop stop
No active teleop or recording runs.

caasi teleop replay

caasi teleop replay BAG [--dry-run] [-- EXTRA…]

Publishes a recorded bag again with ros2 bag play, as a tracked run (kind: replay, backend: ros2). BAG is the path to a recorded bag directory — the dataset directory teleop record wrote works as-is, since ros2 bag record -o pointed at it. Unrecognized arguments are appended to ros2 bag play. Distinct from caasi replay, which opens recorded run artifacts in a viewer.

shellcaasi teleop replay datasets/pick-place-20260907-210512 --dry-run
Dry run — nothing was started:
  command: /opt/ros/jazzy/bin/ros2 bag play datasets/pick-place-20260907-210512
caasi teleop replay /tmp/nonexistent-bag
Error: Bag '/tmp/nonexistent-bag' not found.
exit code: 1

Record → inspect → train → evaluate

The point of datasets being directories with a metadata.json: every producer writes to the same convention, so the consumers need no glue. The chain, in real commands:

1. Produce. Demonstrations: caasi teleop record (above). Synthetic frames and simulation-collected data: caasi dataset generate — the synthetic half is detailed above. Existing data: caasi dataset download.
2. Inspect. caasi teleop stop closes the recording, then caasi dataset inspect latest shows metadata, on-disk contents and the linked run — for SDG sets it also counts produced episodes (above). Gate with caasi dataset validate, whose SDG rules are above.
3. Train. caasi train for any experiment — Isaac Lab ones included (see the lab view) — starts a tracked background run.
4. Evaluate. caasi lab evaluate runs the experiment's evaluate_script against a checkpoint; caasi benchmark start measures sim throughput. Review the recording itself with caasi teleop replay, or run artifacts with caasi replay.

shellcaasi teleop record -t /cmd_vel -t /joint_states --name pick-place
Recording demonstrations: /home/you/demo/datasets/pick-place-20260907-210512
  Follow it with: caasi logs 20260907-210512-pick-place -f
# …drive the robot, then close the recording…
caasi teleop stop
Stopped 20260907-210512-pick-place (bag).
caasi dataset inspect latest
Dataset /home/you/demo/datasets/pick-place-20260907-210512
  name: pick-place
  created: 2026-09-07T21:05:12+08:00
  kind: teleop
  topics: ['/cmd_vel', '/joint_states']
  status: recording
  run_id: 20260907-210512-pick-place
  Run: 20260907-210512-pick-place (stopped)
caasi train experiments/pick.yaml --steps 100000 --device cuda:0
Run 20260907-212000-pick started in the background.
  Follow it with: caasi logs 20260907-212000-pick -f
caasi lab evaluate experiments/pick.yaml --checkpoint runs/20260907-212000-pick/model_000100000.pt
Run 20260907-224500-pick started in the background.
  Follow it with: caasi logs 20260907-224500-pick -f

caasi dataset download

caasi dataset download REF [--backend hf|ngc|url] [--name NAME] [--dry-run] [-- EXTRA…]

Downloads an existing dataset as a tracked run. The backend is auto-detected from the reference's scheme; --backend forces one. Each backend resolves to an upstream download tool and builds its command:

Failures are honest and early (both exit 1):

shellcaasi dataset download ftp://example.com/x
Error: Cannot detect a download backend for 'ftp://example.com/x' (use hf://, ngc://, http(s):// or --backend hf|ngc|url).
caasi dataset download hf://lerobot/aloha_sim
Error: No download tool found for backend 'hf'.

The dataset name is --name, or the last path segment of the source — hf://lerobot/aloha_sim becomes aloha-sim-<timestamp>. The run is tracked with kind: dataset and backend set to the detected download backend, and metadata.json is written at launch:

{
  "name": "aloha-sim",
  "created": "2026-09-07T21:00:00+08:00",
  "source": "hf://lerobot/aloha_sim",
  "backend": "hf",
  "status": "downloading",
  "run_id": "20260907-210000-aloha-sim"
}

shellcaasi dataset download hf://lerobot/aloha_sim --dry-run
Dry run — nothing was started:
  command: hf download lerobot/aloha_sim --local-dir /home/you/.caasi/datasets/aloha-sim-20260907-205053
  dest:    /home/you/.caasi/datasets/aloha-sim-20260907-205053
caasi dataset download https://example.com/data.tar.gz --dry-run
Dry run — nothing was started:
  command: /usr/bin/curl -L -o /home/you/.caasi/datasets/data-tar-gz-20260907-205053/data.tar.gz https://example.com/data.tar.gz
  dest:    /home/you/.caasi/datasets/data-tar-gz-20260907-205053
caasi dataset download hf://lerobot/aloha_sim
Dataset download started: /home/you/.caasi/datasets/aloha-sim-20260907-210000
  Run: 20260907-210000-aloha-sim — follow it with 'caasi logs 20260907-210000-aloha-sim -f'

Unrecognized flags pass straight through to the tool, between the source and the destination flags — caasi dataset download hf://lerobot/aloha_sim --revision main runs hf download lerobot/aloha_sim --revision main --local-dir <dataset-dir>. Dry runs need no tool installed and leave no directories behind.

Once the files land, the directory is an ordinary dataset: dataset inspect, dataset convert and dataset validate all work on it, and dataset list shows its source and status.