Training dataset
Turn the recordings of a session into a training dataset with the specialist repository.
The specialist repository turns the recordings of the cameras into a dataset that a policy can train on. Each session goes through three steps: convert, estimate and state. A fourth step is optional: it registers the dataset on a Rerun catalog.
Before you start
You need:
- A Linux computer with an NVIDIA GPU.
- uv, git, a C++17 compiler and the Boost headers (
libboost-dev). - The recordings of a session, made as Scene setup shows: a hold at the start and taps at the end.
- The IDs of the anchor tags in the scene.
Install
git clone https://github.com/specialist-ai/specialist.git
cd specialist
uv syncuv sync installs the dependencies and builds a native part of the solver. Do it on each computer that processes data.
Copy the recordings
Each camera writes two files for each recording: a file with _00_ in its name and a file with _10_ in its name. Copy both files of each camera from its card. Keep the two files of a camera in the same folder.
1. Convert
uv run python scripts/convert.py \
--input-insvs path/to/left_00.insv path/to/right_00.insv \
--session-id <session-id> --output-dir logs- Give the
_00_file of each camera. The script finds the_10_file. - The first file sets the clock of the session. The script aligns the other recordings to it by their sound.
- The result is
logs/base/<session-id>.rrd: the video of each lens, the IMU data and the lens calibration.
If the recordings share no sound, give the time offset of each recording with --audio-offsets. To label the session, add --task, --date, --operator and --site.
2. Estimate
uv run python scripts/estimate.py --base logs/base/<session-id>.rrd \
--anchor.tag-ids 0 1 --output-dir logs- Give the IDs of the anchor tags. The first ID sets the world frame.
- The script finds the gripper tags of each camera.
- The result is
logs/estimate/<session-id>.rrd: the pose of each gripper, the finger angles, the gripper opening and the poses of the anchor tags. The script also writeslogs/viz/<session-id>.rrd, which only shows the result.
The script does not write over a layer that exists. To run it again, add --overwrite.
3. State
uv run python scripts/state.py --base logs/base/<session-id>.rrd --output-dir logsThe result is the training data: logs/state_s12_r100/<session-id>.rrd. For each hand, it holds:
- The image of the observation lens, 240 × 240 pixels, black outside the fisheye circle.
- The pose and the velocity of the gripper in the world frame.
- The finger angle.
The data has 100 rows each second. A trainer makes the observation of a policy from the images and the past rows, and the action from the future rows.
The script keeps the time between the hold at the start and the taps at the end, and removes 1 second next to each. It finds which gripper was in which hand. To set the hand, add --left <slot>.
| Option | Default | Sets |
|---|---|---|
--scale | 12 | Lens pixels for each stored pixel. 12 gives 240 × 240 images. |
--f-row | 100 | Rows each second: the rate of the control loop of the policy. |
--margin | 1 | Seconds to remove next to the hold and the taps. |
The folder name, state_s12_r100, records the scale and the row rate. Thus datasets with different settings can be in the same logs folder.
4. Register (optional)
uv run python scripts/catalog.py logs/state_s12_r100 \
--catalog-url rerun+http://host:portThis registers all the sessions of one state folder as one dataset on a Rerun catalog, for inspection and training.
Look at a session
uv run rerun logs/blueprints/specialist.rbl logs/base/<session-id>.rrd \
logs/estimate/<session-id>.rrd logs/viz/<session-id>.rrd \
--asset logs/assets/gripper-<sha>-cam00.rrd \
--asset logs/assets/gripper-<sha>-cam01.rrdThe viewer shows the video of each lens, the path of each gripper and the anchor tags. To show the video, ffmpeg must be on PATH.
Problems
| Message | Cause | Remedy |
|---|---|---|
no tracker observes the reference anchor | The solve did not find the first anchor tag. | Make sure that --anchor.tag-ids gives the IDs of the tags in the scene. |
the gripper never slows near an anchor | The recording has no hold at the start. | Record again with the hold. |
the hold ends at … past its first tenth | The hold is too late in the recording. | Hold the grippers immediately after you start the recording. |
no burst of acceleration spikes reaches every camera | The recording has no taps at the end, or one camera did not record them. | Record again. Tap all the grippers on the table. |
the strongest tap burst sits at … before its last tenth | The taps are too early in the recording. | Stop the recording immediately after the taps. |
failed to warm the flow backend | A GPU start-up fault that occurs sometimes. | Run the estimate again. |