ccwatson/gr00t-nth-lr-vpconcat-59000k

GR00T N1.6 PolaRiS NTH-LR VPConcat — checkpoint 59K

This is the complete step-59,000 checkpoint from a selective fine-tune of nvidia/GR00T-N1.6-3B on ccwatson/PnPNext-To-lr-pairs-procedural-concat-benchmark-left-right-lerobot. It consumes two RGB views, Franka joint/gripper state, and an English instruction and predicts a 16-step joint/gripper action chunk.

This model is not a LoRA adapter. It uses NVIDIA Isaac-GR00T N1.6's standard NEW_EMBODIMENT selective fine-tuning path and contains full model weight shards.

Repository-name note: the requested suffix 59000k denotes the saved step-59,000 checkpoint. It does not mean 59,000,000 updates.

This is an intermediate snapshot at 59,000 of 88,000 planned optimizer updates. It is not the final selected policy. Closed-loop MolmoSpaces results have not yet been recorded for this checkpoint.

At a glance

Field	Value
Base model	nvidia/GR00T-N1.6-3B
Checkpoint	update 59,000 of 88,000 planned
Method	N1.6 selective full-checkpoint fine-tuning; no LoRA
Embodiment	NEW_EMBODIMENT (new_embodiment in saved JSON)
Dataset revision	c74ec0af9b73a8ceca5af70e028d247ace6b8d0f
Dataset size	3,940 episodes; 1,082,531 frames; 15 FPS
Cameras	exterior_1_left, wrist_left
State	joint_position (7), gripper_position (1)
Language	annotation.language.language_instruction
Output	16 × (joint_position (7), gripper_position (1))
Global/physical batch	32 / 32; no gradient accumulation
Hardware	one NVIDIA L40 (46,068 MiB), 16 CPUs, 128 GiB RAM
Precision	BF16; TF32 enabled
W&B	ccwatson/finetune-gr00t-n1d6
Step-59,000 training log	loss 0.0139; gradient norm 0.1410307; learning rate 2.6867952e-5

The step-59,000 values are training telemetry from the final log entry saved in trainer_state.json, not validation or benchmark metrics. Online validation was disabled, and closed-loop evaluation remains pending.

Intended use and limitations

This checkpoint is intended for research/evaluation of language-conditioned Franka manipulation with exactly the schema below. It supports direct Gr00tPolicy inference, the GR00T policy server, open-loop dataset inspection, and exact Trainer resume when training state is retained.

It is not a generic Hugging Face text/image pipeline, so hosted inference is disabled. It is not an end-effector-delta policy and must not be connected to a robot or simulator with a different joint order, gripper convention, camera convention, frequency, normalization, or execution semantics. Use strict input validation. No closed-loop success rate is reported; training loss is not a benchmark result. The inherited NVIDIA license restricts use to non-commercial research or evaluation. Read LICENSE.

Download

The full repo is about 22.8 GB: roughly 9.8 GB of model weights plus a roughly 12.9 GB AdamW optimizer and small resume/config files. For inference only:

hf download ccwatson/gr00t-nth-lr-vpconcat-59000k \
  --local-dir ./gr00t-nth-lr-vpconcat-59000k \
  --exclude optimizer.pt \
  --exclude scheduler.pt \
  --exclude rng_state.pth \
  --exclude trainer_state.json \
  --exclude training_args.bin

For exact resume or archival inspection, download everything:

hf download ccwatson/gr00t-nth-lr-vpconcat-59000k \
  --local-dir ./gr00t-nth-lr-vpconcat-59000k

Validated software version

git clone --branch n1.6.1-release https://github.com/NVIDIA/Isaac-GR00T.git
cd Isaac-GR00T
git checkout 5dc80c4afd726b34faad1d8f7e007a13b34e4c88
uv sync --frozen

config.json uses Gr00tN1d6; the Isaac-GR00T package registers that Transformers class. Newer revisions may work but were not used for training or qualification.

Direct inference

The saved processor_config.json contains the complete new_embodiment contract and normalization. Re-registering the training-time Python config is not required merely to load this checkpoint.

import numpy as np
from gr00t.data.embodiment_tags import EmbodimentTag
from gr00t.policy.gr00t_policy import Gr00tPolicy

policy = Gr00tPolicy(
    model_path="./gr00t-nth-lr-vpconcat-59000k",
    embodiment_tag=EmbodimentTag.NEW_EMBODIMENT,
    device="cuda:0",
    strict=True,
)

# exterior_rgb/wrist_rgb: RGB uint8 (H,W,3); q: 7 joints; gripper: scalar.
observation = {
    "video": {
        "exterior_1_left": exterior_rgb[None, None, ...],  # (1,1,H,W,3)
        "wrist_left": wrist_rgb[None, None, ...],
    },
    "state": {
        "joint_position": np.asarray(q, np.float32)[None, None, :],  # (1,1,7)
        "gripper_position": np.asarray([gripper], np.float32)[None, None, :],
    },
    "language": {
        "annotation.language.language_instruction": [[instruction]],
    },
}
action, info = policy.get_action(observation)
assert action["joint_position"].shape == (1, 16, 7)
assert action["gripper_position"].shape == (1, 16, 1)
next_joint_target = action["joint_position"][0, 0]
next_gripper_target = action["gripper_position"][0, 0]

training_recipe/inference_example.py is a runnable version accepting two image files, seven joint values, one gripper value, and an instruction.

Observation contract

Modality	Key	Shape	dtype / meaning
video	exterior_1_left	(B,1,H,W,3)	uint8, RGB exterior view
video	wrist_left	(B,1,H,W,3)	uint8, RGB wrist view
state	joint_position	(B,1,7)	float32, current Franka joints
state	gripper_position	(B,1,1)	float32, current gripper state
language	annotation.language.language_instruction	list (B,1)	English strings

Every input has delta_indices=[0], hence temporal horizon 1. Images must be RGB, not BGR. Joint order, units, and gripper convention must match the dataset.

Action contract

Key	Shape	Training representation	get_action result
joint_position	(B,16,7)	relative to current joint state	decoded absolute joint targets
gripper_position	(B,16,1)	absolute	absolute gripper targets

Training data is 15 FPS and the prediction horizon is 16. Prediction horizon and executed horizon are different: validate how many steps to execute before replanning in the target environment. Do not reinterpret arm output as Cartesian deltas.

The root model config retains padded action_horizon=50 and 128-wide tensors. Those are cross-embodiment padding maxima. The saved new_embodiment modality config is authoritative here: 16 steps, 7 arm values, and 1 gripper value.

Policy server

uv run python gr00t/eval/run_gr00t_server.py \
  --model-path /absolute/path/to/gr00t-nth-lr-vpconcat-59000k \
  --embodiment-tag NEW_EMBODIMENT \
  --device cuda:0 \
  --strict \
  --host 0.0.0.0 \
  --port 5555

Clients send the nested observation above. The server owns the checkpoint and returns a 16-step chunk; the client owns the execution horizon and replanning. There is no published MolmoSpaces/vla-evaluation-harness config yet, so this is the appropriate validated GR00T-native configuration, not a claim that closed-loop integration is complete.

Open-loop dataset check

Register the custom config before open_loop_eval.py parses its CLI:

PYTHONPATH="/path/to/model/training_recipe:${PYTHONPATH:-}" \
uv run python -c '
import runpy
import pnpnext_lr_jointpos_config
runpy.run_module("gr00t.eval.open_loop_eval", run_name="__main__")
' \
  --dataset-path /path/to/PnPNext-To-lr-pairs-procedural-concat-benchmark-left-right-lerobot \
  --embodiment-tag NEW_EMBODIMENT \
  --model-path /path/to/gr00t-nth-lr-vpconcat-59000k \
  --traj-ids 0 \
  --action-horizon 16 \
  --save-plot-path /tmp/gr00t-26k-open-loop.jpeg

This is prediction-vs-ground-truth diagnostics, not closed-loop success.

Dataset and modality configuration

All 3,940 episodes from this exact dataset revision were eligible, matching the corresponding OpenPI pi0.5 data selection:

hf download \
  ccwatson/PnPNext-To-lr-pairs-procedural-concat-benchmark-left-right-lerobot \
  --repo-type dataset \
  --revision c74ec0af9b73a8ceca5af70e028d247ace6b8d0f \
  --local-dir /path/to/PnPNext-To-lr-pairs-procedural-concat-benchmark-left-right-lerobot

For GR00T we explicitly selected seven arm joints and one gripper dimension; we did not assume OpenPI and GR00T share a schema. Normalization is included in statistics.json and experiment_cfg/dataset_statistics.json.

The exact registration file is training_recipe/pnpnext_lr_jointpos_config.py. Its contract is:

PNPNEXT_LR_JOINTPOS_CONFIG = {
    "video": ModalityConfig([0], ["exterior_1_left", "wrist_left"]),
    "state": ModalityConfig([0], ["joint_position", "gripper_position"]),
    "action": ModalityConfig(
        list(range(16)), ["joint_position", "gripper_position"],
        action_configs=[
            ActionConfig(rep=RELATIVE, type=NON_EEF, format=DEFAULT,
                         state_key="joint_position"),
            ActionConfig(rep=ABSOLUTE, type=NON_EEF, format=DEFAULT),
        ],
    ),
    "language": ModalityConfig(
        [0], ["annotation.language.language_instruction"]
    ),
}

Exact training command

At stack revision a5d3a1d, the worker ran the following. /tmp inputs were checksummed copies staged from persistent Kostas GRAID onto node-local NVMe; checkpoint output remained on persistent GRAID.

uv run --frozen --no-sync python gr00t/experiment/launch_finetune.py \
  --base-model-path /tmp/isaac-gr00t-520010/base-model \
  --dataset-path /tmp/isaac-gr00t-520010/lerobot/ccwatson/PnPNext-To-lr-pairs-procedural-concat-benchmark-left-right-lerobot \
  --embodiment-tag NEW_EMBODIMENT \
  --modality-config-path /mnt/kostas-graid/datasets/ccwatson/repos/molmo-robotics-stack/scripts/isaac_gr00t/pnpnext_lr_jointpos_config.py \
  --num-gpus 1 \
  --output-dir /mnt/kostas-graid/datasets/ccwatson/molmo-robotics-stack-state/isaac-gr00t/checkpoints/pnpnext-lr-jointpos \
  --save-total-limit 5 \
  --save-steps 500 \
  --max-steps 88000 \
  --global-batch-size 32 \
  --color-jitter-params brightness 0.3 contrast 0.4 saturation 0.5 hue 0.08 \
  --dataloader-num-workers 4 \
  --experiment-name gr00t-n1d6-pnpnext-lr-jointpos-grasp-l40-20260824 \
  --use-wandb

training_recipe/train_command.sh is the same command with portable path environment variables.

Resolved training configuration

NVIDIA defaults not explicit above are captured in the checkpoint's fully resolved experiment_cfg/conf.yaml.

Area	Resolved value
Optimizer	adamw_torch
LR / schedule	1e-4 peak / cosine
Warmup / weight decay	0.05 ratio / 1e-5
Max grad norm	1.0
Global / physical batch	32 / 32
Gradient accumulation	1
Precision	BF16 train/eval; TF32 on; FP16 off
Updates	88,000 planned; this snapshot at 59,000
Logging / saving	every 10 / 500 updates; retain latest 5
Validation	disabled (eval_strategy: no)
Workers	4
Shuffle / seed / subsample	true / 42 / 1.0
Shard size / episode sampling	1,024 / 0.1
Image path	shortest edge 256, crop 0.95, data crop 244², target 224²
Color jitter	brightness .3, contrast .4, saturation .5, hue .08
Video backend / denoising steps	torchcodec / 4

Trainable modules: top four language-model layers, projector, diffusion model, and VLLN. Frozen: visual backbone and remaining language-model layers. Flags: tune_top_llm_layers=4, tune_llm=false, tune_visual=false, tune_projector=true, tune_diffusion_model=true, tune_vlln=true.

Batch-size and duration qualification

Global = physical batch	Peak L40 memory (MiB)
1	36,018
2	36,148
4	36,118
8	36,274
16	38,214
32	40,666

Batch 32 left 5,402 MiB and met a 4 GiB reserve. A 100-update smoke measured 0.929293 seconds/steady update and 64.7214 seconds/full checkpoint write. Including saves and 12-hour requeues, 88,000 updates projected to 29.97 hours. See training_recipe/batch_probe.csv.

Checkpointing, resume, and validation

GRASP Slurm job 520010 used 12-hour allocations, --requeue, and --signal=B:USR1@300. On signal the worker stops Trainer, requeues the same job, restages inputs locally, and relaunches. Isaac-GR00T uses resume_from_checkpoint=True, selecting the newest checkpoint in the same experiment directory. Job 519994 qualified the path by writing step 50, requeueing under the same ID, and continuing through step 100.

For exact step-59K resume, place this directory at <output-dir>/<experiment-name>/checkpoint-59000, keep optimizer/scheduler/RNG and Trainer files, and relaunch the same command with the same source, dataset, config, arguments, and experiment name. An inference-only download is not resumable.

Qualification also covered dataset/statistics and relative-action round trips, batch-32/100-step smokes, exact loading with no key mismatch, real two-camera inference with finite (1,16,7) and (1,16,1) outputs, and finite nonzero CUDA gradients through action decoding and differentiable Franka FK.

FK calibration used 4,096 recorded joint/TCP pairs. Asset constants are 580 mm pedestal-to-base and 262 mm link-7-to-grasp-site. Fitted offsets were 579.978 mm and 261.977 mm; median/p99 position error was 0.133/0.383 mm and p99 orientation error 0.052°. See training_recipe/fk_calibration.json. These checks do not establish benchmark performance or robot safety.

Provenance

Component	Revision
Dataset	c74ec0af9b73a8ceca5af70e028d247ace6b8d0f
NVIDIA release / source	n1.6.1-release / 5dc80c4afd726b34faad1d8f7e007a13b34e4c88
Stack training workflow	a5d3a1d
OmniGuide-GR00T compatibility	21d9145
Slurm production job	520010
W&B run ID	gr00t-n1d6-pnpnext-lr-grasp-l40-20260824

Structured metadata: training_recipe/provenance.json.

File guide

Path	Purpose
model-*.safetensors, index	complete model weights
config.json	GR00T N1.6 model config
processor_config.json, statistics.json	modality, preprocessing, normalization
experiment_cfg/conf.yaml	human-readable resolved model/data/train config
experiment_cfg/config.yaml	Python-tagged config serialization
experiment_cfg/dataset_statistics.json	run dataset statistics
optimizer.pt, scheduler.pt, rng_state.pth	optimizer/resume state
trainer_state.json, training_args.bin	Trainer progress/config
training_recipe/	portable commands, examples, qualification metadata

Evaluation status

Closed-loop MolmoSpaces evaluation is pending. Future results should identify the exact Hub/checkpoint revision, benchmark leaf, renderer, execution horizon, episode coverage, W&B eval run, and artifact root. Do not infer success from training loss or the open-loop smoke.