poolside

/

Laguna-S-2.1-NVFP4

like 171

Follow

Poolside 1.37k

Hi @karypid , thanks for sharing! I haven't managed to replicate the issue on sm_90 with 8fdfcdc4e7b0c474a0102430a8cae0a3a358669/b3b5921a900b9e0a1e27e50bdaeb480692a6d19b. I will be testing it on sm_121a shortly but may I ask you in the meantime to double-check if the checkpoints used correspond to these commits and some e.g. cached checkpoint hasn't been used? Thanks for your patience 🙏

I haven't manage to reproduce the issue on DGX Spark neither with CUDA 13.0, vLLM 0.25.1, Torch 2.11.0. If the issue persist on your side, I'd appreciate sharing vLLM logs from both nodes, so I can take a closer look.

I hope the comment in the recipe didn't confuse you. The comment is outdated and mentions the size of the older (smaller 71GB) model but it is just the comment. I am definitely running:

$ echo "=== Main model cache ref ===" && cat ~/.cache/huggingface/hub/models--poolside--Laguna-S-2.1-NVFP4/refs/main && echo && echo "=== DFlash model cache ref ===" && cat ~/.cache/huggingface/hub/models--poolside--Laguna-S-2.1-DFlash-NVFP4/refs/main && echo
=== Main model cache ref ===
f8fdfcdc4e7b0c474a0102430a8cae0a3a358669
=== DFlash model cache ref ===
b3b5921a900b9e0a1e27e50bdaeb480692a6d19b

The cache contains two snapshots for the main model:

07614121b31898586430f189d27a25a0be310843 — the old main (15 shards, ~71 GB), from a previous download on Jul 28
f8fdfcdc4e7b0c474a0102430a8cae0a3a358669 — the new main (49 shards, ~93 GB), downloaded Aug 2

The refs/main file points to f8fdfcdc... (the new main), not the old 07614121.... So vLLM loaded the correct checkpoint.

IMPORTANT: For DFlash, there is only ONE snapshot (b3b5921a...) — no stale cache exists. Given the changes in the model (seems like more/different weights and also !M context from 256K) I would expect this needs updating?

I haven't manage to reproduce the issue on DGX Spark neither with CUDA 13.0, vLLM 0.25.1, Torch 2.11.0. If the issue persist on your side, I'd appreciate sharing vLLM logs from both nodes, so I can take a closer look.

I noticed this in my logs:

WARNING [laguna.py:381] Laguna config has sliding_attention layers but neither
`swa_rope_parameters` nor a nested per-layer-type `rope_parameters` — SWA layers
will reuse the global rope. If the checkpoint was trained with distinct SWA rope
(theta / partial_rotary_factor), regenerate its HF config to include either form.

I haven't manage to reproduce the issue on DGX Spark neither with CUDA 13.0, vLLM 0.25.1, Torch 2.11.0. If the issue persist on your side, I'd appreciate sharing vLLM logs from both nodes, so I can take a closer look.

Ok I've compiled tag 0.25.1 of VLLM (in the logs it shows as 0.25.2dev but its that tag + the sparkrun shim for distributing the model).

Here is the log with speculative decoding enabled which as you will see is extremely slow at about 8 tok/s and basically wastes the ~15tok/s the draft model produces. I didn't leave this one running for long as it would take forever.

Here is the log WITHOUT speculative which works at >20tok/sec and I left running for long.

• What I noticed is that the model now thinks a lot, but acts very little.
• My test is to start from as specific tag of my code repository, and ask it to execute a ralph wiggum loop to implement a task list of 16 items.
• Laguna iterated 30 times in 6 hours (overnight) and was stuck in task 1 of the 16-task list.
• Every time the ralph loop iterates and starts a fresh session the loop it thinks again for too long and never gets to writing any code.

I have restarted the loop and saving the output to a file. I will review that later today to see what the thinking is about and whether it is looping or if it is just aloof and unfocused.

EDIT: adding revised recipe below.

# Recipe: Laguna-S-2.1-NVFP4 (dual-node, 2x DGX Spark)
# NVFP4 variant of Laguna S 2.1 — 117.6B total / 8.5B activated, 48 layers
# (12 global + 36 SWA), 256 experts + 1 shared, top-k=16, 1M native context.
#
# Model:  poolside/Laguna-S-2.1-NVFP4  (~93 GB weights, NVFP4 4-bit, 49 shards)
# Draft:  poolside/Laguna-S-2.1-DFlash-NVFP4  (speculative decoding, 15 tokens)
#
# NOTE: The model card states ~71 GB but the actual new main checkpoint
# (f8fdfcdc, August 2026) is 92.85 GiB on disk — 31% larger than the old
# main (07614121, 15 shards, ~71 GB). This is the checkpoint that causes
# OOM on single-node 128 GB setups (see discussion #20).
#
# Weight budget at TP=2: ~48.3 GiB/node of ~111 GiB free -> ~63 GiB/node
# headroom for KV cache (256K ctx), DFlash draft, activations + CUDA graphs.
# Observed: 48.34 GiB model loading, 51.79 GiB KV cache (1.4M tokens),
# 0.9 GiB CUDA graphs. KV cache usage during inference: 0.5-5%.
#
# DFlash acceptance: near-zero on sm_121a with new main (0.0-0.6% acceptance,
# mean acceptance length ~1.0) when running vLLM 0.26.1rc1.dev30. DFlash weights
# (model.safetensors) are identical across all revisions — the draft model was
# not retrained for the new main checkpoint. Switched to vLLM 0.25.1 (stable
# release, matches Poolside's testing) to isolate whether the issue is a
# 0.26.1rc1 regression. See discussion #21 for details.
#
# SWA rope warning: vLLM warns that Laguna config has sliding_attention layers
# but neither swa_rope_parameters nor nested per-layer-type rope_parameters —
# SWA layers reuse the global rope. May affect quality if the checkpoint was
# trained with distinct SWA rope.
#
# This recipe tests the latest main checkpoint (August 2026) with pristine
# sampling config — no --override-generation-config. The checkpoint's
# generation_config.json defaults are authoritative (top_k 20, eval-certified).
#
# Source: https://huggingface.co/poolside/Laguna-S-2.1-NVFP4
#   - Quantization detected automatically from quantization_config.
#   - Do NOT add min_p: vLLM rejects min_p under speculative decoding.
#   - First start ~25 min (13 min cache->container, 7 min GPU load,
#     3 min engine init + CUDA graph capture).

# For speculative decoding add:
#     --speculative-config '{"model":"poolside/Laguna-S-2.1-DFlash-NVFP4","num_speculative_tokens":15}' \

recipe_version: "1"
name: Laguna-S-2.1-NVFP4
model: poolside/Laguna-S-2.1-NVFP4
draft_model: poolside/Laguna-S-2.1-DFlash-NVFP4
container: vllm-node-0.25.1
cluster_only: true

defaults:
  host: 0.0.0.0
  tensor_parallel: 2
  gpu_memory_utilization: 0.85
  max_model_len: 262144
  max_num_seqs: 8

env:
  CUTE_DSL_ARCH: sm_121a
  MAX_JOBS: "4"
  HF_HOME: /cache/huggingface
  HF_HUB_CACHE: /cache/huggingface
  HF_HUB_OFFLINE: "1"
  HF_TOKEN: ${HF_TOKEN}

command: |
  vllm serve poolside/Laguna-S-2.1-NVFP4 \
    --enable-auto-tool-choice \
    --tool-call-parser poolside_v1 \
    --reasoning-parser poolside_v1 \
    --default-chat-template-kwargs '{"enable_thinking":true}' \
    --max-num-seqs {max_num_seqs} \
    --max-num-batched-tokens 4096 \
    --max-model-len {max_model_len} \
    --gpu-memory-utilization {gpu_memory_utilization} \
    -tp {tensor_parallel} \
    --host {host} \
    --port 8888

I haven't manage to reproduce the issue on DGX Spark neither with CUDA 13.0, vLLM 0.25.1, Torch 2.11.0. If the issue persist on your side, I'd appreciate sharing vLLM logs from both nodes, so I can take a closer look.

Ok I have made some progress:

• The model is very chatty but is NOT looping. It is working through the problem and is simply extremely verbose.
• The thinking output is huge. I have set my max output tokens to 8192 and the responses get cut off from vllm mid-reasoning due to reaching that limit.

I think you need to add some logic to your reasoning parser:

def is_reasoning_end(self, input_ids):
    # walks token ids BACKWARD from the end of the output
    for tok_id in reversed(input_ids):
        if tok_id == start_token_id:      # ` thinking`
            return False                  # reasoning NOT ended
        if tok_id == end_token_id:        # ` response`
            return True                   # reasoning ended
        if tok_id == assistant_token_id:  # `<assistant>`
            return False                  # stop scanning, still reasoning
    return False

This does not seem to take into account the max output tokens at all.
I will do another run and report with a 16384 output limit

Can you post if there is a maximum output token budget so I can use that value?

UPDATE: running with 65536 max output tokens in opencode has allowed me to successfully run the latest model!

(truncated)
          "limit": {
            "context": 262143,
            "output": 65536
          }
(truncated)

The ralph wiggum loop is finally making progress:

vscode ➜ /analytics-laguna-21-s (laguna-m1-nvfp4) $ ralph --status

╔══════════════════════════════════════════════════════════════════╗
║                    Ralph Wiggum Status                           ║
╚══════════════════════════════════════════════════════════════════╝

🔄 ACTIVE LOOP
   Iteration:    3 / 30
   Started:      2026-08-04T16:54:05.402Z
   Elapsed:      1h 58m 19s
   Promise:      COMPLETE
   Agent:        OpenCode
   Model:        local_llm/selfhosted/thinking-coding
   Tasks Mode:   ENABLED
   Task Promise: READY_FOR_NEXT_TASK
   Prompt:       Follow the plan in `.ralph/plan.md`.

- Select the highest p...

📋 CURRENT TASKS:
   1. ✅ Task 1: Require and validate symbolBoundaries in the ingest contract
   2. ✅ Task 2: Strengthen tablesToIngest validation (non-empty, unique, known types)
   3. 🔄 Task 3: Validate source and target path existence and permissions
   4. ⏸️  Task 4: Add letter-range table naming utilities
   5. ⏸️  Task 5: Update dateHasData for letter-range tables
   6. ⏸️  Task 6: Rewrite processStream letter-range assignment and naming
   7. ⏸️  Task 7: Materialize empty letter-range tables on successful type completion
   8. ⏸️  Task 8: Uppercase sort-order validation and empty-ticker malformed handling
   9. ⏸️  Task 9: Wire progressInterval and fix start-of-file progress sentinels
   10. ⏸️  Task 10: Enforce date-level source availability (all-missing skip vs partial fatal)
   11. ⏸️  Task 11: Stop the run on permission denied / read errors
   12. ⏸️  Task 12: Process requested table types in operator-supplied order
   13. ⏸️  Task 13: Complete operator feedback (boundaries, idempotent stop, run-complete)
   14. ⏸️  Task 14: Align CLI ingest_tool with required boundaries and date bounds
   15. ⏸️  Task 15: Update unit/integration tests and fixtures for letter-range world
   16. ⏸️  Task 16: End-to-end acceptance pass against PRD v2.0 critical paths

   Progress: 2/16 complete, 1 in progress

📊 HISTORY (2 iterations)
   Total time:   1h 33m 46s

   Recent iterations:
   #1  53m 6s  opencode / local_llm/selfhosted/thinking-coding  no tools
   #2  40m 40s  opencode / local_llm/selfhosted/thinking-coding  no tools

I will let it run to see if it can finish the task list and report back.

Thanks @karypid for the detailed follow-up, I’m glad the run is making progress. We’ve been investigating the long-reasoning and looping behaviour and working on improvements. The exact checkpoint pair works with TP=1 on our DGX Spark - have you observed the same DFlash acceptance issue in that setting on your end? In parallel let me investigate the TP=2 path :)