Update DFlash recommendation
#26
by baranowskiadam - opened
README.md
CHANGED
|
@@ -108,7 +108,7 @@ Laguna S 2.1-NVFP4 is supported in vLLM, Transformers, and TRT-LLM thanks to the
|
|
| 108 |
The full vLLM recipe is on the main [Laguna S 2.1 model card](https://huggingface.co/poolside/Laguna-S-2.1) and on the [vLLM recipes page](https://recipes.vllm.ai/poolside/Laguna-S-2.1). Quantization is detected automatically from `quantization_config` in this checkpoint, so the same command works with `poolside/Laguna-S-2.1-NVFP4` substituted for the model ID. No extra flags required.
|
| 109 |
|
| 110 |
> [!NOTE]
|
| 111 |
-
> **Optional: speculative decoding with DFlash.** Pair with the quantization-matched draft model [poolside/Laguna-S-2.1-DFlash-NVFP4](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-NVFP4) by adding `--speculative-config '{"model":"poolside/Laguna-S-2.1-DFlash-NVFP4","num_speculative_tokens":
|
| 112 |
|
| 113 |
#### DGX Spark & Mac Studio
|
| 114 |
|
|
@@ -177,7 +177,7 @@ export MAX_JOBS=4 # cap JIT fan-out; see warning below
|
|
| 177 |
source ~/venvs/vllm025/bin/activate
|
| 178 |
|
| 179 |
vllm serve poolside/Laguna-S-2.1-NVFP4 \
|
| 180 |
-
--speculative-config '{"model":"poolside/Laguna-S-2.1-DFlash-NVFP4","num_speculative_tokens":
|
| 181 |
--enable-auto-tool-choice \
|
| 182 |
--tool-call-parser poolside_v1 \
|
| 183 |
--reasoning-parser poolside_v1 \
|
|
|
|
| 108 |
The full vLLM recipe is on the main [Laguna S 2.1 model card](https://huggingface.co/poolside/Laguna-S-2.1) and on the [vLLM recipes page](https://recipes.vllm.ai/poolside/Laguna-S-2.1). Quantization is detected automatically from `quantization_config` in this checkpoint, so the same command works with `poolside/Laguna-S-2.1-NVFP4` substituted for the model ID. No extra flags required.
|
| 109 |
|
| 110 |
> [!NOTE]
|
| 111 |
+
> **Optional: speculative decoding with DFlash.** Pair with the quantization-matched draft model [poolside/Laguna-S-2.1-DFlash-NVFP4](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-NVFP4) by adding `--speculative-config '{"model":"poolside/Laguna-S-2.1-DFlash-NVFP4","num_speculative_tokens":7,"method":"dflash"}'` to the serve command.
|
| 112 |
|
| 113 |
#### DGX Spark & Mac Studio
|
| 114 |
|
|
|
|
| 177 |
source ~/venvs/vllm025/bin/activate
|
| 178 |
|
| 179 |
vllm serve poolside/Laguna-S-2.1-NVFP4 \
|
| 180 |
+
--speculative-config '{"model":"poolside/Laguna-S-2.1-DFlash-NVFP4","num_speculative_tokens":7}' \
|
| 181 |
--enable-auto-tool-choice \
|
| 182 |
--tool-call-parser poolside_v1 \
|
| 183 |
--reasoning-parser poolside_v1 \
|