Archive NexaMat with final qualified model card and evidence

Marks the project archived and on ice indefinitely. Preserves the compact CIF-valid generator, broad static evidence, partial 2/64 metastability result, limitations, compendium, and technical paper.

README.md

@@ -1,45 +1,120 @@
 ---
 library_name: pytorch
 tags:
-- materials-science
-- crystal-generation
-- diffusion
-- encoder
-- multimodal
-- gradio
-license: mit
+  - materials-science
+  - crystal-generation
+  - diffusion
+  - inorganic-materials
+  - nexamat
+license: other
 ---
 
-# Nexa_Mat2
+# NexaMat2 Model Card
 
-`AethronPhantom/Nexa_Mat2` is the public artifact repository for the Nexa_Mat Gen Stack. It contains the frozen physics encoder, the constrained diffusion decoder, the experimental multimodal/controller pilot, and the stack manifest consumed by the public Space.
+## Model Summary
 
-Space: https://huggingface.co/spaces/AethronPhantom/nexamat-crystal-viewer
+NexaMat2 is a compact materials-generation stack centered on a frozen physics
+encoder and a native diffusion decoder for inorganic crystal proposals. The
+full compact encoder-decoder stack contains 4,061,859 parameters. The decoder
+contains 1,248,145 trainable parameters.
 
-## Artifacts
+The model was recovered from a low-yield geometry generator through explicit
+native accounting, profile-conditioned chemistry contracts, radius-aware
+periodic geometry, and bounded external evaluation. It is shipped as a
+candidate generator, not as a substitute for relaxation or thermodynamic
+stability calculations.
 
-| Component | Path | Status |
-|---|---|---|
-| Encoder V1 | `encoder/v1/nexa_mat_V1_final.safetensors` | Frozen downstream handoff checkpoint. |
-| Encoder manifest | `encoder/v1/manifest.json` | Architecture, source URI, checksum, and training metadata. |
-| Diffusion V1 | `decoder/diffusion_v1/final_checkpoint.safetensors` | Production diffusion/checkpoint handoff for constrained sampling. |
-| Diffusion manifest | `decoder/diffusion_v1/manifest.json` | Architecture, eval, checksum, and operating-mode metadata. |
-| Controller pilot | `multimodal/controller/nexa_mat_controller_fft_pilot_20260518T234148Z/final_model_merged/model.safetensors` | Experimental Qwen-based controller pilot. |
-| Cross-attention contract | `cross_attention_contract.json` | Interface contract joining encoder, decoder, controller, evidence, and task-head lanes. |
-| Stack manifest | `stack_manifest.json` | Canonical manifest for the public Space and downstream tooling. |
+## Components
+
+| Component | Parameters | Function |
+| --- | ---: | --- |
+| Physics encoder | 2,813,714 | Physics features and plausibility grounding |
+| Diffusion decoder | 1,248,145 | Native atom, coordinate, and lattice generation |
+| Full compact stack | 4,061,859 | Candidate proposal and physics grounding |
+| Controller pilot | Separate Qwen3-14B artifact | Optional evidence routing and explanation |
 
 ## Intended Use
 
-The stack is intended for materials candidate triage. Forward mode proposes constrained candidate structures from a design intent. Reverse mode ranks a candidate pool against a target use case. Generated candidates should be treated as hypotheses for DFT and downstream validation, not as confirmed stable materials.
+- Generate native, unrepaired inorganic candidate structures.
+- Produce diverse candidate pools for downstream screening.
+- Combine with the encoder for physics-aware ranking.
+- Support research and product prototyping with explicit external validation.
+
+## Out-of-Scope Use
+
+- Treating generated structures as experimentally stable without relaxation and
+  hull evaluation.
+- Claiming SOTA parity with MatterGen from the internal static profile suite.
+- Organic, molecular-crystal, polymer, or MOF generation.
+- Safety-critical or autonomous synthesis decisions without expert review.
+
+## Verified Evidence
 
-## Component Roles
+The final non-alloy static suite passed profile-specific gates:
 
-The encoder is the physics-grounding layer. It learned a periodic materials manifold from the Nexa_Mat V1 training surface and is frozen for downstream generative experiments.
+| Profile | Raw | Accepted | Charge | LeMat-distance | Strict | Novelty |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| SSB regression | 1,440 | 513 | 100.0% | 35.7% | 60.4% | 100.0% |
+| Battery ionics | 1,248 | 411 | 100.0% | 33.1% | 60.3% | 100.0% |
+| Oxide catalysts | 1,584 | 424 | 100.0% | 26.8% | 59.7% | 100.0% |
+| Halides/chalcogenides | 2,688 | 436 | 100.0% | 16.3% | 55.1% | 100.0% |
+| Semiconductors | 672 | 256 | profile diagnostic | 38.1% | 65.0% | 100.0% |
+| Broad inorganic | 2,688 | 242 | profile diagnostic | 9.0% | 55.1% | 100.0% |
 
-The diffusion decoder is the proposal layer. It repairs and proposes structures under constraints, but should be used as a best-of-N sampler with filtering instead of a one-shot oracle.
+The strongest bounded external stability anchor produced:
 
-The multimodal/controller layer is the semantic/evidence layer. It connects model outputs to use cases, evidence packets, explanations, and reverse-pool ranking. The published controller checkpoint is a pilot, not a final full fine-tune.
+- 2/64 M.S.U.N. candidates;
+- mean evaluated `e_above_hull = 0.8750 eV/atom`;
+- two individual candidates at 0.1786 and 0.1837 eV/atom;
+- 3/64 strict relaxation convergence.
+
+This proves sparse support in the desired low-hull region and places observed
+evaluated candidates in an approximately 0.8-to-0.18 eV/atom band. It does not
+prove repeatable high-yield metastable generation. The 2/64 hit rate was 3.125%,
+well below the intended 25-50% yield. A broader 512-candidate screen converged
+58/512, had weighted mean hull 1.2330 eV/atom, and produced no M.S.U.N.
+candidates.
 
 ## Limitations
 
-These artifacts do not replace DFT, relaxation, experimental validation, or synthesis review. The decoder is strongest in constrained sampling and weaker in unconditional high-yield generation. The controller should not be trusted for unsupported literature claims unless paired with an evidence retrieval layer.
+- Relaxation convergence and low-hull yield are substantially weaker than
+  static validity.
+- The low-hull anchor did not reproduce reliably enough in later cohorts.
+- Alloys/intermetallics remain outside the passed static profile panel.
+- Internal and public model comparisons use different protocols.
+- The optional 14B controller is a partial pilot and is not part of the compact
+  4.061859M-parameter claim.
+
+## Required Validation
+
+Every generated candidate should pass:
+
+1. profile-specific static validation;
+2. external geometry and post-relax validation;
+3. MLIP or DFT relaxation appropriate to the intended use;
+4. formation-energy and hull evaluation against a compatible reference set;
+5. domain-expert review before synthesis or deployment decisions.
+
+## Artifact Lineage
+
+- Encoder: `encoder/v1/`
+- Original production decoder: `decoder/diffusion_v1/`
+- Recovered native decoder: `decoder/native_v78/`
+- Research initialization: `research/v29_directional_step800/`
+- Final documents: `docs/final_closeout/`
+- Evidence summaries: `eval/final_evidence/`
+
+The V78 checkpoint is the shippable recovered native lineage. V90 is an
+evaluation surface without a train checkpoint at the historically configured
+URI. Later conservative-field checkpoints are archived research evidence and
+are not production promotions.
+
+The metastability program is on ice indefinitely because its duration and cost
+outgrew the reliability gains. It is not classified as a complete scientific
+failure: the archived low-hull candidates provide a real restart anchor.
+
+## Citation
+
+Use the technical paper in
+`docs/final_closeout/NEXA_MAT_FINAL_TECHNICAL_PAPER_2026-07-24.md` and cite the
+exact model-hub revision used for an experiment.

decoder/native_v78/manifest.json

@@ -0,0 +1,12 @@
+{
+  "schema_version": "nexamat.model_artifact/v1",
+  "name": "nexa_mat_v78_detached_global_sampler",
+  "role": "shipped_recovered_native_decoder",
+  "architecture": "encoder_detached_native_generator",
+  "native_latent_dim": 128,
+  "canonical_object_uri": "wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/diffusion_alignment/2026-06-20/nexa_mat_v78_detached_global_sampler_20260620T065315Z/train/best_diffusion_checkpoint.pt",
+  "expected_size_bytes": 7253874,
+  "hub_path": "decoder/native_v78/best_diffusion_checkpoint.pt",
+  "claim_boundary": "production native candidate generation; external relaxation and hull evaluation required",
+  "status": "canonical_object_preserved_in_wasabi_hub_mirror_requested"
+}

docs/final_closeout/ARCHIVE_CLOSEOUT_2026-07-24.md

@@ -0,0 +1,154 @@
+# NexaMat Final Project Archive Closeout
+
+Date: 2026-07-24  
+Status: Archived and on ice indefinitely; model shipped with qualified claims  
+Canonical project root: `docs/projects/Nexa_Mat`  
+Archive protocol: NexaMass-style immutable bundle, checksum, manifest, and
+minimal local index
+
+## Executive Verdict
+
+NexaMat is a successful recovery and an incomplete stability program.
+
+The original decoder was brittle: atom identity and novelty were strong, but
+native geometry validity was too sparse to support downstream science. The
+alignment campaign recovered a compact 4.061859M-parameter stack into a useful
+native/no-repair inorganic candidate generator. The final non-alloy panel passed
+its profile-specific static gates, and the strongest SSB lane produced 595
+accepted candidates from 4,096 raw samples.
+
+The metastability campaign reached meaningful but incomplete evidence. Evaluated
+structures entered an approximately 0.8-to-0.18 eV/atom band, including 2/64
+M.S.U.N. candidates at 0.1786 and 0.1837 eV/atom. That was genuine progress,
+not a complete miss. It was also far below the desired reliable yield: the
+observed hit rate was 3.125%, strict relaxation convergence was 3/64, and the
+campaign's practical ambition was at least 25% and ultimately 50%. Later
+architecture work improved representation AUC, force magnitude, and external
+teacher agreement without making the metastable yield repeatable.
+
+The project stopped because the experiment program had run for weeks and the
+remaining reliability gap no longer justified continuous iteration. This is a
+time, cost, and focus decision, not evidence that NexaMat can never improve.
+
+The honest shipping position is therefore:
+
+> NexaMat is a compact, efficient native inorganic proposal model with sparse
+> demonstrated metastable support. It requires external relaxation and
+> stability screening and is not yet a reliable high-yield metastable
+> generator.
+
+## What Was Achieved
+
+- Recovered native no-repair crystal generation from catastrophic geometry
+  starvation.
+- Separated native, projected, scaffolded, ranked, relaxed, and hull-scored
+  accounting.
+- Preserved raw candidate payloads and explicit failure reasons.
+- Generalized the SSB recovery into profile-conditioned non-alloy inorganic
+  generation.
+- Passed the final static suite for SSB, battery, oxide, halide/chalcogenide,
+  semiconductor, and broad-inorganic profiles.
+- Retained 100% novelty in the final six-profile suite.
+- Produced the first bounded low-hull/M.S.U.N. evidence.
+- Built a source-disjoint trajectory, force, stress, relaxation, and hull
+  evidence stack.
+- Established a real stability representation signal: ROC-AUC 0.70845 and
+  minimum balanced accuracy 0.60048.
+- Established that the frozen MACE-OMat teacher was strong on the comparison
+  bridge: force cosine 0.75450 and force MAE 0.05728 eV/A.
+- Built reliable external gating, provenance, W&B, Wasabi, and failure receipts.
+
+## What Remains On Ice
+
+- High-yield relaxation convergence.
+- Repeatable density near the metastable hull threshold.
+- A native controller that turns local physical signals into globally
+  productive coordinate and lattice motion.
+- Alloys/intermetallics under the same general profile contract.
+- An apples-to-apples public MatterGen/SUN benchmark.
+- A matched generative objective trained directly on a broad low-hull endpoint
+  distribution.
+
+## Final Evidence Table
+
+| Surface | Best verified result | Interpretation |
+| --- | ---: | --- |
+| Full compact stack | 4.061859M parameters | Strong efficiency |
+| Trainable decoder | 1.248145M parameters | Small native generator |
+| V90 SSB native accepted | 595/4096, 14.53% | Useful static yield |
+| Final six-profile accepted | 513, 411, 424, 436, 256, 242 | Non-alloy static panel passed |
+| Final six-profile novelty | 100% per profile | Strong novelty |
+| Full 512 stability screen | 58/512 converged; mean hull 1.2330; 0 M.S.U.N. | Failed stability gate |
+| Best bounded stability anchor | 3/64 converged; mean hull 0.8750; 2 M.S.U.N. | Sparse support, not reliable density |
+| Best individual hull rows | 0.1786 and 0.1837 eV/atom | Desired region is reachable |
+| V32 representation | AUC 0.70845; balanced accuracy 0.60048 | Basin signal exists |
+| V32 controller | negative coordinate, lattice, and pair contraction | Controller failed |
+| Frozen foundation | cosine 0.75450; MAE 0.05728 eV/A | Strong local physical teacher |
+| Residual transfer | approximately 0.75 cosine, no gain | Plateau |
+
+## Canonical Artifact Destinations
+
+### Wasabi
+
+Final project archive prefix:
+
+```text
+wasabi:nexa-mat/project_archives/nexa_mat/2026-07-24_final_closeout/
+```
+
+The prefix contains:
+
+- the complete project documentation bundle;
+- a SHA-256 sidecar;
+- a machine-readable file manifest;
+- the final model publication manifest;
+- archive and upload verification receipts.
+
+Historical run trees remain under:
+
+```text
+wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/diffusion_alignment/
+```
+
+### Hugging Face
+
+Canonical model hub:
+
+```text
+https://huggingface.co/AethronPhantom/Nexa_Mat2
+```
+
+The model hub contains the shipped encoder/decoder stack, curated research
+lineage artifacts, final documentation, figures, evidence summaries, and an
+explicit limitations section. It is a model and evidence hub, not a claim that
+every historical checkpoint is production-ready.
+
+## Archival Policy
+
+This closeout does not delete source code or historical evidence. Code remains
+authoritative in git. Heavy experiment artifacts remain authoritative in
+object storage. Hugging Face holds the curated shippable model surface. Local
+project documentation remains a lightweight navigation and recovery index.
+
+No failed mechanism checkpoint is promoted merely because it is newer. The
+V78 recovered native checkpoint is the shippable decoder lineage; V90 is static
+evaluation evidence; V29 and later checkpoints are research lineage.
+
+## Reopen Decision
+
+Reopen only for one of these reasons:
+
+1. reproduce the July 2 low-hull anchor under frozen lineage;
+2. build a low-hull endpoint or composition-conditioned crystal-structure
+   prediction pilot with a matched diffusion/flow objective;
+3. integrate a stronger external stability model as a transparent product
+   routing stage;
+4. address a concrete product requirement that the shipped static generator
+   does not satisfy.
+
+Do not reopen for another generic loss-weight, learning-rate, router, profile
+adapter, or static-validity sweep.
+
+The project is archived indefinitely rather than terminated permanently. A
+future restart should begin from the frozen evidence and reproduction contract,
+not from the assumption that the metastability campaign was a total failure.

docs/final_closeout/NEXA_MAT_FINAL_MODEL_CARD_2026-07-24.md

@@ -0,0 +1,120 @@
+---
+library_name: pytorch
+tags:
+  - materials-science
+  - crystal-generation
+  - diffusion
+  - inorganic-materials
+  - nexamat
+license: other
+---
+
+# NexaMat2 Model Card
+
+## Model Summary
+
+NexaMat2 is a compact materials-generation stack centered on a frozen physics
+encoder and a native diffusion decoder for inorganic crystal proposals. The
+full compact encoder-decoder stack contains 4,061,859 parameters. The decoder
+contains 1,248,145 trainable parameters.
+
+The model was recovered from a low-yield geometry generator through explicit
+native accounting, profile-conditioned chemistry contracts, radius-aware
+periodic geometry, and bounded external evaluation. It is shipped as a
+candidate generator, not as a substitute for relaxation or thermodynamic
+stability calculations.
+
+## Components
+
+| Component | Parameters | Function |
+| --- | ---: | --- |
+| Physics encoder | 2,813,714 | Physics features and plausibility grounding |
+| Diffusion decoder | 1,248,145 | Native atom, coordinate, and lattice generation |
+| Full compact stack | 4,061,859 | Candidate proposal and physics grounding |
+| Controller pilot | Separate Qwen3-14B artifact | Optional evidence routing and explanation |
+
+## Intended Use
+
+- Generate native, unrepaired inorganic candidate structures.
+- Produce diverse candidate pools for downstream screening.
+- Combine with the encoder for physics-aware ranking.
+- Support research and product prototyping with explicit external validation.
+
+## Out-of-Scope Use
+
+- Treating generated structures as experimentally stable without relaxation and
+  hull evaluation.
+- Claiming SOTA parity with MatterGen from the internal static profile suite.
+- Organic, molecular-crystal, polymer, or MOF generation.
+- Safety-critical or autonomous synthesis decisions without expert review.
+
+## Verified Evidence
+
+The final non-alloy static suite passed profile-specific gates:
+
+| Profile | Raw | Accepted | Charge | LeMat-distance | Strict | Novelty |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| SSB regression | 1,440 | 513 | 100.0% | 35.7% | 60.4% | 100.0% |
+| Battery ionics | 1,248 | 411 | 100.0% | 33.1% | 60.3% | 100.0% |
+| Oxide catalysts | 1,584 | 424 | 100.0% | 26.8% | 59.7% | 100.0% |
+| Halides/chalcogenides | 2,688 | 436 | 100.0% | 16.3% | 55.1% | 100.0% |
+| Semiconductors | 672 | 256 | profile diagnostic | 38.1% | 65.0% | 100.0% |
+| Broad inorganic | 2,688 | 242 | profile diagnostic | 9.0% | 55.1% | 100.0% |
+
+The strongest bounded external stability anchor produced:
+
+- 2/64 M.S.U.N. candidates;
+- mean evaluated `e_above_hull = 0.8750 eV/atom`;
+- two individual candidates at 0.1786 and 0.1837 eV/atom;
+- 3/64 strict relaxation convergence.
+
+This proves sparse support in the desired low-hull region and places observed
+evaluated candidates in an approximately 0.8-to-0.18 eV/atom band. It does not
+prove repeatable high-yield metastable generation. The 2/64 hit rate was 3.125%,
+well below the intended 25-50% yield. A broader 512-candidate screen converged
+58/512, had weighted mean hull 1.2330 eV/atom, and produced no M.S.U.N.
+candidates.
+
+## Limitations
+
+- Relaxation convergence and low-hull yield are substantially weaker than
+  static validity.
+- The low-hull anchor did not reproduce reliably enough in later cohorts.
+- Alloys/intermetallics remain outside the passed static profile panel.
+- Internal and public model comparisons use different protocols.
+- The optional 14B controller is a partial pilot and is not part of the compact
+  4.061859M-parameter claim.
+
+## Required Validation
+
+Every generated candidate should pass:
+
+1. profile-specific static validation;
+2. external geometry and post-relax validation;
+3. MLIP or DFT relaxation appropriate to the intended use;
+4. formation-energy and hull evaluation against a compatible reference set;
+5. domain-expert review before synthesis or deployment decisions.
+
+## Artifact Lineage
+
+- Encoder: `encoder/v1/`
+- Original production decoder: `decoder/diffusion_v1/`
+- Recovered native decoder: `decoder/native_v78/`
+- Research initialization: `research/v29_directional_step800/`
+- Final documents: `docs/final_closeout/`
+- Evidence summaries: `eval/final_evidence/`
+
+The V78 checkpoint is the shippable recovered native lineage. V90 is an
+evaluation surface without a train checkpoint at the historically configured
+URI. Later conservative-field checkpoints are archived research evidence and
+are not production promotions.
+
+The metastability program is on ice indefinitely because its duration and cost
+outgrew the reliability gains. It is not classified as a complete scientific
+failure: the archived low-hull candidates provide a real restart anchor.
+
+## Citation
+
+Use the technical paper in
+`docs/final_closeout/NEXA_MAT_FINAL_TECHNICAL_PAPER_2026-07-24.md` and cite the
+exact model-hub revision used for an experiment.

docs/final_closeout/NEXA_MAT_FINAL_PROJECT_COMPENDIUM_2026-07-24.md

@@ -0,0 +1,258 @@
+# NexaMat Final Project Compendium
+
+Date: 2026-07-24  
+Project state: archived and on ice indefinitely  
+Model state: shipped with qualified claims
+
+## 1. Executive Summary
+
+NexaMat began as a compact but brittle materials generator. Its atom predictions
+were strong, its outputs were novel, and its encoder carried useful physical
+signals, but the decoder placed too little probability mass on valid periodic
+geometry. Early external runs produced only a handful of strict CIFs from
+thousands of samples. This made every downstream stability benchmark appear
+catastrophic because the expensive scientific stack was starved before it could
+evaluate enough usable structures.
+
+The alignment campaign solved that first problem. It introduced honest native
+accounting, explicit failure reasons, profile-conditioned chemistry, radius-aware
+geometry, charge-aware composition logic where appropriate, best-of-N native
+rollouts, and target-gated evaluation. The result is a 4.061859M-parameter stack
+that produces useful native candidate pools across six non-alloy inorganic
+profiles.
+
+The second problem was harder. A statically valid crystal can still sit far from
+a stable energy basin. NexaMat reached an approximately 0.8-to-0.18 eV/atom
+band and produced two M.S.U.N. candidates in a 64-candidate bounded screen.
+That is real progress, but it did not become a repeatable high-density
+metastable distribution. The 3.125% hit rate remained far below the intended
+25-50% yield. Several weeks of force, trajectory, conservative-energy, tensor,
+routed-expert, and foundation-residual experiments improved proxies without
+moving the external objective reliably. The project is therefore put on ice
+indefinitely at a clear plateau rather than being called a complete failure or
+being kept alive through unproductive iteration.
+
+## 2. What The Model Is
+
+NexaMat is a modular materials stack:
+
+```mermaid
+flowchart LR
+    P["Profile and generation request"] --> D["Native diffusion decoder"]
+    D --> C["Native CIF candidate"]
+    C --> S["Static profile gate"]
+    S --> E["Physics encoder and ranking"]
+    E --> R["External relaxation"]
+    R --> H["Hull and metastability evaluation"]
+    H --> O["Evidence-backed candidate set"]
+```
+
+The decoder jointly proposes atom identities, fractional coordinates, and a
+periodic lattice. The encoder supplies physics-aware representation and scoring.
+External relaxation and hull evaluation remain authoritative for stability.
+
+The compact stack excludes the optional Qwen3-14B controller:
+
+```text
+physics encoder:          2,813,714 parameters
+diffusion decoder:        1,248,145 parameters
+compact full stack:       4,061,859 parameters
+```
+
+## 3. The Original Failure
+
+The initial decoder learned chemistry more readily than global crystal geometry.
+Atom vocabulary and novelty saturated while lattice scale, periodic pair
+distances, charge-compatible composition, and coordinate placement remained
+brittle. The conceptual mistake was treating generic denoising loss and
+teacher-forced validation as adequate substitutes for the density of native
+benchmark-eligible CIFs.
+
+The decoder was therefore good at producing different-looking compositions but
+poor at instantiating them as usable crystals. The encoder became a crutch
+because it had to reject nearly everything.
+
+## 4. The Static-Validity Recovery
+
+The recovery succeeded because it changed the contract, not merely the loss:
+
+- raw candidates were retained;
+- native and repaired surfaces were separated;
+- failure reasons became explicit;
+- SSB chemistry became a profile rather than a global hardcoded identity;
+- ionic and non-ionic profiles used different charge policies;
+- radius-aware site assignment and lattice checks attacked periodic geometry;
+- target-gated early stopping reduced wasted benchmark time;
+- checkpoint selection used external native yield instead of final-step loss.
+
+The strongest SSB lane, V90, reached:
+
+| Metric | Result |
+| --- | ---: |
+| Accepted | 595/4096 |
+| Filter accept | 14.53% |
+| Allowed elements | 100.00% |
+| Charge neutrality | 100.00% |
+| LeMat-distance validity | 14.53% |
+| Strict validity | 54.20% |
+| Formula uniqueness | 41.80% |
+| Novelty | 100.00% |
+
+The final six-profile suite accepted 513 SSB, 411 battery, 424 oxide, 436
+halide/chalcogenide, 256 semiconductor, and 242 broad-inorganic candidates. The
+last two were tolerant passes against profile-specific targets. Alloys and
+intermetallics remained unresolved.
+
+## 5. The Stability Gap
+
+Static validity asks whether a crystal is structurally and chemically plausible
+enough to evaluate. Metastability asks whether relaxation places it close to a
+competitive energy basin. The latter is a much narrower target.
+
+The broad 512-candidate screen showed the gap:
+
+```text
+relaxation convergence:      58 / 512
+weighted mean hull:          1.2330 eV/atom
+M.S.U.N. candidates:         0 / 512
+post-relax validity:         high
+```
+
+The best bounded low-hull anchor improved the scientific picture:
+
+```text
+relaxation convergence:      3 / 64
+mean hull:                    0.8750 eV/atom
+M.S.U.N. candidates:         2 / 64
+best individual hull rows:   0.1786 and 0.1837 eV/atom
+```
+
+The result matters because it proves the generator can reach the desired region.
+It is not enough because the probability mass remains sparse, the observed
+2/64 yield is far below the 16-32/64 practical ambition, and the result did not
+reproduce consistently enough for product reliability.
+
+## 6. Metastability Campaign
+
+The project tested a broad set of hypotheses:
+
+1. force and stress auxiliary supervision;
+2. real relaxation trajectories;
+3. normalized geometry and cell-shape context;
+4. scalar conservative energy surfaces;
+5. bounded mobility and symmetric lattice strain;
+6. rank-2 angular refinement;
+7. profile adapters and FiLM;
+8. routed periodic experts;
+9. source scheduling and profile balancing;
+10. MACE relation and force distillation;
+11. a fresh broad force foundation;
+12. frozen MACE-OMat plus a learned residual potential.
+
+Several internal signals became useful. V32 reached true-label ROC-AUC 0.70845
+and minimum balanced accuracy 0.60048. V30 reached force MAE 0.14260 eV/A and
+low-force MAE 0.07715 eV/A. The frozen MACE-OMat foundation reached force cosine
+0.75450 and MAE 0.05728 eV/A.
+
+The generative controller did not improve accordingly. V32 moved coordinates,
+lattice, and pair distances in the wrong direction and became non-finite. V30
+had negative coordinate and pair contraction. The final residual grid stayed
+near its frozen foundation baseline and did not justify mobility or native
+generation stages.
+
+## 7. Why The Redesigns Plateaued
+
+The redesigns attacked useful intermediate mechanisms, but they did not change
+the native endpoint distribution strongly enough. The system repeatedly asked
+a generator trained primarily for denoising and static geometry to behave like
+a low-hull endpoint model after modest continuation.
+
+Four deeper mismatches remained:
+
+- training data was not consistently filtered and weighted around broad relaxed
+  low-hull endpoints;
+- the native sampler was not always the exact reverse process implied by the
+  training corruption;
+- composition, coordinates, and lattice were not coupled strongly enough by
+  one endpoint objective;
+- internal force, critic, and contraction gates delayed direct external
+  metastability feedback.
+
+This explains how proxy metrics improved while metastable yield stayed flat.
+
+## 8. Technical Innovations
+
+The project produced reusable engineering and scientific mechanisms:
+
+- replay-grade candidate JSONL with raw payloads and failure reasons;
+- strict separation of native, projected, scaffolded, ranked, relaxed, and
+  hull-scored surfaces;
+- profile-conditioned element, charge, radius, and benchmark contracts;
+- charge-balanced composition search;
+- radius-aware periodic geometry checks;
+- generated hard-negative replay;
+- external noisy-state training paths;
+- stability evidence ingestion with provenance-safe joins;
+- source-disjoint trajectory and composition-family splits;
+- bounded symmetric lattice updates with positive volume;
+- target-gated 64-to-128-to-512 external promotion;
+- streaming and resumable benchmark operations;
+- object-storage-first lineage, manifests, receipts, and W&B references.
+
+## 9. Honest Product Position
+
+NexaMat is ready to ship as a compact proposal model when the product preserves
+an explicit screening funnel:
+
+```text
+native generation
+-> static profile validation
+-> encoder/ranker
+-> external relaxation
+-> hull and novelty checks
+-> expert review
+```
+
+It should not present raw candidates as stable materials. Its value is efficient
+candidate generation and unusually strong static coverage for its parameter
+count, not SOTA stability proof.
+
+## 10. Lessons
+
+1. Optimize the final scientific outcome from the beginning.
+2. Keep benchmark surfaces separate so repair cannot masquerade as generation.
+3. A good classifier or force model is not automatically a good actuator.
+4. Sparse success proves support, not distribution-level competence.
+5. Freeze evaluator, sampler, seeds, and lineage before interpreting trends.
+6. Use small external probes early; do not wait for a large campaign to reveal
+   objective mismatch.
+7. Preserve negative results. The failed architectures sharply narrowed the
+   remaining problem.
+8. Stop when repeated redesigns move proxies but not the external objective.
+
+## 11. Reopening Strategy
+
+The best future experiment is a composition-conditioned low-hull endpoint pilot:
+
+1. assemble relaxed primitive structures with reliable hull evidence;
+2. separate near-hull, metastable, and higher-hull slices;
+3. train a mathematically matched wrapped-coordinate diffusion or Riemannian
+   flow over coordinates and lattice;
+4. warm-start useful chemistry embeddings but not the failed controller;
+5. evaluate fixed compositions first;
+6. run the same frozen 64-candidate relaxation/hull protocol.
+
+If fixed-composition generation succeeds, composition is the next bottleneck.
+If teacher denoising succeeds but free sampling fails, the sampler is the
+bottleneck. If fixed-composition geometry fails, the periodic generator
+architecture must be replaced.
+
+## 12. Final Verdict
+
+NexaMat did not reach the reliable high-yield stable-material generator
+originally targeted, but it is not a failed project. It transformed a nearly
+unusable decoder into a compact, auditable, multi-profile inorganic candidate
+generator, demonstrated sparse metastable support, and established the exact
+scientific boundary that remained. The project is archived and placed on ice
+indefinitely with a useful shipped model, durable evidence, and a decisive
+restart contract.

docs/final_closeout/NEXA_MAT_FINAL_TECHNICAL_PAPER_2026-07-24.md

@@ -0,0 +1,288 @@
+# NexaMat: Recovering a Compact Native Crystal Generator and Locating the Metastability Boundary
+
+## Abstract
+
+NexaMat is a compact 4.061859M-parameter materials stack comprising a
+2.813714M-parameter physics encoder and a 1.248145M-parameter diffusion decoder.
+The initial decoder learned atom identity and novelty substantially faster than
+periodic geometry, producing very low native benchmark yield. We report a
+multi-stage recovery that introduced replay-grade candidate accounting,
+profile-conditioned chemistry policies, charge-aware composition handling,
+radius-aware periodic geometry, native best-of-N rollouts, and external
+target-gated evaluation. The final native, no-repair suite passed profile-specific
+static gates for six non-alloy inorganic domains and preserved 100% novelty.
+
+We then investigated whether the recovered generator could be aligned to
+relaxable low-hull basins. A bounded experiment produced two M.S.U.N. candidates
+from 64 selected structures and individual hull values of 0.1786 and 0.1837
+eV/atom, placing observed candidates in an approximately 0.8-to-0.18 eV/atom
+band and demonstrating sparse support in the target region. However, only 3/64
+structures met strict relaxation convergence, the 3.125% hit rate remained far
+below the intended 25-50% yield, a 512-candidate screen produced zero M.S.U.N.
+candidates, and subsequent force, trajectory, conservative-field, tensor,
+routed-expert, and foundation-residual redesigns did not produce repeatable
+external improvement. We conclude that NexaMat is a useful compact native
+candidate generator with partial metastability evidence, but not a proven
+high-yield metastable generator. The central remaining issue is generative
+endpoint density rather than static validity or local force magnitude.
+
+## 1. Introduction
+
+Crystal generation couples discrete composition, periodic coordinates, and
+lattice geometry. A model can perform well on atom prediction while failing to
+place those atoms into a physically useful cell. It can also emit a statically
+valid crystal that relaxes into a poor or chemically uncompetitive energy basin.
+These are separate scientific problems and require separate evidence.
+
+NexaMat exposed both boundaries. Its first failure was native geometry density:
+only a tiny fraction of raw samples could be exported as strict CIFs. Its second
+failure emerged after that problem was repaired: static-valid candidates did
+not reliably relax into near-hull structures.
+
+This paper documents the complete system, the alignment mechanisms that worked,
+the metastability experiments that did not, and the evidence boundary of the
+shipped model.
+
+## 2. Model
+
+### 2.1 Physics encoder
+
+The encoder contains 2,813,714 parameters and was trained on mixed relaxed,
+rattled, and off-equilibrium inorganic structures. It provides physics-aware
+representations and plausibility signals. The encoder is frozen during the
+shipped native-generation workflow.
+
+### 2.2 Diffusion decoder
+
+The decoder contains 1,248,145 trainable parameters and jointly predicts:
+
+- atom identities;
+- fractional coordinate updates;
+- lattice updates;
+- optional static and stability diagnostics.
+
+Periodic neighbor information is represented through distance and direction
+features. Later research variants added scalar energy, force, stress, bounded
+mobility, symmetric strain, rank-2 angular state, routed experts, and foundation
+residuals. These variants are research lineage, not shipped promotions.
+
+### 2.3 Candidate surfaces
+
+The final accounting contract preserves distinct surfaces:
+
+1. native candidate;
+2. static-qualified candidate;
+3. projected or scaffolded candidate;
+4. critic-ranked candidate;
+5. relaxed candidate;
+6. hull-scored candidate.
+
+Only native candidates count toward native-generation claims. Projection,
+scaffolding, ranking, or relaxation cannot overwrite native metrics.
+
+## 3. Data And Evidence
+
+The project used canonical crystal structures, profile-specific datasets,
+generated candidate replay, relaxation trajectories, force and stress labels,
+and external hull evidence. Sources included SSB regression data, broad
+inorganic rows, OMat-style force data, MPTrj trajectories, JARVIS/LeMat-style
+evidence where available, and generated NexaMat failures.
+
+Each stability row was designed to preserve:
+
+- source and material identity;
+- composition family;
+- atomic numbers, fractional coordinates, and lattice;
+- trajectory frame and time;
+- force, stress, and energy labels with masks;
+- convergence and post-relax validity;
+- hull evidence;
+- provenance hashes.
+
+Splits were source- and material-disjoint where the mechanism experiments
+required transfer evidence.
+
+## 4. Native-Validity Alignment
+
+### 4.1 Failure diagnosis
+
+The original benchmark showed a severe mismatch: chemical plausibility and
+novelty were much stronger than allowed-element and periodic-distance validity.
+Generic denoising loss did not measure usable native CIF density, and final-step
+checkpoint selection could prefer a model whose free sampler had regressed.
+
+### 4.2 Recovery mechanisms
+
+The recovery introduced:
+
+- candidate JSONL containing raw atoms, coordinates, lattice, and explicit
+  failure reasons;
+- hard-negative replay from generated failures;
+- allowed-element and charge pressure;
+- profile-specific element, charge, and radius policies;
+- radius-aware periodic geometry;
+- charge-balanced composition search for ionic profiles;
+- formula-diverse selection;
+- native best-of-N rollouts;
+- external checkpoint selection by accepted yield and geometry metrics.
+
+### 4.3 Static results
+
+The strongest V90 SSB diagnostic accepted 595/4096 native samples, with 100%
+allowed elements, 100% charge neutrality, 54.20% strict validity, 41.80%
+selected formula uniqueness, and 100% novelty.
+
+The final multi-profile suite produced:
+
+| Profile | Raw | Accepted | Filter accept | LeMat-distance | Strict | Novelty |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| SSB regression | 1,440 | 513 | 35.6% | 35.7% | 60.4% | 100.0% |
+| Battery ionics | 1,248 | 411 | 32.9% | 33.1% | 60.3% | 100.0% |
+| Oxide catalysts | 1,584 | 424 | 26.8% | 26.8% | 59.7% | 100.0% |
+| Halide/chalcogenides | 2,688 | 436 | 16.2% | 16.3% | 55.1% | 100.0% |
+| Semiconductors | 672 | 256 | 38.1% | 38.1% | 65.0% | 100.0% |
+| Broad inorganic | 2,688 | 242 | 9.0% | 9.0% | 55.1% | 100.0% |
+
+These metrics use profile-specific gates and are not public MatterGen-equivalent
+stability metrics.
+
+## 5. External Stability Evaluation
+
+The external funnel consisted of native generation, static qualification,
+selection, short relaxation, post-relax validation, and hull/SUN-style scoring.
+
+The broad 512-candidate selector screen produced:
+
+```text
+converged:                 58 / 512
+weighted mean hull:        1.2330 eV/atom
+M.S.U.N.:                  0 / 512
+post-relax validity:       high
+```
+
+The strongest bounded stable-manifold probe produced:
+
+```text
+converged:                 3 / 64
+mean hull:                 0.8750 eV/atom
+M.S.U.N.:                  2 / 64
+best hull rows:            0.1786, 0.1837 eV/atom
+```
+
+A follow-up force-basin probe improved mean hull to 0.8165 eV/atom and retained
+2/64 hits but converged only 4/64 structures. Later reproduction cohorts did not
+establish stable recurrence. The external evidence therefore shows sparse
+support, not a reliable metastable distribution.
+
+## 6. Mechanism Experiments
+
+### 6.1 Trajectory and force supervision
+
+Real trajectory windows, force, stress, energy descent, and endpoint displacement
+were added to the training contract. V30 reached force MAE 0.14260 eV/A and
+low-force MAE 0.07715 eV/A, but force cosine remained 0.30881, coordinate
+contraction was -2.486%, and pair contraction was -14.555%.
+
+### 6.2 Stability representation
+
+V32 corrected the true-label stability policy and reached ROC-AUC 0.70845 and
+minimum balanced accuracy 0.60048. This demonstrated that the shared
+representation contained a transferable basin-quality signal.
+
+The controller failed. Held-out force MAE was 0.35973 eV/A, force cosine was
+0.23114, one-step coordinate/lattice/pair contractions were all negative, and
+the four-step higher-order rollout became non-finite.
+
+### 6.3 Broader conservative fields
+
+The project tested profile adapters, FiLM, periodic residual experts, tensor
+angular refinement, source scheduling, MACE force mixing, MACE relation
+distillation, and a fresh force foundation. The custom NexaMat force field
+plateaued near 0.306 force cosine on the frozen bridge, while MACE-OMat reached
+approximately 0.776 on the audit surface.
+
+The final foundation-residual campaign used frozen MACE-OMat plus a trainable
+equivariant residual. The frozen baseline reached cosine 0.75450 and MAE 0.05728
+eV/A. Activating the residual gradient path and sweeping residual scale and
+learning rate did not improve held-out direction. The best residual result was
+0.75322, and no controller or native screen was justified.
+
+## 7. Discussion
+
+### 7.1 Why static validity improved
+
+Static recovery used direct, localizable signals and explicit sampler contracts.
+The model could learn allowed chemistry, charge policy, radius compatibility,
+lattice bounds, and candidate rejection. These constraints substantially
+increased benchmark-eligible density.
+
+### 7.2 Why metastability plateaued
+
+Metastability is an endpoint-distribution problem. A local force model may know
+a downhill direction without teaching a generator to initialize near a globally
+competitive basin. A stability critic may rank structures without providing an
+accurate actuator. A controller may improve teacher-forced trajectories while
+failing under free native rollouts.
+
+The campaign retained a decoder whose pretraining contract emphasized denoising
+and conservative lattice behavior. Later continuation attempted to reshape that
+distribution using relatively small trajectory and stability objectives. The
+resulting improvements were real but mostly proxy-level.
+
+The likely remaining requirements are:
+
+- a broad low-hull endpoint training distribution;
+- a corruption and reverse process matched exactly at sampling time;
+- tighter joint modeling of composition, coordinates, and lattice;
+- direct and frequent external endpoint feedback;
+- fixed-composition ablations that separate geometry from composition failure.
+
+## 8. Efficiency
+
+The compact stack is approximately 8.7% of MatterGen's reported 46.8M parameter
+count. This is a meaningful engineering result for static candidate generation.
+It must not be converted into a stability-performance claim because the models
+were not evaluated under the same full protocol.
+
+The optional 14B controller is separate and must not be included in the compact
+parameter-efficiency claim.
+
+## 9. Limitations
+
+- No repeatable high-yield metastable or SUN/MSUN result.
+- No identical public MatterGen benchmark.
+- Alloys/intermetallics remain unresolved.
+- Hull values depend on evaluator, reference data, and relaxation coverage.
+- Sparse low-hull hits are sensitive to checkpoint, sampler, seeds, and
+  evaluator lineage.
+- Recent force-field research checkpoints are not shippable native generators.
+
+## 10. Conclusion
+
+NexaMat demonstrates that a very small materials stack can recover strong native
+inorganic candidate generation when the benchmark and sampler contracts are
+made explicit. It also demonstrates that static validity, representation
+quality, and force accuracy do not automatically produce a metastable
+distribution.
+
+The project closes with a useful shipped model and a clear scientific boundary.
+It is placed on ice indefinitely because the multi-week campaign's remaining
+reliability gap no longer justified continuous iteration. Future work should
+not extend the controller lineage incrementally. It should first reproduce the
+low-hull anchor and then test a matched low-hull endpoint generator under
+fixed-composition and free-composition settings.
+
+## Artifact Availability
+
+Curated model artifacts and final documentation:
+
+```text
+https://huggingface.co/AethronPhantom/Nexa_Mat2
+```
+
+Complete project archive and historical lineage:
+
+```text
+wasabi:nexa-mat/project_archives/nexa_mat/2026-07-24_final_closeout/
+wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/diffusion_alignment/
+```

docs/final_closeout/PROJECT.md

@@ -0,0 +1,93 @@
+# NexaMat Archived Project Record
+
+Project: NexaMat / NexaMat2  
+Status: Archived and on ice indefinitely; model shipped as a qualified research and candidate-generation artifact  
+Archive date: 2026-07-24  
+Canonical source root: `docs/projects/Nexa_Mat`  
+Code authority: Git repository at revision `81185a7cf1f85e63cc8b9ca3b7284a5498d56974` plus the recorded dirty-worktree manifest  
+Model hub: `https://huggingface.co/AethronPhantom/Nexa_Mat2`  
+Historical archive: `wasabi:nexa-mat/project_archives/nexa_mat/2026-07-24_final_closeout/`
+
+## Purpose
+
+NexaMat is a compact materials model stack for proposing and evaluating inorganic
+crystal candidates. The project recovered a diffusion decoder that initially
+produced very low-yield geometry into a useful native, no-repair candidate
+generator across six non-alloy inorganic profiles. It then investigated the
+harder problem of repeatable metastable generation.
+
+## Shipped Claim
+
+The shipped model is suitable for:
+
+- profile-conditioned native inorganic candidate generation;
+- auditable candidate filtering and ranking;
+- chemistry-aware proposal workflows using the frozen physics encoder;
+- research, screening, and application prototyping where external relaxation
+  and stability evaluation remain mandatory.
+
+The shipped model is not claimed to be:
+
+- a high-yield metastable or SUN/MSUN generator;
+- a replacement for relaxation, DFT, MLIP, or hull evaluation;
+- a universally validated generator for alloys, organics, MOFs, polymers, or
+  molecular crystals;
+- directly comparable to MatterGen under an identical public stability protocol.
+
+## Canonical Model State
+
+| Component | Parameters | Role | Status |
+| --- | ---: | --- | --- |
+| Physics encoder | 2,813,714 | Plausibility and physics grounding | Shipped, frozen |
+| Diffusion decoder | 1,248,145 | Native atom, coordinate, and lattice proposal | Shipped |
+| Full compact stack | 4,061,859 | Encoder plus decoder | Shipped |
+| Multimodal controller | Separate 14B pilot | Evidence routing and explanation | Optional, partial pilot |
+
+The production decoder lineage is the recovered V78 native checkpoint. V90 is
+an evaluation artifact produced from a provided checkpoint; its recorded run
+prefix does not contain a trainable `best_diffusion_checkpoint.pt`. Later
+V26-V36 conservative-field experiments are archived as research evidence and
+must not be substituted for the shipped native decoder.
+
+## Final Scientific State
+
+- Static native generation: useful and production-tolerable on SSB, battery,
+  oxide, halide/chalcogenide, semiconductor, and broad-inorganic profiles.
+- Novelty: 100% in the final six-profile static suite.
+- Best bounded metastability anchor: 2/64 M.S.U.N. candidates, mean evaluated
+  `e_above_hull = 0.8750 eV/atom`, with two low-hull candidates at 0.1786 and
+  0.1837 eV/atom. The observed stability band was approximately 0.8 to 0.18
+  eV/atom.
+- Repeatability: not established. The same anchor converged only 3/64 structures
+  at the strict relaxation threshold, and later reproductions did not preserve
+  the result reliably. The 2/64 hit rate was 3.125%, well below the intended
+  25-50% yield.
+- Final campaign verdict: meaningful partial progress followed by a scientific
+  plateau. The project was paused because the campaign had dragged on for weeks
+  and the remaining reliability gap no longer justified continuous iteration,
+  not because the model had no useful metastable support.
+
+## Canonical Documents
+
+1. `ARCHIVE_CLOSEOUT_2026-07-24.md`
+2. `results/papers/NEXA_MAT_FINAL_PROJECT_COMPENDIUM_2026-07-24.md`
+3. `results/papers/NEXA_MAT_FINAL_TECHNICAL_PAPER_2026-07-24.md`
+4. `results/papers/NEXA_MAT_FINAL_MODEL_CARD_2026-07-24.md`
+5. `results/papers/NEXA_MAT_ALIGNMENT_PIPELINE_KNOWLEDGE_PAPER.md`
+6. `results/papers/NEXA_MAT_METASTABLE_ARCHITECTURE_AND_TRAINING.md`
+7. `results/benchmarks/2026-07-24_foundation_residual_metastability_closeout.md`
+
+## Reopen Contract
+
+If the project is resumed, do not restart with another static-validity,
+force-loss, router, adapter, or controller sweep. First reproduce the July 2
+low-hull anchor from the exact checkpoint, sampler, seeds, candidate rows,
+relaxer, and hull evaluator. If it reproduces, train a mathematically matched
+low-hull endpoint generator and measure recurrence density. If it does not,
+classify the result as lineage or evaluator drift before changing the model.
+
+All heavy artifacts remain preserved in Wasabi and the curated shippable model
+surface is mirrored to Hugging Face. Local documentation is an index, not the
+only copy. The project is on ice indefinitely, not abandoned; the archive is
+designed to support a clean restart if the stability work becomes worthwhile
+again.

eval/final_evidence/2026-07-15_v30_tensor_angular_refinement_closeout.md

@@ -0,0 +1,71 @@
+# NexaMat V30 Tensor Angular Refinement Closeout
+
+Date: 2026-07-15
+Verdict: mechanism gate failed; native 64-CIF probe correctly blocked
+Trend classification: `PLATEAU` at `5e-5`, `REGRESS` at `1e-4`
+
+## Objective
+
+Test whether a zero-initialized, rotation-equivariant rank-2 periodic tensor pathway could break the V29 directional ceiling while preserving the recovered V29 force and chemistry behavior. Promotion required held-out force MAE at or below `0.20 eV/A`, force cosine at or above `0.40`, low-force MAE at or below `0.14289 eV/A`, coordinate contraction at or above `25%`, and non-negative lattice and pair-distance contraction.
+
+## Architecture Change
+
+V30 adds `metastable_manifold_v18`, a rank-2 angular refinement after the existing scalar-vector interaction stack. Periodic edge directions form traceless quadrupoles, `u outer u - I/3`, which are aggregated into per-site tensor channels. The tensor context can refine the scalar state and contract with vector channels to refine the trajectory controller. Its output projections are zero-initialized, so a V17 checkpoint reproduces V17 outputs exactly before V30 training.
+
+Local verification passed `106` focused tests and Ruff. The tests cover rotation covariance, exact V17 warm-start behavior, live tensor gradients, the force-only lattice no-op, trainer/grid architecture acceptance, and existing real-trajectory contracts.
+
+## Results
+
+| Arm | Best step | Force MAE | Force cosine | Low-force MAE | Coordinate contraction | Lattice contraction | Pair contraction | Gate |
+|---|---:|---:|---:|---:|---:|---:|---:|---|
+| `tensor_lr5e5` | 1000 | 0.14260 | 0.30881 | 0.07715 | -2.486% | 0.000% | -14.555% | Fail |
+| `tensor_lr1e4` | 1000 | 0.14729 | 0.30188 | 0.08232 | -4.130% | 0.000% | -14.118% | Fail |
+
+For comparison, V29 had already reached approximately `0.304` force cosine and a better trajectory-target cosine without achieving contraction. V30 did not move the held-out force-direction ceiling, and the higher learning rate made coordinate behavior worse.
+
+The tensor path was active. At step 1000, the initially zero vector-gate weight norm was `0.289` for `5e-5` and `0.518` for `1e-4`; the initially zero scalar projection weight norm was `0.226` and `0.376`. The failure therefore cannot be attributed to a disconnected or frozen refinement module.
+
+## Research Verdict
+
+V30 falsifies the narrow hypothesis that a terminal local rank-2 angular correction is sufficient to learn the stable-basin displacement from this teacher contract. V28 separated force from finite displacement, V29 added an explicit directional objective, and V30 added anisotropic tensor capacity; all three retained good magnitude metrics but failed held-out contraction. The repeated bottleneck is now a verified ceiling rather than a tuning gap.
+
+The most likely remaining causes are:
+
+1. The contiguous MPTrj/CHGNet frame displacement is too small, noisy, or weakly identifiable from a single source state for the current one-step target.
+2. Tensor information must participate throughout message passing and energy construction rather than enter as a terminal correction.
+3. The model needs an iterative rollout or endpoint/energy-basin target that is physically identifiable, not another local displacement regression weight.
+
+No 64-CIF relaxation or hull screen was run because the internal causal mechanism did not pass. This preserves the evaluation contract and avoids spending expensive scientific compute on a controller already shown to damage held-out geometry.
+
+## Engineering Closeout
+
+One persistent H100 pod was reused. The productive run sustained about `1,188 samples/s` and both 1,000-step arms completed with W&B online. A stale trainer CLI allowlist caused one pre-GPU startup failure; it was patched once, covered by a CLI contract test, and the same run lineage resumed. Periodic Wasabi sync retried one live-file checksum race successfully. The full remote run produced 119 files totaling 846,862,217 bytes; after intentionally excluding transient W&B and training-selection surfaces, Wasabi contains 106 durable objects totaling 727,465,387 bytes. An `rclone check --one-way --size-only` reported zero differences for every expected artifact under:
+
+`wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/diffusion_alignment/2026-07-15/nexa_mat_v30_tensor_angular_refinement_20260715T134826Z`
+
+## Carry Forward
+
+Preserve V29 step 800 as the better directional initialization and preserve V30 as negative architecture evidence. Do not continue with more V18 learning-rate or loss-weight sweeps. Before another GPU cycle, run a target-identifiability audit that compares source-state features against immediate-frame, multi-frame, final-relaxed, force-integrated, and energy-decrease targets. Only design V31 after one target demonstrates materially higher held-out predictability than the current contiguous-frame displacement.
+
+## End-Of-Day Handoff
+
+The target-identifiability audit is now wired in `scripts/materials/audit_nexamat_trajectory_target_identifiability.py`. It fits force-step calibration on source-disjoint training trajectories, evaluates local, multi-frame, and reconstructed final-relaxed targets on validation trajectories, and reports per-profile contraction and direction evidence. It now fails closed unless at least `4,096` usable training pairs and `512` usable validation pairs are present, and the CLI returns non-zero when the scientific verdict fails. The focused handoff checks pass (`4 passed`, Ruff clean).
+
+The V26 run receipt proves a source-disjoint contract with `18,431` paired training rows, `6,145` paired validation rows, `6,746` training sources, and `2,250` validation sources. Its recorded stream fingerprints are `sha256:b92ac8ae4a4500bd3fc70c5670665bc0aab3069ded5e1cb3af9c0bd2c8ef1a9c` for training and `sha256:c8a8017d42bcc08bb1665bd42150f7f6dc2ea1a9b88cab7afb619cf93c26072b` for validation. However, those exact JSONLs were not copied into the V26 run prefix or the documented dataset prefixes. The next session must therefore hydrate a durable real-trajectory source, verify its fingerprints, and run the target audit before designing or training V31. The durable broad source currently available is `wasabi:nexa-mat/datasets/nexa_mat/metastable_manifold/nexa_mat_metastable_v6_broad16_leftdeform_20260713T060715Z`; it contains `15,020` training and `4,980` validation trajectory pairs and satisfies the required scale. Another architecture or loss sweep is not justified until one target policy passes the identifiability gate.
+
+## Quantified Delta And Trend Snapshot
+
+The current trend is `PLATEAU` with a narrowed causal bottleneck, not `REWARD_DEATH`. Force magnitude and low-force accuracy have crossed their practical gates, but force direction and coordinated geometry transfer have not.
+
+| Mechanism metric | Current V30 | Gate | Delta |
+|---|---:|---:|---:|
+| Force MAE | `0.14260 eV/A` | `<=0.20` | pass by `0.05740` |
+| Low-force MAE | `0.07715 eV/A` | `<=0.14289` | pass by `0.06574` |
+| Force cosine | `0.30881` | `>=0.40` | short by `0.09119` |
+| Coordinate contraction | `-2.486%` | `>=25%` | short by `27.486` percentage points |
+| Lattice contraction | `0.000%` | `>=0%` | boundary only; no learned improvement |
+| Pair-distance contraction | `-14.555%` | `>=0%` | short by `14.555` percentage points |
+
+The best external low-hull anchor remains the July 2 bounded run: mean `e_above_hull=0.8750 eV/atom`, individual candidates at `0.1786` and `0.1837 eV/atom`, `2/64` M.S.U.N. candidates, and only `3/64 = 4.6875%` clean relaxation convergence. The mean-hull delta is `0.125 eV/atom` to the practical `0.75` gate, `0.375` to the `0.50` stretch gate, `0.675` to a `0.20` stable-regime target, and `0.775` to `0.10`. The immediate convergence gate requires `16/64`, so the observed anchor is short by `13` converged structures and `20.3125` percentage points. The July 9 reproduction attempts returned zero converged structures, so the two low-hull hits prove support but not repeatable distribution-level performance.
+
+Ruled out for the next run: more learning-rate sweeps on V18, larger weights on the same contiguous-frame displacement target, another terminal tensor correction, selector-only reranking, and renewed static-validity optimization. The next causal question is whether local, multi-frame, or final-relaxed displacement is identifiable from the source state and source force. Run the target-identifiability audit first. Only a target with force/displacement cosine `>=0.50`, held-out coordinate contraction `>=25%`, and at least `4,096/512` source-disjoint usable rows can authorize V31.

eval/final_evidence/2026-07-16_v32_shared_energy_metastability_closeout.md

@@ -0,0 +1,109 @@
+# NexaMat V32 Shared-Energy Metastability Closeout
+
+Date: 2026-07-16, final rerun 2026-07-17
+
+Run: `nexa_mat_v32_shared_energy_20260716T141000Z`
+
+Wasabi prefix:
+
+`wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/diffusion_alignment/2026-07-16/nexa_mat_v32_shared_energy_20260716T141000Z`
+
+Final rerun: `nexa_mat_v32_shared_energy_pairbudget_eligible_20260717T040000Z`
+
+Final rerun Wasabi prefix:
+
+`wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/diffusion_alignment/2026-07-17/nexa_mat_v32_shared_energy_pairbudget_eligible_20260717T040000Z`
+
+## Final Rerun Verdict
+
+The deterministic pair-budget fix removed the H100 memory blocker and produced the first decisive V32 scientific result. The true-label stability representation passed its staged gate at step 250, but the physical field and shared-energy controller failed. Controller training became non-finite at step 3, and its first finite update moved coordinates, lattice, and pair distances away from the observed relaxed trajectory. The native 64-CIF screen was therefore correctly blocked.
+
+This narrows the bottleneck to controller mechanics and higher-order differentiation through the conservative energy rollout. It is no longer a teacher-coverage, profile-balance, representation-separation, GPU-memory, or static-validity problem.
+
+### Final Gate Readout
+
+| Gate | Target | Result | Verdict |
+| --- | ---: | ---: | --- |
+| Cross-profile ROC-AUC | `>=0.70` | `0.70845` | Pass |
+| Minimum balanced accuracy | `>=0.55` | `0.60048` | Pass |
+| Force MAE | `<=0.20 eV/A` | `0.35973 eV/A` held out | Fail |
+| Force cosine | `>=0.50` | `0.23114` held out; `0.05131` controller step 1 | Fail |
+| Coordinate contraction | `>=25%` | `-2.161%` at controller step 1 | Fail |
+| Lattice contraction | Positive | `-2.050%` at controller step 1 | Fail |
+| Pair-distance contraction | Positive | `-1.399%` at controller step 1 | Fail |
+| Controller numerical stability | Finite | non-finite loss at step 3 | Fail |
+| Native 64-CIF screen | Mechanism pass required | Not run | Correctly blocked |
+
+The first representation evaluation contained `1,798` true-labeled validation rows (`353` positive, `1,445` negative). Every required profile passed the minimum row and class coverage checks. Leave-one-profile-out AUC ranged from `0.62017` for alloys/intermetallics to `0.81977` for battery ionics. This means the representation has a real but uneven cross-domain stability signal; it does not mean the model has learned a usable relaxation vector field.
+
+### Batch And Runtime Contract
+
+- Representation: `8,840` eligible rows; `3` oversized rows excluded.
+- Controller: `8,608` eligible rows; `232` rows excluded by the stricter four-step pair budget.
+- Validation: all `2,199` rows remained eligible.
+- Controller profile coverage remained above the `512`-row floor: alloys `1,616`, battery `1,837`, halides/chalcogenides `1,783`, oxides `2,098`, and semiconductors `1,274`.
+- Peak controller allocation at step 1 was only `2.229 GiB`, confirming that the pair-budget implementation solved the prior 80 GB padding failure.
+
+### Numerical Failure Diagnosis
+
+The controller uses forces and cell responses derived from gradients of one scalar energy, then backpropagates trajectory losses through four unrolled updates. This creates higher-order derivatives through periodic inverses, pair geometry, conservative force gradients, bounded strain, and the matrix-exponential cell update. The forward loss was finite through step 2, but PyTorch gradient clipping previously allowed non-finite gradients by default. A non-finite gradient could therefore corrupt an optimizer update and only appear as a non-finite forward loss on the next step.
+
+The trainer now calls `clip_grad_norm_(..., error_if_nonfinite=True)` and converts that failure into an explicit phase-and-step `FloatingPointError`. This prevents silent parameter corruption and will identify the first unstable backward pass. It does not make V32 scientifically promotable; the wrong-direction contraction and weak force direction existed before the numerical failure.
+
+### Final Closeout
+
+- The failed run receipt, manifests, teacher and batch audits, step-250 checkpoint, representation evaluation, controller pulse, CUDA telemetry, W&B metadata, logs, and failure context are present under the final Wasabi prefix.
+- W&B run: `https://wandb.ai/allanw-mk-none/Nexa_Mat/runs/cz40gh4p`.
+- The only H100 pod was terminated after object-storage verification.
+- Final Prime inventory: zero active pods.
+- Final local verification: `52` focused tests passed, Ruff passed, and Python compilation passed.
+
+### Stop Decision
+
+V32 is blocked at the shared-energy controller boundary. Do not launch another paid run from this exact controller. Any future continuation must first prove finite higher-order gradients and positive one-step coordinate, lattice, and pair contraction on a small local or CPU fixture, then reproduce those properties over four steps on held-out trajectory windows. The static-valid decoder and the step-250 representation checkpoint remain useful assets, but neither satisfies metastability generation.
+
+## Initial Attempt Verdict
+
+V32 is blocked operationally before its scientific mechanism gate. The corrected teacher contract and shared-energy implementation are ready, but the final H100 attempt exhausted memory before the first 250-step held-out representation evaluation. No controller phase or native CIF screen ran, so V32 does not establish improvement or regression in metastability.
+
+The verified blocker is row-count batching for a second-order pairwise energy model. A 16-row batch can be padded to a 136-atom maximum; the V19 force graph scales approximately with `batch_size * max_atoms^2`. Step 1, with a 52-atom maximum, completed and released CUDA allocation to `0.084 GiB`. The next large padded batch required more than the H100's 80 GB. This is a batching-contract failure, not evidence that the shared-energy architecture failed scientifically.
+
+## Completed Work
+
+- Built and audited `11,042` exact four-step trajectory windows: `8,843` train and `2,199` validation.
+- Passed source-disjointness, timestep/sigma consistency, bounded atomic step, bounded strain, positive-volume, profile-row, and per-class label-coverage gates.
+- Covered all required profiles: battery ionics, oxides, halides/chalcogenides, semiconductors, and alloys/intermetallics.
+- Implemented `metastable_manifold_v19` with one scalar energy surface, force and cell gradients, positive bounded mobility, symmetric matrix-exponential lattice updates, and shared four-step rollout semantics.
+- Implemented staged representation/controller training, conjunctive mechanism promotion, true-label checkpoint selection, gradient-allocation reporting, W&B logging, periodic Wasabi sync, failure receipts, and idle teardown protection.
+- Removed two real runtime defects: cached stream batches being moved to CUDA in place, and graph-bearing loop locals surviving into the next forward pass.
+
+## Scientific Gate Status
+
+The planned gates remain unmeasured:
+
+- true-label cross-profile ROC-AUC `>=0.70`: not evaluated;
+- minimum balanced accuracy `>=0.55`: not evaluated;
+- force MAE `<=0.20 eV/A`: not evaluated on held-out data;
+- force cosine `>=0.50`: not evaluated on held-out data;
+- coordinate contraction `>=25%`: controller phase did not run;
+- positive lattice and pair contraction per profile: controller phase did not run;
+- native 64-CIF relaxation/hull gate: correctly blocked.
+
+The only final-attempt training pulse was step 1 (`force_mae=0.2312 eV/A`, `force_cosine=0.3781`). It is a training-batch diagnostic and must not be reported as held-out model performance.
+
+## Final Experiment Evidence
+
+- Teacher set: `nexa_mat_v32_teacher_three_shard_20260716T134500Z`.
+- Initialization: preserved V29 `checkpoint_step_800.pt` lineage.
+- Final receipt: `status=failed`, `exit_code=1`, updated `2026-07-16T14:26:20Z`.
+- Wasabi contains `37` run objects, including all attempt logs, audits, manifests, W&B metadata, CUDA-memory telemetry, and failure receipts.
+- Prime pod `600780323dca4d6187c06acc6c8ec1ce` was terminated after sync.
+- Final Prime inventory: zero active pods.
+
+Local closeout verification passed: `49` focused materials tests, Ruff on the V32 implementation surfaces, Python compilation for materials modules/scripts, and shell syntax checks for both V32 remote launch paths. The only test warning was joblib falling back from physical-core detection to logical-core detection in a deterministic projection fixture.
+
+## Carry Forward
+
+Do not change the teacher labels, V19 energy controller, losses, or scientific gates before exercising the existing design. Replace row-count batching with deterministic atom-count buckets and a hard pair budget based on `batch_size * max_atoms^2`; use microbatches plus gradient accumulation to preserve the intended effective batch and profile/class balance. Apply the same budget to held-out evaluation. Add a preflight that computes the maximum padded pair cost and fails before GPU launch when it exceeds the configured H100 budget.
+
+After that single fix, rerun the representation phase from V29 step 800. Continue to the controller and native 64-CIF screen only if the original conjunctive gates pass. If representation then fails, classify V32 as a data/representation failure; if representation passes and contraction fails, classify it as controller mechanics. No architecture conclusion is justified from this stopped run.

eval/final_evidence/2026-07-24_foundation_residual_metastability_closeout.md

@@ -0,0 +1,104 @@
+# NexaMat Foundation-Residual Metastability Closeout
+
+Date: 2026-07-24  
+Verdict: **PLATEAU / ARCHITECTURE STOP**
+
+## Objective
+
+Prove that a frozen MACE-OMat conservative foundation plus a zero-initialized
+equivariant residual field can reach the force and contraction mechanism gates
+required before native metastability evaluation.
+
+Promotion required aggregate force cosine `>=0.90`, force MAE `<=0.20 eV/A`,
+weakest-profile cosine `>=0.75`, coordinate contraction `>=25%`, and positive
+lattice and pair-distance contraction in every profile. The bounded stop rule
+was two completed force attempts below `0.85` cosine.
+
+## Results
+
+| Surface | Attempt 1 | Attempt 2 | Gate | Verdict |
+|---|---:|---:|---:|---|
+| Aggregate force cosine, best | 0.75450 | 0.75322 | >=0.90 | Failed |
+| Aggregate force cosine, final | 0.75435 | 0.75217 | >=0.90 | Failed |
+| Force MAE, final (eV/A) | 0.05728 | 0.05762 | <=0.20 | Passed |
+| Weakest profile, final | battery 0.72470 | battery 0.72031 | >=0.75 | Failed |
+| Coordinate contraction | Not run | Not run | >=25% | Blocked |
+| Lattice/pair contraction | Not run | Not run | >0% | Blocked |
+| Native 64-CIF screen | Not run | Not run | External gate | Blocked |
+
+Attempt 1 used a zero residual gate. Its scale remained effectively zero and
+the held-out cosine stayed flat from step 1 through step 1500. Attempt 2 fixed
+that causal defect by zero-initializing the additive residual energy heads while
+starting the residual gate at `0.10`. This preserved the foundation output at
+initialization and raised the first-step gradient norm from `0.00012` to `2.50`.
+Despite the active gradient path, validation cosine remained between `0.75169`
+and `0.75322` through 1500 steps while validation loss stayed flat.
+
+## Interpretation
+
+The frozen foundation already models force magnitude well but does not meet the
+required trajectory-state direction contract. The residual branch can now
+receive and apply gradients, but it fits training batches without learning a
+source-disjoint correction that transfers across material families. More
+learning-rate or scale sweeps on this residual architecture are not justified.
+
+Ruled out:
+
+- missing or corrupt teacher/checkpoint hydration;
+- frozen-foundation mutation;
+- zero residual gradient as the sole failure;
+- force-magnitude error as the primary bottleneck;
+- lack of profile coverage or source-disjoint evaluation.
+
+Most likely remaining bottleneck:
+
+- the additive residual potential does not have an identifiable, transferable
+  cross-profile correction target on top of MACE-OMat for these trajectory
+  states; battery ionics remains the clearest failing slice.
+
+The next scientific decision is not another run of this architecture. Either
+use the foundation directly as the physical controller and train only a
+trajectory-conditioned mobility map against real paired displacement, or
+rebuild the residual teacher target explicitly as `DFT force - frozen
+foundation force` and prove that residual target is learnable source-disjoint
+before reconnecting it to the generator.
+
+## Four-GPU Optimization Grid
+
+After the bounded attempts, a user-authorized `2x2` grid used all four supplied
+GPUs to test whether the failure was recoverable through residual gate size or
+learning rate. Every arm used the same frozen cache, data split, seed, and
+500-step evaluation contract.
+
+| Initial residual scale | Learning rate | Final cosine | Final MAE | Weakest profile |
+|---:|---:|---:|---:|---|
+| 0.10 | 5e-4 | 0.75167 | 0.05783 | battery 0.71934 |
+| 0.10 | 1e-3 | 0.74940 | 0.05824 | battery 0.71783 |
+| 0.25 | 5e-4 | 0.75167 | 0.05852 | battery 0.72002 |
+| 0.25 | 1e-3 | 0.74926 | 0.05944 | battery 0.71426 |
+
+The grid rules out a simple optimization-envelope rescue. Higher learning rate
+consistently worsened held-out direction, and a larger residual gate did not
+improve transfer. No grid checkpoint is promotable.
+
+## Operations And Lineage
+
+- Provider: RunPod, user-supplied `4x RTX 4090` Ada node.
+- Topology: independent arms; no DDP or P2P assumption.
+- Canonical tmux owner: `nexa-mat-foundation-residual`.
+- W&B attempt 1: `x86sxjqw`.
+- W&B attempt 2: `82xn3nwd`.
+- W&B grid: `581cqfsr`, `u6mmr4aw`, `pi4zxmvi`, `xt8bw2sz`.
+- Code slice SHA-256: `c7175dc66040beeff17681549016a6ea25e9bd1533a97f063120754c755939bc`.
+- Attempt 1 Wasabi prefix:
+  `wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/diffusion_alignment/2026-07-24/foundation-residual-20260724-a1-r1`
+- Attempt 2 Wasabi prefix:
+  `wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/diffusion_alignment/2026-07-24/foundation-residual-20260724-a2`
+
+Both prefixes contain the run manifest, hydration receipt, foundation cache,
+all seven force checkpoints, metrics history, W&B reference, launch log, and
+closeout JSON. No mobility training, native generation, relaxation, hull, SUN,
+or MSUN work ran because the force gate failed. Each grid prefix under the same
+Wasabi date root contains its run manifest, three force checkpoints, metrics,
+W&B reference, and closeout JSON. The pod was not terminated;
+provider termination remains a user-controlled action.

eval/final_evidence/NEXA_MAT_ALIGNMENT_PIPELINE_KNOWLEDGE_PAPER.md

@@ -0,0 +1,269 @@
+# NexaMat Alignment Pipeline Knowledge Paper
+
+Date: 2026-06-28
+Status: alignment phase closed; no active training
+
+## Abstract
+
+NexaMat's diffusion decoder initially behaved like a low-yield chemistry prior rather than a production crystal generator. It could preserve allowed elements, novelty, and formula diversity, but it repeatedly failed corrected external-native hard-validity gates because the sampled structures did not land on the LeMat-radius-compatible crystal geometry manifold. The recovery campaign converted validity from an after-the-fact report into the primary control surface: native/no-repair accounting, raw candidate payloads, profile-conditioned sampler contracts, target-gated diagnostics, and profile-specific acceptance thresholds. The result is a competent native generator for the original SSB-style lane, strong transfer to battery, oxide, halide, semiconductor, and broad-inorganic profiles under the current production tolerance, and one unresolved weak profile: alloys/intermetallics.
+
+This document is the canonical top-to-bottom picture of the alignment pipeline, the failure mechanism, the redesign, the evidence, and the current operating contract.
+
+## Executive Read
+
+The final alignment result is a `5 / 5` static-native inorganic coverage pass across the final non-alloy panel: SSB, battery, oxide, halide, semiconductor, and broad inorganic all cleared their production/tolerance gates. This is not yet a public SOTA or SUN/stability claim. The decisive improvement was not another generic diffusion continuation. It was a benchmark-contract redesign: native candidate accounting, profile-conditioned element and charge policies, radius-aware composition-to-site assignment, best-of-N native rollout selection, and target-gated profile diagnostics.
+
+The phase should be read as a static native hard-validity recovery. It does not yet prove full relaxation survival, SUN/MSUN, or low hull energy across the full periodic table. MatterGen is still the relevant public bar for stable inorganic generation because it reports stable, unique, novel generation after DFT-style relaxation and broad conditional design. NexaMat's current differentiator is not beating that benchmark; it is a tightly audited native generation pipeline that recovered a brittle decoder and can now emit useful candidate pools without hiding repair behind the metric.
+
+## One-Picture Pipeline
+
+```mermaid
+flowchart TD
+    A["Canonical Materials Data"] --> B["Encoder and Diffusion Decoder Pretraining"]
+    B --> C["Recovered Native Checkpoint Lineage"]
+    C --> D["Profile-Conditioned Native Sampler"]
+    D --> E["Candidate JSONL with Raw Payloads"]
+    E --> F["Native No-Repair Scoring"]
+    F --> G["Profile Gate Summary"]
+    G --> H{"Gate Passed?"}
+    H -- "yes" --> I["Selected CIFs and Manifests"]
+    H -- "no" --> J["Failure-Reason Audit"]
+    J --> K["Targeted Code or Sampler Patch"]
+    K --> D
+    I --> L["Bounded Stability or App Candidate Lane"]
+    I --> M["Wasabi Artifact Closeout"]
+```
+
+The important invariant is that the promotion path is native/no-repair. Projected, scaffolded, relaxed, and stability-scored surfaces can be useful for debugging or downstream product use, but they do not prove native decoder progress unless separately reported.
+
+## Original Failure
+
+The model was not globally useless. The atom/composition lane was strong: allowed elements, novelty, uniqueness, and many charge patterns were recoverable. The hard failure was that the decoder placed too much probability mass in chemically plausible but geometrically invalid regions. The clearest early symptom was strict-filter candidate starvation: one diagnostic produced only a handful of benchmark-eligible CIFs when the downstream LeMat/SUN-style screen needed hundreds. That made ORB, MACE, relaxation, SUN, and hull scoring premature. They were not rejecting many good candidates; they were being starved before the real stability problem could even be measured.
+
+The root cause was the pretraining contract. Lattice behavior had been made too conservative. That helped convergence and atom resolution, but it under-trained global lattice scale, volume-per-atom variation, periodic pair distances, and element-radius-compatible geometry. The decoder learned chemistry more easily than it learned the crystal support.
+
+## Alignment Principle
+
+The campaign's central correction was simple: optimize the surface that matters. A run was not considered progress because loss improved, because a repair layer could save outputs, or because internal proxy metrics looked better. A run mattered only if corrected sampled native CIF density improved under the same accounting used by the benchmark.
+
+The final control contract became:
+
+- raw candidate payloads are persisted
+- native, projected, scaffolded, relaxed, and final-scored candidates remain separated
+- allowed-element, charge, strict geometry, LeMat-radius, novelty, and formula-diversity rates are recorded
+- checkpoint or sampler selection prefers external-native accepted density over generic validation loss
+- profile-specific gates define whether charge neutrality is a hard rejection rule or a diagnostic-only signal
+- every remote run syncs manifests, candidate JSONL, selected CIFs, logs, receipts, and closeout notes to Wasabi before compute is terminated
+
+## Main Innovations
+
+The first innovation is the **native accounting contract**. Earlier generations mixed raw, projected, scaffolded, relaxed, and stability-scored surfaces. That made it too easy to claim progress from a downstream repair layer. The aligned pipeline makes native/no-repair output the promotion surface and keeps every other surface explicitly labeled.
+
+The second innovation is the **profile-conditioned generator contract**. SSB is no longer a hidden spec. Each domain declares its own element universe, charge policy, radius policy, sampler knobs, thresholds, and stability-screen policy. Ionic profiles can require charge neutrality; covalent or alloy-like profiles can treat charge as diagnostic-only.
+
+The third innovation is **radius-aware native site assignment**. The core bottleneck was global lattice scale and LeMat-radius geometry coupling. The successful sampler therefore does not only choose legal atoms; it tries to place those atoms into radius-compatible periodic geometry while preserving atom count, formula counts, and native/no-repair accounting.
+
+The fourth innovation is **best-of-N native rollout selection with formula diversity**. The decoder had valid mass, but a single rollout was too noisy. Multiple native attempts plus formula-diverse selection exposed usable native candidates without pretending that scaffold repair was native model quality.
+
+The fifth innovation is **target-gated benchmarking**. Profiles stop when the evidence is sufficient. This saves compute, prevents broad sweeps from hiding profile-specific failure modes, and makes iteration a scalpel rather than a hammer.
+
+## Architecture
+
+### Data And Profile Layer
+
+The profile system moves domain assumptions out of hardcoded SSB defaults. Each profile defines an element universe, charge policy, radius policy, sampler settings, stability-screen policy, and thresholds. Current profiles include:
+
+- `ssb_regression`
+- `battery_ionics`
+- `semiconductors`
+- `oxide_catalysts`
+- `halide_chalcogenides`
+- `alloys_intermetallics`
+- `broad_inorganic`
+
+The profile layer is intentionally not a new model identity. SSB is one regression profile, not the decoder spec. The same decoder is evaluated under multiple profile contracts.
+
+### Native Sampler Layer
+
+The recovered native sampler does three jobs. First, it keeps atoms within the active profile's element universe. Second, it applies profile-aware charge or compatibility logic, using charge balancing for ionic profiles and non-ionic allowed-only handling for covalent/alloy-like profiles where forcing charge neutrality would be scientifically wrong. Third, it couples composition to site assignment using radius-aware geometry pressure so that selected atoms are more likely to land in LeMat-compatible pair-distance configurations.
+
+The strongest SSB recovery used multiple native rollout attempts and formula-diverse selection. This did not repair geometry after the fact. It exposed valid mass already present in the learned distribution and selected usable native samples from it.
+
+### Model And Training Dynamics
+
+The model retained useful chemistry priors from the original decoder. Atom identity, profile-compatible element selection, novelty, and formula variety were not the central failures by the end of the campaign. The weak mechanism was the continuous crystal lane: lattice scale, fractional-coordinate placement, periodic pair distances, and element-radius coupling.
+
+This matters because the decoder can look excellent internally while failing externally. A low training loss or high atom recovery rate does not imply a material scientist receives a usable crystal. The campaign therefore treated training metrics as debug signals and sampled external-native hard-valid density as the decision metric.
+
+The practical alignment path was to preserve the learned chemistry prior and reshape the sampler/training contract around geometry validity. That is why the winning lane used charge-balanced native assignment, radius-aware site placement, rollout attempts, and formula-diverse selection instead of only adding one more loss term.
+
+### Scoring Layer
+
+The scorer evaluates candidates under the active profile contract:
+
+- allowed elements
+- charge neutrality, when required
+- strict geometry
+- LeMat-radius distance validity
+- benchmark/filter acceptance
+- selected formula uniqueness
+- selected novelty
+
+The scoring patch at closeout fixed an important accounting issue: non-ionic profiles with `require_charge_neutral=false` no longer record `charge_not_neutral` as a failure reason. Acceptance already honored the profile; the patch made the JSONL audit surface match the actual filter.
+
+### Reporting And Closeout Layer
+
+Each run writes candidate JSONL, selected CIFs, manifests, profile gate summaries, parser summaries, receipts, logs, and closeout notes. Wasabi is the durable artifact store. The local repo is now a lightweight index and code surface, not the complete experiment store.
+
+## Benchmark Graphics
+
+The following graphics summarize the final six-profile static-native suite. They are local evidence graphics, not a public leaderboard claim. The unresolved alloy/intermetallic lane is excluded from these final production-tolerance plots and remains a separate future workstream.
+
+![NexaMat profile yield vs targets](../figures/nexamat_profile_yield_vs_targets.png)
+
+![NexaMat profile geometry rates](../figures/nexamat_profile_geometry_rates.png)
+
+![NexaMat inorganic readiness score](../figures/nexamat_inorganic_readiness_score.png)
+
+![NexaMat vs public generators qualitative matrix](../figures/nexamat_vs_public_generators_matrix.png)
+
+![Known-parameter size vs static native coverage](../figures/nexamat_size_performance_known_params.png)
+
+![Public generator evidence score comparison](../figures/nexamat_public_generator_evidence_scores.png)
+
+![Public generator evidence profile](../figures/nexamat_public_generator_evidence_profile.png)
+
+The model-comparison figures are intentionally split. The size/performance plot only uses recorded parameter counts for NexaMat and MatterGen. FlowMM, DiffCSP, and CDVAE are included in evidence-score and evidence-profile plots instead of assigning unverified parameter counts. NexaMat's `5 / 5` score here means final static-native inorganic coverage; SUN/MSUN, hull stability, and full relaxation are separate gates that remain unproven.
+
+## Evidence Timeline
+
+| Stage | Result | Meaning |
+| --- | --- | --- |
+| Early strict screen | only a few strict-filter candidates from thousands | downstream stability was starved |
+| V87 bottleneck grid | geometry improved when sampler pressure changed, charge could collapse | failure was composition/geometry coupling |
+| V88 radius-aware charge sampler | charge and LeMat behavior improved together | charge and geometry were not inherently opposed |
+| V88D rollout attempts | 148/1,024 accepted, 100% charge, 14.45% LeMat | valid native mass existed but needed rollout selection |
+| V89D-F | yield/diversity frontier narrowed | final problem became quota-aware selection |
+| V90 | 595/4,096 accepted, 14.53% accept, 100% charge, 54.20% strict, 100% novelty | corrected SSB native static gate passed |
+| 2026-06-24 profile continuation | SSB and battery held; semiconductors, oxide, halide partial; alloys failed | SSB mechanics transferred unevenly |
+| 2026-06-28 daily multi-domain | SSB 550, battery 418, oxide 414, halide 410 accepted before full sweep | target-gated multi-domain path worked |
+| 2026-06-28 remaining cleanout | semiconductors 250 pass, broad 233 tolerant pass, alloys 18 fail | alloys/intermetallic geometry remains unresolved |
+| 2026-06-28 final six-profile suite | SSB 513, battery 411, oxide 424, halide 436, semiconductor 256, broad 242 accepted | six non-alloy profiles passed native/no-repair gates |
+| 2026-06-28 relaxation retry | MatterSim runtime installed, but artifact restore exited before screens ran | relaxation remains unproven; static validity is the closed result |
+
+## Current Performance Snapshot
+
+The strongest native SSB evidence is V90:
+
+| Metric | Result |
+| --- | ---: |
+| accepted after filters | 595 / 4096 |
+| filter accept | 14.53% |
+| allowed elements | 100.00% |
+| charge neutrality | 100.00% |
+| LeMat-distance validity | 14.53% |
+| strict validity | 54.20% |
+| selected formula uniqueness | 41.80% |
+| selected novelty | 100.00% |
+
+The strongest final multi-domain closeout evidence from 2026-06-28 is the six-profile non-alloy static suite:
+
+| Profile | Raw Samples | Accepted | Target | Filter Accept | Charge | LeMat | Strict | Novelty | Formula Uniqueness |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| `ssb_regression` | 1,440 | 513 | 512 | 35.6% | 100.0% | 35.7% | 60.4% | 100.0% | 40.4% |
+| `battery_ionics` | 1,248 | 411 | 410 | 32.9% | 100.0% | 33.1% | 60.3% | 100.0% | 74.4% |
+| `oxide_catalysts` | 1,584 | 424 | 410 | 26.8% | 100.0% | 26.8% | 59.7% | 100.0% | 47.1% |
+| `halide_chalcogenides` | 2,688 | 436 | 410 | 16.2% | 100.0% | 16.3% | 55.1% | 100.0% | 62.1% |
+| `semiconductors` | 672 | 256 | 240 | 38.1% | diagnostic | 38.1% | 65.0% | 100.0% | 74.4% |
+| `broad_inorganic` | 2,688 | 242 | 230 | 9.0% | diagnostic | 9.0% | 55.1% | 100.0% | 76.4% |
+
+Aggregate verdict: `pass`. SSB regression passed, `5 / 4` required non-SSB profiles passed, median accepted after filters was `417.5`, and the minimum accepted count was `242`. The fair interpretation is: SSB is recovered; the non-alloy inorganic panel is production-tolerable under target-gated native diagnostics; alloys/intermetallics remain a distinct unsolved geometry lane outside this final suite.
+
+## Delta To Production Thresholds
+
+The practical inorganic production target used at closeout was `300 accepted / 4096`, with profile-specific tolerant thresholds when LeMat, strict geometry, novelty, and accounting were otherwise clean. The final six-profile suite used those profile-specific gates rather than the earlier fixed daily-cleanout target.
+
+| Profile | Accepted | Target | Delta vs Target | Delta vs 300 Production Goal | Verdict |
+| --- | ---: | ---: | ---: | ---: | --- |
+| `ssb_regression` | 513 | 512 | +1 | +213 | pass |
+| `battery_ionics` | 411 | 410 | +1 | +111 | pass |
+| `oxide_catalysts` | 424 | 410 | +14 | +124 | pass |
+| `halide_chalcogenides` | 436 | 410 | +26 | +136 | pass |
+| `semiconductors` | 256 | 240 | +16 | -44 | tolerant pass |
+| `broad_inorganic` | 242 | 230 | +12 | -58 | tolerant pass |
+
+This is why the static-native inorganic coverage score is now `5 / 5`: the final non-alloy static suite passed, including tolerant passes for semiconductor and broad inorganic. This should not be read as a SOTA claim because full relaxation, hull stability, SUN/MSUN, and the alloy/intermetallic lane are not solved by this closeout.
+
+## Comparison To Public Generators
+
+The closest public comparison set is not an exact apples-to-apples benchmark. MatterGen, CDVAE, DiffCSP, and FlowMM report public crystal-generation metrics under their own datasets, sampling policies, stability definitions, and relaxation workflows. NexaMat's archived claim is narrower: corrected profile-conditioned native/no-repair static validity.
+
+MatterGen is the strongest public reference point. Its Nature paper frames the problem as stable, diverse inorganic generation across the periodic table and reports that MatterGen structures are substantially more likely to be stable, unique, and new than prior generative baselines, with generated structures close to DFT local minima. CDVAE established diffusion/VAE-style periodic material generation. DiffCSP and FlowMM push equivariant diffusion and Riemannian flow matching for crystal generation. NexaMat should be positioned as an internally recovered and audited native generator, not as a proven SOTA stable generator.
+
+The parameter-scale comparison is the clearest efficiency win. MatterGen's public model card reports `46.8M` parameters. NexaMat's archived stack reports `4.061859M` total parameters and `1.248145M` trainable diffusion-decoder parameters. That makes MatterGen about `11.5x` larger than the full NexaMat stack and about `37.5x` larger than the NexaMat generator head.
+
+Because NexaMat did not run the same full public stability/SUN benchmark, the honest comparison must separate static-native coverage from stability evidence. On static-native inorganic coverage, MatterGen and NexaMat are both shown at `5.0 / 5`: MatterGen as the public SOTA reference, and NexaMat because the final non-alloy static suite passed all six profile gates. On stability evidence, MatterGen remains ahead because it reports SUN/stability evidence and NexaMat does not yet have a completed SUN/MSUN pass. At `4.061859M` parameters versus MatterGen's `46.8M`, NexaMat reaches the static coverage bar at only `8.7%` of the parameter count. That is the efficiency win. It is not a measured NexaMat SUN result.
+
+MatterGen still has stronger public stability evidence, but NexaMat's performance-per-parameter is unusually strong: the aligned stack became a credible native inorganic generator at roughly one-tenth the model scale.
+
+| Model family | Public strength | NexaMat read |
+| --- | --- | --- |
+| MatterGen | stable, diverse inorganic generation with broad conditioning and SUN-style evaluation | above NexaMat on public stability evidence |
+| FlowMM | flow-matching crystal generation with strong geometric modeling direction | relevant architecture bar |
+| DiffCSP | joint equivariant diffusion for crystal structure prediction | relevant diffusion/geometry bar |
+| CDVAE | early periodic material diffusion/VAE generation baseline | useful historical baseline |
+| NexaMat aligned | audited native/no-repair profile-conditioned candidate generation | strong internal recovery; stability still to prove |
+
+## What Worked
+
+Native accounting worked. The pipeline became auditable once candidate stages were separated and raw payloads were retained.
+
+Profile conditioning worked. SSB stopped being silently hardcoded into every domain, and non-ionic profiles stopped being punished for failing an ionic charge assumption.
+
+Radius-aware native site assignment worked for oxide, halide, battery, and SSB. It directly attacked global lattice scale and LeMat-radius geometry coupling.
+
+Best-of-N native rollout selection worked. It increased yield without claiming scaffold repair as decoder progress.
+
+Target-gated benchmarks worked. They reduced paid runtime by stopping once a profile had enough evidence, and they exposed failure modes earlier when projected accepted count fell below the useful threshold.
+
+## What Did Not Fully Work
+
+Alloys/intermetallics did not clear the mini-goal. The failure is not allowed elements or charge. It is LeMat-radius-compatible geometry for metallic/intermetallic compositions. Those systems likely need a different profile contract: metallic radii, packing priors, lattice prototypes, or alloy-specific candidate source/template logic.
+
+The decoder is still not a uniformly high-yield generator. The strongest lanes are good enough for production-style candidate filtering, but not all domains are equal and not every generated sample is valid.
+
+The current evidence is mostly static hard-validity. Stability, relaxation, e_above_hull, SUN/MSUN, and application-specific downstream utility should be treated as follow-on screens, not already solved by this alignment phase.
+
+## Limits
+
+The first limit is stability evidence. Static native validity is necessary but insufficient. A candidate that passes allowed elements, charge policy, strict geometry, and LeMat-radius checks may still fail relaxation or hull screening.
+
+The second limit is domain unevenness. Oxide, halide, battery, semiconductor, and broad inorganic behavior is good enough under current tolerance. Alloys/intermetallics are not. They probably require metallic radii, packing priors, prototype-aware sampling, or a separate alloy-specific source/template path.
+
+The third limit is yield. The model is now useful, but it is not a uniformly high-yield generator. Best-of-N rollout and target-gated selection are still doing meaningful work.
+
+The fourth limit is comparability. The public literature often reports relaxed stability or SUN under specific datasets and reference hulls. NexaMat's closeout reports native static validity under profile-conditioned gates. Those should not be conflated.
+
+## Robustness Contract Going Forward
+
+Any future claim about NexaMat should state which surface it came from:
+
+- native/no-repair
+- profile-conditioned native
+- projected
+- scaffolded
+- relaxed
+- stability-scored
+
+Claims should include the profile, checkpoint lineage, candidate count, early-stop settings, rollout attempts, selected CIF count, novelty, formula uniqueness, Wasabi prefix, and whether charge was a hard gate or diagnostic-only.
+
+## Final Verdict
+
+The alignment phase succeeded at recovering NexaMat from a low-yield native geometry failure into a usable native profile-conditioned generator. It did not make the model universal across all inorganic chemistry. The next honest milestone is not more generic realignment. It is either downstream stability screening for the passing profiles or a separate alloy/intermetallic geometry program.
+
+## References
+
+- MatterGen: [A generative model for inorganic materials design](https://www.nature.com/articles/s41586-025-08628-5)
+- CDVAE: [Crystal Diffusion Variational Autoencoder for Periodic Material Generation](https://arxiv.org/abs/2110.06197)
+- DiffCSP: [Crystal Structure Prediction by Joint Equivariant Diffusion](https://arxiv.org/abs/2309.04475)
+- FlowMM: [Generating Materials with Riemannian Flow Matching](https://arxiv.org/abs/2406.04713)

eval/final_evidence/NEXA_MAT_STABILITY_SELECTOR_CALIBRATED_CLOSEOUT_2026-07-01.md

@@ -0,0 +1,413 @@
+# NexaMat Stability Selector Calibrated Closeout
+
+Date: 2026-07-01
+
+Run: `nexa_mat_stability_selector_calibrated_fixed_20260701T094441Z`
+
+Wasabi prefix: `wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/diffusion_alignment/2026-07-01/nexa_mat_stability_selector_calibrated_fixed_20260701T094441Z`
+
+## Verdict
+
+Miss, but the failure is now cleanly isolated. The code-side starvation issue was fixed: the bounded external screen evaluated `512` selected native/no-repair CIFs across `4` inorganic profiles, with `128` selected per profile and `0` dropped by max-element filtering. Static behavior remained strong after relaxation, but the stability target failed: only `58 / 512` candidates converged under the short relaxation screen, weighted mean evaluated `e_above_hull` was `1.2330 eV/atom`, and stable/M.S.U.N. count remained `0`.
+
+## Gate Result
+
+| Metric | Target | Result | Verdict |
+| --- | ---: | ---: | --- |
+| Selected native/no-repair CIFs | `>=512` | `512` | pass |
+| Profiles screened | `>=3` | `4` | pass |
+| Short-relax convergence | `>=50%` | `11.33%` (`58 / 512`) | fail |
+| Post-relax validity | `>=90%` | `96.88-100%` by profile | pass |
+| Mean `e_above_hull` | `<=0.75 eV/atom` | `1.2330 eV/atom` weighted over MatterGen-evaluated rows | fail |
+| Stable candidates | `>=1` practical evidence | `0` | fail |
+| M.S.U.N. candidates | `>=26 / 512` | `0 / 512` | fail |
+| Active pods after closeout | `0` | `0` verified by Prime | pass |
+
+## Profile Results
+
+| Profile | Converged | Convergence | Post-relax benchmark-valid | Mean hull | Stable | M.S.U.N. |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| `battery_ionics` | `7 / 128` | `5.47%` | `97.66%` | `1.3786` | `0` | `0` |
+| `broad_inorganic` | `10 / 128` | `7.81%` | `96.88%` | `1.1083` | `0` | `0` |
+| `oxide_catalysts` | `8 / 128` | `6.25%` | `100.00%` | `1.1198` | `0` | `0` |
+| `semiconductors` | `33 / 128` | `25.78%` | `100.00%` | `1.4124` | `0` | `0` |
+
+## What Changed
+
+- Added a hard minimum selected-CIF guard to the bounded relaxability screen.
+- Fixed observed-label matching so profile-scoped labels can match full CIF paths without leaking bare basenames.
+- Removed stale `stability_prior_scores.jsonl` hydration from the remote runner.
+- Forwarded `RELAXABILITY_MIN_SELECTED_CIFS` through the launch path.
+- Removed the max-elements default that starved `battery_ionics`.
+
+## Interpretation
+
+The prior run failed because the selector could not even assemble a meaningful pool. This run proves that issue is fixed. The current failure is not static validity, charge, allowed elements, or hidden repair accounting. The selected CIFs remain static-valid after relaxation at high rates, but most do not enter a low-force basin and the structures that MatterGen can hull-score are still too high above hull. The next bottleneck is therefore stability-basin modeling and ranking, not native static validity.
+
+Semiconductors are the only profile showing useful relaxability signal (`25.78%` convergence), but their hull score is still poor. Battery, broad inorganic, and oxides preserve geometry after relaxation yet rarely converge. This suggests the selector is choosing candidates that are geometrically legal but not close enough to stable local minima.
+
+## Next Experiment
+
+Do not run another broad static-validity continuation. Build a stability-basin selector/ranker calibration pass from the completed `512`-CIF screen:
+
+1. Train or tune a selector on the actual convergence, residual-force, stress, volume-per-atom, radius-margin, profile, formula-complexity, and hull outcomes from this run plus prior labeled screens.
+2. Use a cheap pre-relax scoring stage to predict convergence probability before spending full MatterSim relaxation.
+3. Select by expected low-hull relaxability, not just static validity or formula diversity.
+4. Re-run the bounded screen only when a `64`-CIF probe reaches `>=30%` convergence and improves mean hull versus this run.
+
+Promotion target remains unchanged: `>=26 / 512` M.S.U.N. candidates, `>=50%` convergence, post-relax validity `>=90%`, and mean `e_above_hull <=0.75 eV/atom`.
+
+## Tomorrow: Stability-Basin Selector Pass
+
+The next pass should stop optimizing static validity and target the metastability gap directly. NexaMat can emit valid native CIFs, but the selected structures are not close enough to low-force, low-hull basins. Build a calibrated stability selector from completed relaxation and hull evidence, use it to rank native/no-repair candidates before full relaxation, and only scale when a small probe proves better convergence and lower hull energy.
+
+Implementation order:
+
+1. Build `stability_basin_selector_labels.jsonl` from completed bounded relaxability summaries and MatterGen metrics. Each row must preserve full CIF path, profile-scoped CIF key, profile, formula, convergence, post-relax validity, force/stress, hull, stable/metastable, and M.S.U.N. fields.
+2. Add a `selection_policy=stability_basin` path to the bounded relaxability screen. It should rank by observed stability-basin outcome first, then expected low force/stress, low hull bin, and static prior score. It must not scaffold, project, repair, or relabel native validity.
+3. Run a `64`-CIF probe before any `512` screen. Promote only if the probe reaches `>=30%` convergence, post-relax validity `>=90%`, and mean hull below this run's `1.2330 eV/atom`.
+4. If the probe passes, run the `512` selected-CIF screen and evaluate against the unchanged stability target: `>=26 / 512` M.S.U.N. candidates, `>=50%` convergence, post-relax validity `>=90%`, and mean `e_above_hull <=0.75 eV/atom`.
+
+For the metastable-only hunt, disable formula diversity during the probe so ranking is purely low-hull-basin directed: set `RELAXABILITY_SELECTION_POLICY=stability_basin`, `RELAXABILITY_STABILITY_BASIN_DIVERSITY_POLICY=none`, `RELAXABILITY_MAX_CIFS=64`, and point `RELAXABILITY_STABILITY_BASIN_LABELS_URI` at the calibrated fixed run prefix above. Formula-diverse selection remains available as `formula_first` for later product screens after at least one metastable candidate is externally verified.
+
+Default profile priority for the probe is semiconductors first, then broad inorganic, oxide catalysts, and battery ionics. Semiconductors showed the strongest relaxability signal (`25.78%`), but they still need low-hull filtering, so they should not be treated as solved.
+
+Code safeguards:
+
+- Label matching must remain profile-scoped; never match by bare CIF basename.
+- Native, projected, scaffolded, relaxed, and hull-scored accounting must remain separate.
+- Static guardrails remain active: allowed/profile-compatible `>=98%`, charge `>=95%` where applicable, uniqueness `>=85%`, and novelty `>=85%`.
+- If the `64`-CIF probe misses the probe gate, do not run the expensive `512` screen; close out with the dominant failure mode and update the selector.
+
+## 2026-07-02 SSB 512 Scale Attempt
+
+Run: `nexa_mat_msun_ssb512_20260702T032455Z`
+
+Wasabi prefix: `wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/diffusion_alignment/2026-07-02/nexa_mat_msun_ssb512_20260702T032455Z`
+
+This run tested whether the calibrated `stability_basin` selector could scale directly to a corrected native/no-repair `512`-CIF SSB screen using the archived V78 static source. The screen successfully assembled the full `512` selected-CIF pool from native static outputs with `0` candidates dropped by element-count limits, and it used the fresh metastable `64`-CIF probe labels as preferred selector evidence. It was stopped early after `88` processed CIFs because the best-first selected prefix converged only `10 / 88 = 11.36%`, far below the `50%` practical gate. Since this selector orders candidates by its strongest evidence first, continuing the remaining `424` CIFs was unlikely to rescue the gate and would have spent compute on a low-information failure.
+
+| Metric | Target | Observed at stop | Verdict |
+| --- | ---: | ---: | --- |
+| Selected native/no-repair CIF pool | `512` | `512` | pass |
+| Processed before stop | checkpoint | `88 / 512` | evidence sufficient |
+| Short-relax convergence | `>=50%` | `10 / 88` (`11.36%`) | fail |
+| M.S.U.N. recovery gate | `>=26 / 512` | not evaluated; relaxation stopped before hull pass | fail |
+| Active pods after closeout | `0` | `0` verified by Prime | pass |
+
+The interpretation is now sharper: static validity and candidate accounting are not blocking the SSB scale screen, but the current selector is not actually ranking structures into relaxable basins. The previous oxide `64`-CIF probe found one metastable candidate, so the project should keep that positive evidence, but it should not extrapolate that hit into a broad `512` run without a selector that shows materially better convergence in the ranked prefix.
+
+Next action: stop direct `512` scaling from this selector. Build the next selector/training iteration around actual relaxability labels, with an explicit ranked-prefix gate: a profile cannot promote to `512` unless its first `64` selected CIFs reach at least `30%` convergence and improve mean hull below the current selector baseline. If the goal is M.S.U.N. count versus SOTA, the immediate recovery target remains `>=26 / 512`, but the next experiment should first prove the selector can rank the first `64` into a low-force basin.
+
+## 2026-07-02 Oxide 64 Prefix-Gated Probe And Rejected-CIF Inspection
+
+Run: `nexa_mat_msun_oxide64_prefixgate_20260702T041308Z`
+
+Wasabi prefix: `wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/diffusion_alignment/2026-07-02/nexa_mat_msun_oxide64_prefixgate_20260702T041308Z`
+
+The automated ranked-prefix gate worked as intended. The oxide probe processed the first `64 / 64` selected CIFs, stopped after the prefix failed the `>=30%` convergence gate, synced artifacts, and terminated the pod. Static post-relax validity remained perfect, but metastability still failed: `7 / 64` converged (`10.94%`), mean `e_above_hull` improved only slightly to `1.0830 eV/atom`, and stable/metastable/M.S.U.N. counts remained `0`.
+
+| Metric | Probe Gate | Result | Verdict |
+| --- | ---: | ---: | --- |
+| Processed selected CIFs | `64` | `64` | pass |
+| Short-relax convergence | `>=30%` | `7 / 64` (`10.94%`) | fail |
+| Post-relax benchmark validity | `>=90%` | `64 / 64` (`100%`) | pass |
+| Mean `e_above_hull` | `<1.11996 eV/atom` baseline | `1.0830 eV/atom` | weak pass |
+| Stable candidates | `>=1` | `0` | fail |
+| M.S.U.N. candidates | `>=1` probe | `0` | fail |
+
+One rejected CIF was inspected directly: `nexamat_rank_00018_sample_000200.cif`, formula `ZnCu(Mo2O7)2` (`Zn1 Cu1 Mo4 O14`). It passed native/static scoring after sampler projection and remained post-relax benchmark-valid, charge-neutral, and LeMat-radius-valid. It failed metastability because MatterSim did not converge within `1200` steps, final max force remained about `0.184 eV/A`, stress norm about `0.806`, and MatterGen hull was about `1.151 eV/atom`, far above the metastable threshold. The important audit finding is provenance: the candidate was a projection-heavy composition transplant. The native sampler metadata showed the original atoms were not allowed/profile-compatible and not charge-neutral, the template/source lineage was `GePd6Sb3` / `Ge2Pd12Sb6`, and `9 / 20` sites were changed to produce a charge-balanced oxide. Static validity therefore hid the real problem: the geometry was locally legal but not a coherent low-energy oxide basin.
+
+Code response: the bounded relaxability selector now loads sibling `candidate_scores.jsonl` metadata for selected CIFs, extracts native sampler provenance by `sample_index`, records projection-risk features, and penalizes unlabeled or failed projection-heavy transplants during `stability_prior` and `stability_basin` ranking. The next training/selection pass should use these rows as hard-negative evidence: teach the model to emit native composition-geometry pairs that are already basin-coherent, not just candidates that can be made static-valid by atom rewrites.
+
+Next action: rerun only a `64`-CIF oxide or semiconductor probe with the projection-risk selector enabled. Do not scale to `512` until the ranked prefix clears `>=30%` convergence and produces at least one metastable or M.S.U.N. candidate. If the projection-risk selector improves convergence but still misses metastability, the training set should be rebuilt around low-hull/metastable positives versus projection-heavy high-hull negatives.
+
+## 2026-07-02 Projection-Risk Selector Probe
+
+Run: `nexa_mat_msun_oxide64_projectionrisk_20260702T045948Z`
+
+Wasabi prefix: `wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/diffusion_alignment/2026-07-02/nexa_mat_msun_oxide64_projectionrisk_20260702T045948Z`
+
+This run repeated the oxide `64`-CIF probe after adding selector access to `candidate_scores.jsonl` and penalizing projection-heavy composition transplants. The operational path worked: the pod launched, hydrated the V78 static source, used `selection_policy=stability_basin`, selected `64 / 512` candidates, wrote the ranked-prefix early-stop payload, synced artifacts, and terminated the pod. The scientific gate still failed.
+
+| Metric | Probe Gate | Result | Verdict |
+| --- | ---: | ---: | --- |
+| Selected native/no-repair CIFs | `64` | `64` | pass |
+| Short-relax convergence | `>=30%` | `7 / 64` (`10.94%`) | fail |
+| Post-relax benchmark validity | `>=90%` | `64 / 64` (`100%`) | pass |
+| Mean `e_above_hull` | `<1.0830 eV/atom` prior probe | `1.0809 eV/atom` | negligible pass |
+| Stable candidates | `>=1` | `0` | fail |
+| Metastable candidates | `>=1` | `0` | fail |
+| M.S.U.N. candidates | `>=1` probe | `0` | fail |
+
+Selector audit: the patch successfully exposed the hidden provenance problem, but it also proved ranking alone is insufficient for oxides. All `512 / 512` oxide pool rows still had native composition provenance issues before projection, meaning the native atom/composition head is not naturally emitting charge-neutral oxide-basin candidates. The selected top `64` reduced the average projection-risk penalty versus the full pool, but every selected candidate still depended on native charge correction, and all selected rows remained high-hull after relaxation. The best hull value was about `0.603 eV/atom`, still above the metastable threshold, and the mean remained about `1.081 eV/atom`.
+
+Conclusion: this is no longer a selector-only failure. The selector can avoid the worst transplants, but the candidate pool itself is not populated with native low-hull oxide basin structures. The next realignment step should train on this exact distinction: positive rows are low-hull/metastable or near-metastable structures from historical labels and any recovered probe hits; hard negatives are static-valid but projection-heavy, nonconvergent, high-hull rows. The target is native composition-geometry coherence before projection, especially native charge-balanced oxide formulas and lattice/site patterns that relax toward low hull.
+
+Immediate next experiment: build a metastable-basin teacher/replay table from `candidate_scores.jsonl`, `stability_basin_selector_scores.jsonl`, bounded relaxation summaries, and MatterGen hull metrics. Then run a short native atom/composition plus geometry continuation, not another pure selector probe. The continuation should optimize native charge-balanced composition, reduced projection changed-site fraction, low initial-force proxy, and low-hull/metastable ranking. A new `64`-CIF probe should only scale to `512` if it reaches `>=30%` convergence and at least one metastable/M.S.U.N. candidate.
+
+Derived teacher proof: `wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/diffusion_alignment/2026-07-02/nexa_mat_msun_oxide64_projectionrisk_20260702T045948Z/derived/metastable_basin_teacher_probe`
+
+The stable-manifold teacher builder was patched after this probe to avoid ambiguous label joins. Candidate rows use `sample_index`; relaxation rows often use only CIF path plus `structure_index`. The builder now parses `_sample_XXXXXX` from CIF filenames and no longer matches candidate `sample_index=0` to relaxation `structure_index=0` by accident. Rebuilding the probe teacher table produced `1,584` candidate rows, exactly `64` relaxation/hull-labeled rows, `64` trainable hull-energy rows, `1,226` hard negatives, `358` near positives, and `0` positives. This is the correct training signal for the next pass: the oxide pool contains many static-valid structures, but the externally evaluated selected prefix contains no low-hull positives and many projection-heavy hard negatives.
+
+## 2026-07-02 Stable-Manifold Oxide Training Probe
+
+Run: `nexa_mat_msun_stablemanifold_oxide_train_20260702T054825Z`
+
+Wasabi prefix: `wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/diffusion_alignment/2026-07-02/nexa_mat_msun_stablemanifold_oxide_train_20260702T054825Z`
+
+This pass tested the new premise directly: if the model needs to emit metastable materials, the continuation must train on metastable/near-metastable basin evidence rather than only static validity. The run built a broad stable-manifold teacher set from prior native static candidates, relaxation outcomes, MatterGen hull labels, and projection-risk hard negatives, then trained a short `2,000`-step native continuation from the recovered V78 lineage before running an oxide native diagnostic and `64`-CIF bounded relaxation screen.
+
+Teacher evidence was materially better than the previous oxide-only proof: `10,320` teacher rows, `402` positives, `1,358` near positives, `8,560` hard negatives, `626` hull-labeled rows, `2,108` relaxation-labeled rows, and non-SSB positive/near-positive coverage across battery ionics, broad inorganic, halides/chalcogenides, oxide catalysts, and semiconductors. The label join fix stayed active, so candidate provenance and relaxation/hull labels were not joined by ambiguous bare `structure_index`.
+
+| Metric | Probe Gate / Target | Result | Verdict |
+| --- | ---: | ---: | --- |
+| Native raw samples | diagnostic | `1,024` | pass |
+| Native accepted after filters | `>=410` formal profile target | `287 / 1,024` | fail formal target, pass probe sufficiency |
+| Native filter accept | `>=10%` | `28.03%` | pass |
+| Native charge neutrality | `>=95%` | `100%` | pass |
+| Native strict validity | `>=45%` | `62.30%` | pass |
+| Native LeMat-distance validity | `>=12%` | `28.03%` | pass |
+| Selected native CIFs | `>=64` for probe | `128` | pass |
+| Relaxed/evaluated selected CIFs | `64` | `64` | pass |
+| Short-relax convergence | `>=30%` probe | `3 / 64` (`4.69%`) | fail |
+| Post-relax benchmark validity | `>=90%` | `64 / 64` (`100%`) | pass |
+| Mean `e_above_hull` | below prior `1.0809 eV/atom` preferred | `0.8750 eV/atom` | pass |
+| Metastable candidates | `>=1` probe | `2 / 64` | pass |
+| M.S.U.N. candidates | `>=1` probe | `2 / 64` | pass |
+
+Interpretation: this was the first clear evidence that training on the stability basin moves the model in the right scientific direction. The previous projection-risk selector probe found `0 / 64` metastable and mean hull around `1.0809 eV/atom`; this pass recovered `2 / 64` metastable/M.S.U.N. candidates and reduced mean hull to `0.8750 eV/atom`. That is not enough for the practical `26 / 512` recovery gate, but it breaks the zero-metastable loop and validates the decision to train on metastable/near-metastable labels instead of running more static-validity-only continuations.
+
+The remaining bottleneck is now force-basin convergence. Only `3 / 64` candidates converged at `fmax=0.05`, even though `19 / 64` reached final force `<=0.25` and `39 / 64` reached `<=0.50`. In other words, the selected structures are closer to useful low-hull regions, but the native generator is still not consistently emitting low-force basin interiors. The next pass should keep the stable-manifold teacher, increase direct force/residual supervision from relaxation-labeled rows, use the near-force rows as positive curriculum examples, and select checkpoints by `metastable_count`, mean hull, and convergence together. Do not return to a pure selector-only run unless the candidate pool already contains enough low-force positives.
+
+## 2026-07-02 Force-Basin Scalar Supervision Probe
+
+Run: `nexa_mat_msun_forcebasin_oxide_train_20260702T073329Z`
+
+Wasabi prefix: `wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/diffusion_alignment/2026-07-02/nexa_mat_msun_forcebasin_oxide_train_20260702T073329Z`
+
+This pass added explicit relaxation-outcome supervision on top of the stable-manifold teacher: convergence classification, log max-force regression, and log stress-norm regression. The code path worked: scalar relaxation labels were exposed by the teacher builder, collated into `MaterialBatch`, trained through new decoder heads, synced to Wasabi, and the secure H100 pod terminated with zero active pods remaining. The new heads resumed cleanly from the recovered V78 lineage with expected missing-key initialization for the added heads.
+
+| Metric | Probe Gate / Target | Result | Verdict |
+| --- | ---: | ---: | --- |
+| Native raw samples | diagnostic | `1,024` | pass |
+| Native accepted after filters | `>=410` formal profile target | `277 / 1,024` | fail formal target, pass probe sufficiency |
+| Native filter accept | `>=10%` | `27.05%` | pass |
+| Native charge neutrality | `>=95%` | `100%` | pass |
+| Native strict validity | `>=45%` | `62.40%` | pass |
+| Native LeMat-distance validity | `>=12%` | `27.05%` | pass |
+| Selected native CIFs | `>=64` for probe | `128` | pass |
+| Relaxed/evaluated selected CIFs | `64` | `64` | pass |
+| Short-relax convergence | `>=30%` probe | `4 / 64` (`6.25%`) | fail |
+| Post-relax benchmark validity | `>=90%` | `64 / 64` (`100%`) | pass |
+| Mean `e_above_hull` | below prior `0.8750 eV/atom` preferred | `0.8165 eV/atom` | pass |
+| Metastable candidates | `>=1` probe | `2 / 64` | pass |
+| M.S.U.N. candidates | `>=1` probe | `2 / 64` | pass |
+
+Interpretation: the model is now learning from metastability-adjacent labels, and the result confirms the direction without clearing the gate. Mean hull improved again, from `0.8750` to `0.8165 eV/atom`, and the run preserved `2 / 64` metastable/M.S.U.N. hits. However, convergence remains the blocker: only `4 / 64` structures converged after `1200` MatterSim steps, so the selected candidates are still mostly near lower-hull regions rather than inside low-force relaxation basins. This is not a static-validity problem and not an SSB regression problem; it is a native force-basin emission problem.
+
+The operational audit also exposed a benchmark-loop issue: hydrated relaxation label paths are recursively rediscovered and passed as hundreds of `--observed-relaxability-labels` arguments. The earlier environment-size failure was fixed by not exporting those large path lists, but the next code cleanup should replace recursive label-path spray with a compact curated label manifest. That will reduce command-line bloat, sync noise, and relaxation-loop overhead without changing benchmark semantics.
+
+Next action: build a cleaner metastable teacher curriculum where positives are explicitly `converged + low hull` or `near-force + low hull`, hard negatives are `static-valid + nonconverged + high hull`, and neutral examples are static-valid rows without external stability evidence. Then run a short continuation that selects by the joint key `convergence_rate`, `metastable_count`, `mean_e_above_hull`, and static non-regression. Do not scale to `512` until a `64`-CIF probe reaches at least `30%` convergence while preserving at least one metastable/M.S.U.N. candidate.
+
+## 2026-07-02 MSUN Curriculum Oxide Probe Early Closeout
+
+Run: `nexa_mat_msun_curriculum_oxide_probe_20260702T091746Z`
+
+Wasabi prefix: `wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/diffusion_alignment/2026-07-02/nexa_mat_msun_curriculum_oxide_probe_20260702T091746Z`
+
+This pass tightened the teacher role assignment so static-valid rows alone no longer count as metastability positives. Positives required low-hull plus converged or near-force evidence, while hard negatives captured static-valid structures that failed relaxation or sat too high above hull. The stable-manifold teacher gate passed with `10,320` rows: `384` positives, `8,582` hard negatives, and `1,354` unlabeled context rows across battery ionics, broad inorganics, halides/chalcogenides, oxide catalysts, and semiconductors. The H100 continuation trained for `3,000` steps from the recovered V78/V90 native lineage and wrote checkpoints at `1,000`, `2,000`, `3,000`, plus best/final checkpoints.
+
+| Metric | Probe Gate / Target | Result | Verdict |
+| --- | ---: | ---: | --- |
+| Native raw samples | diagnostic | `1,024` | pass |
+| Native accepted after filters | `>=128` for probe | `282 / 1,024` | pass |
+| Native filter accept | `>=10%` | `27.54%` | pass |
+| Native charge neutrality | `>=95%` | `100%` | pass |
+| Native strict validity | `>=45%` | `62.40%` | pass |
+| Native LeMat-distance validity | `>=12%` | `27.54%` | pass |
+| Selected native CIFs | `>=64` for probe | `128` | pass |
+| Short-relax convergence | `>=30%` probe | partial `2 / 22` (`9.09%`) | fail / unfinished |
+| Metastable or M.S.U.N. candidates | `>=1` | not available | unfinished |
+
+Interpretation: static native generation remained solid, so the model did not regress the repaired native-valid lane. The decisive signal is still relaxation: the bounded MatterSim/MatterGen probe was CPU-bound and had only completed one visible batch by the user-requested `19:30 JST` wrap. That batch converged only `2 / 22` candidates, below the `30%` probe gate, and no final hull/M.S.U.N. summary was available before shutdown. This run therefore cannot be claimed as a metastability pass even though the static diagnostic was healthy.
+
+Closeout: artifacts were copied to the Wasabi prefix above, an early-closeout note was written into the remote run root, and the secure-cloud H100 pod `523d0f5b57314fb4bce5362d82e67136` was terminated. `prime pods list` returned zero active pods after shutdown.
+
+Carry-forward: the next useful change is not another static-valid continuation. Add streaming per-CIF relaxation outputs so interrupted screens preserve every completed CIF, shorten the first probe to `16-32` stratified candidates while tuning the selector, and train/rank against explicit `converged + low hull` and `near-force + low hull` labels. The selector should be judged by convergence, metastable/M.S.U.N. count, and mean hull together; static validity remains only a guardrail.
+
+## 2026-07-02 Force-Basin Probe32 Plumbing Closeout
+
+Run: `nexa_mat_msun_forcebasin_probe32_20260702T120228Z`
+
+Wasabi prefix: `wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/diffusion_alignment/2026-07-02/nexa_mat_msun_forcebasin_probe32_20260702T120228Z`
+
+This was the final H100 attempt for the session. It did not reach the native diagnostic or relaxation screen, but it resolved the most important upstream plumbing failure. The teacher builder now prioritizes candidate-score evidence before broad Parquet rows, the launcher no longer suppresses stability-label defaults with an empty `STABLE_MANIFOLD_LABEL_URI_LIST`, and checkpoint caching is keyed by a URI hash so same-basename checkpoints cannot collide.
+
+The stable-manifold evidence gate finally passed:
+
+| Evidence | Result |
+| --- | ---: |
+| Teacher rows | `200,000` |
+| Positive / near-positive rows | `378` |
+| Required positive / near-positive rows | `250` |
+| Non-SSB positive/near profiles | `5` |
+| Required non-SSB positive/near profiles | `3` |
+| Hull-labeled rows | `567` |
+| Relaxation-labeled rows | `1,985` |
+| Trainable energy-labeled rows | `2,347` |
+| Force-vector labeled rows | `189,680` |
+| Stress-tensor labeled rows | `189,680` |
+
+The run exited with code `1` after the teacher gate and before a visible native-training or relaxation result. The pod `b3be6644a7894bb9a4b8339c0643c240` was terminated by the launcher, and `prime pods list` returned zero active pods afterward. Local Wasabi retrieval was blocked by DNS timeout when attempting to pull the final remote log, but the launcher showed periodic sync during the run and the gate payload above was captured in the controller output.
+
+Carry-forward: do not return to V78 for this stable-manifold config; V78 was confirmed shape-incompatible with the current model heads. The next session should pull the synced `v23_remote.log` from Wasabi when DNS is healthy, identify the post-gate exit-code-1 source, and then relaunch from the force-basin checkpoint only after that failure is patched. The teacher evidence gate itself is no longer the blocker.
+
+## 2026-07-02 Force-Basin Probe32 Resume-Compatibility Closeout
+
+Run: `nexa_mat_msun_forcebasin_probe32_resumecompat_20260702T123227Z`
+
+Wasabi prefix: `wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/diffusion_alignment/2026-07-02/nexa_mat_msun_forcebasin_probe32_resumecompat_20260702T123227Z`
+
+This pass patched two train-start hazards before launch: model-only resume can now explicitly skip same-name checkpoint tensors whose shapes no longer match the current architecture, and the V23 stability lane now enables the relaxation surrogate by default instead of passing relaxation weights into a disabled objective. Focused local verification passed before launch: `38` stability/materials tests, shell syntax checks, and Ruff on the touched Python files.
+
+The corrected cycle launched on one secure-cloud `H200 141GB` because no 1x H100 configuration was available in Prime at launch time. The pod `a65ec339ee42436e90f64f7185da7ff0` hydrated the full dataset (`~170G`, `8,562` files), installed the MatterSim/MatterGen runtime, hydrated the latest July 2 stability-label evidence, and passed the same stable-manifold evidence gate:
+
+| Evidence | Result |
+| --- | ---: |
+| Teacher rows | `200,000` |
+| Positive / near-positive rows | `378` |
+| Required positive / near-positive rows | `250` |
+| Non-SSB positive/near profiles | `5` |
+| Required non-SSB positive/near profiles | `3` |
+| Hull-labeled rows | `567` |
+| Relaxation-labeled rows | `1,985` |
+| Trainable energy-labeled rows | `2,347` |
+| Force-vector labeled rows | `189,680` |
+| Stress-tensor labeled rows | `189,680` |
+
+The run still exited with code `1` immediately after the teacher gate and before the visible `teacher ready` line, native training, diagnostic, or relaxation screen. That narrows the likely failing surface to the intermediate `sync_dir "$STABLE_MANIFOLD_ROOT" "$WORK_CLOSEOUT_URI/replay/stable_manifold"` call, not teacher construction and probably not the training command. Local Wasabi retrieval was again blocked by DNS resolution timeout for `nexa-mat.s3.wasabisys.com`, so the full synced `v23_remote.log` could not be pulled from the controller at closeout time.
+
+Code carry-forward from this closeout: the intermediate teacher-registry sync is now best-effort and emits `[stable-manifold][warn] intermediate teacher sync failed; continuing to training` instead of aborting the run before training. Mandatory final closeout sync remains unchanged. Prime confirmed zero active pods after termination.
+
+Ops carry-forward: the next attempt should not pay another full 170G fresh hydration if avoidable. The current launcher has only `full`, `ssb_core`, and `SKIP_TRAIN` val-only hydration modes; there is no narrow stable-manifold/profile probe hydration mode yet. Add one or use a persistent/cached dataset disk before relaunching, otherwise the H100/H200 will sit idle for roughly twenty minutes before any model work starts.
+
+### Follow-Up Patch: Stable-Manifold Probe Hydration
+
+After the resume-compat closeout, the remote script gained `DATASET_FILTER_MODE=stable_manifold_probe`. This mode copies only the metadata/manifest surface, the validation split, and the first sorted `STABLE_MANIFOLD_SOURCE_PARQUET_LIMIT` train parquet shards through `--files-from-raw`, then records the parquet limit in the dataset receipt key. It preserves the teacher-builder contract while avoiding the full `170G` hydration tax on fresh pods. Use this mode for the next force-basin probe unless a persistent dataset disk is attached.
+
+## 2026-07-02 Force-Basin Fastdata H100 Closeout
+
+Run: `nexa_mat_msun_forcebasin_probe32_fastdata_20260702T131452Z`
+
+Wasabi prefix: `wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/diffusion_alignment/2026-07-02/nexa_mat_msun_forcebasin_probe32_fastdata_20260702T131452Z`
+
+This final controller pass proved the fastdata lane and exposed the exact training-start bug. The run used one secure-cloud H100 pod, `68f641404a454afbab9d30405b38ca73`, with `DATASET_FILTER_MODE=stable_manifold_probe`, `STABLE_MANIFOLD_SOURCE_PARQUET_LIMIT=96`, and the force-basin checkpoint as the resume source. The narrow train hydration path worked well enough to reach Python runtime setup, MatterSim/MatterGen install, teacher-label assembly, teacher-set construction, and diffusion-training startup. The remaining hydration overhead is that the current probe mode still copies the full validation split, which was roughly `84G`; the next optimization should add a validation parquet limit or cached validation disk.
+
+The run failed before productive training because the diffusion-generation resume path did not receive the mismatch-skip option even though the helper and launcher flag existed. The remote error was a checkpoint/model head mismatch:
+
+| Tensor | Checkpoint Shape | Current Shape |
+| --- | ---: | ---: |
+| `site_mlp.0.weight` | `[192, 585]` | `[192, 776]` |
+| `composition_head.0.weight` | `[192, 197]` | `[192, 576]` |
+| `lattice_head.0.weight` | `[192, 403]` | `[192, 768]` |
+| `volume_per_atom_head.0.weight` | `[192, 403]` | `[192, 768]` |
+
+Closeout patch: the diffusion-generation phase now passes `ignore_mismatched_shapes=bool(resume_model_only and args.materials_resume_ignore_mismatched_shapes)` into `_load_checkpoint`, and a regression test guards that exact resume block. Focused checks passed locally after the patch: three targeted pytest cases, Ruff on the touched backend/tests, and shell syntax checks for both remote launch wrappers. `prime pods list` returned zero active pods after the failed H100 run, so there is no ongoing GPU spend.
+
+Carry-forward: the next run should not revisit teacher-gate plumbing. It should relaunch the same force-basin probe after syncing this patch, preferably with a smaller validation hydration surface, and then judge only by relaxation convergence, hull movement, and at least one metastable/M.S.U.N. hit.
+
+## 2026-07-02 Fastdata Resume Regression and Diagnostic Failure
+
+Runs:
+
+- Training: `nexa_mat_msun_forcebasin_probe32_fastdata_fix_20260702T134105Z`
+- Diagnostic retry: `nexa_mat_msun_forcebasin_probe32_diagfix_20260702T142913Z`
+
+The fastdata patch worked operationally. Stable-manifold probe hydration copied capped training and validation shards, and diagnostic-only hydration now supports `DIAGNOSTIC_VAL_PARQUET_LIMIT` so smoke diagnostics do not pay the full validation-copy tax. The training run sustained roughly `420 samples/sec` with the 5-step native rollout profile; the dataloader fraction was near zero, so the runtime bottleneck was model/loss compute, not CPU streaming. A future fast-probe profile should target `NATIVE_ROLLOUT_STEPS=2-3` and only modest batch-size increases because the live run had about `12 GB` CUDA headroom.
+
+The model result failed. The trained checkpoint produced only `11 / 1024` accepted native/no-repair candidates on the smoke diagnostic, with `filter_accept=1.07%`, `allowed_elements=96.78%`, `charge_neutral=18.65%`, `strict_valid=26.86%`, `LeMat-distance=4.49%`, formula uniqueness `37.5%`, and novelty `100%`. This is below the static guardrails and therefore not eligible for the metastability screen.
+
+Root cause: the checkpoint resume was too permissive. `--materials-resume-ignore-mismatched-shapes` let a broad architecture mismatch proceed, skipping many core same-name tensors and leaving too much of the current model randomly initialized. The run therefore did not preserve the recovered native generator; it effectively trained a partially reset architecture for only `2500` steps and collapsed charge/static validity before stability ranking could matter.
+
+Code guard added after the failure: `_load_checkpoint` now enforces `--materials-resume-max-mismatched-shape-fraction` with a default cap of `0.05`. Shape-skipping remains available for controlled small head changes, but broad checkpoint/model incompatibility now fails before training. Focused tests, Ruff, shell syntax checks, and `prime pods list` passed after the patch; Prime reported zero active pods.
+
+Carry-forward: do not continue from the failed fastdata checkpoint. The next stability attempt must start from an architecture-compatible recovered checkpoint or explicitly rebuild the model with flags matching the checkpoint family. Speed tuning should happen after compatibility is proven; otherwise it only makes a bad partial restart fail faster.
+
+## 2026-07-02 Pause Note: Compatible Detached Probe Deferred
+
+Attempted run: `nexa_mat_msun_v90_detached_probe64_20260702T144615Z`
+
+Pod: `01b04e2d2217465088b647b441e2f565`
+
+This lane was opened only after local safety checks passed: targeted checkpoint-resume tests passed, Ruff passed on the touched backend/tests, shell syntax checks passed for the remote launch wrappers, and Prime reported zero active pods. The planned probe was an architecture-compatible detached V90/V78-lineage diagnostic, not a continuation from the failed force-basin fastdata checkpoint. It used capped diagnostic hydration, detached-native flags, global charge-balanced native sampling, `DIAG_BATCH_SIZE=128`, `RAW_SAMPLES=1024`, `TARGET_CIF_COUNT=128`, and a 64-CIF oxide stability-basin probe.
+
+The user paused the session during SSH bring-up, before benchmark execution, candidate generation, relaxation, or hull scoring began. The launcher cleanup path terminated the pod successfully, and `prime --plain pods list --output json` returned zero active pods afterward. There are therefore no new model-quality results from this attempted lane.
+
+Carry-forward for tomorrow: start from the same compatible detached-lineage path, not the failed force-basin checkpoint. First prove static guardrails on the 1,024-sample smoke, then run the 64-CIF metastability probe only if the selected pool is healthy. For throughput, the measured bottleneck is not data loading; it is model/loss compute from the multi-step native rollout. Keep the fast probe profile at one rollout attempt for diagnostics, use capped hydration, avoid full validation-copy startup, and only raise batch size if GPU memory headroom remains after compatibility is proven.
+
+## 2026-07-02 Ops Rule Update: Persistent Astral Forge Pod
+
+The materials H100 launch path now follows the Mintlify infrastructure convention instead of opening and closing timestamped pods per iteration. GPU machines belong to Astral Forge and should be named `astral-forge-XX` with zero-padded indexes, for example `astral-forge-01`, `astral-forge-02`, and `astral-forge-03`. The default materials launcher now uses `POD_INDEX=01` and `POD_NAME=astral-forge-01`. The default launcher behavior is now `KEEP_POD=1`, and the base V23 launcher reuses an already-active pod with the requested `POD_NAME` instead of forcing a new create path. Final closeout still requires explicit sync and termination, but intermediate iteration failures should reuse the same pod surface.
+
+Carry-forward: for the next compute session, launch or reuse `astral-forge-01`, keep it alive across smoke, diagnostic, selector tuning, and reruns, and terminate it only at user-requested closeout or if it is idle/stale with no productive process.
+
+## 2026-07-03 Pre-VM SMART Goal: Metastable / SUN Only
+
+Fresh goal: focus only on metastable and SUN/M.S.U.N. evidence. Before opening any VM, verify the stability-ranking code, launch discipline, checkpoint compatibility guards, and artifact wiring. Then use one persistent `astral-forge-01` H100 session to diagnose and improve metastable/SUN yield. The first success gate is a corrected native/no-repair 64-CIF stability probe with at least one metastable or SUN/M.S.U.N. candidate, post-relax validity `>=90%`, and mean `e_above_hull <=0.75 eV/atom`. If that gate is clean, scale to a 512-CIF stability screen targeting at least `26 / 512` M.S.U.N. as the recovery gate.
+
+Pre-VM review result: Prime reported zero active pods, the Mintlify pod convention is now wired as `astral-forge-XX`, and focused local checks passed. The metastability probe gate was tightened so lower mean hull plus convergence is not enough to promote; a 64-CIF probe now also requires a metastable signal before scale-up. This prevents another compute cycle from optimizing a low-hull but non-metastable pool.
+
+Verification:
+
+```bash
+uv run pytest -q tests/materials/test_nexamat_metastability_probe_gate.py tests/materials/test_nexamat_relaxability_selection.py tests/materials/test_nexamat_stable_manifold_teacher_set.py tests/materials/test_materials_training_adapter.py::test_materials_resume_model_only_can_skip_mismatched_checkpoint_shapes tests/materials/test_materials_training_adapter.py::test_materials_diffusion_generation_resume_path_skips_mismatched_shapes_when_requested
+uv run ruff check scripts/materials/nexamat_metastability_probe_gate.py scripts/materials/run_nexamat_bounded_relaxability_screen.py scripts/materials/build_nexamat_stability_basin_selector_labels.py tests/materials/test_nexamat_metastability_probe_gate.py tests/materials/test_nexamat_relaxability_selection.py tests/materials/test_nexamat_stable_manifold_teacher_set.py nexa_train/materials/backend.py
+bash -n scripts/materials/launch_nexamat_v23_prime_spot_h100.sh scripts/materials/launch_nexamat_v77_prime_secure_h100_encoder_detached_native.sh scripts/materials/launch_nexamat_v78_prime_secure_h100_detached_global_sampler.sh scripts/materials/launch_nexamat_v79_prime_secure_h100_diverse_sampler_diagnostic.sh scripts/materials/run_nexamat_v23_exact_sampler_rollout_remote.sh
+uv run python -m py_compile scripts/materials/nexamat_metastability_probe_gate.py scripts/materials/build_nexamat_stability_basin_selector_labels.py scripts/materials/run_nexamat_bounded_relaxability_screen.py
+```
+
+Carry-forward: do not open a new timestamped pod. Launch or reuse `astral-forge-01`, keep it alive across iterations, and start with a 64-CIF metastability probe from the compatible recovered lineage. If the 64-CIF probe has no metastable/SUN signal, do not scale to 512; inspect rejected and near-hit CIFs, rebuild selector labels around `converged + low hull` and `near-force + low hull`, then retry on the same pod.
+
+## Data-First Stable-Manifold Contract
+
+The next pass should move more slowly and make the data artifact explicit before any training loop starts. The training loop must consume a prebuilt `stable_manifold_teacher.jsonl` and its `stable_manifold_input_manifest.json`, not discover candidates and labels ad hoc during launch. The manifest freezes the exact staged inputs: canonical source JSONL/Parquet rows, native `candidate_scores.jsonl`, MatterSim single-point rows, bounded relaxation summaries, MatterGen hull metrics, profile set, role weights, and row caps.
+
+The label contract is:
+
+- `stable_positive`: `e_above_hull <= 0.10 eV/atom`, stress acceptable, and no failed relaxation evidence.
+- `metastable_positive`: `e_above_hull <= 0.25 eV/atom`, stress acceptable, and either converged or not explicitly nonconvergent.
+- `relaxable_positive`: low force/stress or converged relaxation evidence without hull labels.
+- `near_hull_positive`: `e_above_hull <= 0.75 eV/atom` with converged or near-force evidence.
+- `hard_negative`: static-valid but high-hull, nonconvergent, invalid-after-relax, or projection-heavy rows.
+- `unlabeled_context`: structure-valid rows without enough external stability signal.
+
+Before training, the dataset gate must report at least: nonzero stable/metastable positives if available, `>=128` positive-or-near rows, `>=3` non-SSB positive-or-near profiles, `>=64` hull-labeled rows, `>=64` trainable energy rows, candidate provenance coverage, and a quarantine file for rows missing structure or stability signal. The current code path writes this through `scripts/materials/write_nexamat_stable_manifold_input_manifest.py` followed by `scripts/materials/build_nexamat_stable_manifold_teacher_set.py --input-manifest ...`. The remote launcher now freezes `stable_manifold_input_manifest.json` first and builds the teacher set from that manifest, so reruns are reproducible and the training loop no longer depends on live recursive path discovery.
+
+Do not train from this data unless the summary shows the actual bottleneck labels are present: `converged + low hull`, `near-force + low hull`, and `static-valid + high-hull/nonconverged/projection-heavy` hard negatives. If those are missing, build more labels first; do not compensate with longer training.
+
+## 2026-07-09 Metastable Active-Label Queue
+
+The first data-first active-label queue is now built from the stable-manifold teacher artifact rather than from another broad static-valid sweep. Source teacher prefix: `wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/stable_manifold_teacher/nexa_mat_stable_manifold_teacher_20260709T1010Z`. Queue prefix: `wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/metastable_active_label_queue/nexa_mat_metastable_active_queue_20260709T184500Z`.
+
+The queue selected `64` probe CIFs from `2,480` eligible teacher rows, with `64 / 64` prior relax-converged rows, `6` near-hull positives, `58` relaxable positives, `selected_mean_known_hull=0.4770 eV/atom`, and profile coverage across battery ionics, broad inorganic, halide/chalcogenides, oxide catalysts, and semiconductors. The builder now derives queue formulas from atomic numbers so stale pseudo-formula labels do not leak into probe file names. This is not yet metastability proof: `selected_known_metastable_count=0`, so the next step is a bounded 64-CIF relaxation/hull probe against this queue. Promote only if it produces at least one metastable or M.S.U.N. hit; otherwise harvest the new hull/relaxation labels back into the teacher set and rebuild the queue.
+
+## 2026-07-09 Active Queue Probe Closeout
+
+Remote pod: `astral-forge-01` / `c69d55c312d84a67a12496fc3d74eefe` (`H100_80GB x1`). The pod was terminated after closeout and `prime --plain pods list --output json` returned zero active pods.
+
+Wasabi closeout root: `wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/metastability/2026-07-09`.
+
+Runs:
+
+- `nexa_mat_metastable_active_queue_probe_20260709T104500Z`: original 64-CIF active queue.
+- `nexa_mat_metastable_active_queue_simple10_probe_20260709T111000Z`: narrowed queue with `max_atoms=24`, `max_unique_elements=10`.
+- `nexa_mat_metastable_seed16_atoms20_e4_probe_20260709T113000Z`: simple 16-CIF queue with `max_atoms=20`, `max_unique_elements=4`.
+- `nexa_mat_metastable_seed4_atoms20_e4_long_probe_20260709T114000Z`: 4-CIF long-step check at `1000` MatterSim steps.
+- `nexa_mat_prior_msun_seed8_repro_20260709T115000Z`: prior low-hull seed reproduction attempt, capped during CPU-bound MatterGen scoring.
+
+Result: the July 9 active-label queue did not transfer to the current MatterSim force-basin contract. The original `64`-CIF queue, the narrowed `64`-CIF queue, the simple `16`-CIF queue, and the `4`-CIF `1000`-step check all returned `0` converged structures. This means the `prior_relax_converged` labels in the new teacher queue are not sufficient selectors for the current bounded screen. The failure is not static validity: post-relax rows stayed charge-neutral, LeMat-radius-valid, and benchmark-valid in the inspected summaries. The failure is force-basin entry.
+
+The recovered positive anchor remains the July 2 stable-manifold run: `wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/diffusion_alignment/2026-07-02/nexa_mat_msun_stablemanifold_oxide_train_20260702T054825Z`. That run produced `2 / 64` M.S.U.N. candidates, mean `e_above_hull=0.8750 eV/atom`, and low-hull rows at selected indices `48` (`0.1786 eV/atom`) and `63` (`0.1837 eV/atom`). It also exposed the real remaining bottleneck: only `3 / 64` structures converged at `fmax=0.05`, while many near-hull rows had low but nonzero residual forces.
+
+Carry-forward: do not repeat the July 9 proxy-label active queue. Build the next training set from the July 2 low-hull/M.S.U.N. positives and near-force low-hull rows, paired against July 9 nonconvergent hard negatives. The model target should be explicit: emit native oxide/broad-inorganic structures inside the low-force basin, not merely static-valid or prior-labeled relaxable structures. The next probe should first reproduce the July 2 positive anchor, then train on `metastable_positive + near_hull_low_force` versus `static_valid_nonconvergent` before any 512-CIF scale attempt.

file_manifest.json

@@ -0,0 +1,127 @@
+{
+  "file_count": 24,
+  "files": [
+    {
+      "path": "README.md",
+      "sha256": "f1966fdaffd5b2815480c9c277230c8922150e97fb655b8d41cc5f206fa6acf8",
+      "size_bytes": 4817
+    },
+    {
+      "path": "archive/archive_manifest.json",
+      "sha256": "995fd674dce1b26f2609348a354818cd1739864be0a6d8d81ee682849e45bbde",
+      "size_bytes": 57458
+    },
+    {
+      "path": "archive/nexamat_final_project_archive_20260724T135317Z.tar.gz",
+      "sha256": "5be28c38a5975e010d2e77fce48899989de371e8e3f9f81944447ac1217b2c84",
+      "size_bytes": 7304231
+    },
+    {
+      "path": "archive/nexamat_final_project_archive_20260724T135317Z.tar.gz.sha256",
+      "sha256": "d8ab98cbcf8d904f9b9a0de9d017d1a87c22ec4b5e9aade0b32accaa2557bf15",
+      "size_bytes": 125
+    },
+    {
+      "path": "decoder/native_v78/manifest.json",
+      "sha256": "3b4ab4ed55d0118beadda85664f1ce831bd6bffe1ea9c6a9d129c427aa6bd526",
+      "size_bytes": 715
+    },
+    {
+      "path": "docs/final_closeout/ARCHIVE_CLOSEOUT_2026-07-24.md",
+      "sha256": "d666b851659920b7a68556b2b80e37fdb443695d365dfb9a8993603f48798e15",
+      "size_bytes": 6716
+    },
+    {
+      "path": "docs/final_closeout/NEXA_MAT_FINAL_MODEL_CARD_2026-07-24.md",
+      "sha256": "f1966fdaffd5b2815480c9c277230c8922150e97fb655b8d41cc5f206fa6acf8",
+      "size_bytes": 4817
+    },
+    {
+      "path": "docs/final_closeout/NEXA_MAT_FINAL_PROJECT_COMPENDIUM_2026-07-24.md",
+      "sha256": "e4a34c1c119539aeaf1768252dbb19e56b7166283753c4c12e85a05759f06e31",
+      "size_bytes": 10726
+    },
+    {
+      "path": "docs/final_closeout/NEXA_MAT_FINAL_TECHNICAL_PAPER_2026-07-24.md",
+      "sha256": "2aae262037ac0c9ee23ae267b444e150ac74b62547eee5a6ef82f25b1918a08d",
+      "size_bytes": 12258
+    },
+    {
+      "path": "docs/final_closeout/PROJECT.md",
+      "sha256": "40daedb938ae5807f1d5c450f5da23bf677acbd725288d08d3cdde4dc527e75e",
+      "size_bytes": 4712
+    },
+    {
+      "path": "eval/final_evidence/2026-07-15_v30_tensor_angular_refinement_closeout.md",
+      "sha256": "d833787c46c0dc04a2390e97413ca23969d50fb58aa71fe8be4233ddfad8cd22",
+      "size_bytes": 8649
+    },
+    {
+      "path": "eval/final_evidence/2026-07-16_v32_shared_energy_metastability_closeout.md",
+      "sha256": "eab9494ba5bb006fb6b0cbf6caa56425d1cd5a26bd973ea5f2ffaefe1f13e96d",
+      "size_bytes": 9510
+    },
+    {
+      "path": "eval/final_evidence/2026-07-24_foundation_residual_metastability_closeout.md",
+      "sha256": "11e271ec8170f8baf3d605f9d0938b2e06a8b5bf42b082d998547ec2809f4228",
+      "size_bytes": 5107
+    },
+    {
+      "path": "eval/final_evidence/NEXA_MAT_ALIGNMENT_PIPELINE_KNOWLEDGE_PAPER.md",
+      "sha256": "0584e01419f78db2e38a4d50251410e527e1071fe3507e1a54531b528e7d5b86",
+      "size_bytes": 23071
+    },
+    {
+      "path": "eval/final_evidence/NEXA_MAT_STABILITY_SELECTOR_CALIBRATED_CLOSEOUT_2026-07-01.md",
+      "sha256": "0b2caad983f31b8c1b9cb2c46ce93cdcae0d8ab08d0e80d581632ea09cb95e90",
+      "size_bytes": 48928
+    },
+    {
+      "path": "figures/nexamat_inorganic_readiness_score.png",
+      "sha256": "546c2c78d2b5dd7362257c7db4facb3a3e73203bb912ed83654ba272108ee2f7",
+      "size_bytes": 85026
+    },
+    {
+      "path": "figures/nexamat_profile_geometry_rates.png",
+      "sha256": "24e242889d65c9ec60a20454694a55324addb1b34252a7b3fc0c1e4eb04748f5",
+      "size_bytes": 115263
+    },
+    {
+      "path": "figures/nexamat_profile_yield_vs_targets.png",
+      "sha256": "609984d3796152b68c11504906541c829ae2525277f680a858ff4ddff4275636",
+      "size_bytes": 123667
+    },
+    {
+      "path": "figures/nexamat_public_generator_evidence_profile.png",
+      "sha256": "d57864742a65bf35249aa168fc00bc4971b59baf8029e8dfcd051be19b1af7d5",
+      "size_bytes": 181140
+    },
+    {
+      "path": "figures/nexamat_public_generator_evidence_scores.png",
+      "sha256": "8641161b28323104f0fd21797bcf34e21f7e4a2d416d50f16ebe86824ea1455c",
+      "size_bytes": 88017
+    },
+    {
+      "path": "figures/nexamat_size_performance_known_params.png",
+      "sha256": "9165a019aed11088c8d1b27aa053c6c535d449c35e7c714c9bafb165a026bffa",
+      "size_bytes": 185184
+    },
+    {
+      "path": "figures/nexamat_vs_public_generators_matrix.png",
+      "sha256": "15ba70a6fe635ccb3cbb07362b1ba8aa64dc9de6d4120e18aa8ae3e62437c938",
+      "size_bytes": 120347
+    },
+    {
+      "path": "final_publication_manifest.json",
+      "sha256": "ebdecc1409c4c70b916ec0d8303d729f065ea8ef066274c224e1544d82c2fe0c",
+      "size_bytes": 1159
+    },
+    {
+      "path": "research/v29_directional_step800/manifest.json",
+      "sha256": "248b4ec760e45cdbbbdf81d1518668cc73f5cf518b25afcdfa076fb4cea60b7f",
+      "size_bytes": 616
+    }
+  ],
+  "schema_version": "nexamat.hf_file_manifest/v1",
+  "total_bytes": 8402259
+}

final_publication_manifest.json

@@ -0,0 +1,27 @@
+{
+  "schema_version": "nexamat.final_publication/v1",
+  "project": "NexaMat",
+  "archive_status": "archived_on_ice_indefinitely",
+  "publication_date": "2026-07-24",
+  "model_hub": "AethronPhantom/Nexa_Mat2",
+  "compact_stack_parameters": 4061859,
+  "encoder_parameters": 2813714,
+  "decoder_parameters": 1248145,
+  "shipped_decoder_lineage": "v78_detached_global_sampler",
+  "static_evidence_lineage": "v90_native_evaluation_surface",
+  "metastability_evidence": {
+    "best_bounded_msun_count": 2,
+    "bounded_pool_size": 64,
+    "strict_relaxation_converged": 3,
+    "mean_e_above_hull_ev_per_atom": 0.875,
+    "best_e_above_hull_ev_per_atom": [
+      0.1786,
+      0.1837
+    ],
+    "interpretation": "partial low-hull support with low reliability; below the intended 25-50 percent yield"
+  },
+  "canonical_archive": "wasabi:nexa-mat/project_archives/nexa_mat/2026-07-24_final_closeout/",
+  "full_history": "wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/diffusion_alignment/",
+  "claim": "compact native inorganic candidate generator with external screening required",
+  "excluded_claim": "repeatable high-yield metastable or SUN/MSUN generation"
+}

research/v29_directional_step800/manifest.json

@@ -0,0 +1,10 @@
+{
+  "schema_version": "nexamat.model_artifact/v1",
+  "name": "nexa_mat_v29_directional_trajectory_step800",
+  "role": "research_chemistry_and_directional_initialization",
+  "canonical_object_uri": "wasabi:nexa-mat/artifacts/active_worksets/nexa_mat/diffusion_alignment/2026-07-15/nexa_mat_v29_directional_trajectory_20260715T132827Z/arms/trajectory_direction_w16/train/checkpoint_step_800.pt",
+  "expected_size_bytes": 27268123,
+  "hub_path": "research/v29_directional_step800/checkpoint_step_800.pt",
+  "promotable": false,
+  "status": "research_lineage_canonical_object_preserved_in_wasabi_hub_mirror_requested"
+}