Instructions to use mahiyama/sae-splade-ja-310m with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use mahiyama/sae-splade-ja-310m with sentence-transformers:
from sentence_transformers import SparseEncoder model = SparseEncoder("mahiyama/sae-splade-ja-310m") queries = ["Which planet is known as the Red Planet?"] documents = [ "Venus is often called Earth's twin because of its similar size and proximity.", "Mars, known for its reddish appearance, is often referred to as the Red Planet.", "Jupiter, the largest planet in our solar system, has a prominent red spot.", ] query_embeddings = model.encode_query(queries) document_embeddings = model.encode_document(documents) similarities = model.similarity(query_embeddings, document_embeddings) print(similarities) - Notebooks
- Google Colab
- Kaggle
sae-splade-ja-310m
English | 日本語
A Japanese learned sparse retrieval model built on sbintuitions/modernbert-ja-310m. Like SPLADE, it encodes text into a sparse vector with only a few non-zero dimensions and ranks documents by the dot product of the query and document vectors.
In standard SPLADE, each output dimension corresponds to a token in the vocabulary. In this model, each dimension instead corresponds to one of 65,536 features learned by a sparse autoencoder (SAE). As in SAE-SPLADE (Zong et al., 2026), the SAE encoder replaces the MLM head as the output layer, and as in SPLARE (Formal et al., 2026), the SAE weights are kept frozen while the model is trained for retrieval.
Model Overview
| Property | Value |
|---|---|
| Base model | sbintuitions/modernbert-ja-310m |
| Architecture | Transformer + SAETopKPooling (Top-K SAE encoding and max pooling) |
| Parameters | About 365M (base model about 315M, SAE encoder about 50M) |
| Output dimensions | 65,536 (the number of SAE features) |
| Maximum input length | 8,192 tokens (trained with queries of up to 256 tokens and documents of up to 1,024 tokens) |
| Similarity | Dot product |
| Framework | SparseEncoder in sentence-transformers (tested with 6.2.0) |
An SAE decomposes the hidden states of a model into a large number of features, only a few of which are active at a time. This model uses, as its output layer, a Top-K SAE pre-trained on the final-layer hidden states of the base model. For each token, only the 16 features with the largest values are kept, and the output sparse vector takes the maximum of each feature over the whole text.
Usage
The recommended setting keeps only the top 40 dimensions (by value) of a query and the top 400 dimensions of a document as non-zero. Calling encode_query for queries and encode_document for documents applies this automatically.
from sentence_transformers import SparseEncoder
model = SparseEncoder("mahiyama/sae-splade-ja-310m", trust_remote_code=True)
# Queries: "What is the capital of Japan?", "What is machine learning?"
queries = [
"日本の首都はどこですか",
"機械学習とは何ですか",
]
# Documents: on Tokyo (the capital of Japan), Kyoto (an ancient capital), and machine learning
documents = [
"東京は日本の首都であり、政治と経済の中心地である。",
"京都は古都として知られ、神社仏閣が多くある。",
"機械学習は人工知能の一分野で、データからパターンを学習する手法である。",
]
# Encode into 65,536-dimensional sparse tensors
q_emb = model.encode_query(queries)
d_emb = model.encode_document(documents)
# Compute similarity scores by dot product
scores = model.similarity(q_emb, d_emb)
print(scores)
Unlike in SPLADE, the dimensions do not correspond to tokens, so the output vector cannot be read as a list of words. Each dimension (SAE feature) responds to a particular kind of content that tokens express in context, such as specific words or topics (e.g., military and national defense), expressions (e.g., requests for information), or sentence forms (e.g., sentence-final periods and question marks). interpretability.md (in Japanese) analyzes what each dimension responds to (feature labels assigned by a language model, and how well they fit) and which kinds of features support the retrieval rankings.
Inference Settings
As loaded, the model keeps only the top 40 dimensions of a query and the top 400 dimensions of a document, setting the rest to 0, to reduce the cost of retrieval. The output still has 65,536 dimensions. This top-k limit is applied by encode_query and encode_document, but not by encode. The table below compares it with no limit.
| Setting | nDCG@10 | FLOPs | Avg. non-zero dimensions (query / document) |
|---|---|---|---|
| Top-k limit (query 40, document 400) | 75.35 | 12.1 | 40.0 / 301.1 |
| No limit | 75.96 | 45.1 | 92.4 / 395.5 |
nDCG@10 is the average over the 11 retrieval tasks of JMTEB v2 lite described below. FLOPs is the expected number of dimensions that are non-zero in both the query and the document, an estimate of the computational cost of retrieval with an inverted index. Lower values mean faster retrieval.
The limit can be removed after loading by changing attributes of the output layer.
pooling = model[-1] # Output layer (SAETopKPooling)
# 0 removes the limit
pooling.query_top_k = 0
pooling.document_top_k = 0
Evaluation
nDCG@10 was measured on the 11 retrieval tasks of JMTEB v2 lite (named JMTEB-lite(v1) in MTEB).
Documents were encoded with up to 8,192 tokens, and queries were not truncated. The top 1,000 results for each query were scored in the same way as in MTEB. MultiLongDocRetrieval uses the dev split, and all other tasks use the test split.
| Task | Top-k limit (query 40, document 400) | No limit |
|---|---|---|
| JaqketRetrievalLite | 79.72 | 81.63 |
| MrTyDiJaRetrievalLite | 55.66 | 57.17 |
| JaGovFaqsRetrieval | 76.49 | 76.39 |
| NLPJournalTitleAbsRetrieval.V2 | 97.58 | 97.67 |
| NLPJournalTitleIntroRetrieval.V2 | 97.48 | 97.69 |
| NLPJournalAbsIntroRetrieval.V2 | 99.25 | 99.29 |
| NLPJournalAbsArticleRetrieval.V2 | 99.66 | 99.59 |
| JaCWIRRetrievalLite | 82.39 | 81.10 |
| MIRACLJaRetrievalLite | 68.99 | 70.74 |
| MintakaRetrieval | 22.70 | 23.09 |
| MultiLongDocRetrieval | 48.95 | 51.21 |
| Average | 75.35 | 75.96 |
| Avg. non-zero dimensions (query / document) | 40.0 / 301.1 | 92.4 / 395.5 |
Comparison with Published Results
The table below compares this model with published results on the average nDCG@10 over the 7 tasks shared by JMTEB v2 lite and JMTEB v2 (JaGovFaqs, the four NLP Journal tasks, Mintaka, and MultiLongDocRetrieval). Only models with 1B parameters or fewer are shown. The published results were measured under each model's own settings (input prefixes, maximum input length, and so on), which differ from the settings used for this model.
| Model | Type | 7-task average | Source |
|---|---|---|---|
| sionic-ai/comsat-embed-ja-0.3b-preview | Dense | 79.68 | Link |
| cl-nagoya/ruri-v3-130m | Dense | 78.40 | Link |
| cl-nagoya/ruri-v3-310m | Dense | 78.29 | Link |
| mahiyama/sae-splade-ja-310m | Sparse | 77.44 | |
| codefuse-ai/F2LLM-v2-0.6B | Dense | 76.86 | Link |
| mahiyama/splade-ja-310m-v2 | Sparse | 76.53 | Link |
| BAAI/bge-m3 | Dense | 74.50 | Link |
| microsoft/harrier-oss-v1-270m | Dense | 72.48 | Link |
Training
After the SAE was pre-trained, the retrieval model was trained in two stages: contrastive learning (stage 1) and distillation (stage 2). In both stages, the SAE weights were frozen and only the weights of the base model were updated. All training was done on a single NVIDIA RTX PRO 6000 Blackwell Server Edition GPU (Google Colab).
Training Data
| Dataset | Config | Domain | Stage 1 | Stage 2 |
|---|---|---|---|---|
| mahiyama/auto-wiki-qa | pairs | Wikipedia QA | 662,505 | 71,748 |
| mahiyama/quiz-no-mori | pairs | Wikipedia QA | 17,219 | 1,577 |
| mahiyama/quiz-works | pairs | Wikipedia QA | 14,744 | 1,363 |
| mahiyama/miracl-retrieval | pairs | Wikipedia QA | 3,036 | 245 |
| mahiyama/mrtydi | pairs | Wikipedia QA | 2,496 | 199 |
| mahiyama/mqa-ja | pairs | General web | 450,000 | 31,242 |
| mahiyama/mmarco-ja | pairs | General web | 250,000 | 28,140 |
| sbintuitions/JMTEB | jaqket-query train (positive: article body) | Knowledge QA | 29,427 | 1,656 |
| sbintuitions/JMTEB | jaqket-query train (positive: article title) | Knowledge QA | 11,110 | 1,831 |
| AmazonScience/mintaka | ja train (positive: answer entity name) | Knowledge QA | 6,548 | 1,046 |
| mahiyama/anlp-meeting-retrieval | abs-intro_pairs | Academic | 8,460 | 122 |
| mahiyama/anlp-meeting-retrieval | title-intro_pairs | Academic | 8,052 | 708 |
| mahiyama/anlp-meeting-retrieval | title-abs_pairs | Academic | 6,117 | 711 |
| mahiyama/amagasaki-qna | pairs | Public sector | 16,938 | 11,674 |
| mahiyama/kosodate-faq-pairs-ja | default | Public sector | 203 | 106 |
| Non-public government FAQ data | - | Public sector | 99,489 | 47,928 |
| Total | 1,586,344 | 200,296 |
In stage 1, to include more pairs with long documents, the JAQKET article-body pairs were used 3 times, the abs-intro and title-intro pairs of anlp-meeting-retrieval 4 times, and the title-abs pairs 3 times. The counts in the table include these repetitions.
SAE Pre-training
A Top-K SAE (Gao et al., 2024) was trained on the final-layer hidden states (768 dimensions) of the base model, with 65,536 features and 16 active features per token. Training covered 1 billion tokens (about 4.9 hours). The corpus has 846 million tokens: the documents and queries of the datasets in the table above, plus Japanese Wikipedia articles (20231101.ja of wikimedia/wikipedia).
The trained SAE is released as mahiyama/modernbert-ja-310m-sae, and this model uses its encoder as-is as the output layer. See that model card for the training settings, the breakdown of the corpus, and the reconstruction performance.
Stage 1: Contrastive Learning
The model was trained with contrastive learning on (query, positive) pairs, using the positives of the other queries in the same batch as negatives. To make the output sparse, FLOPS regularization (Paria et al., 2020), which reduces FLOPs, was applied to both queries and documents.
| Setting | Value |
|---|---|
| Learning rate | 2e-5 (linear decay, warmup over the first 5%) |
| Optimizer | AdamW (weight decay 0.01) |
| Batch size | 256 (computed 32 at a time with gradient caching) |
| Max sequence length | 256 tokens for queries, 1,024 tokens for documents |
| FLOPS regularization weight | 3.33e-5 for both queries and documents (increased quadratically from 0 over the first 30% of steps) |
| Epochs | 1 (6,197 steps) |
| Training time | About 3.8 hours |
Stage 2: Distillation
Queries were selected per domain from the same datasets as in stage 1, and tuples of (query, positive, 5 negatives) with teacher scores were built as follows.
- Collecting negative candidates: The stage-1 model searched the positives of the training data in the same domain (up to 600,000 for the Wikipedia QA and general web domains), and the top 30 results for each query became candidates. When the n-tuples subset of a dataset provides negatives (mined with cl-nagoya/ruri-v3-310m), those were added to the candidates as well.
- Removing false negatives: Candidates were scored with cl-nagoya/ruri-v3-reranker-310m, and those scoring at least as high as the positive, or 0.5 or higher after a sigmoid, were removed. For the general web, public sector, and knowledge QA domains, candidates were also scored with Cloudflare/clef-flash, and those scoring at least as high as the positive, or with a relevance probability of 0.3 or higher, were removed.
- Selecting negatives and teacher scores: Five negatives were selected from the remaining candidates (2 from the n-tuples negatives and 3 mined with the stage-1 model), and queries with fewer than five were dropped. Teacher scores came from ruri-v3-reranker-310m for the Wikipedia QA and academic domains, and mainly from clef-flash for the general web, public sector, and knowledge QA domains (over 90% of the tuples; clef-flash probabilities were converted to logits).
The loss is the sum of a contrastive loss and a distillation loss. The contrastive loss uses all documents in the batch (the positives and negatives of every query). The distillation loss is the KL divergence that brings the distribution of scores over the positive and the five negatives closer to the distribution of the teacher scores. FLOPS regularization was also applied with the same weights as in stage 1.
| Setting | Value |
|---|---|
| Learning rate | 1e-5 (linear decay, warmup over the first 5%) |
| Optimizer | AdamW (weight decay 0.01) |
| Batch size | 32 queries (1 positive and 5 negatives per query) |
| Max sequence length | 256 tokens for queries, 1,024 tokens for documents |
| Loss weights | Contrastive 1.0, distillation 1.0 |
| Distillation temperature | 1.5 for this model (student), 2.0 for ruri-v3-reranker-310m, 1.25 for clef-flash |
| FLOPS regularization weight | 3.33e-5 for both queries and documents |
| Epochs | 1 (6,259 steps) |
| Training time | About 2.3 hours |
Decontamination
The queries and relevant documents used in the JMTEB v2 lite evaluation were removed from the training data, the negative candidates, and the SAE corpus. For the evaluation corpora with few documents (JaGovFaqs, NLP Journal, and MultiLongDocRetrieval), all documents were removed, whether relevant or not. Matches were detected by exact match of normalized strings and by approximate match based on character 8-gram overlap.
License
The model weights are released under the MIT License. The base model, sbintuitions/modernbert-ja-310m, is also released under the MIT License.
However, the training data includes the following data that require attention:
- Data released for non-commercial use: mmarco-ja (a Japanese translation of MS MARCO), quiz-no-mori, and JAQKET
- Non-public government FAQ data: the author's non-public data based on FAQs from national government agencies and local governments
If you use this model, especially for commercial purposes, please take these points into account and make your own judgment at your own responsibility.
- Downloads last month
- 66
Model tree for mahiyama/sae-splade-ja-310m
Base model
sbintuitions/modernbert-ja-310m