sae-splade-ja-310m

English | 日本語

A Japanese learned sparse retrieval model built on sbintuitions/modernbert-ja-310m. Like SPLADE, it encodes text into a sparse vector with only a few non-zero dimensions and ranks documents by the dot product of the query and document vectors.

In standard SPLADE, each output dimension corresponds to a token in the vocabulary. In this model, each dimension instead corresponds to one of 65,536 features learned by a sparse autoencoder (SAE). As in SAE-SPLADE (Zong et al., 2026), the SAE encoder replaces the MLM head as the output layer, and as in SPLARE (Formal et al., 2026), the SAE weights are kept frozen while the model is trained for retrieval.

Model Overview

Property Value
Base model sbintuitions/modernbert-ja-310m
Architecture Transformer + SAETopKPooling (Top-K SAE encoding and max pooling)
Parameters About 365M (base model about 315M, SAE encoder about 50M)
Output dimensions 65,536 (the number of SAE features)
Maximum input length 8,192 tokens (trained with queries of up to 256 tokens and documents of up to 1,024 tokens)
Similarity Dot product
Framework SparseEncoder in sentence-transformers (tested with 6.2.0)

An SAE decomposes the hidden states of a model into a large number of features, only a few of which are active at a time. This model uses, as its output layer, a Top-K SAE pre-trained on the final-layer hidden states of the base model. For each token, only the 16 features with the largest values are kept, and the output sparse vector takes the maximum of each feature over the whole text.

Usage

The recommended setting keeps only the top 40 dimensions (by value) of a query and the top 400 dimensions of a document as non-zero. Calling encode_query for queries and encode_document for documents applies this automatically.

from sentence_transformers import SparseEncoder

model = SparseEncoder("mahiyama/sae-splade-ja-310m", trust_remote_code=True)

# Queries: "What is the capital of Japan?", "What is machine learning?"
queries = [
    "日本の首都はどこですか",
    "機械学習とは何ですか",
]
# Documents: on Tokyo (the capital of Japan), Kyoto (an ancient capital), and machine learning
documents = [
    "東京は日本の首都であり、政治と経済の中心地である。",
    "京都は古都として知られ、神社仏閣が多くある。",
    "機械学習は人工知能の一分野で、データからパターンを学習する手法である。",
]

# Encode into 65,536-dimensional sparse tensors
q_emb = model.encode_query(queries)
d_emb = model.encode_document(documents)

# Compute similarity scores by dot product
scores = model.similarity(q_emb, d_emb)
print(scores)

Unlike in SPLADE, the dimensions do not correspond to tokens, so the output vector cannot be read as a list of words. Each dimension (SAE feature) responds to a particular kind of content that tokens express in context, such as specific words or topics (e.g., military and national defense), expressions (e.g., requests for information), or sentence forms (e.g., sentence-final periods and question marks). interpretability.md (in Japanese) analyzes what each dimension responds to (feature labels assigned by a language model, and how well they fit) and which kinds of features support the retrieval rankings.

Inference Settings

As loaded, the model keeps only the top 40 dimensions of a query and the top 400 dimensions of a document, setting the rest to 0, to reduce the cost of retrieval. The output still has 65,536 dimensions. This top-k limit is applied by encode_query and encode_document, but not by encode. The table below compares it with no limit.

Setting nDCG@10 FLOPs Avg. non-zero dimensions (query / document)
Top-k limit (query 40, document 400) 75.35 12.1 40.0 / 301.1
No limit 75.96 45.1 92.4 / 395.5

nDCG@10 is the average over the 11 retrieval tasks of JMTEB v2 lite described below. FLOPs is the expected number of dimensions that are non-zero in both the query and the document, an estimate of the computational cost of retrieval with an inverted index. Lower values mean faster retrieval.

The limit can be removed after loading by changing attributes of the output layer.

pooling = model[-1]  # Output layer (SAETopKPooling)

# 0 removes the limit
pooling.query_top_k = 0
pooling.document_top_k = 0

Evaluation

nDCG@10 was measured on the 11 retrieval tasks of JMTEB v2 lite (named JMTEB-lite(v1) in MTEB).

Documents were encoded with up to 8,192 tokens, and queries were not truncated. The top 1,000 results for each query were scored in the same way as in MTEB. MultiLongDocRetrieval uses the dev split, and all other tasks use the test split.

Task Top-k limit (query 40, document 400) No limit
JaqketRetrievalLite 79.72 81.63
MrTyDiJaRetrievalLite 55.66 57.17
JaGovFaqsRetrieval 76.49 76.39
NLPJournalTitleAbsRetrieval.V2 97.58 97.67
NLPJournalTitleIntroRetrieval.V2 97.48 97.69
NLPJournalAbsIntroRetrieval.V2 99.25 99.29
NLPJournalAbsArticleRetrieval.V2 99.66 99.59
JaCWIRRetrievalLite 82.39 81.10
MIRACLJaRetrievalLite 68.99 70.74
MintakaRetrieval 22.70 23.09
MultiLongDocRetrieval 48.95 51.21
Average 75.35 75.96
Avg. non-zero dimensions (query / document) 40.0 / 301.1 92.4 / 395.5

Comparison with Published Results

The table below compares this model with published results on the average nDCG@10 over the 7 tasks shared by JMTEB v2 lite and JMTEB v2 (JaGovFaqs, the four NLP Journal tasks, Mintaka, and MultiLongDocRetrieval). Only models with 1B parameters or fewer are shown. The published results were measured under each model's own settings (input prefixes, maximum input length, and so on), which differ from the settings used for this model.

Model Type 7-task average Source
sionic-ai/comsat-embed-ja-0.3b-preview Dense 79.68 Link
cl-nagoya/ruri-v3-130m Dense 78.40 Link
cl-nagoya/ruri-v3-310m Dense 78.29 Link
mahiyama/sae-splade-ja-310m Sparse 77.44
codefuse-ai/F2LLM-v2-0.6B Dense 76.86 Link
mahiyama/splade-ja-310m-v2 Sparse 76.53 Link
BAAI/bge-m3 Dense 74.50 Link
microsoft/harrier-oss-v1-270m Dense 72.48 Link

Training

After the SAE was pre-trained, the retrieval model was trained in two stages: contrastive learning (stage 1) and distillation (stage 2). In both stages, the SAE weights were frozen and only the weights of the base model were updated. All training was done on a single NVIDIA RTX PRO 6000 Blackwell Server Edition GPU (Google Colab).

Training Data

Dataset Config Domain Stage 1 Stage 2
mahiyama/auto-wiki-qa pairs Wikipedia QA 662,505 71,748
mahiyama/quiz-no-mori pairs Wikipedia QA 17,219 1,577
mahiyama/quiz-works pairs Wikipedia QA 14,744 1,363
mahiyama/miracl-retrieval pairs Wikipedia QA 3,036 245
mahiyama/mrtydi pairs Wikipedia QA 2,496 199
mahiyama/mqa-ja pairs General web 450,000 31,242
mahiyama/mmarco-ja pairs General web 250,000 28,140
sbintuitions/JMTEB jaqket-query train (positive: article body) Knowledge QA 29,427 1,656
sbintuitions/JMTEB jaqket-query train (positive: article title) Knowledge QA 11,110 1,831
AmazonScience/mintaka ja train (positive: answer entity name) Knowledge QA 6,548 1,046
mahiyama/anlp-meeting-retrieval abs-intro_pairs Academic 8,460 122
mahiyama/anlp-meeting-retrieval title-intro_pairs Academic 8,052 708
mahiyama/anlp-meeting-retrieval title-abs_pairs Academic 6,117 711
mahiyama/amagasaki-qna pairs Public sector 16,938 11,674
mahiyama/kosodate-faq-pairs-ja default Public sector 203 106
Non-public government FAQ data - Public sector 99,489 47,928
Total 1,586,344 200,296

In stage 1, to include more pairs with long documents, the JAQKET article-body pairs were used 3 times, the abs-intro and title-intro pairs of anlp-meeting-retrieval 4 times, and the title-abs pairs 3 times. The counts in the table include these repetitions.

SAE Pre-training

A Top-K SAE (Gao et al., 2024) was trained on the final-layer hidden states (768 dimensions) of the base model, with 65,536 features and 16 active features per token. Training covered 1 billion tokens (about 4.9 hours). The corpus has 846 million tokens: the documents and queries of the datasets in the table above, plus Japanese Wikipedia articles (20231101.ja of wikimedia/wikipedia).

The trained SAE is released as mahiyama/modernbert-ja-310m-sae, and this model uses its encoder as-is as the output layer. See that model card for the training settings, the breakdown of the corpus, and the reconstruction performance.

Stage 1: Contrastive Learning

The model was trained with contrastive learning on (query, positive) pairs, using the positives of the other queries in the same batch as negatives. To make the output sparse, FLOPS regularization (Paria et al., 2020), which reduces FLOPs, was applied to both queries and documents.

Setting Value
Learning rate 2e-5 (linear decay, warmup over the first 5%)
Optimizer AdamW (weight decay 0.01)
Batch size 256 (computed 32 at a time with gradient caching)
Max sequence length 256 tokens for queries, 1,024 tokens for documents
FLOPS regularization weight 3.33e-5 for both queries and documents (increased quadratically from 0 over the first 30% of steps)
Epochs 1 (6,197 steps)
Training time About 3.8 hours

Stage 2: Distillation

Queries were selected per domain from the same datasets as in stage 1, and tuples of (query, positive, 5 negatives) with teacher scores were built as follows.

  1. Collecting negative candidates: The stage-1 model searched the positives of the training data in the same domain (up to 600,000 for the Wikipedia QA and general web domains), and the top 30 results for each query became candidates. When the n-tuples subset of a dataset provides negatives (mined with cl-nagoya/ruri-v3-310m), those were added to the candidates as well.
  2. Removing false negatives: Candidates were scored with cl-nagoya/ruri-v3-reranker-310m, and those scoring at least as high as the positive, or 0.5 or higher after a sigmoid, were removed. For the general web, public sector, and knowledge QA domains, candidates were also scored with Cloudflare/clef-flash, and those scoring at least as high as the positive, or with a relevance probability of 0.3 or higher, were removed.
  3. Selecting negatives and teacher scores: Five negatives were selected from the remaining candidates (2 from the n-tuples negatives and 3 mined with the stage-1 model), and queries with fewer than five were dropped. Teacher scores came from ruri-v3-reranker-310m for the Wikipedia QA and academic domains, and mainly from clef-flash for the general web, public sector, and knowledge QA domains (over 90% of the tuples; clef-flash probabilities were converted to logits).

The loss is the sum of a contrastive loss and a distillation loss. The contrastive loss uses all documents in the batch (the positives and negatives of every query). The distillation loss is the KL divergence that brings the distribution of scores over the positive and the five negatives closer to the distribution of the teacher scores. FLOPS regularization was also applied with the same weights as in stage 1.

Setting Value
Learning rate 1e-5 (linear decay, warmup over the first 5%)
Optimizer AdamW (weight decay 0.01)
Batch size 32 queries (1 positive and 5 negatives per query)
Max sequence length 256 tokens for queries, 1,024 tokens for documents
Loss weights Contrastive 1.0, distillation 1.0
Distillation temperature 1.5 for this model (student), 2.0 for ruri-v3-reranker-310m, 1.25 for clef-flash
FLOPS regularization weight 3.33e-5 for both queries and documents
Epochs 1 (6,259 steps)
Training time About 2.3 hours

Decontamination

The queries and relevant documents used in the JMTEB v2 lite evaluation were removed from the training data, the negative candidates, and the SAE corpus. For the evaluation corpora with few documents (JaGovFaqs, NLP Journal, and MultiLongDocRetrieval), all documents were removed, whether relevant or not. Matches were detected by exact match of normalized strings and by approximate match based on character 8-gram overlap.

License

The model weights are released under the MIT License. The base model, sbintuitions/modernbert-ja-310m, is also released under the MIT License.

However, the training data includes the following data that require attention:

  • Data released for non-commercial use: mmarco-ja (a Japanese translation of MS MARCO), quiz-no-mori, and JAQKET
  • Non-public government FAQ data: the author's non-public data based on FAQs from national government agencies and local governments

If you use this model, especially for commercial purposes, please take these points into account and make your own judgment at your own responsibility.

Downloads last month
66
Safetensors
Model size
0.3B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for mahiyama/sae-splade-ja-310m

Finetuned
(25)
this model

Datasets used to train mahiyama/sae-splade-ja-310m

Collection including mahiyama/sae-splade-ja-310m

Papers for mahiyama/sae-splade-ja-310m