AyoubChLin/distilbert-hc3-human-vs-ai

DistilBERT HC3 Human vs AI Detector

This is a fully fine-tuned DistilBERT binary sequence classifier for distinguishing human-written answers from AI-generated answers. It was trained on the English all configuration of Hello-SimpleAI/HC3, using answer text only.

The data pipeline isolates the original HC3 questions across train, validation, and test splits before balancing. Consequently, answers derived from the same question cannot appear in more than one split.

Important: this model measures similarity to the human and early-ChatGPT writing patterns represented in HC3. Its output is a model score—not proof of authorship. Do not use it alone for grading, discipline, hiring, moderation, or accusations of AI use.

Model details

Item	Value
Model	AyoubChLin/distilbert-hc3-human-vs-ai
Base model	distilbert/distilbert-base-uncased
Architecture	DistilBertForSequenceClassification
Language	English
Task	Binary text classification
Input	A standalone passage of text
Maximum input length	512 tokens; longer inputs are truncated
Trainable parameters	66,955,010
Fine-tuning method	Full fine-tuning; not LoRA/PEFT
Output labels	HUMAN, AI_GENERATED
Training dataset	Hello-SimpleAI/HC3, configuration all
Framework	PyTorch + Hugging Face Transformers
License	CC BY-SA 4.0

Label mapping

ID	Label	Meaning
0	HUMAN	Text resembles the human-answer class in HC3
1	AI_GENERATED	Text resembles the early-ChatGPT-answer class in HC3

Held-out test results

The final model was evaluated once on a balanced, question-group-isolated test set containing 5,306 examples: 2,653 human answers and 2,653 AI-generated answers.

Metric	Score
Test loss	0.017504
Accuracy	0.993592
Precision — AI_GENERATED	0.989533
Recall — AI_GENERATED	0.997738
F1 — AI_GENERATED	0.993619
ROC AUC	0.999862

Per-class results

Class	Precision	Recall	F1	Support
HUMAN	0.9977	0.9894	0.9936	2,653
AI_GENERATED	0.9895	0.9977	0.9936	2,653
Macro average	0.9936	0.9936	0.9936	5,306
Weighted average	0.9936	0.9936	0.9936	5,306

Confusion matrix

Rows are true labels; columns are predicted labels.

	Predicted HUMAN	Predicted AI_GENERATED
True HUMAN	2,625	28
True AI_GENERATED	6	2,647

These results describe the held-out HC3 split only. They must not be interpreted as expected performance on newer language models, edited text, other languages, or unrelated domains.

Run test cells

The following cells load the published checkpoint, run single/batch inference, return both class probabilities, and perform a small smoke test.

Cell 1 — Install dependencies

%pip install -q "transformers==4.57.1" "accelerate>=1.2,<2"

If the notebook runtime asks for a restart after installation, restart it once before continuing.

Cell 2 — Load the model

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

MODEL_ID = "AyoubChLin/distilbert-hc3-human-vs-ai"
MAX_LENGTH = 512

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, use_fast=True)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID)
model.to(device)
model.eval()

print(f"Loaded {MODEL_ID} on {device}")
print("Label mapping:", model.config.id2label)

Cell 3 — Reliable inference function

def predict_texts(texts, batch_size=16, ai_threshold=0.50):
    """Classify one string or a list of strings.

    `ai_threshold` applies to the AI_GENERATED score. The default 0.50
    threshold is suitable for a basic demo; recalibrate it on representative
    in-domain validation data before deploying the model.
    """
    if isinstance(texts, str):
        texts = [texts]

    if not isinstance(texts, list) or not texts:
        raise ValueError("texts must be a non-empty string or list of strings")
    if not all(isinstance(text, str) and text.strip() for text in texts):
        raise ValueError("every input must be a non-empty string")
    if not 0.0 <= ai_threshold <= 1.0:
        raise ValueError("ai_threshold must be between 0 and 1")

    ai_id = int(model.config.label2id.get("AI_GENERATED", 1))
    human_id = int(model.config.label2id.get("HUMAN", 0))
    results = []

    for start in range(0, len(texts), batch_size):
        batch = texts[start : start + batch_size]
        inputs = tokenizer(
            batch,
            padding=True,
            truncation=True,
            max_length=MAX_LENGTH,
            return_tensors="pt",
        ).to(device)

        with torch.inference_mode():
            probabilities = model(**inputs).logits.softmax(dim=-1).cpu()

        for text, probs in zip(batch, probabilities):
            human_score = float(probs[human_id])
            ai_score = float(probs[ai_id])
            prediction = "AI_GENERATED" if ai_score >= ai_threshold else "HUMAN"

            results.append({
                "text": text,
                "prediction": prediction,
                "confidence": max(human_score, ai_score),
                "human_probability": human_score,
                "ai_probability": ai_score,
                "truncated_to_max_length": len(
                    tokenizer(text, add_special_tokens=True)["input_ids"]
                ) > MAX_LENGTH,
            })

    return results

Cell 4 — Single-text test

result = predict_texts(
    "I tried the recipe yesterday. It was a little too salty, "
    "but my family still finished everything."
)[0]

print(f"Prediction:       {result['prediction']}")
print(f"Confidence:       {result['confidence']:.4f}")
print(f"Human score:      {result['human_probability']:.4f}")
print(f"AI score:         {result['ai_probability']:.4f}")
print(f"Input truncated:  {result['truncated_to_max_length']}")

Cell 5 — Batch test with validation checks

samples = [
    "I missed the bus this morning, so I walked to work in the rain.",
    "Machine learning is a field of study that enables systems to learn patterns from data and make predictions.",
    "The first version failed twice. I changed the parser, reran it, and finally got the output I expected.",
]

results = predict_texts(samples, batch_size=8)

assert len(results) == len(samples)
for item in results:
    assert item["prediction"] in {"HUMAN", "AI_GENERATED"}
    assert 0.0 <= item["confidence"] <= 1.0
    assert abs(
        item["human_probability"] + item["ai_probability"] - 1.0
    ) < 1e-5

    print("-" * 80)
    print(f"Prediction: {item['prediction']} | confidence={item['confidence']:.4f}")
    print(f"HUMAN={item['human_probability']:.4f} | AI_GENERATED={item['ai_probability']:.4f}")
    print(item["text"])

print("\nSmoke test passed.")

The sample labels are intentionally not asserted: an inference smoke test should verify that the checkpoint loads and returns valid probabilities, not treat a few hand-written sentences as ground truth.

Cell 6 — Optional Transformers pipeline

from transformers import pipeline

classifier = pipeline(
    task="text-classification",
    model=MODEL_ID,
    tokenizer=MODEL_ID,
    device=0 if torch.cuda.is_available() else -1,
)

classifier(
    "Paste the passage you want to inspect here.",
    truncation=True,
    max_length=512,
)

Training data

HC3 contains questions paired with lists of human answers and answers generated by an early ChatGPT system. The all configuration combines five English sources:

• finance
• medicine
• open_qa
• reddit_eli5
• wiki_csai

Cleaning and leakage prevention

The preparation pipeline used the following procedure:

1. Expanded every question row into standalone human and AI answer records.
2. Normalized repeated whitespace and removed answers shorter than 20 characters.
3. Compared normalized, lower-cased text to remove exact duplicates globally.
4. Removed ambiguous text that appeared under both labels.
5. Assigned every answer from the same original question a shared group_id.
6. Split question groups 80/10/10, stratified by source, before balancing.
7. Balanced HUMAN and AI_GENERATED independently inside each split.

There were 79,325 usable unique answers before per-split balancing.

Split	Total examples	HUMAN	AI_GENERATED	Represented question groups
Train	41,924	20,962	20,962	18,983
Validation	5,256	2,628	2,628	2,373
Test	5,306	2,653	2,653	2,379

Training procedure

The classification head was initialized for the downstream binary task, and all model parameters were updated.

Hyperparameter	Run value
Maximum sequence length	512
Epoch limit	5
Epoch reached	3.0
Learning rate	2e-5
Per-device train batch size	128
Per-device evaluation batch size	256
Gradient accumulation steps	1
Effective train batch size	128
Weight decay	0.01
Warmup ratio	0.10
Padding	Dynamic, to a multiple of 8
Evaluation strategy	Every epoch
Checkpoint strategy	Every epoch
Best-model metric	Validation F1, higher is better
Load best model at end	Yes
Early-stopping patience	2 evaluation rounds
Maximum retained checkpoints	2
Logging interval	50 steps
Random/data seed	42
Precision	BF16
Training hardware	One NVIDIA A100-SXM4 80 GB
Transformers version	4.57.1
Experiment tracking	Disabled

The configuration selected batch sizes from available GPU memory:

Detected VRAM	Train batch	Evaluation batch
At least 70 GiB	128	256
At least 35 GiB	64	128
Less than 35 GiB / fallback	16	32

Recorded training statistics

Statistic	Value
Training runtime	233.6299 seconds
Samples/second	897.231
Steps/second	7.020
Training loss	0.061305
Total FLOPs	1.655368e16

The notebook requested up to five epochs and reached epoch 3.0 with early stopping enabled. The final exported checkpoint was loaded from the best validation-F1 checkpoint.

Intended use

This model is suitable for:

• research and educational experiments on HC3-style AI-text detection;
• benchmarking binary text-classification pipelines;
• exploratory screening where every result is reviewed by a human;
• serving as a baseline before training or calibrating on newer, in-domain data.

It is not suitable as a standalone authorship verifier or as the sole basis for consequential decisions.

Limitations and failure modes

• Dataset age: HC3's synthetic class represents an early ChatGPT system, not the full range of current generators.
• Domain shift: performance can fall sharply on domains and writing styles that are not represented in HC3.
• English only: the model and training corpus are English-focused. Results for other languages are unsupported.
• Editing and paraphrasing: human editing, paraphrasing, translation, deliberate evasion, or mixed human/AI authorship can change the prediction substantially.
• False positives and false negatives: polished human prose may resemble the AI class, while generated text may resemble the human class.
• Short text: short passages provide little stylistic evidence and are intrinsically harder to classify.
• Long text: inputs beyond 512 tokens are truncated; the default score therefore does not represent the entire document.
• Scores are not calibrated proof: softmax values are confidence scores under this model and dataset. They are not the real-world probability that an author used AI.
• Balanced evaluation: the test set is class-balanced. Precision and predictive value will change when the real deployment prevalence differs.
• No question context: the model receives only answer text, even though the original HC3 records also contain questions.

For deployment, collect recent human and AI examples from the target domain, preserve author/source groups across splits, evaluate subgroup error rates, calibrate the threshold, and monitor drift and false positives continuously.

Ethical considerations

AI-text detectors can incorrectly accuse people, and stylistic differences may produce uneven error rates across writers, language backgrounds, accessibility needs, and levels of writing experience. Predictions should be treated as weak supporting signals and reviewed alongside transparent, independent evidence. Users should have a meaningful way to contest any consequential decision.

Reproducibility notes

• Random seed and data seed: 42
• Full balanced training corpus: MAX_SAMPLES_PER_CLASS = None
• Tokenization: fast DistilBERT tokenizer, truncation at 512 tokens
• Evaluation: binary precision/recall/F1 for AI_GENERATED, plus accuracy and ROC AUC
• Held-out evaluation split: 5,306 examples, isolated at the original-question level

Citation

If you use this checkpoint, cite the HC3 dataset and the DistilBERT work in addition to referencing this model repository.

@inproceedings{guo2023hc3,
  title     = {How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection},
  author    = {Guo, Biyang and Zhang, Xin and Wang, Ziyuan and Jiang, Minqi and Nie, Jinran and Ding, Yuxuan and Yue, Jianwei and Wu, Yupeng},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year      = {2023}
}

@inproceedings{sanh2019distilbert,
  title     = {DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter},
  author    = {Sanh, Victor and Debut, Lysandre and Chaumond, Julien and Wolf, Thomas},
  booktitle = {NeurIPS EMC2 Workshop},
  year      = {2019}
}

License

This model repository is released under CC BY-SA 4.0. Users are responsible for complying with the licenses and terms of the base model and HC3 dataset.