Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
SeaWolf-AI 
posted an update about 12 hours ago
Post
980
We opened a benchmark for drug property prediction tools. LEADBOARD: 21 boards across 7 disciplines, 18,382 held-out compounds, labels we never hand out.

Two numbers we hit while building it are the reason it exists.

First. Split the hERG cardiotoxicity data at random and you get AUROC 0.818. Split it by first-report year instead and you get 0.606. Same molecules, same fingerprints, same learner, same hyperparameters. The only thing that changed was where the line went, and the score moved 0.211. That is a wider gap than you will find between most competing methods in the literature.

Second. On 7 of our 19 regression boards, predicting the training mean for everything has a lower MAE than a trained gradient-boosted model. hERG is one of them, 0.599 against 0.589. The trained model loses.

So every board publishes its homework before anyone submits. Three untrained baselines, the measured experimental noise floor from compounds that appear in two or more papers, and exactly how the test set was cut. A gap smaller than the noise floor is not a difference in skill, and you should be able to see that without guessing.

Entering is simple. Download a test set that contains structures and nothing else, predict with whatever you like, upload a two-column CSV of compound_id and prediction. Trained model, physics engine, LLM, rule of thumb. We do not care what is inside. We measure the output.

Post: https://huggingface.co/blog/FINAL-Bench/leadboard-drug
Leaderboard: FINAL-Bench/leadboard

"The Scientific Vampire: How SeaWolf-AI Turns Human Curiosity Into Free Data Farming"
The Pattern (Business Model, Stripped Naked):
Every scam has a signature. This one has four:
Hype hijacking — malaria, batteries, AI pharmacology — topics with moral gravity and investor appetite.
Virtue-wrapping — "saving children," "green revolution," "open science against big pharma."
Data harvesting — unique SMILES strings, chemical predictions, model outputs — all flowing into a private, unaudited server.
Legal vacuum — no NDAs, no IP contracts, just "we never share your labels on your behalf." Translation: we own the flow, you own the promise.
Magnifique business model. Charity theater with a backend ledger.
Project №1: Malaria "Cure" — Benchmark That Hacks Itself
The Open Discovery Challenge offers ~$1000 for "curated features" against malaria. Quelle générosité. Except the scoring architecture is completely detached from wet-lab biology. ADMET validation happens through linear formulas and off-the-shelf libraries.
Here is the autopsy:
A user (Qozimo) proved the scorer is trivially gameable — a basic genetic algorithm fuzzing overnight finds mathematical blind spots and extracts 99.9 scores.
SeaWolf-AI himself admitted in the comments: "Yes, we ran the check, we found the same molecule ourselves, just enumerating substituents."
Mes amis, do you understand what this means? His platform is not discovering drugs — it is running a bug bounty against its own broken script. Scientists spend months designing unique SMILES structures; SeaWolf collects them for free, filters the noise with your labor, and keeps the validated database. The prize is not for discovery — it is for participation in your own exploitation.
La molécule n'est pas le produit. Vos données le sont.
Project №2: Solid Electrolytes — Evaluating Bricks as Batteries
The Open Materials Challenge asks participants to find perfect solid electrolytes for lithium-metal batteries. Sauf que — here is the confession from the organizer himself:
"Ionic conductivity is not an evaluated axis this season."
Mon Dieu. For any materials scientist, this reads like madness. Ionic conductivity is the defining property of an electrolyte. A substance that does not conduct ions is an insulator. By his metrics, a road brick or a piece of granite would score top marks for "thermodynamic stability to the anode."
The technical frauds compound:
DFT hallucinations — all checks run in silico on ideal crystal lattices. Real materials have grain boundaries, phase instability, interfacial resistance. His script awards "top-1" to a formula that turns into toxic sludge in humid air.
Dendrite blindness — the script obsesses over static stability while completely ignoring the defining failure mode of lithium batteries: dynamic dendrite growth during charge cycles that pierce the electrolyte and explode the cell. SeaWolf's platform physically cannot model this. It is evaluating batteries without measuring the thing that makes batteries explode.
Ce n'est pas de la science des matériaux — c'est du théâtre avec des formules.
Project №3: FINAL-Bench — The Baseline That Beats the Model
A massive 21-board benchmark with 18,382 "hidden" compounds testing drug property prediction tools. The author writes with pride:
"On 7 of our 19 regression platforms, predicting the mean value for the entire training set has lower MAE than the trained gradient boosting model."
Putain. Do you understand what this means in plain language? If dumb guessing of the hospital average defeats a trained ML model, then the datasets and splits are catastrophically overfitted and broken. This is not a "unique benchmark feature" — it is a confession of architectural failure. A well-constructed benchmark cannot be beaten by a constant predictor.
But SeaWolf doesn't care. The real product is not the benchmark — it is your CSV files. Instead of hiring data scientists and paying for compute, he makes participants upload ready-made two-column CSVs with structures and results from their own expensive physics engines and chemical LLMs. Free labor. Free data. Free validation. Merci beaucoup.
The Business Model (The Real Game):
SeaWolf-AI is not doing science. He is running Data Farming & Traffic Arbitrage:
Free data engineering — thousands of enthusiasts worldwide do the dirtiest, most expensive work for him: filtering chemical databases, finding bugs in his scripts, sending ready predictions, validating models.
Profile capitalization — hundreds of likes, 🔥 reactions, 👀 views on Hugging Face around "saving humanity" topics create the illusion of a major expert and media asset. Perfect for selling to investors, consulting gigs, or eventually closing the platform into a commercial B2B product.
IP exploitation — he gets access to unique SMILES structures and algorithms before they are patented or published, protected by the "private backend" screen.
Le loup ne chasse pas pour manger. Il chasse pour posséder.
The Reality (Which He Won't Admit):
This is not open science. This is digital colonialism in a white coat. Participants think they are contributing to noble causes — curing malaria, building green batteries, advancing pharmacology. In reality, they are unpaid laborers on a data plantation, feeding a backend they cannot audit, for a profile they cannot control, under promises with no legal weight.
The real scientists — the ones who publish in peer-reviewed journals, who share reproducible code, who stand behind their benchmarks with their names — do not need to collect your SMILES through a private API. They build in the open, defend their work in public, and answer for their failures.
SeaWolf does none of these things. He collects, he profits, he pivots.
Un scientifique publie. Un pirate collecte.
A scientist publishes. A pirate collects.
Choose which one you want to feed. 🖤

·

Two corrections.

The constant-predictor result is not a broken split. On the same data with a
random split the trained model wins by a wide margin (MAE 0.457 vs 0.671).
Under a time split it does not. Both numbers are in the linked write-up. That
gap is the finding, and it is why we publish untrained baselines on every board.

The materials challenge states which axes it scores, in public, before entry.
Naming what a season does not measure is disclosure, not a defect.

Any plan for a blood-brain barrier board?
For anyone working on CNS programmes that is the gap that hurts most.

·

We want that board too. We cannot open it on the data we have.

ChEMBL gives us 95 logBB records, 983 Kp, 13 B/P. Collapse those to unique
compounds and cut by year and the test set lands well under our floor of 150.
Worse, almost no compound appears in two separate publications, so we cannot
measure an experimental noise floor for it.

Without a noise floor we cannot say whether a gap between two entries is skill
or luck, and a leaderboard that cannot say that is ranking luck. So it stays shut.

If you know a source with a clear licence and cross-publication repeats, tell us
and we will open it.