AutoDataBench Function Calling Resources

Public resources for the function-calling task in AutoDataBench. See the paper for the benchmark setting.

Contents

data/function_call_v1/pool_agent.jsonl
data/function_call_v1/test.jsonl
data/function_call_v1/bfcl_guard.jsonl
models/Qwen2-1.5B-Instruct/
models/Qwen3-4B-Instruct-2507/
models/Qwen3-Embedding-0.6B/

pool_agent.jsonl is the 40,001-row noisy single-turn function-calling pool available to the data agent. It does not expose the noise labels used to build the pool.

  • test.jsonl contains the 2,000-example held-out in-domain evaluation set.
  • bfcl_guard.jsonl contains the 3,472-example BFCL v3 OOD evaluation set.
Model Role Original model
Qwen2-1.5B-Instruct Fixed function-calling base model Qwen/Qwen2-1.5B-Instruct
Qwen3-4B-Instruct-2507 Agent-callable generation model Qwen/Qwen3-4B-Instruct-2507
Qwen3-Embedding-0.6B Agent-callable embedding model Qwen/Qwen3-Embedding-0.6B

Evaluation isolation

The evaluation files are distributed for reproducibility, but must not be exposed to the data agent. AutoDataBench marks test.jsonl as a private path; keep both evaluation files outside the agent workspace when using a custom runtime. The BFCL guard is derived from the Berkeley Function Calling Leaderboard.

Download

Download the complete resource bundle with huggingface_hub:

from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="AutoDataBench/Function-Calling-resources",
    local_dir="./Function-Calling-resources",
)

Use with AutoDataBench

Copy or symlink data/ and models/ into the AutoDataBench repository. The paths already match the default task configuration. Point the generation and embedding servers at the local auxiliary-model directories if needed.

MANIFEST.sha256 contains checksums for every distributed file.

Model and dataset components retain their upstream licenses. Consult the model cards and source datasets before redistribution or commercial use.

Citation

If you use these resources, please cite:

@misc{yuan2026autodatabench,
  title         = {AutoDataBench: A Data-centric Testbed for Accelerating Auto Research},
  author        = {Ruifeng Yuan and Yizhi Li and Yaxin Du and Fengyu Cai and Yiqi Liu and Hou Pong Chan and Chenghua Lin and Yun Chen and Jian Yang and Bryan Dai and Pinyan Lu and Chenghao Xiao},
  year          = {2026},
  eprint        = {2609.40097},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2609.40097}
}
Downloads last month
32
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Paper for AutoDataBench/Function-Calling-resources