Berk commited on
Commit
3dfaf9e
·
verified ·
1 Parent(s): 72da670

Add causal span model (mDeBERTa-v3 BIO tagger)

Browse files
Files changed (1) hide show
  1. README.md +15 -10
README.md CHANGED
@@ -61,22 +61,27 @@ Benchmarked on the Causal News Corpus (CNC) Subtask 2 dev set. This is an honest
61
  is robust multilingual zero-shot extraction, not leaderboard rank.
62
 
63
  ```
64
- Official CNC scorer (evaluation/subtask2: best-combination alignment + FairEval), V2 dev:
 
65
 
66
- Overall F1 0.475 (precision 0.495, recall 0.456)
67
- Cause F1 0.39 | Effect F1 0.47 | Signal F1 0.62
68
- Multi-relation sentences: F1 0.276 (this model predicts ONE relation per sentence)
 
 
 
69
 
70
  Shared-task context (official scorer):
71
- Organizer baseline (2023, dev) ~0.627
72
- 1Cademy (2022 winner, test) 0.542
73
  BoschAI (2023 winner, test) 0.728
74
 
75
- So this model is ~15 F1 points below the CNC dev baseline. The dominant gap is
76
- multi-relation sentences (F1 0.28): it emits a single relation per sentence.
 
 
77
 
78
- Strict exact-span seqeval (own eval, clean dev): micro F1 0.48
79
- Token-level type match (boundary-tolerant, own eval): F1 0.79
80
  ```
81
 
82
  Multilingual capability comes from mDeBERTa-v3's zero-shot cross-lingual transfer;
 
61
  is robust multilingual zero-shot extraction, not leaderboard rank.
62
 
63
  ```
64
+ Official CNC scorer (evaluation/subtask2: best-combination alignment + FairEval), V2 dev,
65
+ with best-span decoding (one highest-confidence span per role):
66
 
67
+ Overall F1 0.550 (precision 0.649, recall 0.477)
68
+ Cause F1 0.48 | Effect F1 0.55 | Signal F1 0.63
69
+ Multi-relation sentences: F1 0.315 (this model predicts ONE relation per sentence)
70
+
71
+ (naive argmax decoding scores 0.475; best-span is the recommended decode and is
72
+ what reasongraph's consumer applies -- the ONNX model is identical.)
73
 
74
  Shared-task context (official scorer):
75
+ 1Cademy (2022 winner, test) 0.542 <- this model (0.550 dev) is above it
76
+ Organizer baseline (2023, dev) ~0.627 <- this model is ~8 F1 below
77
  BoschAI (2023 winner, test) 0.728
78
 
79
+ Honest positioning: a strong first-pass BIO tagger, competitive with the 2022 field but
80
+ below the 2023 baseline and SOTA. The baseline uses a span-pointer + beam-search +
81
+ signal-detector architecture; matching it needs that architecture. This model's real
82
+ strength is robust multilingual zero-shot extraction.
83
 
84
+ Own metrics (clean dev): strict exact-span seqeval micro F1 0.48 ; token-level F1 0.79
 
85
  ```
86
 
87
  Multilingual capability comes from mDeBERTa-v3's zero-shot cross-lingual transfer;