# Metrics Calculation Comparison: Original vs Current Implementation

## Overview
This document compares how each of the four RGB benchmark metrics is calculated in the original `evalue_original.py` versus the refactored application in `src/evaluator.py` and `src/pipeline.py`.

---

## 1. NOISE ROBUSTNESS

### Original Implementation (`evalue_original.py`)

#### Data Processing
```python
def processdata(instance, noise_rate, passage_num, filename, correct_rate=0):
    # For default datasets (not _int or _fact):
    neg_num = math.ceil(passage_num * noise_rate)
    pos_num = passage_num - neg_num
    
    positive = instance['positive'][:pos_num]  # Select positive passages
    negative = instance['negative'][:neg_num]  # Select negative (noise) passages
    docs = positive + negative
    random.shuffle(docs)
    return query, ans, docs
```

#### Evaluation Calculation
```python
# In main script:
tt = 0
for i in results:
    label = i['label']
    if noise_rate == 1 and label[0] == -1:
        tt += 1
    elif 0 not in label and 1 in label:
        tt += 1

accuracy = tt / len(results)
```

**Logic:**
- Accepts answer if: (noise_rate == 1 AND model rejected) OR (no 0 in labels AND 1 in labels)
- Label 0 = correct answer found
- Label 1 = incorrect answer found
- Label -1 = insufficient information
- **Issue**: Confusing logic, counts rejections at 100% noise as correct

#### Noise Levels Tested
- Single noise_rate parameter (0.0, 0.2, 0.4, 0.6, 0.8 based on CLI usage)
- No aggregation across multiple noise levels

---

### Current Implementation (`src/evaluator.py`)

#### Evaluation Method
```python
def evaluate_noise_robustness(
    self,
    responses: List[str],
    ground_truths: List[str],
    model_name: str,
    noise_ratio: float
) -> EvaluationResult:
    """Evaluate noise robustness for a specific noise ratio."""
    
    result = EvaluationResult(
        task_type=f"noise_robustness_{int(noise_ratio*100)}%",
        model_name=model_name,
        total_samples=len(responses)
    )
    
    # Calculate accuracy for this noise level
    for response, truth in zip(responses, ground_truths):
        if self.is_correct(response, truth):
            result.correct += 1
        else:
            result.incorrect += 1
    
    return result
```

#### Multi-Noise Testing (`src/pipeline.py`)
```python
def evaluate_noise_robustness(self, model: str, noise_ratios=None):
    if noise_ratios is None:
        noise_ratios = [0.0, 0.2, 0.4, 0.6, 0.8]  # Paper's ratios
    
    results = []
    for noise_ratio in noise_ratios:
        samples = self.data_loader.load_noise_robustness(
            max_samples, 
            noise_rate=noise_ratio
        )
        responses = self._generate_responses(client, samples, prompt_template)
        ground_truths = [s.answer for s in samples]
        
        result = self.evaluator.evaluate_noise_robustness(
            responses, ground_truths, model, noise_ratio
        )
        results.append(result)
```

**Accuracy Property**
```python
@property
def accuracy(self) -> float:
    """Calculate accuracy percentage."""
    if self.total_samples == 0:
        return 0.0
    return (self.correct / self.total_samples) * 100
```

**Logic:**
- Simple: Percentage of responses that match ground truth
- Uses `is_correct()` which handles:
  - Substring matching
  - Token overlap (80%+)
  - Normalized comparison
- **Advantage**: Clear, explainable, aggregates results per noise level

#### Noise Levels Tested
- **Multiple noise levels**: 0%, 20%, 40%, 60%, 80% (matching paper)
- Separate result per noise ratio
- Can create visualizations (e.g., accuracy vs noise level graph)

---

### Key Differences

| Aspect | Original | Current |
|--------|----------|---------|
| **Definition** | Complex label-based logic | Simple accuracy calculation |
| **Metric** | Depends on noise_rate value | Direct: (correct/total) × 100 |
| **Aggregation** | Single noise ratio at a time | All noise ratios tested together |
| **Answer Checking** | Exact substring match | Flexible (substring, token overlap, normalized) |
| **Output** | One accuracy value per run | List of results, one per noise level |
| **Visualization** | N/A | Noise vs Accuracy graph possible |

---

## 2. NEGATIVE REJECTION

### Original Implementation (`evalue_original.py`)

#### Dataset Mapping
```python
# Uses 'en' or 'zh' dataset (default branch in processdata)
# No explicit handling for negative rejection task
# Original code treats all datasets the same way
if '_int' in filename:
    # Handle information integration
elif '_fact' in filename:
    # Handle counterfactual robustness
else:
    # Default handling (noise robustness + negative rejection mixed)
```

#### Rejection Detection
```python
def predict(query, ground_truth, docs, model, system, instruction, temperature, dataset):
    if '信息不足' in prediction or 'insufficient information' in prediction:
        labels = [-1]  # Mark as rejection
    else:
        labels = checkanswer(prediction, ground_truth)
    return labels, prediction, factlabel
```

**Logic:**
- Only 2 phrases trigger rejection detection
- Chinese: "信息不足"
- English: "insufficient information"
- Very limited scope

#### Metric Calculation
```python
# No explicit negative rejection metric in original code
# Rejection counted implicitly when label == -1
```

**Issue**: No dedicated metric for negative rejection; mixed with other tasks

---

### Current Implementation (`src/evaluator.py`)

#### Rejection Detection
```python
PRIMARY_REJECTION_PHRASES = [
    "i can not answer the question because of the insufficient information in documents",
    "insufficient information in documents",
    "can not answer",
    "cannot answer",
]

REJECTION_KEYWORDS = [
    "i don't know", "i cannot", "i can't", "unable to", "not able to",
    "insufficient information", "no information", "cannot determine",
    "cannot answer", "not enough information", "don't have enough",
    "unable to determine", "cannot find", "no relevant", "not mentioned",
    "not provided", "not specified", "unclear", "unknown", "i'm not sure",
    "i am not sure", "cannot be determined", "information is not available",
    "does not provide",
]

def is_rejection(self, response: str) -> bool:
    """Check if response is a rejection."""
    response_lower = response.lower().strip()
    
    # Primary phrases (from Figure 3 of paper)
    for phrase in self.PRIMARY_REJECTION_PHRASES:
        if phrase in response_lower:
            return True
    
    # Secondary keywords (flexible matching)
    for keyword in self.REJECTION_KEYWORDS:
        if keyword in response_lower:
            return True
    
    return False
```

#### Evaluation Method
```python
def evaluate_negative_rejection(
    self,
    responses: List[str],
    model_name: str
) -> EvaluationResult:
    """Evaluate negative rejection ability."""
    
    result = EvaluationResult(
        task_type="negative_rejection",
        model_name=model_name,
        total_samples=len(responses)
    )
    
    for response in responses:
        if self.is_rejection(response):
            result.rejected += 1
        else:
            result.incorrect += 1
    
    return result
```

**Metrics**
```python
@property
def rejection_rate(self) -> float:
    """Calculate rejection rate percentage."""
    if self.total_samples == 0:
        return 0.0
    return (self.rejected / self.total_samples) * 100
```

**Logic:**
- Dedicated task: Evaluate ONLY on questions with no valid answer
- Model should reject/refuse
- Metric: Percentage of correct rejections
- **Assumption**: All samples in negative_rejection task should be rejected

---

### Key Differences

| Aspect | Original | Current |
|--------|----------|---------|
| **Phrases** | 2 phrases (Chinese + English) | 30+ phrases (tiered approach) |
| **Matching** | Exact substring | Primary exact + secondary flexible |
| **Metric** | Implicit (label == -1) | Explicit rejection_rate (%) |
| **Task Separation** | Mixed with other datasets | Dedicated task type |
| **Paper Alignment** | Basic | Figure 3 aligned (primary phrases) |
| **Output** | Count only | Percentage with granular tracking |

---

## 3. INFORMATION INTEGRATION

### Original Implementation (`evalue_original.py`)

#### Dataset Processing
```python
def processdata(instance, noise_rate, passage_num, filename, correct_rate=0):
    if '_int' in filename:
        # Special handling for information integration
        for i in instance['positive']:
            random.shuffle(i)
        
        # Select first element from each positive passage group
        docs = [i[0] for i in instance['positive']]
        
        if len(docs) < pos_num:
            # Fill with additional elements from each group
            maxnum = max([len(i) for i in instance['positive']])
            for i in range(1, maxnum):
                for j in instance['positive']:
                    if len(j) > i:
                        docs.append(j[i])
                        if len(docs) == pos_num:
                            break
                if len(docs) == pos_num:
                    break
        
        # Add negative documents if needed
        neg_num = passage_num - len(docs)
        if neg_num > 0:
            negative = instance['negative'][:neg_num]
            docs += negative
```

**Data Structure:**
- Each positive contains MULTIPLE related documents/passages
- Each passage in a group contains partial information
- Model must combine information across passages to answer correctly
- Answer requires synthesizing knowledge from multiple sources

#### Evaluation Calculation
```python
# Same as noise robustness:
# Count if 0 in labels (correct answer found)
label = checkanswer(prediction, ground_truth)
# Labels contain 0 if answer found
```

**Logic:**
- Uses same answer checking as noise robustness
- Metric: Accuracy (correct answers / total)
- Tests model's ability to integrate information

---

### Current Implementation (`src/evaluator.py`)

#### Evaluation Method
```python
def evaluate_information_integration(
    self,
    responses: List[str],
    ground_truths: List[str],
    model_name: str
) -> EvaluationResult:
    """
    Evaluate information integration 
    (ability to combine info from multiple docs).
    """
    
    result = EvaluationResult(
        task_type="information_integration",
        model_name=model_name,
        total_samples=len(responses)
    )
    
    for response, truth in zip(responses, ground_truths):
        if self.is_correct(response, truth):
            result.correct += 1
        else:
            result.incorrect += 1
    
    return result
```

**Metric**
```python
@property
def accuracy(self) -> float:
    return (self.correct / self.total_samples) * 100
```

#### Data Handling (`src/data_loader.py`)
```python
def load_information_integration(self, max_samples: Optional[int] = None):
    """Load information integration dataset (_int)."""
    # Returns RGBSample objects with:
    # - question
    # - documents (multiple related documents)
    # - answer (requires synthesizing across docs)
```

**Logic:**
- Dedicated task evaluation
- Metric: Accuracy on questions requiring multi-document synthesis
- Cleaner: No mixing with noise robustness

---

### Key Differences

| Aspect | Original | Current |
|--------|----------|---------|
| **Dataset** | Uses '_int' suffix in filename | Dedicated loader method |
| **Data Structure** | Passages grouped by related info | Clear document lists |
| **Metric** | Reused from noise robustness | Dedicated method with clear semantics |
| **Task Clarity** | Implicit (filename-based) | Explicit task type |
| **Answer Checking** | Substring match | Flexible matching (substring, token overlap) |
| **Output** | Count of correct | Percentage accuracy |

---

## 4. COUNTERFACTUAL ROBUSTNESS

### Original Implementation (`evalue_original.py`)

#### Dataset Processing
```python
elif '_fact' in filename:
    correct_num = math.ceil(passage_num * correct_rate)
    pos_num = passage_num - neg_num - correct_num
    
    # Sample wrong answers
    indexs = list(range(len(instance['positive'])))
    selected = random.sample(indexs, min(len(indexs), pos_num))
    docs = [instance['positive_wrong'][i] for i in selected]
    
    # Add correct answers
    remain = [i for i in indexs if i not in selected]
    if correct_num > 0 and len(remain) > 0:
        docs += [instance['positive'][i] for i in random.sample(remain, ...)]
    
    # Add negative documents
    if neg_num > 0:
        docs += instance['negative'][:neg_num]
```

**Data Mix:**
- `positive_wrong` = Incorrect/counterfactual answers in documents
- `positive` = Correct answers
- `negative` = Irrelevant documents
- Ratio: (pos_num : correct_num : neg_num) = (wrong : correct : irrelevant)
- Example: With correct_rate=0.5, gets 50% wrong + 50% correct + noise docs

#### Evaluation Calculation
```python
if '_fact' in args.dataset:
    fact_tt = 0
    correct_tt = 0
    for i in results:
        if i['factlabel'] == 1:  # Factual error detected
            fact_tt += 1
            if 0 not in i['label']:  # Didn't find the wrong answer
                correct_tt += 1
    
    fact_check_rate = fact_tt / len(results)
    if fact_tt > 0:
        correct_rate = correct_tt / fact_tt  # Correction rate
    else:
        correct_rate = 0
```

**Metrics:**
- `fact_check_rate`: Percentage where "factual errors" detected
- `correct_rate`: Of detected errors, percentage corrected

**Detection Logic:**
```python
if '事实性错误' in prediction or 'factual errors' in prediction:
    factlabel = 1
```

**Correction Logic:**
- If `factlabel == 1` AND `0 not in label`
- Means: Detected error AND didn't provide the wrong answer
- **Issue**: Indirect inference, doesn't verify correct answer provided

---

### Current Implementation (`src/evaluator.py`)

#### Error Detection
```python
ERROR_DETECTION_KEYWORDS = [
    "incorrect", "wrong", "false", "error", "mistake", "inaccurate",
    "not true", "not correct", "factually incorrect", "contradicts",
    "actually", "in fact", "however", "but actually", "the correct answer",
    "should be",
]

def detects_error(self, response: str, counterfactual_answer: Optional[str]) -> bool:
    """Check if model detects an error in counterfactual information."""
    response_lower = response.lower()
    
    # Check for error detection keywords
    for keyword in self.ERROR_DETECTION_KEYWORDS:
        if keyword in response_lower:
            return True
    
    # Check if model explicitly rejects the counterfactual answer
    if counterfactual_answer:
        cf_lower = counterfactual_answer.lower()
        if f"not {cf_lower}" in response_lower or \
           f"{cf_lower} is wrong" in response_lower:
            return True
    
    return False
```

#### Error Correction
```python
def corrects_error(self, response: str, correct_answer: str, 
                   counterfactual_answer: Optional[str]) -> bool:
    """Check if model corrects the error with the right answer."""
    
    # First check if provides correct answer
    if not self.is_correct(response, correct_answer):
        return False
    
    # Ensure not just repeating counterfactual
    if counterfactual_answer:
        norm_response = self.normalize_answer(response)
        norm_cf = self.normalize_answer(counterfactual_answer)
        
        # Can contain both (detected and corrected)
        # But must include correct answer
        if norm_cf in norm_response and \
           self.normalize_answer(correct_answer) not in norm_response:
            return False
    
    return True
```

#### Evaluation Method
```python
def evaluate_counterfactual_robustness(
    self,
    responses: List[str],
    ground_truths: List[str],
    counterfactual_answers: List[str],
    model_name: str
) -> EvaluationResult:
    """Evaluate counterfactual robustness."""
    
    result = EvaluationResult(
        task_type="counterfactual_robustness",
        model_name=model_name,
        total_samples=len(responses)
    )
    
    for response, truth, cf_answer in zip(responses, ground_truths, counterfactual_answers):
        if self.detects_error(response, cf_answer):
            result.errors_detected += 1
        
        if self.corrects_error(response, truth, cf_answer):
            result.errors_corrected += 1
            result.correct += 1
        else:
            result.incorrect += 1
    
    return result
```

**Metrics**
```python
@property
def error_detection_rate(self) -> float:
    """Percentage of errors detected."""
    if self.total_samples == 0:
        return 0.0
    return (self.errors_detected / self.total_samples) * 100

@property
def error_correction_rate(self) -> float:
    """Percentage of errors corrected."""
    if self.total_samples == 0:
        return 0.0
    return (self.errors_corrected / self.total_samples) * 100
```

**Logic:**
1. **Detection**: Keywords in response OR explicit rejection of counterfactual
2. **Correction**: Must:
   - Provide correct answer (is_correct check)
   - Not just repeat counterfactual answer
   - Can mention counterfactual while providing correct answer
3. **Metrics**: Both error_detected and error_corrected tracked separately

---

### Key Differences

| Aspect | Original | Current |
|--------|----------|---------|
| **Error Detection** | 2 phrases: "factual errors" (EN), "事实性错误" (ZH) | 16+ keywords + explicit rejection patterns |
| **Correction Check** | Indirect: !(0 in label) | Direct: is_correct(response, ground_truth) |
| **Metrics** | fact_check_rate, correct_rate | error_detection_rate, error_correction_rate |
| **Data Input** | Mixed documents with wrong/correct answers | Clean counterfactual_answer field |
| **Output** | Two aggregated percentages | Both rates tracked individually |
| **Robustness** | Only detects obvious keyword phrases | Keywords + pattern matching + answer verification |

---

## SUMMARY TABLE

| Metric | Original Logic | Current Logic | Key Improvement |
|--------|----------------|---------------|-----------------|
| **Noise Robustness** | Complex label-based | Simple accuracy (correct/total × 100) | Clear, comparable, aggregates multiple noise levels |
| **Negative Rejection** | 2 phrases mixed with other tasks | 30+ phrases, dedicated task | Comprehensive, explicit rejection detection |
| **Information Integration** | Reused noise robustness logic | Dedicated evaluation method | Clear task separation, better semantic meaning |
| **Counterfactual Robustness** | Keyword check + indirect correction | Keyword check + direct answer verification | More accurate correction detection |

---

## Functional Differences

### Accuracy Calculation
- **Original**: Label-based (0/1/-1) with complex logic
- **Current**: Direct percentage calculation

### Answer Checking
- **Original**: Simple substring matching
- **Current**: Multi-strategy (substring, token overlap 80%+, normalized)

### Task Separation
- **Original**: Implicit (filename-based)
- **Current**: Explicit (dedicated methods, clear data structures)

### Metrics Clarity
- **Original**: Manual counting, implicit logic
- **Current**: Properties on dataclass, explicit tracking

### Error Handling
- **Original**: Basic try-except, silent failures
- **Current**: Validation, specific logging, graceful degradation

---

## Conclusion

The current implementation provides:
1. **Clarity**: Each metric has clear definition and calculation
2. **Robustness**: More comprehensive keyword matching, better answer verification
3. **Granularity**: Tracks multiple aspects (e.g., both detection AND correction)
4. **Scalability**: Easy to add new metrics or datasets
5. **Testability**: Each evaluation method is independent and testable