| --- |
| license: mit |
| library_name: pytorch |
| tags: |
| - vision-language |
| - clip |
| - qwen2.5 |
| - predictive-coding |
| - clevr |
| --- |
| |
| # PCN Bridge Lab Connector Weights |
|
|
| Connector-only weights for CLIP ViT-B/32 to Qwen2.5-0.5B-Instruct CLEVR experiments. |
|
|
| Code, training scripts, results, and plots: https://github.com/REDDITARUN/pcn-bridge-lab |
|
|
| This is an AI-assisted research/engineering experiment with a small human-in-the-loop workflow for direction, review, and decisions. |
|
|
| Base models are not included: |
|
|
| - Vision encoder: `openai/clip-vit-base-patch32` |
| - Language model: `Qwen/Qwen2.5-0.5B-Instruct` |
|
|
| Each `.pt` file contains only: |
|
|
| - connector state dict |
| - run config |
| - best validation loss |
| - candidate-choice test metrics |
|
|
| Optimizer state and full frozen model weights are intentionally omitted. |
|
|
| ## Results |
|
|
| | Rank | Run | Choice Accuracy | File | |
| |---:|---|---:|---| |
| | 1 | `pcn-depth3-s1-eqprop-momentum` | `47.2%` | `connectors/pcn-depth3-s1-eqprop-momentum.pt` | |
| | 2 | `pcn-depth3-s1-adamw` | `46.3%` | `connectors/pcn-depth3-s1-adamw.pt` | |
| | 3 | `pcn-depth3-s6-adamw` | `45.9%` | `connectors/pcn-depth3-s6-adamw.pt` | |
| | 4 | `pcn-depth3-s12-eqprop-momentum` | `45.5%` | `connectors/pcn-depth3-s12-eqprop-momentum.pt` | |
| | 5 | `pcn-depth3-s6-sgd` | `45.5%` | `connectors/pcn-depth3-s6-sgd.pt` | |
| | 6 | `pcn-depth3-s1-sgd` | `45.2%` | `connectors/pcn-depth3-s1-sgd.pt` | |
| | 7 | `linear-adamw` | `45.0%` | `connectors/linear-adamw.pt` | |
| | 8 | `pcn-depth3-s12-sgd` | `45.0%` | `connectors/pcn-depth3-s12-sgd.pt` | |
| | 9 | `mlp-depth3-adamw` | `43.8%` | `connectors/mlp-depth3-adamw.pt` | |
| | 10 | `pcn-depth3-s6-eqprop-momentum` | `43.2%` | `connectors/pcn-depth3-s6-eqprop-momentum.pt` | |
| | 11 | `pcn-depth3-s12-adamw` | `43.0%` | `connectors/pcn-depth3-s12-adamw.pt` | |
|
|
| ## Loading |
|
|
| Use the code in the GitHub repository to instantiate `ConnectorVLM`, then load the `connector` state dict from a file in `connectors/`. |
|
|
| ```python |
| import torch |
| from src.model import ConnectorVLM |
| |
| ckpt = torch.load('connectors/pcn-depth3-s1-eqprop-momentum.pt', map_location='cpu') |
| model = ConnectorVLM(ckpt['config']) |
| model.connector.load_state_dict(ckpt['connector']) |
| ``` |
|
|
| ## Caveat |
|
|
| The primary metric is candidate-choice accuracy over valid CLEVR answers. Free-form generations remain noisy and should be treated as diagnostic only. |
|
|
