Zero-Shot Image Classification
Transformers
Safetensors
tipsv2
feature-extraction
vision
image-text
contrastive-learning
zero-shot
custom_code
Instructions to use google/tipsv2-g14 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use google/tipsv2-g14 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("zero-shot-image-classification", model="google/tipsv2-g14", trust_remote_code=True) pipe( "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/parrots.png", candidate_labels=["animals", "humans", "landscape"], )# Load model directly from transformers import AutoProcessor, AutoModel processor = AutoProcessor.from_pretrained("google/tipsv2-g14", trust_remote_code=True) model = AutoModel.from_pretrained("google/tipsv2-g14", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Gabriele commited on
Commit ·
c7d0623
1
Parent(s): 8e6f865
Update README
Browse files
README.md
CHANGED
|
@@ -10,7 +10,7 @@ library_name: transformers
|
|
| 10 |
pipeline_tag: zero-shot-image-classification
|
| 11 |
---
|
| 12 |
|
| 13 |
-
# TIPSv2 —
|
| 14 |
|
| 15 |
TIPSv2 (Text-Image Pre-training with Spatial awareness) is a family of contrastive vision-language models that produce spatially rich image features aligned with text embeddings. This is the Giant variant with 1.1B vision params and 389M text params.
|
| 16 |
|
|
@@ -74,6 +74,17 @@ similarity = cls @ text_emb.T
|
|
| 74 |
prediction = similarity.argmax(dim=-1)
|
| 75 |
```
|
| 76 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 77 |
### GPU inference
|
| 78 |
|
| 79 |
```python
|
|
|
|
| 10 |
pipeline_tag: zero-shot-image-classification
|
| 11 |
---
|
| 12 |
|
| 13 |
+
# TIPSv2 — g/14
|
| 14 |
|
| 15 |
TIPSv2 (Text-Image Pre-training with Spatial awareness) is a family of contrastive vision-language models that produce spatially rich image features aligned with text embeddings. This is the Giant variant with 1.1B vision params and 389M text params.
|
| 16 |
|
|
|
|
| 74 |
prediction = similarity.argmax(dim=-1)
|
| 75 |
```
|
| 76 |
|
| 77 |
+
### Zero-shot segmentation
|
| 78 |
+
|
| 79 |
+
```python
|
| 80 |
+
import numpy as np
|
| 81 |
+
from sklearn.decomposition import PCA
|
| 82 |
+
|
| 83 |
+
spatial = out.patch_tokens.reshape(1, 32, 32, 1536)
|
| 84 |
+
feat = spatial[0].detach().numpy().reshape(-1, 1536)
|
| 85 |
+
rgb = PCA(n_components=3).fit_transform(feat).reshape(32, 32, 3)
|
| 86 |
+
```
|
| 87 |
+
|
| 88 |
### GPU inference
|
| 89 |
|
| 90 |
```python
|