Instructions to use microsoft/Florence-2-large with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use microsoft/Florence-2-large with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="microsoft/Florence-2-large", trust_remote_code=True)# Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("microsoft/Florence-2-large", trust_remote_code=True) model = AutoModelForMultimodalLM.from_pretrained("microsoft/Florence-2-large", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use microsoft/Florence-2-large with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "microsoft/Florence-2-large" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "microsoft/Florence-2-large", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/microsoft/Florence-2-large
- SGLang
How to use microsoft/Florence-2-large with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "microsoft/Florence-2-large" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "microsoft/Florence-2-large", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "microsoft/Florence-2-large" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "microsoft/Florence-2-large", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use microsoft/Florence-2-large with Docker Model Runner:
docker model run hf.co/microsoft/Florence-2-large
Fix AttributeError from _supports_sdpa read before language_model exists
Browse filesFlorence2PreTrainedModel._supports_sdpa/_supports_flash_attn_2 are properties that forward to self.language_model._supports_sdpa. But self.language_model is only assigned partway through Florence2ForConditionalGeneration.__init__, after super().__init__(config) runs -- and this transformers version's PreTrainedModel.__init__ now reads self._supports_sdpa during that super().__init__() call, before language_model exists.
The resulting AttributeError from the inner self.language_model access gets silently reinterpreted by Python's descriptor protocol (property getter raises AttributeError -> falls through to nn.Module.__getattr__ -> raises a new AttributeError naming the outer attribute) -- hence the confusing "no attribute '_supports_sdpa'" message instead of the real cause.
Fall back to True (the constant Florence2LanguageForConditionalGeneration always declares for both flags) when language_model isn't set yet.
- modeling_florence2.py +13 -4
|
@@ -2335,17 +2335,26 @@ class Florence2PreTrainedModel(PreTrainedModel):
|
|
| 2335 |
def _supports_flash_attn_2(self):
|
| 2336 |
"""
|
| 2337 |
Retrieve language_model's attribute to check whether the model supports
|
| 2338 |
-
Flash Attention 2 or not.
|
|
|
|
|
|
|
|
|
|
| 2339 |
"""
|
| 2340 |
-
|
|
|
|
|
|
|
|
|
|
| 2341 |
|
| 2342 |
@property
|
| 2343 |
def _supports_sdpa(self):
|
| 2344 |
"""
|
| 2345 |
Retrieve language_model's attribute to check whether the model supports
|
| 2346 |
-
SDPA or not.
|
| 2347 |
"""
|
| 2348 |
-
|
|
|
|
|
|
|
|
|
|
| 2349 |
|
| 2350 |
|
| 2351 |
FLORENCE2_INPUTS_DOCSTRING = r"""
|
|
|
|
| 2335 |
def _supports_flash_attn_2(self):
|
| 2336 |
"""
|
| 2337 |
Retrieve language_model's attribute to check whether the model supports
|
| 2338 |
+
Flash Attention 2 or not. PreTrainedModel.__init__ reads this before
|
| 2339 |
+
self.language_model is assigned (it's set partway through this class's
|
| 2340 |
+
own __init__, after super().__init__() runs), so fall back to the
|
| 2341 |
+
constant Florence2LanguageForConditionalGeneration always declares.
|
| 2342 |
"""
|
| 2343 |
+
language_model = getattr(self, "language_model", None)
|
| 2344 |
+
if language_model is None:
|
| 2345 |
+
return True
|
| 2346 |
+
return language_model._supports_flash_attn_2
|
| 2347 |
|
| 2348 |
@property
|
| 2349 |
def _supports_sdpa(self):
|
| 2350 |
"""
|
| 2351 |
Retrieve language_model's attribute to check whether the model supports
|
| 2352 |
+
SDPA or not. Same early-access issue as _supports_flash_attn_2 above.
|
| 2353 |
"""
|
| 2354 |
+
language_model = getattr(self, "language_model", None)
|
| 2355 |
+
if language_model is None:
|
| 2356 |
+
return True
|
| 2357 |
+
return language_model._supports_sdpa
|
| 2358 |
|
| 2359 |
|
| 2360 |
FLORENCE2_INPUTS_DOCSTRING = r"""
|