โ† AI Tools
Image AIIntermediate

LocateAnything-3B

<!

NVIDIA's LocateAnything-3B, released in July 2026, is a 3-billion-parameter open-vocabulary vision-language model (VLM) designed to precisely locate specific objects, text, or user interface (UI) elements within an image using bounding box coordinates. Built on a highly integrated architecture combining the MoonViT-SO-400M visual encoder and the Qwen2.5-3B-Instruct language model, it instantly infers the visual location of a target specified by the user in natural language. By merging the language model's exceptional contextual understanding with high-resolution image processing capabilities, it goes beyond simple object detection, excelling in tasks such as providing visual grounding that aligns with complex descriptive sentences and inferring the location of GUI elements.

Existing large-scale vision-language models have inherent structural limitations, including slow inference speeds and high computational resource consumption, due to their adoption of an autoregressive approach that sequentially predicts text tokens to derive the coordinates of a target. To overcome this bottleneck, LocateAnything-3B introduces a parallel box decoding (PBD) method. Similar to how a high-performance ultrasound diagnostic device scans the entire contrast profile of biological tissue at once, it extracts a large number of location area coordinates at ultra-high speed with just a single parallel computation step. Despite its lightweight 3B architecture, the parallel computation mechanism enables significantly faster processing compared to conventional autoregressive models. It also fully supports ComfyUI custom nodes and ggml C++ porting, providing excellent operational accessibility, allowing it to run quickly and easily in general research environments or on local workstations.

In biomedical and biotechnology research, LocateAnything-3B serves as a powerful automation tool for detecting fine structures such as cells, protein aggregates, and fluorescent staining areas in microscopic images or histology slide images using only natural language queries. When a researcher inputs a somewhat complex referring expression, such as "the green fluorescent apoptotic area around the cell nucleus" or "mitochondrial aggregates within the cytoplasm," the model instantly specifies the precise coordinates of the target object without requiring fine-tuning. This enables efficient pre-screening by automatically selecting regions of interest (ROI) in high-resolution bio-image datasets in real-time and connecting them to subsequent quantitative analysis pipelines.

Furthermore, it plays a crucial role in building automated bio-labs and automated control systems for laboratory equipment. It can accurately infer the location of reagent container labels, pipette tip boxes, and control buttons on equipment control software in real-time camera footage or software interface screenshots, and then pass the precise coordinates to a robotic arm or control script. In the process of virtualizing the experimental environment and automating robotic control, it functions as a visual detection and location tracking (UI/Object Grounding) engine, maximizing experimental reproducibility and research productivity.

๐Ÿ’ป System Requirements

๐Ÿง RAM

์ตœ์†Œ 8GB NVIDIA GPU (RTX 3060/4060 ์ด์ƒ ๊ถŒ์žฅ, FP16/BF16 ๊ธฐ์ค€ ์•ฝ 6-8GB ์†Œ๋ชจ)

๐Ÿ’พStorage

๋ชจ๋ธ ๊ฐ€์ค‘์น˜ ์•ฝ 6.5GB, ์ „์ฒด ํ™˜๊ฒฝ ๊ตฌ์ถ•์šฉ 10GB ์ด๋‚ด

โšก Installation

4-1. Quick Start

PyTorch ๋ฐ Transformers ๊ธฐ๋ฐ˜ ๊ธฐ๋ณธ ์‹คํ–‰ ํŒจํ‚ค์ง€ ์„ค์น˜

pip install torch torchvision transformers accelerate Pillow

4-2. ์ƒ์„ธ ์„ค์น˜

GitHub ์ €์žฅ์†Œ ํด๋ก  ๋ฐ Embodied ํŒจํ‚ค์ง€ ์ด๋™

git clone https://github.com/NVlabs/Eagle.git cd Eagle/Embodied

์˜์กด์„ฑ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ ๋ฐ Flash-Attention ์„ค์น˜

pip install -r requirements.txt pip install flash-attn --no-build-isolation

๐Ÿงฌ Bio Use Cases

๐Ÿ”ฌ

Automated Segmentation of Microstructures and Extraction of ROI Regions in Cell Microscopy Images

Given a high-resolution fluorescence/brightfield cell microscopy image (1024x1024 or higher), the system receives a specific descriptive phrase such as "apoptotic cells marked with green fluorescence" or "mitochondrial aggregates within the cytoplasm." Within 1-2 seconds, it generates precise bounding box coordinates and passes them to a quantitative analysis pipeline.

๐Ÿงฌ

Automated Control of Biotechnology Equipment Screenshots and UI Elements

To automate laboratory robotic arm integration and equipment control, the system identifies GUI/experimental equipment elements such as reagent container labels, pipette tip locations, and software control buttons using natural language. It then transmits the coordinates to a control script.

๐Ÿ’Š

Detection of Schemas and Diagrams within Large-Scale Biomedical Papers

The system automatically classifies and infers the location of visual elements such as "experimental procedure flowcharts," "Western blot band regions," and "statistical graph axes" in academic paper PDFs and research document images, and uses this information to build a research knowledge base.

FAQ

What is LocateAnything-3B?

NVIDIA's LocateAnything-3B, released in July 2026, is a 3-billion-parameter open-vocabulary vision-language model (VLM) designed to precisely locate specific objects, text, or user interface (UI) elements within an image using bounding box coordinates. Built on a highly integrated architecture combining the MoonViT-SO-400M visual encoder and the Qwen2.5-3B-Instruct language model, it instantly infers the visual location of a target specified by the user in natural language. By merging the language model's exceptional contextual understanding with high-resolution image processing capabilities, it goes beyond simple object detection, excelling in tasks such as providing visual grounding that aligns with complex descriptive sentences and inferring the location of GUI elements. Existing large-scale vision-language models have inherent structural limitations, including slow inference speeds and high computational resource consumption, due to their adoption of an autoregressive approach that sequentially predicts text tokens to derive the coordinates of a target. To overcome this bottleneck, LocateAnything-3B introduces a parallel box decoding (PBD) method. Similar to how a high-performance ultrasound diagnostic device scans the entire contrast profile of biological tissue at once, it extracts a large number of location area coordinates at ultra-high speed with just a single parallel computation step. Despite its lightweight 3B architecture, the parallel computation mechanism enables significantly faster processing compared to conventional autoregressive models. It also fully supports ComfyUI custom nodes and ggml C++ porting, providing excellent operational accessibility, allowing it to run quickly and easily in general research environments or on local workstations. In biomedical and biotechnology research, LocateAnything-3B serves as a powerful automation tool for detecting fine structures such as cells, protein aggregates, and fluorescent staining areas in microscopic images or histology slide images using only natural language queries. When a researcher inputs a somewhat complex referring expression, such as "the green fluorescent apoptotic area around the cell nucleus" or "mitochondrial aggregates within the cytoplasm," the model instantly specifies the precise coordinates of the target object without requiring fine-tuning. This enables efficient pre-screening by automatically selecting regions of interest (ROI) in high-resolution bio-image datasets in real-time and connecting them to subsequent quantitative analysis pipelines. Furthermore, it plays a crucial role in building automated bio-labs and automated control systems for laboratory equipment. It can accurately infer the location of reagent container labels, pipette tip boxes, and control buttons on equipment control software in real-time camera footage or software interface screenshots, and then pass the precise coordinates to a robotic arm or control script. In the process of virtualizing the experimental environment and automating robotic control, it functions as a visual detection and location tracking (UI/Object Grounding) engine, maximizing experimental reproducibility and research productivity.

When should I use LocateAnything-3B?

<!

What is a biomedical use case for LocateAnything-3B?

Automated Segmentation of Microstructures and Extraction of ROI Regions in Cell Microscopy Images: Given a high-resolution fluorescence/brightfield cell microscopy image (1024x1024 or higher), the system receives a specific descriptive phrase such as "apoptotic cells marked with green fluorescence" or "mitochondrial aggregates within the cytoplasm." Within 1-2 seconds, it generates precise bounding box coordinates and passes them to a quantitative analysis pipeline.

๐Ÿ“„ Official Docs๐Ÿ™ GitHub

๐Ÿ“ Update Notes

No update notes yet.

๐Ÿงช Related Code of Life

No related Code of Life posts yet.