OmniParser, updated to v2.0 and released by Microsoft Research in February 2025, is an innovative screen parsing engine that enables AI agents to visually understand and manipulate computer screens in the same way as humans. OmniParser accurately detects the location of all interactive elements, such as text, icons, tabs, and checkboxes, within a given UI screenshot image, and automatically maps them to semantic captions that describe the function of each element, much like a human intuitively identifies the location of a specific menu or button on a monitor and moves the mouse accordingly. To achieve this, OmniParser adopts a dual-pipeline architecture that combines YOLOv8, an object detection model specialized in capturing fine-grained icons and buttons on the screen, with Florence-2, a vision-language model that generates high-level text descriptions of their functional roles. Existing general-purpose vision-language models (VLMs) were good at reading and analyzing the overall context of an entire monitor screen, but when it came to the physical computer manipulation stage, where they needed to find the exact pixel coordinates of a specific button and initiate a click, they often suffered from geometric resolution distortion or hallucination, where they pointed to the wrong location. OmniParser acts like a pair of precision corrective glasses, focusing the vision of an agent with astigmatism, thereby fundamentally solving this chronic misoperation problem. Instead of letting the screen be interpreted randomly, the detector, based on YOLOv8 and trained in the first stage of the pipeline, clearly identifies the bounding box of the click area as if cutting it with a knife, generates visual markers, and assigns numbers. The Florence-2-based captioner in the second stage of the pipeline then organizes the semantic information hidden behind each number into unique descriptive texts and reports it to the large model. As a result, backend brain models such as OpenAI's GPT-4o or Anthropic's Claude 3.5 Sonnet can receive precisely targeted mouse control sequences without errors, even within complex web UIs or legacy software. From general IT researchers who want to build software automation and data integration pipelines to bio scientists who want to automate microscopic analysis systems for drug screening, OmniParser is the key to unmanned control of closed desktop applications and complex web portals that do not offer official APIs. For example, when a bioinformatics researcher needs to control an hourly-changing genomic database or high-resolution molecular dynamics simulation software, they can simply provide a single screen to a visual agent equipped with OmniParser v2.0, and a complete UI layout tree will be provided immediately, eliminating the need to build complex and cumbersome Selenium code. Because it can perfectly transform a 1024x1024 resolution computer screen screenshot into geometric and text structures with a short latency of only 0.8 seconds on a GPU-accelerated system, it delivers absolute economic value by completing a real-time responsive data processing pipeline without bottlenecks when operating large-scale unmanned automation loops (Human-in-the-loop), and drastically reducing the cost of developing automation solutions and the workload of researchers by more than half.

OmniParser, updated and released by Microsoft Research in February 2025 as version 2.0, is an innovative screen parsing engine that enables AI agents to visually understand and manipulate computer screens in the same way as humans. OmniParser, like a human looking at a monitor, intuitively determines the location of specific menus or buttons and moves the mouse. It accurately detects the location of all interactive elements, such as text, icons, tabs, and checkboxes, within the input UI screenshot image, and identifies the function of each element.

🔬 [Automated Collection of Bioinformatics Data from GUI-Based Databases]: By combining OmniParser v2.0 and the GPT-4o vision agent, the system extracts the precise bounding box coordinates of 34 clickable icons and input fields from the complex UI of the NCBI ClinVar website (which lacks a web browser API) with 98% accuracy, using the best.pt detection model and a Florence-2-based caption model. This enables automated data crawling at a rate of 1.2 times per second, reducing the construction time of a clinical genomics variation analysis pipeline by 75%.