Developed by Microsoft Research and released in February 2025 with a major v2.0 update, OmniParser v2 is a state-of-the-art, vision-based UI parsing tool that precisely converts complex graphical user interface (GUI) screenshots of computer screens into machine-readable, structured pixel data. It acts as a visual translator, enabling large vision-language models (VLMs) to perfectly recognize the location and meaning of screen objects, much like a person visually identifies buttons, icons, and text boxes on a monitor screen and clicks them with a mouse. OmniParser v2 operates through the organic combination of two core deep learning architectures: an icon detection model that accurately detects bounding boxes of individual UI elements on the screen, and an icon captioning model that describes the functional meaning of each element's visual icon in text. Through this pipeline, a single input image screenshot is easily and automatically converted into structured JSON data containing the relative coordinates of each element, its clickability, text labels, and visual function captions. Traditional methods of building computer manipulation agents (GUI Agents) have been limited by their excessive reliance on the Document Object Model (DOM) tree of web browsers or specific proprietary APIs. This has often resulted in agents failing to properly identify the location or click area of buttons in dedicated desktop software with undisclosed source code or in virtual desktop environments rendered with images, causing frequent interruptions in the agent's operation. Just as GPT uses embeddings to understand complex contexts in natural language processing, OmniParser v2 extracts all clickable geometric objects from the screen pixel array and converts them into visual tokens that VLMs can immediately manipulate. In particular, version 2.0 achieves a low-latency inference processing speed that is more than 60% faster than previous versions, enabling real-time analysis of high-resolution screens within a delay of less than one second. Furthermore, it incorporates a finely tuned vision attention dataset and an optimized bounding box regression algorithm, which accurately recognizes even small or complex icons, fundamentally solving the chronic problem of graphic icon misidentification and malfunction that has plagued traditional OCR-based approaches. This pixel-based UI parsing technology provides an innovative solution for building automated pipelines for dedicated electronic lab notebooks (ELNs) and bio-research equipment control software, which have long been in the blind spot of automation in the fields of biotechnology and multidisciplinary research. For example, in the case of a three-dimensional confocal microscope control program or automated liquid handler software that operates on a closed, native Windows platform, the lack of internal APIs requires a person to manually enter numerical parameters and press the capture button each time. By combining OmniParser v2 with existing VLM agents and the OmniTool computer automation framework, it is possible to extract the coordinates of clickable buttons and the location of numerical input forms within high-resolution graphic interfaces of 1024x1024 or higher with a high accuracy of 95% or more, using only a single screen screenshot. The collected screen coordinates are immediately converted into mouse clicks and keyboard events, contributing to the complete automation of multi-sample data collection procedures without researcher intervention during nighttime or large-scale bio-screening experiments. Furthermore, OmniParser v2 goes beyond simple screenshot parsing and enables cross-platform agent orchestration across heterogeneous platforms, including web, desktop, and mobile apps. Implemented in PyTorch, it enables fast pass-through inference even in a single consumer GPU environment, and through tensor-based pipeline optimization, it supports immediate connection with various open-source and commercial LLM/VLM prompt chains. Research and development organizations can easily integrate all GUI-based software as targets for intelligent agents by using OmniParser v2, without the need to modify existing offline legacy systems or develop new, costly APIs, thereby dramatically improving the productivity of digital transformation and the overall research data collection process.

Developed by Microsoft Research and released with a major v2.0 update in February 2025, OmniParser v2 is a state-of-the-art vision-based UI parsing tool that precisely converts complex graphical user interface (GUI) screenshots of computer screens into structured pixel data that machines can understand. Just as a person visually identifies buttons, icons, and text boxes on a monitor screen and clicks them with a mouse, a large vision-language model (VLM) can identify the location and meaning of objects on the screen.

Automated Legacy Software for Controlling Laboratory Equipment: Automated analysis of screenshots from native Windows microscope/spectrometer software that does not support APIs → Automatically identify UI button coordinates (parsing at 1024x1024 resolution within 1 second) and numerical input fields, and link to mouse/keyboard automation for control.