OmniParser v2, released by Microsoft Research in February 2025, is a general-purpose screen parsing tool that converts computer screen screenshots into machine-readable, structured data. It was developed to fundamentally solve the positional error problem that existing large vision language models (VLMs) faced when they could not accurately calculate mouse click coordinates based solely on computer screen images. Similar to how a large language model (LLM) precisely processes natural language text by breaking it down into tokens, OmniParser v2 divides and processes every graphical element on the screen into independent visual tokens, each mapped with geometric coordinates and functional descriptions. Internally, it integrates an object detection model that detects all interactive objects on the screen and an icon captioning model that provides natural language descriptions of the detected icons, using a dual-stage architecture to achieve perfect screen decoding capabilities. This significantly reduces the rate of agent malfunctions across various operating system interfaces. Traditional screen control methods have relied heavily on the source code within applications or the Document Object Model (DOM) tree of web browsers, making it nearly impossible to design automation scenarios for closed-source software, mobile apps, or virtual machine environments where the source code is hidden. In contrast, OmniParser v2 identifies the location of buttons, input fields, and checkboxes in milliseconds based solely on the pixel values on the screen, without requiring any internal system APIs or code sources. This tool maximizes the visual object bounding box detection technology, allowing it to respond in real-time to environmental variables such as changes in resolution or window size without distortion. In particular, by identifying the visual form of an icon and then supplementing it with a textual functional description, it perfectly guides the vision language model to ensure that it does not miss the mouse click area, and it continues uninterrupted automation even when websites are redesigned or UI layouts change, by capturing elements based solely on their visual characteristics. In actual biotechnology research settings, tedious GUI repetitive tasks often occur, requiring simultaneous manipulation of web browsers and local legacy analysis programs during gene sequence analysis or protein structure visualization. By linking OmniParser v2 with OmniTool, a tool suite for controlling Windows 11, and high-performance visual models, an automated protein structure design pipeline can be established. For example, if a 1920x1080 screenshot of an analysis program is sent to a YOLOv9-E-based detector, dozens of fine protein rendering option buttons on the screen are detected within an average of 1-2 seconds. Based on the detected target click point data, the automation agent automatically searches for and selects the target protein from the protein database (RCSB PDB), opens the downloaded molecular docking data file, and performs mouse operations to occupy a specific active site without errors, quantitatively maximizing the data acquisition time and analytical productivity of researchers.

Microsoft Research released OmniParser v2 in February 2025. It is a general-purpose screen parsing tool that converts computer screen screenshots into machine-readable, structured data. It was developed to fundamentally solve the positional error problem that existing large vision language models (VLMs) faced, where they could not accurately calculate mouse click coordinates by only looking at computer screen images. It leverages large language models (LLMs) to process natural language text.

🔬 Automated Web-Based PDB Molecular Search and Analysis: Using the OmniParser v2 (YOLOv9-E) + Claude 3.5 Sonnet combination, visually track the process of entering a specific target protein ID on the RCSB PDB website and manipulating the 3D viewer, achieving 100% visual tracking within 5 seconds to automate the extraction and capture of active site information → Accelerate the preprocessing stage of virtual screening pipelines.