browser-use is an open-source AI agent framework that combines large language models (LLMs) and vision language models (VLMs) with Playwright, a web automation library, to autonomously control web browsers. Released by its developers in November 2024, this tool enables users to navigate dynamic websites and autonomously perform target tasks using only natural language instructions, without having to manually write complex programming code or web element selectors. The agent simultaneously analyzes the browser's DOM (Document Object Model) structure and real-time screen screenshot visual information to precisely understand the current page's state, and then infers and applies the optimal browser action to be executed next. Traditional Selenium or Playwright-based web scraping scripts heavily rely on the CSS selectors or HTML structure of web pages, which has a critical limitation: even minor changes to the website design or the application of dynamic layouts can break the entire script. Just as a person can ride in a self-driving car and simply state the destination, allowing the car to autonomously drive by recognizing road conditions and signs in real-time, browser-use equips the web browser with an AI driver that recognizes and flexibly handles complex interactive web interfaces, form inputs, multi-tab navigation, and login processes, just like a human. In particular, by applying a DOM refinement tree structure and visual element tagging technology that significantly reduces the number of browser elements beyond simple text prompting, it minimizes token consumption while accurately clicking and manipulating complex elements such as CAPTCHAs or drop-down menus. In the fields of biotechnology and biomedical research, browser-use demonstrates powerful performance in automatically collecting research data from multiple databases that do not provide a separate REST API or require complex session logins. For example, a researcher can automate the process of accessing multiple web portals such as NCBI PubMed, UniProt, and ClinicalTrials.gov, and manually searching for and collecting specific gene variants or phase 3 clinical trial information using only natural language instructions. It is also fully compatible with local Ollama environments and vision models, as well as OpenAI GPT-4o and Anthropic Claude 3.5 Sonnet, so it is possible to build an autonomous browser agent within a secure hospital or research institute's internal network to explore data without external data leakage, and connect it directly to a data analysis pipeline.

browser-use is an open-source AI agent framework that combines large language models (LLMs) and vision language models (VLMs) with the Playwright web automation library to autonomously control web browsers. Released by its developers in November 2024, this tool enables users to navigate dynamic websites and autonomously perform target tasks using only natural language instructions, without having to write complex programming code or web element selectors manually. The agent uses the browser's DOM.

🔬 Autonomous Exploration of Web-Based Biological Databases and Automated Metadata Collection: Accesses biomedical databases with API limitations, such as NCBI PubMed, UniProt, and ClinicalTrials.gov, to collect target gene variant and phase 3 clinical trial data using natural language instructions, and refines the data into a Pandas DataFrame format.