Skip to main content
Set up Agent S3, the open-source computer-use agent by Simular AI, against your own laptop or against an Orgo computer. Agent S3 is the current release of the framework, distributed as the gui-agents package. It drops the manager and worker hierarchy Agent S2 used, adds a native coding agent that can write and run code alongside GUI actions, and adds Behavior Best-of-N, which runs several rollouts and picks the best one.
Agent S2 code imported from gui_agents.s2. Agent S3 lives under gui_agents.s3 and renames the top-level class to AgentS3. Update both when you upgrade.

Coordinate space

A Linux computer created without a template or a resolution boots at 1280x720. Some templates and Macs are 1920x1080. Agent S3 never hands you a coordinate to translate. It emits pyautogui code with the coordinates already in it, which RemoteExecutor runs inside the computer through POST /computers/{id}/exec. The scaling happens inside Agent S3: OSWorldACI multiplies each x the grounding model returns by width / grounding_width, and each y by height / grounding_height. Three things follow from that:
  • OSWorldACI’s width and height must be the size of the screen the clicks land on. They default to 1920x1080. The script below reads them from each executor’s screen_size().
  • grounding_width and grounding_height must describe the space the grounding model’s coordinates come in. With a model that answers in screenshot pixels, such as UI-TARS-1.5-7B, that is the size of the screenshot it receives, so the script measures them from the same screenshot() it sends.
  • Any resolution you mention in a prompt must match the computer’s real one.
grounding_width and grounding_height are not always your computer’s resolution. UI-TARS-1.5-7B, the default grounding model here, returns absolute pixel coordinates on the screenshot it receives, so for it they are the screenshot’s size. The Agent S README pairs UI-TARS-1.5-7B with 1920x1080, which is that size on a 1920x1080 screen. Keep 1920x1080 on a 1280x720 computer while passing width=1280 and every click lands at two thirds of its target. A model that answers on a fixed grid takes that grid instead: the Agent S README gives 1000x1000 for UI-TARS-72B.
GET /computers/{id}/screens reports the current width and height.

Setup

Install the required packages. gui-agents supports Python 3.9 through 3.12. Use Orgo’s Python SDK 0.0.47 or later: earlier versions ask for 2 GB of RAM by default, which POST /computers rejects.
Set up your API keys. HF_TOKEN is for a grounding model hosted on Hugging Face Inference Endpoints:

Simple usage

Run Agent S3 with natural language commands:
The package also ships its own CLI, agent_s, which drives the laptop it runs on:
With UI-TARS-1.5-7B, 1920 and 1080 fit a 1920x1080 screen. The CLI sends screenshots at the screen size pyautogui reports, scaled down to fit 2400 pixels on each side, so on another screen pass that size instead.

Complete example

Pass a LocalEnv from gui_agents.s3.utils.local_env as OSWorldACI(env=...) to turn on the coding agent. It executes code on the host running the script, which for RemoteExecutor is your laptop rather than the Orgo computer. Leave it None unless that is what you want.

Platform requirements

These apply to local mode. Remote mode runs the actions on the Orgo computer, and RemoteExecutor installs pyautogui there when it starts.

macOS

Grant Terminal access: System Settings, then Privacy & Security, then Accessibility

Windows

May require running Terminal as Administrator

Linux

Install dependencies:

Environment variables

Architecture

Agent S3 replaced Agent S2’s hierarchical planner with a flatter design: Native coding agent: writes and runs code alongside GUI actions, so tasks better solved in a shell stop being click sequences Behavior Best-of-N: runs several rollouts of a task, summarizes each as a behavior narrative, and has a judge pick the one that completed it Mixture of grounding: routes actions to specialized visual grounding models for precise UI localization Cross-platform support: works on macOS, Windows, and Linux For current benchmark results, see the Agent S3 writeup and the OSWorld leaderboard.

Resources

Video tutorial

The video was recorded against Agent S2. The imports and class names above are the Agent S3 ones.