Legacy tools, API-less web apps, multi-step wizards — describe the goal in a sentence and the agent handles every click, keystroke, and page load. No connector, no script, no browser extension required.
Actionist runs a tight perceive-decide-act loop for every step. It captures the screen, sends it to a vision model, receives an action instruction, executes it, then loops back. Repeat until done.
Screenshot sent to vision model
Model plans the next action
Click, type, or navigate
repeats until done
A screenshot of the active display is captured and sent to a vision model. Every loop starts fresh — the agent sees the actual current state, not a cached assumption.
The vision model interprets the screenshot against your task goal and produces a precise action instruction: click a coordinate, type text, press a key, or call a wait.
Actionist executes the instruction through OS-level input — the same accessibility APIs a human uses. The screen changes, a new screenshot is captured, and the loop runs again.
Every team has tools that resist automation — a vendor portal, an internal dashboard, a multi-step wizard. Computer Use dissolves that wall. If it is visible on screen, it is automatable.
Before Computer Use
Legacy desktop apps, sites without APIs, and multi-step web flows stay manual. Every repetitive task costs a human hour. Building a connector takes an engineer.
With Computer Use
Actionist captures the screen, sends it to a vision model, receives action instructions, and executes them through OS-level input. Describe the goal — the agent handles every click.
Computer Use runs on the macOS desktop app. The web app shows a desktop-only notice when you attempt to use it.
Actionist needs Screen Recording (to see your screen) and Accessibility (to control UI elements). Both must be granted before any automation runs.
Red bannerappears
PermissionsRequired section
Screen Recordinggrant access
Accessibilitygrant access
All systemsactive
Look for the red banner
If either permission is missing, Actionist shows: “Critical system permissions missing. Core app functions are disabled.” Click Fix in Settings.
Open Permissions Required
Settings opens to the Permissions Required section. The subtitle reads: “Actionist needs Screen Recording and Accessibility permissions to observe your screen and perform actions.”
Grant Screen Recording
Click Grant access next to Screen Recording. Actionist opens System Settings > Privacy & Security > Screen Recording automatically. Toggle Actionist on. macOS may ask you to restart the app.
Grant Accessibility
Return to Actionist. Click Grant access next to Accessibility. Repeat the same System Settings toggle step.
Confirm success
A toast appears: “Permissions granted! All system features are now active.” You are ready to automate.
Settings — Permissions Required
Permissions Required
Actionist needs Screen Recording and Accessibility permissions to observe your screen and perform actions.
Screen Recording
Allows Actionist to see your screen
Granted
Accessibility
Allows Actionist to control UI elements
Grant access
In chat, if you trigger an automation before permissions are granted, a Grant Permission button appears inline in the input area — tapping it opens the same Settings flow described above.
Send any task in plain language. Actionist classifies the request, opens a Computer Use session, and starts working. The AI Process panel feeds every step label in real time — you never wonder what the agent is doing.
AI Process — session · mayarunning
task received · 09:41:22
youOpen Notion, find my task list, and mark Weekly report as done.
computer use session started
step 1Opening Safari — bringing window to front.
step 2Clicking address bar. Typing app.notion.so. Pressing Return.
step 3waitingIntelligent wait — page loading. Holding for content to settle.
step 4Navigating to My Tasks in left sidebar. Clicking.
step 5Scanning task list. Found Weekly report — row 7.
step 6Clicking checkbox next to Weekly report.
step 7Confirming check state — item marked complete.
session complete · 09:41:51
mayadoneDone. Weekly report is checked off in your Notion task list.
Input locked while running. The chat input shows “Automation in progress…” and is non-interactive. A Stop button is the only control — preventing accidental overlap with new tasks.
Intelligent waits. Actionist inserts automatic pauses after navigation, form submissions, and search requests. These let the page settle — they are not lag, they are deliberate pacing.
One task at a time. Only one Computer Use session can run simultaneously. If you send a new task while one is running, it will fail. Click Stop first.
Open an agent's chat
Select any agent from the chat selector. Make sure it has Screen Automation (Actionist) enabled on its Tools tab.
Describe the task
Type what you want — for example: “Open Safari, go to app.notion.so, find my task list, and mark the item called ‘Weekly report’ as done.”
Watch it work
The input area shows “Automation in progress…” and is non-interactive while the session runs. Open the AI Process tab in the right sidebar to see a live feed of every step the agent takes.
Receive the result
When the task is complete, the agent sends a message and the input area returns to normal. The AI Process tab keeps the full step history for review.
chat · mayarunning
S
Open Safari, find my Notion task list, and mark Weekly report as done.
Actionist’s agent is explicitly instructed never to proceed autonomously when it encounters a password field, payment form, or ambiguous choice. It stops and asks you instead. Delegation stays safe.
chat · mayarunning
S
Log in to the vendor portal and download the latest invoice.
M
On it. Opening the vendor portal now.
Automation in progress…Stop
chat · mayawaiting
S
Log in to the vendor portal and download the latest invoice.
M
On it. Opening the vendor portal now.
M
Please enter your login password for vendor.example.com to continue.
Waiting for your input.
Reply to continue…Send
The AI Process feed is frozen. No further tool calls run until you reply. No credential is ever sent to the model — your reply goes directly to the session continuation.
chat · mayadone
M
Please enter your login password for vendor.example.com to continue.
S
●●●●●●●●
M
Logged in. Navigating to invoices. Found the latest — INV-2041. Downloading now.
Message Maya…Send
The session resumes from exactly where it paused — no context lost. The agent uses your reply for the current field and continues the task.
Agent reaches a credential or payment field
The Computer Use session pauses. No tool call is made — the agent sends a text-only question in chat, for example: “Please enter your password for app.notion.so”.
Input area changes to 'Reply to continue...'
The input placeholder changes to “Reply to continue…” and a card in the chat shows “Waiting for your input.” The AI Process feed is frozen.
Type your answer and press Enter
Type the credential or your choice and press Enter. The response is queued for the agent’s next turn.
Agent proceeds
The session resumes automatically. The agent uses the input for the current field and continues the task.
chat · mayawaiting
M
Log in to the vendor portal and download the latest invoice.
M
Please enter your login password for vendor.example.com to continue.
Waiting for your input.
Reply to continue…Send
Never include passwords, API keys, or payment card numbers in the task description itself. The agent is designed to stop and ask — let it. Putting credentials in the task message is unsafe and unnecessary.
While “Automation in progress…” is shown, the chat input area gives you a live stop control — and the agent itself will pause and ask whenever it needs you.
Stop
Agent-initiated pauses
Clicking the Stop button in the chat input area cancels the in-flight API request immediately, then exits the loop. The input area shows “Stopping…” and then “Stopping automation…” while teardown completes. After teardown, the session closes and the input area returns to normal.Use Stop when you want to redirect the task or start a completely different one.
The agent pauses itself whenever it needs your input — a password field, a payment form, or a genuinely ambiguous decision. It sends a text-only question in chat and waits. Reply in the input area and the session continues from exactly where it stopped, with no context lost.
chat · mayarunning
S
Open the vendor portal and export the Q2 report.
M
On it. Navigating to the vendor portal now.
Automation in progress…Stop
AI Process
Opening Safari
Navigating to portal
Locating export button
Only one Computer Use task can run at a time. If you send a new task while one is already running, it will fail. Click Stop to end the current session before starting another.
Every team has a handful of portals and tools that resist automation — no API, no integration, just a browser and a grind. Computer Use dissolves that wall in plain language.
Before — manual entry
09:05Open vendor portal in browser
09:07Log in, navigate to New Submission
09:09Copy invoice number from spreadsheet
09:12Paste into 14 individual form fields
09:19Attach PDF, scroll to confirm checkboxes
09:23Submit — wait for confirmation screen
09:25Screenshot confirmation, paste into Slack
After — one sentence to CUA
Describe the task once in plain language
Agent opens the portal, logs in, fills every field
Attaches the PDF from your Downloads folder
Confirms all checkboxes, submits the form
Returns a confirmation number in chat
saves ~18 min per submission
The pattern works for any portal — here is what the same flow looks like over a phone-style channel.
L
@lena_actionist_bot
ONLINE
Post the Senior Backend Engineer role to Northwind Careers. Title: Senior Backend Engineer. Team: Platform. Location: Remote — EU. Use the standard JD from the shared doc.
10:14
TOOL · SCREEN_AUTOMATION
Got it. Opening Northwind Careers portal now. I will navigate to Post a Job, fill in the title, team, location, and paste the JD. I will pause if a login or confirmation step needs you.
10:14
SEEN · READING CONTEXT
Done. The listing is live. Here is the confirmation:
10:17
Northwind Careers · Ref NW-2041 · Remote EU · Active
View listingTake down
The Northwind Careers portal has no public API. The agent navigates it the same way Lena would — opening the browser, clicking through the form, and pasting the job description — in about 3 minutes instead of 15.
Computer Use interprets your intent from plain language. A few habits make the difference between a task that completes in one shot and one that needs redirecting.
Name the target app explicitly
Say “Open Safari, go to…” rather than just “go to…”. The agent sees the entire screen and needs to know which window or app to bring forward. Naming the app removes ambiguity at the first step, where mistakes compound.
Describe the goal, not every click
Write what you want the result to be, not a step-by-step click script. “Mark ‘Weekly report’ as done in my Notion task list” is better than “click the checkbox next to the third row.” The agent plans the clicks — you set the destination.
Use chat for one-offs; use workflows for recurring tasks
A chat message is the fastest path to a single automation run. For tasks you repeat daily — morning triage, end-of-day reports, status updates — put the same task description in a workflow node and schedule it. Workflow nodes are reviewable, editable, and schedulable; a chat session is ephemeral.
Minimize the Actionist window, do not close it
The agent minimizes Actionist automatically if the chat window is obstructing its target area. Closing Actionist while an automation is running terminates the session immediately. Keep the app running in the background.
Expect short pauses after key actions — this is intentional
Actionist inserts automatic waits after navigation, form submissions, and search requests. These pauses let the page settle before the next action fires. They are not lag — they are deliberate pacing that prevents the agent from acting on a partially-loaded screen.
Check the AI Process panel if the agent appears stuck
The AI Process tab in the right sidebar shows a live feed of every step label. If a session seems frozen, open this panel to see the last action the agent attempted — it tells you exactly what happened before the pause.
Screen automation is a built-in agent tool. It is scoped per agent so you can restrict which of your agents can operate the screen.
Built-inScreen Automation (Actionist) ships as a first-party tool — no install, no connector.
Per-agentToggle it independently on each agent’s Tools tab. Off by default for pure data-processing agents.
Approval-awareRespects the agent’s approval mode. “Guarded auto” or lower still prompts on sensitive screen actions.
The tool
Built-in
The tool is named Screen Automation (Actionist) and described as “Click, type, navigate via CUA.” Toggle it on the agent’s Tools tab in the studio.
Scoping
Turn the tool off for agents that should never touch the screen — a pure data-processing agent or a Slack bot, for example. An agent without this tool cannot initiate a Computer Use session even if asked.
The Screen Automation (Actionist) tool respects the agent’s approval mode. With approval mode set to “Guarded auto” or lower, sensitive screen actions will still prompt for your confirmation.
