Skip to content

Browser Use and Computer Use ​

Memoh can give a bot a visible workspace desktop and a headed browser inside its workspace container. This is different from running a headless Playwright script: the bot can inspect and operate the same graphical browser that you can see in the app's Desktop pane.

Concepts ​

CapabilityBest forHow it works
Headless browser commandsFast scripted automation inside a workspaceRun Playwright or other browser tooling as normal workspace commands.
Browser UseWeb pages, forms, navigation, screenshots, accessibility-tree inspectionOperates the headed workspace Chrome/Chromium instance over CDP.
Computer UseNative dialogs, broken browser state, non-browser GUIReads the desktop accessibility tree (AT-SPI) for refs, falls back to coordinates and screenshots when accessibility is unavailable.

Prefer Browser Use for web pages. Use Computer Use when the task depends on GUI state that CDP cannot reach.

Workspace display and VNC ​

Workspace display is the desktop environment inside the bot's workspace container. VNC/RFB is the display and input transport behind that desktop, while WebRTC is used by the app's desktop session.

The main value is not VNC by itself. The important capability is that the workspace can run a headed Chrome/Chromium browser for sites and login flows that do not work well in headless mode.

Preparing a bot desktop ​

  1. Open the bot detail page.
  2. Go to the Desktop tab.
  3. Prepare or enable the workspace display runtime.
  4. Open a display session from the bot settings page or from the chat workspace.

The display runtime installs or uses the workspace desktop, VNC server, browser, and fonts needed for the visible session. Availability depends on the workspace backend and image.

Agent tools ​

When workspace desktop is enabled, the agent can use browser and computer tools:

  • browser_observe inspects the current browser page (snapshot, get_content, screenshot, evaluate, and so on).
  • browser_action clicks, fills, types, presses, and navigates in the headed browser.
  • browser_remote_session exposes the browser CDP endpoint for code-driven sessions.
  • computer_observe returns an accessibility-tree snapshot of the desktop (refs like e3) or a saved screenshot path.
  • computer_action drives the desktop. It prefers a ref from the snapshot and falls back to (x, y) coordinates when no ref is available or the accessibility action fails.

Screenshots are not auto-attached ​

Both browser_observe and computer_observe save screenshots to a workspace path such as /data/computer-screenshots/1716200000.jpg and return that path in the tool result. The image is not pushed into the conversation. Read the path with the workspace file read tool when you actually need the visual; that keeps observation cheap and lets the model decide when an image is worth the token cost.

Accessibility helper ​

Computer Use depends on the a11y-cli binary installed under /opt/memoh/toolkit/display/bin/a11y-cli and the workspace at-spi2-core package. The display runtime probe reports a11y_available so the app can surface whether the accessibility path is healthy. When AT-SPI is unavailable, computer_action still works with raw coordinates and computer_observe screenshot remains usable.

These tools are workspace runtime features. They do not automate the Electron desktop app itself.

Published under AGPLv3