Click a page in the sandbox browser
Drive a desktop browser through MCP, using element references from its accessibility tree. This program serves a small page inside the sandbox, clicks its button and checks that the page changed before saving a screenshot.
The sandbox image must include the desktop capability. Desktop services share the sandbox’s memory; the example requests enough for the browser and display.
Run the program
Set TANGLE_API_KEY from your API keys.
Save the program below as browser-check.mts:
npm install @tangle-network/sandbox @modelcontextprotocol/sdk
npm install --save-dev tsx
npx tsx browser-check.mtsimport assert from "node:assert/strict";
import { writeFile } from "node:fs/promises";
import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js";
import { Sandbox } from "@tangle-network/sandbox";
const client = new Sandbox({
apiKey: process.env.TANGLE_API_KEY!,
baseUrl: "https://sandbox.tangle.tools",
});
const box = await client.create({
capabilities: ["computer_use"],
resources: { memoryMB: 4096 },
maxLifetimeSeconds: 300,
});
const desktop = new Client({ name: "browser-check", version: "1.0.0" });
try {
await box.write("index.html", `<!doctype html>
<html lang="en"><meta charset="utf-8"><title>Report</title>
<body><h1>Sales report</h1>
<button onclick="const result=document.getElementById('result'); result.textContent='Report ready'; result.setAttribute('aria-label', result.textContent)">Generate report</button>
<p id="result" role="status" aria-label="Waiting">Waiting</p></body></html>`);
await box.process.spawnExact("python3", ["-m", "http.server", "8000", "--bind", "127.0.0.1"], {
timeoutMs: 240_000,
});
const ready = await box.exec(`python3 - <<'PY'
import time
import urllib.request
deadline = time.monotonic() + 10
while True:
try:
with urllib.request.urlopen("http://127.0.0.1:8000/", timeout=1) as response:
assert response.status == 200
break
except OSError:
if time.monotonic() >= deadline:
raise
time.sleep(0.1)
PY`);
assert.equal(ready.exitCode, 0, ready.stderr);
const endpoint = await box.getMcpEndpoint();
await desktop.connect(new StreamableHTTPClientTransport(new URL(endpoint.url), {
requestInit: { headers: { Authorization: `Bearer ${endpoint.authToken}` } },
}));
const call = async (name: string, args: Record<string, unknown> = {}) => {
const result = await desktop.callTool({ name, arguments: args });
assert.notEqual(result.isError, true, `MCP tool ${name} failed`);
return result;
};
await call("open_url", { url: "http://127.0.0.1:8000/" });
const screen = await call("read_screen");
const text = screen.content
.filter((part) => part.type === "text")
.map((part) => part.text)
.join("\n");
const button = text.match(/\[([^\]]+)\] button "Generate report"/);
assert(button, "Generate report button was absent from the accessibility tree");
assert(text.includes('status "Waiting"'), "Waiting status was absent from the accessibility tree");
await call("click_element", { ref: button[1] });
// read_screen includes accessible names; the handler names the status from its rendered text.
const deadline = Date.now() + 10_000;
while (true) {
const changed = await call("read_screen");
if (changed.content.some((part) => part.type === "text" && part.text.includes("Report ready"))) break;
assert(Date.now() < deadline, "The page did not update after clicking");
await new Promise((resolve) => setTimeout(resolve, 100));
}
const captured = await call("computer_use", { action: "screenshot" });
const image = captured.content.find((part) => part.type === "image");
assert(image && image.type === "image" && image.mimeType === "image/png");
const png = Buffer.from(image.data, "base64");
assert.equal(png.subarray(0, 8).toString("hex"), "89504e470d0a1a0a");
await writeFile("browser.png", png);
console.log("Report ready; saved browser.png.");
} finally {
try {
await desktop.close();
} finally {
await box.delete();
}
}Open browser.png to inspect the result. Replace the local URL with your own page
to automate a browser task. The scoped MCP token stays in memory and grants access
only to this sandbox’s computer-use tools.
See MCP clients to connect the same desktop to a coding agent.