#The Basics

This page provides a conceptual overview of Midscene, including how to use an Agent, its architecture and abstractions, and the boundaries of its capabilities. You will learn how an Agent connects AI models to a target interface and how to choose among planning and interaction, instant interaction, and Insight.

#Plan and interact

#aiAct

aiAct accepts a goal described in natural language. It observes the interface, plans the next steps, locates the target elements, and executes the actions until the goal is complete. The prompt can also include assertions. Midscene verifies these assertions during execution and throws an error promptly if one fails.

aiAct is flexible and autonomous, so it works well when a task has multiple steps, conditional branches, or an uncertain execution path. During execution, aiAct continuously uses AI to plan from the latest interface state, which means it usually takes more time and tokens than an instant interaction.

Typical usage:

await agent.aiAct(
  'Search for headphones, add the first item to the cart, and confirm that the cart count changes to 1',
);

To give every subsequent aiAct call more business context, use agent.setAIActContext():

agent.setAIActContext(
  'Close the cookie consent dialog first if it appears. Prices are shown in USD.',
);

The key per-call options are:

• deepThink: focuses more on task decomposition and separates planning from element localization. It can make complex tasks more stable, but increases model calls and latency.
• deepLocate: uses an additional model call to improve element localization. Enable it when a target is small or difficult to distinguish from nearby elements.
• context: provides business knowledge or other background for this call only. For aiAct, it overrides the Agent-level aiActContext, including when it is explicitly set to an empty string.

await agent.aiAct('Complete the checkout form and stop before placing the order', {
  deepThink: true,
  deepLocate: true,
  context: 'If an address confirmation dialog appears, select the default shipping address.',
});

#Instant interactions

Instant interaction APIs perform one specified action. Their main job is to locate a UI element and execute a fixed operation on it.

These APIs do not plan a sequence of steps. A request such as “close the popup if it appears, then click the checkout button” should use aiAct; an instant interaction treats its prompt as a description of the target element, not as a workflow.

#aiTap

aiTap locates and taps or clicks one element.

Typical usage:

await agent.aiTap('The checkout button in the shopping cart');

Use deepLocate when the target is small or visually ambiguous:

await agent.aiTap('The cart icon in the upper-right corner', {
  deepLocate: true,
});

#aiInput

aiInput locates an input field and enters a specified value. Its default replace mode clears the existing content before entering the new value.

Typical usage:

await agent.aiInput('The email address input', {
  value: 'user@example.com',
});

Other input modes are typeOnly, which preserves the existing content, and clear, which only clears the field.

Other instant interaction APIs include aiHover, aiClearInput, aiKeyboardPress, aiScroll, aiPinch, aiLongPress, aiDoubleClick, and aiRightClick. Platform support varies; see Planning and interaction in the API reference.

#Insight

Insight APIs observe the interface and return an analysis result without interacting with it. They use the current screenshot by default. On web pages, you can also pass domIncluded when the task requires DOM information that is not visible in the screenshot.

#aiAssert

aiAssert checks a condition described in natural language. It resolves when the condition is true. When the condition is false, it throws an error that includes the reason returned by the model.

Typical usage:

await agent.aiAssert('The shopping cart contains one item and shows a subtotal');

#aiQuery

aiQuery extracts structured data from the interface. Describe both the required data and its expected type or shape in the prompt.

Typical usage:

const items = await agent.aiQuery<
  Array<{ name: string; price: number }>
>('The products in the shopping cart, {name: string, price: number}[]');
// Example items value: [{ name: 'Wireless headphones', price: 99.9 }]

#aiBoolean

aiBoolean answers a question about the interface and returns a boolean.

Typical usage:

const loginDialogVisible = await agent.aiBoolean(
  'Is the login dialog visible?',
);
// Example loginDialogVisible value: true

Related convenience methods include aiNumber for numbers and aiString or aiAsk for strings.

#Orchestrate workflows with JavaScript

Midscene offers two basic ways to orchestrate automation: use aiAct, or orchestrate the workflow with JavaScript. The following examples complete the same task using each approach.

Using aiAct:

await agent.aiAct(
  'Check every record in the list and mark any incomplete record as completed',
);

Using JavaScript orchestration:

const recordNames = await agent.aiQuery<string[]>('All record names in the list');

for (const recordName of recordNames) {
  const completed = await agent.aiBoolean(
    `Is the record named "${recordName}" marked as completed?`,
  );

  if (!completed) {
    await agent.aiTap(`The record named "${recordName}"`);
  }
}

aiAct delegates the execution path to the Agent, while JavaScript orchestration keeps conditions, loops, and step order in code. In the JavaScript example above, Insight APIs provide state to the control flow, and instant interaction APIs execute specified actions.

JavaScript orchestration is a practical and highly deterministic approach. Because the execution path is explicit, developers can use familiar debugging tools and control how each branch behaves.

JavaScript orchestration can respond only to situations handled in the code. A resolution change may require an extra scroll, and a temporary popup may cover the target element. If the code does not account for these changes, the workflow fails. aiAct, by contrast, observes the latest interface state and adjusts its next actions accordingly.

Use the following guidelines when choosing an approach:

1. Use aiAct by default to drive operation goals. Letting the Agent choose the specific steps from the latest interface state makes the workflow more adaptable to UI changes.
2. Use JavaScript orchestration only when the operation flow is well understood and stable. In that case, write the known conditions, loops, and step order directly in code.