CAPTCHA Handling

NEW in v0.13.0 - Built-in CAPTCHA detection with configurable handling strategies for browser automation agents.

Why Sentience Does Not Solve CAPTCHAs

Sentience is designed as infrastructure for token-efficient browser automation, not as a full-service automation provider. We deliberately limit our scope to detection and verification for two reasons:

Sentience detects CAPTCHAs, pauses execution, and verifies clearance. You decide how to resolve them using your own workflows or external systems that comply with your organization's security policies.

How CAPTCHA Handling Works

When the Sentience SDK detects a CAPTCHA during browser automation, the following flow occurs:

1. Detection: The Chrome extension scans for CAPTCHA signals (provider iframes, container selectors, keywords) and reports snapshot.diagnostics.captcha with a confidence score.
2. Policy check: If captcha.detected == true and confidence >= minConfidence, the runtime invokes your configured policy.
3. Action dispatch: Based on the policy, the runtime either aborts, retries with a new session, or waits for clearance.
4. Verification loop: For wait_until_cleared, the runtime continuously re-snapshots until captcha.detected == false or timeout is reached.

Concepts and Terminology

Policies

Policies determine what happens when a CAPTCHA is detected:

Actions

Actions are the runtime behaviors your handler can request:

Configuration Options

Default Strategies (Helpers, Not Solvers)

Sentience provides three built-in strategy helpers. These are not solvers; they configure the runtime's response to CAPTCHA detection and rely on external systems or humans for actual resolution.

Quick Start

Option 1: Abort Policy (Safest Default)

Immediately stop when CAPTCHA is detected. No resolution attempted.

from sentience import AgentRuntime, CaptchaOptions

runtime.set_captcha_options(
    CaptchaOptions(
        policy="abort",
        min_confidence=0.7,
    )
)

Option 2: Human-in-Loop (Recommended for Live Sessions)

When CAPTCHA is detected, the runtime pauses and waits for a human to solve it in the live browser session.

from sentience import CaptchaOptions, HumanHandoffSolver

runtime.set_captcha_options(
    CaptchaOptions(
        policy="callback",
        handler=HumanHandoffSolver(),
        min_confidence=0.7,
        timeout_ms=120_000,  # Wait up to 2 minutes
        poll_ms=1_000,       # Re-check every second
    )
)

Option 3: Vision-Only Verification

Uses vision to confirm the CAPTCHA has cleared. Does not click or type. Useful after an external system has already solved the CAPTCHA.

from sentience import CaptchaOptions, VisionSolver

runtime.set_captcha_options(
    CaptchaOptions(policy="callback", handler=VisionSolver())
)

Option 4: External Resolver Orchestration

Calls your webhook/service when CAPTCHA is detected, then waits for clearance. Your external system performs the actual resolution.

from sentience import CaptchaOptions, ExternalSolver

async def notify_webhook(ctx) -> None:
    """
    Example hook: send context to your external system.
    Replace with your own client / queue / webhook call.
    Sentience does NOT implement solver logic.
    """
    print(f"[captcha] external resolver notified: url={ctx.url} run_id={ctx.run_id}")
    # Example: await your_http_client.post(webhook_url, json={...})

runtime.set_captcha_options(
    CaptchaOptions(
        policy="callback",
        handler=ExternalSolver(lambda ctx: notify_webhook(ctx)),
        timeout_ms=180_000,  # 3 minutes for external service
    )
)

Complete Example

import asyncio
import os

from sentience import (
    AgentRuntime,
    AsyncSentienceBrowser,
    CaptchaOptions,
    ExternalSolver,
    HumanHandoffSolver,
    VisionSolver,
)
from sentience.tracing import JsonlTraceSink, Tracer


async def notify_webhook(ctx) -> None:
    """
    Example hook: send context to your external system.
    Replace with your own client / queue / webhook call.
    Sentience does NOT implement solver logic.
    """
    print(f"[captcha] external resolver notified: url={ctx.url} run_id={ctx.run_id}")
    # Example: await your_http_client.post(webhook_url, json={...})


async def main() -> None:
    tracer = Tracer(run_id="captcha-demo", sink=JsonlTraceSink("trace.jsonl"))

    async with AsyncSentienceBrowser() as browser:
        page = await browser.new_page()
        runtime = await AgentRuntime.from_sentience_browser(
            browser=browser,
            page=page,
            tracer=tracer,
        )

        # ---------------------------------------------------------------------
        # Option 1: Human-in-loop (recommended for live sessions)
        # ---------------------------------------------------------------------
        # When CAPTCHA is detected, the runtime pauses and waits for a human
        # to solve it in the live browser session.
        runtime.set_captcha_options(
            CaptchaOptions(
                policy="callback",
                handler=HumanHandoffSolver(),
                min_confidence=0.7,
                timeout_ms=120_000,  # Wait up to 2 minutes
                poll_ms=1_000,       # Re-check every second
            )
        )

        # ---------------------------------------------------------------------
        # Option 2: Vision-only verification (no DOM actions)
        # ---------------------------------------------------------------------
        # Uses vision to confirm clearance. Does not click or type.
        runtime.set_captcha_options(
            CaptchaOptions(policy="callback", handler=VisionSolver())
        )

        # ---------------------------------------------------------------------
        # Option 3: External resolver orchestration
        # ---------------------------------------------------------------------
        # Calls your webhook/service, then waits for clearance.
        runtime.set_captcha_options(
            CaptchaOptions(
                policy="callback",
                handler=ExternalSolver(lambda ctx: notify_webhook(ctx)),
                timeout_ms=180_000,  # 3 minutes for external service
            )
        )

        # ---------------------------------------------------------------------
        # Option 4: Abort policy (safest for production)
        # ---------------------------------------------------------------------
        # Immediately stop when CAPTCHA is detected. No resolution attempted.
        runtime.set_captcha_options(
            CaptchaOptions(policy="abort", min_confidence=0.7)
        )

        # Navigate and take snapshot (CAPTCHA handling triggers automatically)
        await page.goto(os.environ.get("CAPTCHA_TEST_URL", "https://example.com"))
        runtime.begin_step("Captcha-aware snapshot")
        await runtime.snapshot()


if __name__ == "__main__":
    asyncio.run(main())

CAPTCHA Detection

Detected Providers

Sentience detects the following CAPTCHA providers with high confidence:

CaptchaDiagnostics Schema

The snapshot.diagnostics.captcha object contains:

type CaptchaDiagnostics = {
  detected: boolean;
  provider_hint?: "recaptcha" | "hcaptcha" | "turnstile" | "arkose" | "awswaf" | "unknown";
  confidence: number; // 0..1
  evidence: {
    text_hits: string[];       // Matched text keywords
    selector_hits: string[];   // Matched CSS selectors
    iframe_src_hits: string[]; // Matched iframe sources
    url_hits: string[];        // Matched URL patterns
  };
};

Confidence Scoring

The detection uses a multi-signal scoring system:

• Provider iframe hit: +0.7 confidence
• Provider container selector hit: +0.5 confidence
• Keyword text hit: +0.3 confidence
• URL pattern hit: +0.2 confidence

A CAPTCHA is considered detected when confidence >= 0.7 (configurable via minConfidence).

External Resolution Guidance

If you integrate an external provider (e.g., 2captcha, Anti-Captcha) or your own internal system:

1. Sentience Only Detects and Verifies

The external system performs the actual resolution. Sentience monitors for clearance.

2. Webhook Payload Structure

Your handler receives a context object with:

3. Policy Compliance

Keep audit logs and ensure your resolution approach complies with your organization's policies (consent, allowed domains, rate limits).

4. Handler Return Value

Your handler should return or allow the default wait_until_cleared action. The runtime then confirms clearance before resuming.

5. Timeout Configuration

Set appropriate timeouts based on your external service's SLA. External services may take 30-180 seconds to resolve.

Trace Integration

CAPTCHA events are automatically emitted as verification events to the tracer:

{
  "type": "verification",
  "data": {
    "kind": "captcha",
    "label": "captcha_detected",
    "passed": false,
    "reason": "CAPTCHA detected: recaptcha (confidence: 0.85)",
    "details": {
      "provider_hint": "recaptcha",
      "confidence": 0.85,
      "evidence": {
        "selector_hits": [".g-recaptcha"],
        "iframe_src_hits": ["https://www.google.com/recaptcha/..."]
      }
    }
  },
  "step_id": "abc-123"
}

Best Practices

Related Documentation

• Agent Runtime - Full AgentRuntime API reference
• Tracing & Debugging - Trace events and debugging
• Debugging Agent Failures - Post-mortem analysis