frog-runner: Remote and Local Usage¶

frog-runner is a FHIR Frog service that orchestrates test execution across one or more FHIR servers. You can run it locally in Docker (managed by the MCP) or connect to a remote instance.

When to use frog-runner vs. the local engine¶

Scenario	Tool	Pros
Quick local test during development	`frog_run_local` (MCP's in-process engine)	Fast, no Docker setup
Run a Test IG package	`frog_runner_start_local` + `frog_run_remote`	Test IGs are preloaded in the runner image
Shared CI/remote testing	Remote runner URL	No Docker on your machine; CI infrastructure manages it
Complex multi-server tests	`frog_runner` (v0.2)	Matrix runs, bulk operations

Local runner setup¶

Start the runner and SUT¶

# Start a local HAPI FHIR server (the system under test)
# Note: starts plain HAPI without any IG preloaded
frog_server_start_local(confirm: true)

# Start the runner
frog_runner_start_local(confirm: true)

The MCP will: 1. Create a Docker network named frog-net (if it does not exist) 2. Start a HAPI FHIR container (frog-sut) on loopback port 8081 3. Start the runner container (frog-runner) on loopback port 8080 4. Poll readiness endpoints until both are ready

Typical startup time: - SUT (HAPI FHIR): 40–60 seconds for first start; subsequent starts are faster - Runner: 3–12 seconds

Check status¶

frog_run_status(runId: "<id>")

Polls the runner to check if a run is still in progress. The runner reports COMPLETED, IN_PROGRESS, or FAILED.

Stop the runner and SUT¶

frog_runner_stop(confirm: true)
frog_server_stop(confirm: true)

Removes the containers (but keeps the frog-net network for reuse).

Running tests with the local runner¶

Basic run¶

frog_run_remote(
  testPackage: "fhirfrog.au-core-tests#0.1.0",
  server: "test-sut",
  runnerUrl: "http://127.0.0.1:8080"
)

Or, simpler (if the MCP knows your runner):

frog_run_remote(
  testPackage: "fhirfrog.au-core-tests#0.1.0",
  server: "test-sut"
)

Run a Test IG package¶

Test IG packages (e.g. fhirfrog.au-core-tests#0.1.0) are preloaded in the runner image. Install a concrete version to your local cache, then reference it:

frog_package_install(
  packageId: "fhirfrog.au-core-tests",
  version: "0.1.0",
  confirm: true
)

frog_run_remote(
  testPackage: "fhirfrog.au-core-tests#0.1.0",
  server: "test-sut"
)

Remote runner setup¶

If you have a runner URL (e.g., from your CI infrastructure or a shared environment), point the MCP to it:

export FROG_RUNNER_URL=https://runner.example.org

Then use frog_run_remote without runnerUrl:

frog_run_remote(
  testPackage: "fhirfrog.au-core-tests#0.1.0",
  server: "production-fhir-server"
)

frog-runner HTTP API reference¶

The runner exposes a REST API at /api/v1/. All requests are JSON.

Version (GET /api/v1/version)¶

GET /api/v1/version

Response:

{
  "version": "0.1.0",
  "commit": "abc123...",
  "buildTime": "2026-10-21T12:00:00Z"
}

Create a run (POST /api/v1/runs)¶

Start a new test run.

Request body:

{
  "scriptPath": "patient-search.json",
  "serverUrl": "http://fhir-server:8080",
  "fixturesPath": "./fixtures",
  "igPackage": "fhirfrog.au-core-tests#0.1.0"
}

Response (HTTP 202):

{
  "runId": "run-abc123-def456",
  "status": "QUEUED",
  "createdAt": "2026-10-21T12:34:56Z"
}

Poll run status (GET /api/v1/runs/)¶

GET /api/v1/runs/run-abc123-def456

Response:

{
  "runId": "run-abc123-def456",
  "status": "IN_PROGRESS",
  "progress": {
    "current": 3,
    "total": 10
  },
  "createdAt": "2026-10-21T12:34:56Z"
}

Status values: QUEUED, IN_PROGRESS, COMPLETED, FAILED.

Get run result (GET /api/v1/runs//result)¶

GET /api/v1/runs/run-abc123-def456/result

Response (HTTP 200 if COMPLETED, HTTP 202 if still running):

{
  "runId": "run-abc123-def456",
  "status": "COMPLETED",
  "testScriptUrl": "http://example.org/TestScript/patient-search",
  "serverUrl": "http://fhir-server:8080",
  "result": {
    "resourceType": "TestReport",
    "id": "...",
    "status": "completed",
    "testScriptReference": "...",
    "result": "passed",
    "score": 1.0,
    "tally": {
      "passed": 3,
      "failed": 0,
      "skipped": 0
    }
  }
}

Known limit: The runner returns HTTP 200 even when tests fail. Check result.result for pass/fail status.

Export run (GET /api/v1/runs//export?format=)¶

Export the run result in a different format.

GET /api/v1/runs/run-abc123-def456/export?format=junit-xml
GET /api/v1/runs/run-abc123-def456/export?format=testscript-json
GET /api/v1/runs/run-abc123-def456/export?format=excel

Supported formats (v0.1): - testscript-json — TestReport as JSON - testscript-xml — TestReport as XML - junit-xml — JUnit XML format for CI systems

Response: file download (HTTP 200, with appropriate Content-Type).

Docker setup (local runner and SUT)¶

Network creation¶

The MCP creates the network automatically on first use:

docker network create frog-net

Once created, the network persists until you explicitly remove it:

docker network rm frog-net

If you remove the network while containers are running, you must restart them.

Port binding¶

By default: - SUT: 127.0.0.1:8081 → container port 8080 - Runner: 127.0.0.1:8080 → container port 8080

Both are bound to localhost only (no remote access). Override the host ports with environment variables:

export FHIR_SERVER_PORT=9000
export FROG_RUNNER_PORT=9001
frog_server_start_local(confirm: true)
frog_runner_start_local(confirm: true)

Container-to-container communication¶

On the frog-net network, containers refer to each other by hostname: - Runner refers to SUT as frog-sut:8080 - The SUT does not know about the runner

This means you cannot test by hitting 127.0.0.1:8081 from inside the runner. The runner's tests must target the SUT by name on the internal network.

macOS Docker Desktop notes¶

On macOS, Docker Desktop runs a lightweight Linux VM. Some caveats:

File mounting: Paths are mounted from the macOS host into the Linux VM, then into the container. Home directory (~) works by default; other paths may need sharing in Docker Desktop preferences.
Socket mounting: The Docker socket mount (/var/run/docker.sock) works through a proxy that Docker Desktop provides.
Memory: Give Docker Desktop at least 4 GB when running a SUT + runner together.
FROG_WORKSPACE_HOST: When the MCP itself runs in a container, set this variable to the macOS host path (not the path inside the MCP container).

Remote server registration¶

Register a server so you can refer to it by name:

frog_server_register(
  name: "production",
  url: "https://fhir.example.org/fhir",
  conformsTo: "http://example.org/ImplementationGuide/au-core",
  confirm: true
)

Then use it in runs:

frog_run_remote(
  testPackage: "fhirfrog.au-core-tests#0.1.0",
  server: "production"
)

Troubleshooting¶

Docker not available¶

If the MCP cannot contact Docker, frog_doctor will report docker: FAIL with a hint. Common causes: - Docker daemon is not running - Socket is not mounted (if MCP runs in a container) - DOCKER_HOST is not set (for remote Docker)

Runner unreachable¶

error: network_error, message: "http://127.0.0.1:8080: connection refused"

Check: - Is the runner running? (docker ps | grep frog-runner) - Is the port correct? (check $FROG_RUNNER_PORT) - Is the URL correct? (check $FROG_RUNNER_URL)

Run completes but says `FAILED` to start¶

If the runner returns HTTP 500 with an error like "could not find package", the Test IG package is missing or has the wrong version:

# List available packages
frog_package_list(includeRunner: true)

# Install the missing package
frog_package_install(
  packageId: "fhirfrog.au-core-tests",
  version: "0.1.0",
  confirm: true
)

SUT not ready¶

If HAPI takes longer than 60 seconds to start, the MCP times out. Retry:

frog_server_start_local(confirm: true)

Or check the container logs:

docker logs frog-sut