Workflows¶
Two walk-throughs, one per persona, both using the hello Test IG that ships in the MCP repository (examples/hello-test-ig). Every step below was run for real against a stock HAPI server and the frog-runner image. Tool calls are shown in the form the assistant makes them; you normally just state the goal in plain language and approve the plans.
IG developer: from nothing to a passing test¶
Goal: a Test IG with a working test, compiled from shorthand, run against a local server.
1. Check the machine.
Fix anything marked fail. Warnings for sushi or the IG Publisher only matter when you build or publish an IG.
2. Scaffold the Test IG. The first call returns a plan; repeat with confirm: true after you approve it.
frog_scaffold_test_ig(dir: "hello-ig", packageId: "example.hello.test", canonical: "http://example.org/hello-test", name: "HelloTest", title: "Hello Test", publisher: "Example", subjectIg: "hl7.fhir.au.base#6.0.0", confirm: true)
frog_check_ig(dir: "hello-ig")
The scaffold writes the IG configuration, publisher files, a starter TestScript (example-hello-test-patient-create-starter.json-style) and a fixture. frog_check_ig reports known publishing traps; a fresh scaffold has no errors.
3. Write a test in shorthand. Ask for what you want to test ("create a Patient and search for it"); the assistant writes .tsh using the grammar in the shorthand reference and checks it before compiling:
frog_tsh_validate(path: "hello-ig/src/hello.tsh")
frog_tsh_compile(path: "hello-ig/src/hello.tsh", format: "json", output: "hello-ig/input/resources/hello.json")
frog_testscript_lint(path: "hello-ig/input/resources/hello.json")
Lint flags problems the compiler accepts but the engine would reject, such as a resource written as Patient?family=x (use params), or a sourceId that names a file instead of a fixture id.
3b. Know the limits. Shorthand cannot express search parameters or reads by id; for those the assistant writes JSON TestScript directly. See choosing a format.
4. Start a local server and run.
frog_server_start_local(confirm: true)
frog_run_local(scriptPath: "hello-ig/input/resources/hello.json", server: "frog-sut")
HAPI takes about a minute to start; the tool returns starting and you call it again until it says running. server: "frog-sut" selects the managed local server (Server specifications).
5. Read the result. A failing assertion carries its message; frog_report_summarise groups many failures into causes with hints. Then fix the test or the server and re-run.
6. Clean up.
Publishing the Test IG as a package is planned for v0.2 (frog_test_ig_publish, not available yet).
System tester: run a known test set against a server¶
Goal: run published tests against a server you do not own, repeatably, and understand the failures.
1. Check the machine and bring up the runner.
The runner container serves the preloaded AU test packages. Skip this if your team operates a shared runner: pass its URL as runnerUrl in the run calls.
2. Register the server once. A secret is given as an environment variable reference, never pasted:
frog_server_register(name: "qld-ereferral", url: "https://example.org/fhir", spec: "au-core", authType: "client-credentials", tokenUrl: "https://auth.example.org/token", clientId: "frog", clientSecret: "${FROG_SECRET}", confirm: true)
frog_server_check(name: "qld-ereferral")
3. See which test packages are available.
4. Run. Tests can write to the server, so a non-loopback target needs confirm: true after you have looked at the plan:
frog_run_remote(testPackage: "fhirfrog.au-core-tests#0.1.0", server: "qld-ereferral", confirm: true)
frog_run_status(runId: "<id from the previous result>")
Large packages take longer than one tool call may block; the result says so and you poll frog_run_status.
5. Interpret.
frog_report_summarise(runId: "<id>")
frog_run_report(runId: "<id>", format: "junit-xml", outputPath: "reports/qld.xml")
The summary separates assertion failures (the server behaved differently from the test's expectation) from execution errors (the test could not run: unreachable server, missing fixture, bad script), because they need different fixes. See interpreting failures.
6. Bring your own scripts. To run scripts you wrote against the local runner, lay them out as a bundle (scripts at the top, fixtures under fixtures/), mount it when starting the runner and run it:
frog_runner_start_local(confirm: true, bundlePath: "bundle")
frog_run_remote(bundlePath: "bundle", server: "frog-sut", confirm: true)
See the runner.
When something goes wrong¶
The assistant should stop and tell you when a tool returns an error. The stable error codes and their fixes are in Troubleshooting. Do not let it work around a refused action (for example by pasting a secret or confirming without showing you the plan); the safety model explains what each refusal protects.