Skip to content

TestReport Overview

TestReport is a FHIR resource that captures the results of executing TestScripts. FHIR Frog builds TestReports incrementally during execution as first-class resources in the client-side ResourceSpace.

How TestReports Are Built

Unlike approaches that assemble a report after execution from a shadow data structure, FHIR Frog builds the TestReport as a live FHIR resource inside the ResourceSpace:

  1. Before execution — A TestReport is created with testScript set to a FHIR Reference pointing to the TestScript in the same space.
  2. During setup — Each operation/assertion result is appended to TestReport.setup.action[] as it completes.
  3. During tests — Results are appended to the corresponding TestReport.test[].action[].
  4. During teardown — Results are appended to TestReport.teardown.action[].
  5. After execution — The report's status and result are finalized. The completed TestReport is returned in ExecutionResult.getTestReport().

This means the TestReport always has a proper FHIR Reference to its TestScript, and action results contain live detail/request/response URIs — not reconstructed approximations.

Legacy Path

The older TestResult DTO and TestReportRenderer still work for backward compatibility but are deprecated. New code should use ExecutionResult.getTestReport() directly.

What is a TestReport?

A TestReport documents:

  • Test execution results - Pass/fail status
  • Timing information - Execution duration
  • Assertions - Which passed and which failed
  • Messages - Error details and diagnostics
  • Test hierarchy - Setup, tests, teardown results

Basic Structure

{
  "resourceType": "TestReport",
  "status": "completed",
  "testScript": {
    "reference": "TestScript/patient-crud"
  },
  "result": "pass",
  "score": 100.0,
  "setup": {},
  "test": [],
  "teardown": {}
}

Automatic Generation

FHIR Frog automatically generates TestReports after test execution:

@ExtendWith(FhirFrogExtension.class)
@FhirServerUrl("http://localhost:8080/fhir")
@GenerateTestReport(format = "json", outputDir = "target/reports")
class PatientTests {

    @TestScript("classpath:fhir/patient-crud.json")
    void testPatient() {
        // TestReport automatically generated
    }
}

Report Contents

Test Results

Each test includes:

  • Result - pass, fail, pending, skip
  • Score - Percentage of assertions passed
  • Duration - Execution time
  • Messages - Failure details

Assertion Results

{
  "assert": {
    "result": "pass",
    "message": "Response code is 201",
    "detail": "Expected: 201, Actual: 201"
  }
}

Failed Assertions

{
  "assert": {
    "result": "fail",
    "message": "Patient name mismatch",
    "detail": "Expected: Smith, Actual: Jones"
  }
}

Storage Options

File System

FileSystemReportStorage storage = new FileSystemReportStorage(Path.of("reports"));
storage.store(testReport, "json");

FHIR Server

ServerReportStorage storage = new ServerReportStorage(fhirClient);
storage.store(testReport);

Both

CompositeReportStorage storage = new CompositeReportStorage(
    new FileSystemReportStorage(Path.of("reports")),
    new ServerReportStorage(fhirClient)
);
storage.store(testReport);

Report Formats

JSON

mvn test -Dreport.format=json

XML

mvn test -Dreport.format=xml

Both

mvn test -Dreport.format=both

CI/CD Integration

GitLab CI

test:
  script:
    - mvn test
  artifacts:
    reports:
      junit: target/surefire-reports/*.xml
    paths:
      - target/fhir-reports/*.json

GitHub Actions

- name: Run tests
  run: mvn test

- name: Upload test reports
  uses: actions/upload-artifact@v3
  with:
    name: fhir-test-reports
    path: target/fhir-reports/

Report Analysis

Summary Statistics

TestReport report = renderer.render(testResult);
System.out.println("Result: " + report.getResult());
System.out.println("Score: " + report.getScore() + "%");
System.out.println("Tests: " + report.getTest().size());

Failed Tests

report.getTest().stream()
    .filter(t -> t.getResult() == TestReportActionResult.FAIL)
    .forEach(t -> System.out.println("Failed: " + t.getName()));

Best Practices

Store Reports

Always store TestReports for audit trails and compliance documentation.

Large Reports

For test suites with many tests, consider storing reports on a FHIR server rather than in files.

  1. Version control - Store reports with test results
  2. Trend analysis - Track test success over time
  3. Compliance - Use for regulatory documentation
  4. Debugging - Review failed assertion details

Next Steps