Test cost discipline

Plan 113 WI-4 — every test that talks to a real, paid external service (LLM provider, Azure resource) must be marked so the default pytest invocation never burns cost.

Markers

Marker	Env opt-in	Use for
real_llm	RUN_REAL_LLM_TESTS=1	Anthropic / Azure OpenAI / OpenAI calls
real_azure	RUN_REAL_AZURE_TESTS=1	Live Azure resource calls (Key Vault etc.)
slow	pytest -m slow	Legacy — kept for the v0.11.x eval tests

pyproject.toml deselects real_llm / real_azure / slow from the default addopts, so pytest, uv run pytest, uv run pytest tests/, and CI's pytest tests/unit all skip them automatically.

tests/conftest.py adds a belt-and-braces skip guard: even an explicit pytest -m real_llm invocation skips the tests when RUN_REAL_LLM_TESTS is not 1. The env opt-in must be conscious — otherwise a colleague running -m real_llm from a stale shell with the production Anthropic key in env would burn cost unintentionally.

Marking a test

import pytest

@pytest.mark.real_llm
@pytest.mark.asyncio
async def test_my_anthropic_integration() -> None:
    # talks to a real Anthropic endpoint with a real key
    ...

For Azure:

import pytest

@pytest.mark.real_azure
def test_my_keyvault_write() -> None:
    # talks to a real Azure Key Vault
    ...

Opting in

Local nightly runs:

RUN_REAL_LLM_TESTS=1 uv run pytest -m real_llm tests/eval/
RUN_REAL_AZURE_TESTS=1 uv run pytest -m real_azure tests/integration/

CI does not set these env vars in the default .github/workflows/ci.yml on push to main — they're reserved for the future nightly cost-burn workflow.

What NOT to mark

• Anything that uses the MockClient / MockExtractor LLM stubs — those are deterministic and cost-free, so they're regular unit tests.
• Anything that talks to local Postgres / Redis / Docker — those are integration tests (already covered by tests/integration/), not real-money tests.
• Anything that calls Microsoft Entra ID's /oauth2/v2.0/authorize endpoint with a test client — that's free.

Audit cadence

Every time you add a new third-party paid service:

1. Add a new markers = [...] entry in pyproject.toml.
2. Add a corresponding env-gate stanza in tests/conftest.py.
3. Document in this file.
4. Don't add the env var to the default CI workflow — that's a separate, conscious decision (engineering manager + finance sign-off).