Tests for code that never runs: how we check CI, infra and store metadata

Some of the most important files in ID Caller never run in a test. A GitHub Actions workflow, a docker-compose file, a Prometheus alert rule, the privacy policy on the website, the App Store description: none of them compiles, and none of them fails a unit test when it goes wrong. They fail later, on deploy, during an incident or in App Review.

ID Caller is our app for private calls without a phone number: a Kotlin/Compose Multiplatform client and a Go backend. We treat the configuration and text around it as code. Every such surface has a check-*.sh script that asserts its invariants, and every script has a suite that copies the files into a fixture, breaks one thing and requires the check to fail.

The inventory

The repository has eight check scripts, one per surface, plus check-all.sh, which chains them locally together with the Go and KMP gates. Next to them, scripts/tests/ holds ten suites: eight for the check scripts, one for the backup, restore and secret rotation scripts, and one for the generator that builds the iOS Localizable.xcstrings from Compose resources. When we ran them for this article, they added up to 300 passing cases.

The rule is written into the project instructions: a new non-executable surface means a new check script on the same pattern, a suite with a fixture and a damage helper, and a step in check-all.sh.

Anatomy of a check script

All eight share one template. Here is check-store.sh, trimmed to the skeleton, the length limits and one App Review rule:

#!/usr/bin/env bash
# Usage: check-store.sh [ROOT]  (defaults to the repository root)
set -euo pipefail

ROOT="${1:-$(cd "$(dirname "$0")/.." && pwd)}"
fail=0

err() {
  echo "FAIL: $*" >&2
  fail=1
}

require_grep() { # file pattern message [grep-flags]
  local file="$1" pattern="$2" message="$3" flags="${4:--q}"
  [[ -f "$ROOT/$file" ]] || return 0 # a missing file is reported by require_file
  grep $flags "$pattern" "$ROOT/$file" || err "$file: $message"
}

IOS_META=docs/store/metadata
AND_META=docs/store/metadata/android
# ... require_file for every metadata file

# 2. Store field length limits (in characters, via python3)
if command -v python3 >/dev/null 2>&1; then
  check_len() { # file max
    local file="$1" max="$2" n
    [[ -f "$ROOT/$file" ]] || return 0
    n="$(python3 -c 'import sys; print(len(open(sys.argv[1], encoding="utf-8").read().strip()))' "$ROOT/$file")"
    [[ "$n" -le "$max" ]] || err "$file: $n characters (limit $max)"
  }
  for locale in ru en-US; do
    check_len "$IOS_META/$locale/name.txt"          30
    check_len "$IOS_META/$locale/keywords.txt"      100
    # ...
  done
  # ... the same for other locales and Google Play
fi

# 3b. App Review 2.3.10: no third-party platforms or their stores in App Store
#     metadata (rejection of 07.08.2026). The android/ directory is Google Play
#     metadata, the ban does not apply there.
ios_only_texts=()
while IFS= read -r f; do
  ios_only_texts+=("$f")
done < <(find "$ROOT/$IOS_META" -type f -name '*.txt' -not -path "$ROOT/$AND_META/*" 2>/dev/null)
if [[ "${#ios_only_texts[@]}" -gt 0 ]]; then
  source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib/forbidden-platforms.sh"
  for pattern in "${FORBIDDEN_PLATFORMS[@]}"; do
    hits="$(grep -lE "$pattern" "${ios_only_texts[@]}" 2>/dev/null || true)"
    if [[ -n "$hits" ]]; then
      err "third-party platform mention (App Review 2.3.10), pattern '$pattern': $hits"
    fi
  done
fi

# 6. Review Notes: VoIP justification, seed ID, video demo
rn=docs/store/review-notes.md
require_grep "$rn" 'voip'                  "no VoIP justification"            -qi
require_grep "$rn" 'reportNewIncomingCall' "no reportNewIncomingCall (invariant 5 in CLAUDE.md)"
# ...

if [[ "$fail" -ne 0 ]]; then
  # ...
  exit 1
fi

Four decisions make the template work:

• ✦The root is an argument. By default the script checks the repository, but it can be pointed at any directory, and that one line is what makes the check testable.
• ✦err records a failure and carries on, so one run prints every broken invariant instead of stopping at the first.
• ✦require_grep stays silent when the file is missing, because require_file has already reported it. One mistake, one message.
• ✦Optional tools degrade gracefully. YAML validation runs only when python3 with PyYAML is available, and the CI jobs that rely on it install PyYAML explicitly.

Pattern lists that several checks need live in scripts/lib/. The forbidden "absolute anonymity" phrasings are sourced by both the website check and the store check, so a pattern fixed once applies to both.

What the eight checks guard

• ✦check-ci.sh: workflows exist and parse; the server pipeline still runs golangci-lint, go test and govulncheck; the app pipeline still builds iOS on a macOS runner. It catches a vulnerability scan deleted "temporarily" and a broken renovate.json.
• ✦check-infra.sh: Terraform, compose, TURN, alerting, backups. It rejects an *.enc.yaml without SOPS markers (a plaintext secret), a returning ssh deploy step (we deploy only through the Dokploy API) and a /metrics endpoint open to the outside.
• ✦check-e2e.sh: all seven Maestro flows exist and start the app, the orchestrator is executable and calls each flow, and the nightly workflow fails when staging is unreachable.
• ✦check-privacy.sh: no absolute anonymity promises on the website; the privacy policy lists exactly what the server stores (ID, public key, push token, links between IDs) in both languages, with the 12-month retention and the note that a P2P call shows your IP to the other person; the iOS privacy manifest declares no tracking and is actually included in the app target.
• ✦check-store.sh: metadata files are present and non-empty, fields fit store limits, Review Notes explain the VoIP usage, App Store texts name no other platforms.
• ✦check-release.sh: signing and credentials come only from properties and environment variables, and fastlane, removed on 19 August, does not come back to overwrite metadata we now maintain in the store consoles.
• ✦check-deeplinks.sh: the pairing universal link is consistent across the backend, the Android manifest, the iOS entitlement, shared navigation and the website's association files. A gap in any layer means a dead link.
• ✦check-bootstrap.sh: scaffold invariants such as pinned Go and Java versions.

Several rules are scar tissue. On 7 August the App Store rejected build 1.0 (2) under Guideline 2.3.10 for an onboarding card reading "Android & iOS". Now check-store.sh bans platform names in App Store metadata, and a Kotlin test does the same for UI strings. On 20 August an alert raised a false critical on eight calls because unanswered calls counted as failures. Now the check requires the rule to subtract unanswered calls, gate on a minimum volume and use or vector(0), without which deploying the rule before the backend would silently mute it. On 23 September we found Grafana panels showing staging data, so every dashboard query must now target production.

Testing the checks: fixture plus damage

A check that never fails is worse than none: it gives a green tick for nothing. So every script has a suite, and apart from the initial scaffold, the suite always came first: commits like "failing tests for privacy texts" land with the whole suite red, because the script does not exist yet.

assert_ok() { # name cmd...
  local name="$1"; shift
  if "$@" >/dev/null 2>&1; then
    echo "ok   - $name"; pass=$((pass + 1))
  else
    echo "FAIL - $name (expected exit 0)"; failures=$((failures + 1))
  fi
}
# ... assert_fail is the mirror image; STORE_FILES lists the store surface

# Fixture: a minimal copy of the store surface in a temporary directory
make_fixture() {
  local dir f
  dir="$(mktemp -d)"
  for f in "${STORE_FILES[@]}"; do
    [[ -f "$REPO_ROOT/$f" ]] && { mkdir -p "$dir/$(dirname "$f")"; cp "$REPO_ROOT/$f" "$dir/$f"; }
  done
  echo "$dir"
}

damage() { # file pattern: delete lines matching pattern (no-op if the file is missing)
  local file="$1" pattern="$2"
  [[ -f "$file" ]] || return 0
  grep -v -- "$pattern" "$file" > "$file.tmp" && mv "$file.tmp" "$file"
}

cleanup_dirs=()
trap 'rm -rf "${cleanup_dirs[@]:-}"' EXIT

assert_ok "the real repository passes" "$CHECK" "$REPO_ROOT"

fx="$(make_fixture)"; cleanup_dirs+=("$fx")
assert_ok "a valid fixture passes" "$CHECK" "$fx"

for f in "${STORE_FILES[@]}"; do
  fx="$(make_fixture)"; cleanup_dirs+=("$fx")
  rm -f "$fx/$f"
  assert_fail "missing $f is caught" "$CHECK" "$fx"
done

fx="$(make_fixture)"; cleanup_dirs+=("$fx")
printf '\nWorks on Android too.\n' >> "$fx/$IOS_META/en-US/description.txt"
assert_fail "\"Android\" in en description (App Store) is caught" "$CHECK" "$fx"

fx="$(make_fixture)"; cleanup_dirs+=("$fx")
printf '\nЗвоните между Android и iOS.\n' \
  >> "$fx/$AND_META/ru-RU/full_description.txt"
assert_ok "\"Android\" in Google Play metadata is allowed" "$CHECK" "$fx"

fx="$(make_fixture)"; cleanup_dirs+=("$fx")
damage "$fx/docs/store/review-notes.md" 'reportNewIncomingCall'
assert_fail "review-notes without reportNewIncomingCall is caught" "$CHECK" "$fx"
# ...

Every suite has the same shape:

1. 01Two baseline cases: the real repository passes, and a fresh fixture passes. The second proves the fixture is a valid starting point, so a later failure comes from the damage, not from a file we forgot to copy.
2. 02One damage per fixture. Each case breaks exactly one invariant in a fresh copy and expects a non-zero exit: deleted lines, an appended forbidden phrase, broken JSON or YAML, a removed file, chmod -x on a script.
3. 03Explicit allowed cases. "Android" in Google Play metadata must pass; placeholder fingerprints in assetlinks.json only warn. Without such cases an overeager pattern fails on legitimate content, and people learn to ignore the check.

In effect, this is hand-written mutation testing for configuration: each damage is a mutant, and the check has to kill it.

When a damage test exposed a weak pattern

The backup check once looked for age -r in backup.sh to make sure dumps are encrypted. In code review we noticed that the script's own error message about a missing recipient also contains age -r, so deleting the real encryption pipe would leave the check green. The fix was a tighter pattern, | age -r, plus a damage case that removes only the pipe line and keeps the message. A grep check is only as good as its pattern, and the suite is where patterns get sharpened.

One runner and a CI step that guards itself

The runner is eleven lines:

#!/usr/bin/env bash
# scripts/tests/run-all.sh: single entry point for the check-script shell tests.
set -euo pipefail

ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
cd "$ROOT"

for test_script in scripts/tests/*.test.sh; do
  echo "==> $test_script"
  "$test_script"
done

It runs in the contract workflow on every pull request:

  dto-sync:
    runs-on: ubuntu-latest
    steps:
      # ... checkout, Go setup, golden-file roundtrip
      - name: CI config invariants
        run: ./scripts/check-ci.sh
      - name: Privacy texts invariants (T-041, R-08)
        run: ./scripts/check-privacy.sh
      - name: Shell validator test suites
        run: ./scripts/tests/run-all.sh

Since every suite starts with "the real repository passes", this step also runs all eight checks against the actual tree. The deploy and E2E workflows additionally call check-infra.sh and check-e2e.sh directly.

The runner arrived in a code-review fix in June; before that, the suites existed but nothing in CI ran them. To keep the step from disappearing, check-ci.sh requires it, and its suite deletes that line from a fixture copy of the workflow and expects a failure:

require_grep .github/workflows/contract.yml 'scripts/tests/run-all\.sh' \
  "the check-script shell tests are not run"

The same review fixed a quieter problem. The nightly E2E and weekly load test workflows used to skip softly when staging was unreachable, so a run could look green without executing a single flow. Now they fail, and the checks require the exit 1 guard in both.

Where this stops

These checks prove presence, not meaning: require_grep 'govulncheck' passes even if the word survives only in a comment. We accept that. The scripts are tripwires for regressions people actually make (a deleted step, a reverted setting, a banned phrase), while real execution stays with terraform validate, the OpenAPI validator and the nightly Maestro run. Changing a decision means changing the check and its test in the same commit, and that friction is intended.

How to start in your own project

• ✦Pick one surface that has already hurt you: a workflow, a compose file, store texts.
• ✦Write its invariants down as plain sentences, then turn each one into require_file or require_grep.
• ✦Let the script take the root directory as an argument and collect all failures before exiting.
• ✦Write the suite first: a fixture that copies only what the check reads, baseline cases for the real repository and the fresh fixture, then one damage per invariant.
• ✦Add allowed cases wherever a pattern could fire on legitimate content.
• ✦Keep shared pattern lists in one sourced file.
• ✦Run every suite through a single runner in CI, and make a check assert that this step exists.
• ✦When an incident or a rejection teaches you a rule, add the invariant and its damage test in the same change.