Pilot Proof Mode (v0.2.3+)

PayFlux includes a minimal pilot proof system for capturing "warning → outcome" evidence during a 90-day pilot.

Overview

When enabled, pilot mode provides:

Warning records for elevated+ risk events
Outcome annotation endpoint for event-driven capture
Minimal dashboard for pilot champions
Prometheus metrics for outcome tracking
Stdout proof capture for log pipeline persistence

Enabling Pilot Mode

PAYFLUX_PILOT_MODE=true

When enabled, PayFlux will:

Create in-memory warning records for events with processor_risk_band ∈ {elevated, high, critical}
Register pilot routes (auth required)
Emit pilot outcome annotations to stdout

Pilot Routes

All routes require the same Authorization: Bearer <API_KEY> header as the ingest endpoint.

| Route | Method | Description | |-------|--------|-------------| | /pilot/dashboard | GET | HTML dashboard with outcome annotation | | /pilot/warnings | GET | JSON list of recent warnings | | /pilot/warnings/{id} | GET | Single warning by ID | | /pilot/warnings/{id}/outcome | POST | Set outcome for a warning |

Outcome Annotation

Endpoint

POST /pilot/warnings/{warning_id}/outcome
Authorization: Bearer <API_KEY>
Content-Type: application/json

Request Body

{
  "outcome_type": "throttle",
  "observed_at": "2026-01-11T08:00:00Z",
  "source": "manual",
  "notes": "Processor reduced approval rate by 15%"
}

Outcome Types

| Type | Description | |------|-------------| | throttle | Processor throttled transaction volume | | review | Account flagged for manual review | | hold | Funds or account placed on hold | | auth_degradation | Approval rates declined | | rate_limit | Transaction rate limiting applied | | other | Other observed outcome | | none | No outcome observed (marks as not-observed) |

Example curl

curl -X POST http://localhost:8080/pilot/warnings/1768097434029-0/outcome \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{"outcome_type": "throttle", "observed_at": "2026-01-11T08:30:00Z", "source": "manual", "notes": "15% approval rate drop"}'

Outcome Sources

| Source | Description | |--------|-------------| | manual | Manual annotation via dashboard or API (default) | | stripe_webhook | Outcome detected via Stripe webhook | | adyen_webhook | Outcome detected via Adyen webhook | | other_webhook | Outcome detected via other webhook integration | | other | Other source |

Proof Capture

When an outcome is set, PayFlux emits a JSON line to stdout:

{
  "type": "pilot_outcome_annotation",
  "warning_id": "1768097434029-0",
  "event_id": "550e8400-...",
  "processor": "stripe",
  "risk_band": "high",
  "risk_score": 0.72,
  "warning_at": "2026-01-11T08:00:00Z",
  "outcome_type": "throttle",
  "outcome_timestamp": "2026-01-11T08:30:00Z",
  "outcome_source": "manual",
  "outcome_notes": "15% approval rate drop",
  "lead_time_seconds": 1800,
  "annotated_at": "2026-01-11T08:31:00Z"
}

This can be shipped via Vector/Fluent Bit to persist pilot evidence even if the process restarts.

Prometheus Metrics

| Metric | Type | Description | |--------|------|-------------| | payflux_warning_outcome_set_total{outcome_type,source} | Counter | Outcomes annotated by type and source | | payflux_warning_outcome_lead_time_seconds | Histogram | Time between warning and outcome |

Storage

Pilot mode uses in-memory storage only:

Cap: 1000 warnings (LRU eviction)
Lifetime: process restart clears storage
Redis persistence: not implemented (pilot scope)

This is intentional for pilot simplicity. Proof capture via stdout ensures no data loss.

Guardrails

[!IMPORTANT]

Labels use "Observed Outcome" — not "Caused Outcome"

No causal claims — only "outcome observed after warning"

No recommendations — champions decide their own actions

Language remains probabilistic for Tier 2 context

Security

[!NOTE] All /pilot/* routes require API key authentication via Authorization: Bearer <API_KEY> header. Unauthenticated requests return 401.

Pilot Workflow

Enable pilot mode: PAYFLUX_PILOT_MODE=true
Run PayFlux with Vector shipping logs to your data warehouse
Access /pilot/dashboard with your API key
When processor outcomes occur, annotate them immediately
After 90 days, analyze pilot_outcome_annotation events for proof

Verification Suite

PayFlux includes a full-flow verification script at scripts/verify_pilot.sh. Run it from the repository root:

./scripts/verify_pilot.sh

Checkpoints

| Checkpoint | What it validates | |------------|-------------------| | A: False-Negative | Bad patterns (retry storms, failure spikes) emit warnings | | B: Tier 1 Schema | Gated keys (processor_playbook_context, risk_trajectory) never appear in Tier 1 | | C: Metrics Stability | Metric names and label keys unchanged across container restart |

Metrics stability refers to metric names and label sets, not counter values. Counter values are expected to reset across restarts.

| D: Pilot Containment | /pilot/* routes return 404 when PAYFLUX_PILOT_MODE=false | | E: Log Redaction | No secrets, API keys, or raw payloads in logs | | F: Language Audit | No overclaiming terminology in docs | | G: False Positive | Normal traffic (successful payments) does NOT generate warnings |

Verification Report

On success, a machine-readable report is generated:

Location: verification-reports/verification-report-<timestamp>.json
Contents: timestamp, git commit hash, pass/fail per checkpoint

What a Passing Run Guarantees

A passing verification run confirms:

Risk scoring correctly distinguishes bad patterns from normal traffic
Tier 1 exports do not leak Tier 2 fields
Pilot routes respect the PAYFLUX_PILOT_MODE setting
Logs are safe for observability pipelines (no secrets)
Documentation does not make unsupported claims

It does not guarantee:

Correctness of risk thresholds for specific use cases
Behavior under production load
External system integration correctness

Future Enhancements (Not Implemented)

Redis-backed persistent warning storage
Lead time dashboards and trend analysis
Automated webhook outcome detection