OpenSat Mission Lab v2.6 Runtime Observability and Resilience
Service-level objectives, error budgets, alerts, incident response, bounded chaos experiments, worker recovery, storage fallback, and automated rollback.
Validation14/14
SLO breaches3
Alerts6
Incidents3
Chaos3
Rollbacks1
Max MTTR9.0 min
Final healthHEALTHY
The demonstration intentionally breaches short-window SLOs. The release gate passes because detection, containment, rollback, recovery, and evidence generation behave as designed.
Runtime health
Validation controls
| Control | Status | Evidence |
|---|---|---|
| Runtime metrics collected | PASS | 60 one-minute samples |
| Availability burn detected | PASS | critical alert expected |
| Latency regression detected | PASS | critical alert expected |
| Worker stall detected | PASS | heartbeat alert expected |
| Object-store failure detected | PASS | storage alert expected |
| SLO breach quantified | PASS | error-budget evidence expected |
| Automated rollback executed | PASS | one rollback |
| Post-rollback service recovered | PASS | minute 28 healthy |
| Worker lease recovery completed | PASS | replacement worker healthy |
| Storage fallback recovered | PASS | object writes healthy |
| Incidents closed | PASS | 3 incidents |
| Chaos experiments bounded | PASS | 3 experiments |
| MTTR guardrail | PASS | all incidents <= 10 minutes |
| Stable final state | PASS | last 10 minutes stable |
SLO evaluation
| SLO | Target | Achieved | Status | Budget burn |
|---|---|---|---|---|
| API availability | 99.900% | 99.212% | BREACH | 7.88x |
| API latency <= 800 ms | 99.000% | 88.965% | BREACH | 11.03x |
| Worker health | 99.000% | 95.000% | BREACH | 5.00x |
Alerts
| ID | Rule | Severity | Minutes | Status |
|---|---|---|---|---|
| ALT-001 | availability-burn | critical | 18-25 | RESOLVED |
| ALT-002 | availability-burn | critical | 45-46 | RESOLVED |
| ALT-003 | latency-saturation | critical | 18-24 | RESOLVED |
| ALT-004 | worker-heartbeat-stale | critical | 34-36 | RESOLVED |
| ALT-005 | queue-backlog | warning | 34-36 | RESOLVED |
| ALT-006 | object-store-errors | warning | 45-47 | RESOLVED |
Incidents
| ID | Incident | Severity | MTTR | Automated response |
|---|---|---|---|---|
| INC-001 | Canary API regression | SEV-1 | 9.0 min | rollback to v2.5-stable |
| INC-002 | Worker heartbeat loss | SEV-2 | 5.0 min | expire lease, requeue job, start replacement worker |
| INC-003 | Object-store write failures | SEV-2 | 3.0 min | activate local spool and retry after recovery |