Production Incident Postmortem — Incomplete Logs

An outage took down checkout for 40 minutes at ShopFast. Logs are partial — a log-rotation bug lost 15 minutes of data during the exact window the trigger likely occurred. The manager must reconstruct the causal chain manually.

# Aplly.xyz Case Study Submission

## Title
Production Incident Postmortem — Incomplete Logs

## Type
Technology/IT

## Difficulty
Advanced

## Estimated Time
90 minutes

## Overview
An outage took down checkout for 40 minutes at ShopFast. Logs are partial — a log-rotation bug lost 15 minutes of data during the exact window the trigger likely occurred. The manager must reconstruct the causal chain manually.

## Case Details

Function Focus: Root-cause reconstruction under incomplete evidence

Scenario:
The CEO wants a 5-whys postmortem by tomorrow. The missing 15-minute log window contains the likely trigger event, so the manager must reason from surrounding evidence (deploy history, error-rate trends before/after the gap, and on-call chat transcripts) rather than from a complete log trail.

Dataset Structure:
- Timestamped log excerpts before and after the 15-minute gap
- Deploy event log (what was shipped and when)
- On-call chat transcript excerpts
- Error-rate trend description (before, during partial data, after)

Tasks:
1. Reconstruct the most probable causal chain despite the gap, using the surrounding evidence
2. Apply a 5-whys analysis by hand
3. Explicitly flag which conclusions are confirmed by direct evidence versus inferred from circumstantial evidence
4. Recommend exactly 2 preventive actions, each tied directly to a specific point in the causal chain (not generic "add more monitoring" answers)

Expected Output:
A causal chain timeline/diagram + written 5-whys analysis + confirmed-vs-inferred labeling for each claim + 2 specific preventive actions.

Evaluation Criteria:
Logical soundness of reasoning under uncertainty, honesty in labeling confirmed vs. inferred conclusions, and specificity (not genericness) of the preventive actions.

## Data Sources

Log excerpts (09:58–10:05):
```
09:58:02 checkout-svc INFO request rate nominal (420 req/min)
09:59:14 checkout-svc INFO deploy_marker: build #4471 rolling out (canary 10%)
10:01:40 checkout-svc WARN p99 latency elevated: 1.2s (baseline 300ms)
10:04:55 checkout-svc WARN connection pool utilization: 78%
```

[LOG GAP: 10:05:00–10:20:00 — log rotation failure, no data captured]

Log excerpts (10:20–10:40):
```
10:20:03 checkout-svc ERROR connection pool exhausted, 502s returned
10:20:03 checkout-svc ERROR 502 rate: 94% of requests
10:31:12 checkout-svc INFO manual rollback initiated: build #4471 -> #4470
10:38:47 checkout-svc INFO connection pool recovering, 502 rate: 12%
10:40:02 checkout-svc INFO service nominal, 502 rate: 0%
```

Deploy log:
```
09:59:14 build #4471 deployed to 10% canary — includes new "saved payment methods" query
10:12:00 build #4471 auto-promoted to 100% (canary metrics looked nominal at time of promotion)
10:31:12 manual rollback to build #4470
```

On-call chat transcript excerpts:
```
[10:06] eng_priya: latency's creeping up on checkout, keeping an eye on it
[10:13] eng_priya: canary looked fine so autopromote went through
[10:21] eng_priya: 502 storm, pool exhausted, paging now
[10:22] eng_marcus: joining. pool exhaustion smells like a connection leak
[10:29] eng_marcus: new payment-methods query doesn't appear to close its DB connection on the timeout path
[10:31] eng_priya: rolling back #4471 now
```

## Solution Frameworks
5-whys, timeline reconstruction, confirmed/inferred evidence labeling

## Solver Guidance & Tutorials
Link to: "Postmortems Under Incomplete Data" tutorial

## What You'll Learn
- Root-cause analysis when evidence is incomplete
- Calibrated confidence — knowing what you actually know vs. what you're inferring
- Writing preventive actions specific enough to actually prevent recurrence

## Tags
incident postmortem, root cause analysis, forensic reasoning, technology/IT

## Registration Links
- Register as Solver
- Register as Evaluator