Investigate a failing app
Rather not read it all? Ask Nova this. It walks you through each step and does the ones it can when you approve.
The cause of a failing app, quoted from what Nova read, then the fix you approve and the proof it worked.
This is a real incident, INC-79. pods-probe-web restarts in a loop, and an investigation opens at 23:51 PDT. Dana, the on-call engineer, works it in Nova > Investigations. Bob's team owns the app and sees it recover.
1. Nova checks the first facts
Before it forms a cause, Nova reads the app's status, events, deploy history and configuration, and the past incidents on this app.
2. Dana allows the log read
App logs are a sensitive read, so Nova asks first under Needs you. Dana allows it at 23:52 PDT. Declining is fine too: Nova carries on without them (Allow or decline a read).
3. Nova quotes the evidence
Cause found, with two quoted reads, each with its source and time. The first is the line that proves the cause:
nginx: [emerg] chown("/var/cache/nginx/client_temp", 101) failed (1: Operation not permitted) App Logs, pods-probe-web, 23:53 PDT
The second is the last deploy, two days before it started. Three other causes are ruled out: a probe on a path or port nginx does not serve, a missing secret or config map, and a bad image reference.
4. Nova says what it could not check
The amber line is what Nova did not confirm: "The container might be missing necessary permissions or capabilities beyond just running as a non-root user." It is in the record, not hidden.
5. Nova names the cause
"The container runs as a non-root user and cannot write to a path nginx requires at startup." How Nova decides a cause is found and not just likely: How Nova investigates.
6. Nova proposes the fix
Redeploy pods-probe-web with the nginx-unprivileged image: low risk, reversible, three steps, and the undo, "Redeploy pods-probe-web with the original image docker.io/library/nginx:1.27-alpine if the unprivileged image causes other issues." Nova also says it could not verify a command for step 3.
7. Dana approves it
On the card, Dana selects Let Nova fix it at 23:55 PDT. Nova applies exactly that change (Fixes and mitigations).
8. Nova checks the app's health
The health check reads "pods-probe-web recovered: RUNNING, no warnings in the last 5 min", and the investigation is Resolved.
9. Who did what
What happened lists every decision with its time: the log read allowed at 23:52, the fix approved at 23:55, and each health check.
10. The postmortem
Nova drafts it from the record: severity, when it opened, the investigation's duration (17 minutes), the cause, the fix and the impact. Where it has no evidence, it says so: user impact is marked unconfirmed.
Result: the cause with the line that proves it, a fix a person approved, a health check that confirms it, and a postmortem, all from one record. Sam, the SRE, publishes it: Write the postmortem from the record.