Skip to main content

Nova fixes a crash-looping deploy

NovaNova
Why is pods-probe-web crash-looping?
Ask Nova

Rather not read it all? Ask Nova this. It walks you through each step and does the ones it can when you approve.

A real incident, INC-79, from the first read to the postmortem.

pods-probe-web, an app Bob's team owns, restarts in a loop. An investigation opens at 23:51 PDT, and Dana, the on-call engineer, picks it up.

1. Nova finds the cause​

Nova reads the app's status, events, deploy history and configuration, asks before reading the logs (Dana allows it at 23:52), tests 4 possible causes and rules out 3.

Open zoom view

Cause found with the quoted nginx chown error from App Logs, pods-probe-web at 23:53 PDT and the last deploy from deploy history, the Not checked warning, three ruled-out causes, and How Nova got here with its 8 reads

Cause found with the quoted nginx chown error from App Logs, pods-probe-web at 23:53 PDT and the last deploy from deploy history, the Not checked warning, three ruled-out causes, and How Nova got here with its 8 readsCause found with the quoted nginx chown error from App Logs, pods-probe-web at 23:53 PDT and the last deploy from deploy history, the Not checked warning, three ruled-out causes, and How Nova got here with its 8 reads

Cause found: "The container runs as a non-root user and cannot write to a path nginx requires at startup", proved by the chown error it quotes from the app's logs.

2. Dana approves the fix​

Open zoom view

The fix: Redeploy pods-probe-web with the nginx-unprivileged image, low risk, reversible, Fix applied, its three steps, the note that Nova could not verify a command for step 3, and the undo

The fix: Redeploy pods-probe-web with the nginx-unprivileged image, low risk, reversible, Fix applied, its three steps, the note that Nova could not verify a command for step 3, and the undoThe fix: Redeploy pods-probe-web with the nginx-unprivileged image, low risk, reversible, Fix applied, its three steps, the note that Nova could not verify a command for step 3, and the undo

Redeploy pods-probe-web with the nginx-unprivileged image, low risk and reversible. Dana selects Let Nova fix it at 23:55.

3. Nova checks the app​

Open zoom view

The resolved investigation: the cause, the quoted nginx chown error, the fix marked Fix applied, and the health check reading pods-probe-web recovered: RUNNING, no warnings in the last 5 min

The resolved investigation: the cause, the quoted nginx chown error, the fix marked Fix applied, and the health check reading pods-probe-web recovered: RUNNING, no warnings in the last 5 minThe resolved investigation: the cause, the quoted nginx chown error, the fix marked Fix applied, and the health check reading pods-probe-web recovered: RUNNING, no warnings in the last 5 min

"pods-probe-web recovered: RUNNING, no warnings in the last 5 min". Bob's team has its app back.

4. Sam reads the postmortem​

Open zoom view

The postmortem for pods-probe-web: SEV1, opened Sep 26, 23:51:40 PDT, investigation duration 17 minutes, the executive summary with the cause and the applied fix, and the impact, unconfirmed because no user-facing telemetry was available

The postmortem for pods-probe-web: SEV1, opened Sep 26, 23:51:40 PDT, investigation duration 17 minutes, the executive summary with the cause and the applied fix, and the impact, unconfirmed because no user-facing telemetry was availableThe postmortem for pods-probe-web: SEV1, opened Sep 26, 23:51:40 PDT, investigation duration 17 minutes, the executive summary with the cause and the applied fix, and the impact, unconfirmed because no user-facing telemetry was available

Result: a 17-minute investigation, a fix a person approved, a verified recovery and a postmortem from the record. Every step in detail: Investigate a failing app.

Next​

Investigate a failing app