Skip to main content

Restore service first, then approve the fix

NovaNova
pods-probe-web broke after the last release. Restore service first, then fix the cause.
Ask Nova

Rather not read it all? Ask Nova this. It walks you through each step and does the ones it can when you approve.

Service back first with a rollback, then the real fix and a health check, each one approved.

Dana, the on-call engineer, has the cause of the pods-probe-web crash loop, INC-79 (the full investigation).

1. Dana approves the rollback, when Nova offers one​

When a recent release lines up with the incident, Nova first offers to roll it back under Needs you while it keeps looking, and Dana selects Approve or Reject (when Nova offers a rollback). In INC-79 the last deploy was two days old, so Nova went straight to the fix.

2. Dana approves the fix​

On Redeploy pods-probe-web with the nginx-unprivileged image, Dana selects Let Nova fix it.

3. Nova applies it and checks the app​

Open zoom view

The resolved investigation: the cause, the quoted nginx chown error from the app logs, the fix marked Fix applied, and the health check reading pods-probe-web recovered: RUNNING, no warnings in the last 5 min

The resolved investigation: the cause, the quoted nginx chown error from the app logs, the fix marked Fix applied, and the health check reading pods-probe-web recovered: RUNNING, no warnings in the last 5 minThe resolved investigation: the cause, the quoted nginx chown error from the app logs, the fix marked Fix applied, and the health check reading pods-probe-web recovered: RUNNING, no warnings in the last 5 min

Result: "pods-probe-web recovered: RUNNING, no warnings in the last 5 min". Every result a fix can end in: Fixes and mitigations.

Next​

Work an incident from Slack or the terminal