Skip to main content

Operate with Nova

NovaNova
How do I put Nova on call for my team?
Ask Nova

Rather not read it all? Ask Nova this. It walks you through each step.

A platform engineer's real job isn't building the platform once; it's carrying it every day after. The pager goes at 2am, and the first hour goes on digging through logs, events, recent deploys and config to find what changed, while the same questions arrive from every team and postmortems get written days later from memory. Nova takes that first hour: it reads the evidence, says what changed and why, proposes the fix and waits for a person to approve it, then checks the app is healthy and writes the postmortem from the record. Your platform team decides what Nova may touch and who approves.

What you're getting into​

Your platform team sets once what Nova may do and who approves. When an app breaks, the on-call engineer opens to a cause with its evidence, approves the fix, and Nova checks the app afterwards. Every answer is recorded with who gave it, and the postmortem is drafted from that record.

What Nova does for you​

When something breaks

While you build

Every day

With guardrails

Who does what​

  • Alice, the platform team: sets what Nova may do and who approves, and builds with Nova.
  • Sam, the SRE: routes alerts to Nova, and owns the postmortem and the fix that sticks.
  • Dana, the on-call engineer: answers the page, works the incident with Nova, restores service and approves the fix.
  • Bob, on a team that uses the platform: owns an app and sees it fail and recover.

Nova investigates an incident​

A real incident, INC-79: pods-probe-web restarts in a loop, and an investigation opens at 23:51 PDT.

  1. Nova reads first. The app's status, events, deploy history and configuration, then past incidents on this app.
  2. It asks before reading logs. App logs are a sensitive read; Dana, on call, allows it at 23:52.
  3. It names the cause, with the line that proves it. It rules out 3 other causes and says what it could not check.
Open zoom view

Cause found with two quoted reads: the nginx chown error from App Logs, pods-probe-web at 23:53 PDT, and the last deploy from deploy history; the warning Not checked: the container might be missing permissions or capabilities beyond running as a non-root user; three ruled-out causes; and How Nova got here, 8 reads

Cause found with two quoted reads: the nginx chown error from App Logs, pods-probe-web at 23:53 PDT, and the last deploy from deploy history; the warning Not checked: the container might be missing permissions or capabilities beyond running as a non-root user; three ruled-out causes; and How Nova got here, 8 readsCause found with two quoted reads: the nginx chown error from App Logs, pods-probe-web at 23:53 PDT, and the last deploy from deploy history; the warning Not checked: the container might be missing permissions or capabilities beyond running as a non-root user; three ruled-out causes; and How Nova got here, 8 reads
  1. It proposes a fix, with its risk and undo. Dana approves it at 23:55.
Open zoom view

The fix: Redeploy pods-probe-web with the nginx-unprivileged image, low risk, reversible, Fix applied, its three steps, a note that Nova could not verify a command for step 3, and the undo

The fix: Redeploy pods-probe-web with the nginx-unprivileged image, low risk, reversible, Fix applied, its three steps, a note that Nova could not verify a command for step 3, and the undoThe fix: Redeploy pods-probe-web with the nginx-unprivileged image, low risk, reversible, Fix applied, its three steps, a note that Nova could not verify a command for step 3, and the undo
  1. It applies the fix and checks the app's health.
Open zoom view

The resolved investigation: the cause, the quoted nginx chown error from the app logs, the fix marked Fix applied, and the health check reading pods-probe-web recovered: RUNNING, no warnings in the last 5 min

The resolved investigation: the cause, the quoted nginx chown error from the app logs, the fix marked Fix applied, and the health check reading pods-probe-web recovered: RUNNING, no warnings in the last 5 minThe resolved investigation: the cause, the quoted nginx chown error from the app logs, the fix marked Fix applied, and the health check reading pods-probe-web recovered: RUNNING, no warnings in the last 5 min
  1. It writes the postmortem from the record.
Open zoom view

The postmortem for pods-probe-web: SEV1, opened Sep 26, 23:51:40 PDT, investigation duration 17 minutes, the executive summary with the cause and the applied fix, and the impact, unconfirmed because no user-facing telemetry was available

The postmortem for pods-probe-web: SEV1, opened Sep 26, 23:51:40 PDT, investigation duration 17 minutes, the executive summary with the cause and the applied fix, and the impact, unconfirmed because no user-facing telemetry was availableThe postmortem for pods-probe-web: SEV1, opened Sep 26, 23:51:40 PDT, investigation duration 17 minutes, the executive summary with the cause and the applied fix, and the impact, unconfirmed because no user-facing telemetry was available

The whole incident: Nova fixes a crash-looping deploy. Each step as the on-call engineer works it: Investigate a failing app.

Platform team: Alice​

For whoever runs the platform and decides what Nova may touch.

  1. Set what Nova may do and who approves
  2. Build with Nova

SRE: Sam​

For whoever owns reliability, alert rules and postmortems.

  1. Put Nova on call
  2. Route alerts to Nova
  3. Write the postmortem from the record

On-call engineer: Dana​

For whoever answers the page.

  1. Investigate a failing app
  2. Restore service, then approve the fix
  3. Work it from Slack or the terminal

Your teams: Bob​

For the teams that ship apps on the platform.

  1. Ask Nova about the platform

Next​

Nova fixes a crash-looping deploy