Skip to main content

Alerts

When an alert fires, Nova can start investigating before anyone opens a laptop. Point Alertmanager or Grafana at Nova and each firing alert starts an investigation that only reads, posts what it finds to your on-call Slack channel, and waits for a person before anything changes.

When you need it​

  • Your on-call starts from alerts and you want the first look done by the time someone picks it up.
  • You want one investigation per incident, not one per notification, even when an alert keeps firing or a whole app is alerting at once.

Before you start​

  • An organization owner or admin to set it up.
  • Units for the investigations: an alert starts a deep investigation, 10 units, or 1 when Nova can read nothing. Nova Units
  • To get results in Slack: the Enterprise plan, Slack connected under Skills, and Nova invited to the channel. Without Slack, results reach the console.

In the console​

  1. In Nova, open Settings (the gear at the bottom of the sidebar) and choose Incidents.

    Open zoom view

    Nova Settings, Incidents: the webhook URL, the alert token and the Slack channel for results

    Nova Settings, Incidents: the webhook URL, the alert token and the Slack channel for resultsNova Settings, Incidents: the webhook URL, the alert token and the Slack channel for results

  2. Under Alerts from Alertmanager or Grafana, select Create alert token. Copy the token now: it is shown once.

  3. Optionally, in Post results to Slack channel, enter a channel such as #incidents.

  4. Point your alerting at Nova with the Webhook URL the panel shows:

    Alertmanager. Add a receiver like the one the panel shows, with your token:

    YAML
    receivers:
      - name: nova
        webhook_configs:
          - url: https://astropulse.io/api/alerts/alertmanager
            send_resolved: false
            http_config:
              authorization:
                credentials: <your alert token>

    Grafana. Add a webhook contact point with the same URL, and send the token in the Authorization: Bearer <your alert token> header. Grafana's webhook sends the same payload as Alertmanager.

    Any other system can send an Alertmanager (version 4) payload to the same URL with the same header.

You should see​

The panel shows when the last alert arrived. When an alert fires, Nova posts in your channel: "<alert>: Nova is investigating. It only reads, and posts what it finds in this thread.", with a link to the investigation. When it concludes, it replies in the thread with the cause and whether it is found or likely, what it could not check, and any recent deploy or unhealthy neighbour. The investigation also appears under Investigations in Nova, shared with your organization, so whoever is on call can open it, approve or decline its fix, and continue the conversation.

Send a test alert​

Nova answers 202 with whether the alert started an investigation, or why not. A test alert starts a real investigation and is charged like one.

A test alert with curl
Terminal
curl -X POST https://astropulse.io/api/alerts/alertmanager \
  -H "Authorization: Bearer $NOVA_ALERT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "version": "4",
    "status": "firing",
    "groupKey": "{}:{alertname=\"HighErrorRate\"}",
    "groupLabels": { "alertname": "HighErrorRate" },
    "commonLabels": { "alertname": "HighErrorRate", "app": "payments-api", "severity": "critical" },
    "commonAnnotations": { "summary": "5xx above 5% for 10m" },
    "alerts": [{
      "status": "firing",
      "labels": { "alertname": "HighErrorRate", "app": "payments-api", "severity": "critical" },
      "annotations": { "summary": "5xx above 5% for 10m" },
      "startsAt": "2026-09-18T02:10:00Z"
    }]
  }'

Label alerts with what they are about​

Nova starts from the target your alert's labels name. The first label found wins:

  • App: app, service, application, deployment or addon.
  • Namespace: namespace.
  • Cluster: cluster or cluster_name. Add it when the same app runs on more than one cluster, or when the alert is about a cluster add-on.

What Nova does with an alert​

  • It only reads. Anything that would change something waits for a person, under your guardrails. Reading logs needs Sensitive Data Access, which you grant from the investigation.
  • It runs as the admin who created the token, and is billed like any other investigation.
  • Alert text is data, not instructions. Labels and annotations reach Nova as quoted data, so an alert cannot redirect the investigation.

Limits that protect you​

  • Resolved alerts are ignored. Only firing alerts start an investigation.
  • The same alert again: Alertmanager re-sends a group while it fires. The re-send is recognized and nothing new starts.
  • The same alert group fires again: it joins its investigation while that incident is open, within the grouping window (4 hours by default). Once the incident has ended, a new firing starts a new investigation.
  • A storm on one app: from the third alert group on the same target within the grouping window (by default), each joins the first investigation instead of starting another.
  • Many alerts at once: at most 10 alert investigations per organization per hour by default. Alerts past the limit start nothing.
  • Fewer than 10 units left: nothing runs, because an alert investigation is a deep one.
  • Wrong or missing token: refused before any work starts.

Owners and admins change the grouping window, storm threshold and hourly limit in Settings › Incidents (see Configure). The same page lists, under Alerts that started no investigation, every alert that started none, with the reason (Already investigating, Grouped, Storm, Hourly limit or No units) and the investigation it joined.

Answer from the alert's thread​

  • Allow, on a read Nova asks for: Nova reads it. Only people who may approve reads can allow one.
  • A button or a reply to Nova's question: your answer is weighed like a read.
  • Steer, while Nova investigates: type what to check first, for example "check the database first", in the box it opens. Nova reads that next. Steering never grants access.
  • Why? Show the evidence, on Nova's conclusion: the cause and the evidence from the record, shown only to you. Nothing new is read.
  • Acknowledge or I'll fix it myself: recorded with who answered and when.
  • Approve, on a low-risk change: applied, if your organization allows Slack approvals and you may approve changes. Otherwise the button opens the console. See Fixes and mitigations.

Stop is in the console. Your Slack account is matched to your AstroPulse account by email. While service is impaired, the thread also gets a one-line stakeholder update at each change, and at least every 30 minutes by default.

Your organization's instructions​

In Settings › Instructions, under Instructions for Nova, admins can tell Nova where services live, who to page and what never to touch (up to 4,000 characters). Nova uses this as context in chat, investigations and postmortems. It never unlocks an action: guardrails still decide what runs. Share what Nova learns with the team, on by default, lets one engineer's ruled-out causes help the next person on call. Every change to these settings is recorded.

Open zoom view

Nova Settings, Instructions: the organization's instructions for Nova

Nova Settings, Instructions: the organization's instructions for NovaNova Settings, Instructions: the organization's instructions for Nova

How the alert token is kept safe​

  • Shown once. The token appears only when you create it; after that the panel never shows it again.
  • Rotate at once. Rotate token issues a new token and the old one stops working immediately. Remove turns alerts off.
  • Tied to an admin. A token stops working if the admin who created it leaves the organization or stops being an admin; create a new one.

If it's refused​

Settings › Incidents shows why the last result could not be posted to Slack. The investigation still runs, and its result is in the console.

"Nova is not in that channel. Invite it there, then try again."
Cause
Nova has not been invited to the Slack channel.
Fix
Invite Nova to the channel.
"Slack rejected the connection. Reconnect Slack."
Cause
Slack no longer accepts Nova's connection, for example after it was revoked.
Fix
Reconnect Slack under Skills.
"Slack is not connected for this organization."
Cause
A Slack channel is set, but Slack is not connected.
Fix
Connect Slack under Skills.
"Send the alert token as Authorization: Bearer <token>"
Cause
The request to the webhook URL had no alert token.
Fix
Send the token in the Authorization header, as in the receiver above.

Next​