Skip to main content

Troubleshoot

When an application is not Running, AstroPulse says why on its page: a status reason, its events and its logs. Read them in that order, then find the message below for its cause and fix. Ask Nova on the application's page reads the same signals and says the likely cause.

Before you start​

  • The application's name, and access to its organization.
  • A model has its own list of refusals: Operate a model.

In the console​

  1. Open the application from Workloads > Applications. When it is not running, the Overview shows the status reason, with See events and See logs.

  2. Select Events. Deployment Events are AstroPulse's view of the deploy; Kubernetes Events are the cluster's (scheduling, image pulls, probes); Platform Events are the platform's own. Select Connect to stream new events. A Warning row usually names the cause.

    Open zoom view

    The Events tab: Deployment Events with FailedScheduling warnings, then Application is starting and Application is ready

    The Events tab: Deployment Events with FailedScheduling warnings, then Application is starting and Application is readyThe Events tab: Deployment Events with FailedScheduling warnings, then Application is starting and Application is ready
  3. Select Logs, then Start streaming, and pick a pod and a container. For a container that keeps restarting, open Stream Settings and set Previous: ON to read the run that crashed.

    Open zoom view

    The Logs tab ready to stream, with Start streaming

    The Logs tab ready to stream, with Start streamingThe Logs tab ready to stream, with Start streaming
  4. Find the message below. When the specification is right but the cluster drifted from it, select Resync in the header: it reapplies only the resources that drifted, and the application keeps serving.

You should see​

The application's page shows Running. While a new version deploys, the Overview says which version is serving, for example "Deploying v5; v4 keeps serving until it's ready."

Open zoom view

An application's Overview reading Running, with its current version and recent deploys

An application's Overview reading Running, with its current version and recent deploysAn application's Overview reading Running, with its current version and recent deploys

The image does not start​

"The image <image> could not be pulled: the registry does not have it. Check the image name and tag. Its events say why."
Cause
The repository or tag does not exist, or is misspelled.
Fix
Fix source.image.repository or tag. On Source, Validate checks that the image and tag exist.
"The image <image> could not be pulled: the registry refused access. Check the registry credentials (the image pull secret)."
Cause
The image is in a private registry the cluster's nodes cannot read.
Fix
Give the cluster read access, keyless or with a pull Secret: Registry access. If the first pull already failed: If the first pull already failed.
"The image <image> could not be pulled: the registry could not be reached. Check that the cluster can reach the registry."
Cause
The cluster has no network path to the registry.
Fix
Check the cluster's outbound access to the registry host.
"The image name <image> is not valid. Check the image name and tag. Its events say why."
Cause
The repository or tag has characters an image reference cannot have.
Fix
Fix the repository and tag, for example astropulse/latency and v1.0.0.
"The image <image> runs as root, which the cluster does not allow; use an image that runs as a numeric non-root user."
Cause
Containers must run as a non-root user.
Fix
Use a non-root image, for example nginxinc/nginx-unprivileged for nginx. See What the container runs with.
"The container's configuration refers to something the cluster does not have, such as a Secret or ConfigMap…"
Cause
The application names a Secret, ConfigMap or key that is not in its namespace.
Fix
Create each Secret and ConfigMap the application names in its namespace, then Resync.

The container keeps restarting​

"The application keeps exiting with code <n> and restarting; its logs say why."
Cause
The process exits on start: a missing setting, a failed connection, or a crash.
Fix
Read Logs with Previous: ON, or astroctl app logs <app>, fix the cause, and deploy again.
"The application keeps being stopped by signal <n> and restarting; the application's events say why."
Cause
Something outside the process stops it: a failed probe, running out of memory, or an eviction.
Fix
Read Kubernetes Events. For an image application, a larger availability tier gives it more memory.

Pods wait as Pending​

A FailedScheduling warning in Kubernetes Events, such as "0/6 nodes are available: 1 Insufficient cpu, 1 Insufficient memory…"
Cause
No node has room for the pods' requests.
Fix
Add nodes or a node group, or turn on node autoscaling. An image application's requests come from its availability tier; a chart's or manifest's from its own values.
"The previous copy of this application's namespace is still being deleted. It deploys when that finishes."
Cause
The application was deleted and deployed again before its old namespace was gone.
Fix
Wait. It deploys on its own.

The deploy or sync fails​

"The sync failed before anything was applied: …"
Cause
The cluster refused the rendered resources. The rest of the message is the cluster's reason.
Fix
Fix the chart, manifest or setting it names, and apply again.
"Application <app> already has a deployment in progress. Use --force to terminate it and start a new sync."
Cause
A resync was asked for while a sync is still running.
Fix
Wait for it, or run astroctl app resync <app> --force for a sync that is itself stuck. It stops the sync in progress and reapplies every resource, never deleting and recreating.
"That update was not applied, so <version> keeps serving: …"
Cause
The last update failed; the earlier version is still running.
Fix
Read the reason after the colon, fix it, and apply again, or roll back.

An update is refused​

"Cannot update application '<app>': Source type '<type>' cannot be changed (source type is immutable once set)"
Cause
The update changes source.type.
Fix
Keep the source type, or delete the application and deploy it again with the new one.
"Cannot update application '<app>': Application configuration cannot be changed (some fields are immutable once set)"
Cause
The update changes a field fixed at the first deploy, such as externalAccess.accessType or the namespace.
Fix
Keep those fields as they are, or delete the application and deploy it again.
"This version cannot be made rollbackable because a value may contain a secret; use secret references and redeploy."
Cause
The specification contains what may be a secret value.
Fix
Use a secret reference instead of the value. See Releases and rollback.

It runs but does not answer​

The application shows Running, but its endpoint does not answer
Cause
For an image application, traffic goes to Container Port (8080 by default). A container listening on another port answers nothing.
Fix
Set Container Port to the port the image listens on. For a domain, see Custom domain.

When a recent deploy caused it​

If the application was healthy before the last deploy, roll back to the previous good version while you investigate: its History tab, Roll back. See Releases and rollback.

Other ways in​

  • Nova: ask it, for example "why is hello-world unhealthy?" or "show me the errors in hello-world's logs". It can also run a deeper cluster diagnosis when the cause is the cluster.
  • SDK and API: the same operations, in the API reference.

With astroctl​

Terminal
astroctl app status hello-world # status (add -w to keep watching)
astroctl app get hello-world -o yaml # full detail, with servingNote
astroctl app events hello-world # deployment events
astroctl app events hello-world --k8s # Kubernetes events (add -w to stream)
astroctl app logs hello-world # logs across all pods, follows by default
astroctl app resync hello-world # reapply only what drifted

servingNote says which version is serving and why, when it is not the one you last applied, and is empty when they match. A watch ends with "The watch stopped: the connection to AstroPulse closed." when the connection closes; nothing failed, run the command again. Every flag: astroctl app.

Next​