Troubleshooting
The commands to run first, what a node says when it will not start, a Migration that failed, and what the System page reports
This page helps you find the cause when the site is not ready.
Start here
- Run
awhere statusto see which container is not ready. - Read the log of that container, such as
awhere logs core-services -f. - Run
awhere preflightto see all the problems at once.
To restart one container, run awhere stop <component> and then awhere start <component>.
A node will not start
If a node sees it cannot run safely, it writes one block to its log and stops with exit code 1.
REFUSED TO START: core-services (release 5.0.0)
why: <what is wrong>
what to do: <what to do>why says | What to do |
|---|---|
N preflight checks failed: followed by one failed check per line | Fix the env file by the fix: line, then run awhere start <component>. For what each check means, see All Preflight checks. |
database `<db>` has Migration `<id>`, applied by release R ... | A newer Release has already changed the data. Run Release R or newer. To go back to an older Release you must restore from a backup (Roll back). |
database `<db>` has data but has not applied Migration `<id>`, which release S no longer carries ... | The site skipped an upgrade stop. Upgrade first to the Release the message names. |
Migration `<id>` failed in database `<db>`: <message> | See A Migration failed. |
database `<db>` is not on the MongoDB Provider ... | The connection string must start with mongodb:// or mongodb+srv://. |
port <N> is already in use | Only core-services and authorization-server can show this. Stop the program that uses that port, or set PORT to another port. |
the session database did not answer after 30 attempts | Only authorization-server can show this. Check SESSION_STORE_URL. |
Is the node ready
While a Migration runs, the node is up but not yet ready to serve. Wait a moment and run awhere status again.
If a node stays in the state waiting, the node itself is fine. There are two possible causes.
- Another replica of the same service is running that Migration. If that replica has stopped, the lease it holds on the Migration expires by itself within 15 seconds.
- The Migration needs another service that has not started. Start that service.
An app counts as ready when it answers HTTP on port 5000.
A Migration failed
The System page shows that row as Failed, with the error (System). In the log you see [migrations] <id>: failed: <message>.
- A Blocking Migration makes the node refuse to start. Fix the cause named in the message. Common causes are a database that does not answer, or a user without the privilege. Then run
awhere start <component>. The Migration continues from where it stopped. - A Background Migration retries by itself, waiting twice as long each time, from 5 seconds up to 5 minutes. The node keeps serving as usual in the meantime.
You can always run a Migration again. It never damages data.
What the System page says
The System page uses the word "Migration" too.
| The page says | What to do |
|---|---|
"Migration <id> failed in <database>." | See A Migration failed. |
"<node> refused to start." | Read the log with awhere logs <node>. |
| "Nodes are on different Releases" | Normal during an upgrade. If you are not upgrading, run awhere upgrade <version> again with the same Release. |
"Migration <id> is paused ..." | Start a node of that service. |
"<node> does not answer." | Check the container of that node. |
"Migration <id> skipped <count> entries." | Open the row and read each one. |
"Migration <id> is running ..." | Do nothing. It is only progress. |
Log lines you may see
| Log line | What it means |
|---|---|
JetStream is not available (...): activity is not recorded; retrying every 5s | NATS has no JetStream on, or cannot be reached. When JetStream answers again, the node starts recording Activities by itself. See Requirements. |
apps: PLATFORM_URL is unset, so apps get relative urls | Set PLATFORM_URL, then correct the address of each app on the Apps page. |
apps: <id> is not installed: ... | An OAuth client that a user made uses the same id as that app. Have its owner make a new client of their own and delete the old one permanently. |
audit retention waits: the copy of the legacy logs is not done | Normal until the copy of the v4 logs is done. |
An app that answers 500 from its first page is almost always caused by a wrong PLATFORM_URL. This value must be the address of the gateway that core-services serves.