Forge Recovery Guide
Step-by-step instructions to fix anything. Assumes zero technical knowledge. Follow the numbered steps exactly.
Jump to Section
Where to Start (Read This First) Step 1: Get Into the Server Step 2: Figure Out What is Wrong Fix: A Single Service is Down Fix: Bad Code Was Deployed Fix: Everything is Broken (Nuclear Option) Fix: Database (Supabase) Issues Fix: Dashboard is Down DEFCON System Important Links and FilesWhere to Start (Read This First)
!
Do not panic. Almost everything is recoverable.
Forge has automatic backups, safe mode, and rollback scripts. The worst case is "stop everything, roll back, restart." That is a 5-minute fix. Just follow the steps below in order.
What computer do you need?
Any computer with internet access. Windows, Mac, or Linux. You will be typing commands into a terminal (a window where you type text commands and press Enter to run them).
What is "the server" or "VPS"?
Forge runs on a rented computer in the cloud (a Virtual Private Server at Hetzner). Its address is 178.156.253.142. You connect to it over the internet using SSH (a secure remote connection tool that is already built into Windows and Mac).
Step 1: Get Into the Server
Option A: From a Windows Computer
1
Open PowerShell
a. Look at the bottom-left corner of your screen. Click the Windows Start button (the Windows logo).
b. A search box will appear. Type the word: PowerShell
c. You will see "Windows PowerShell" appear in the results. Click on it.
d. A dark blue window will open with white text. This is your terminal. You will type commands here.
b. A search box will appear. Type the word: PowerShell
c. You will see "Windows PowerShell" appear in the results. Click on it.
d. A dark blue window will open with white text. This is your terminal. You will type commands here.
2
Connect to the Forge server
a. Click inside the dark blue PowerShell window so it is active.
b. Click the green box below to copy the command (or highlight it and press Ctrl+C):
b. Click the green box below to copy the command (or highlight it and press Ctrl+C):
ssh root@178.156.253.142
c. Right-click inside PowerShell to paste the command. (In PowerShell, right-click = paste.)
d. Press Enter on your keyboard.
d. Press Enter on your keyboard.
If this is your first time connecting, it will ask: "Are you sure you want to continue connecting (yes/no)?" Type yes and press Enter.
If it asks for a password, type your VPS root password and press Enter. Important: You will NOT see any characters appear as you type the password. This is normal security behavior. Just type it and press Enter.
Success! When you see something like
root@ubuntu-16gb-ash-1:~# you are connected to the server. Everything you type now runs on the Forge server, not your local computer.Option B: From a Mac
1
Open Terminal
a. Press Cmd + Space at the same time (the Command key and the spacebar). A search box called Spotlight will appear.
b. Type the word: Terminal
c. Press Enter. A window will open. This is your terminal.
b. Type the word: Terminal
c. Press Enter. A window will open. This is your terminal.
2
Connect to the Forge server
Click the green box below to copy the command, then paste it into Terminal (Cmd+V) and press Enter:
ssh root@178.156.253.142
Same as Windows: it may ask about connecting (type yes) and may ask for your password (type it, you will not see characters).
When you see
root@ubuntu-16gb-ash-1:~# you are in.Option C: SSH Does Not Work / Forgot Password
1
Use the Hetzner browser console instead
a. Open your web browser (Chrome, Edge, Safari, Firefox).
b. Go to this address: https://console.hetzner.cloud
c. Log in with your Hetzner email and password.
d. You will see a list of servers. Click on: ubuntu-16gb-ash-1
e. At the top of the page, click the tab that says Console.
f. A terminal will appear right in your browser. Type root and press Enter to log in.
g. You are now connected. No SSH needed.
b. Go to this address: https://console.hetzner.cloud
c. Log in with your Hetzner email and password.
d. You will see a list of servers. Click on: ubuntu-16gb-ash-1
e. At the top of the page, click the tab that says Console.
f. A terminal will appear right in your browser. Type root and press Enter to log in.
g. You are now connected. No SSH needed.
Option D: Server is Completely Frozen
1
Hard reboot from Hetzner
a. Go to https://console.hetzner.cloud and log in.
b. Click your server: ubuntu-16gb-ash-1
c. Click the Power tab.
d. Click Reset. This is like pulling the power cord and plugging it back in.
e. Wait 60 seconds (count to 60 slowly).
f. Now try connecting again using Option A, B, or C above.
b. Click your server: ubuntu-16gb-ash-1
c. Click the Power tab.
d. Click Reset. This is like pulling the power cord and plugging it back in.
e. Wait 60 seconds (count to 60 slowly).
f. Now try connecting again using Option A, B, or C above.
Once Connected: Start tmux
tmux keeps your work safe even if your internet disconnects. Run this command:
tmux new -s recovery
The screen may flash. You will see a green bar at the bottom of the screen. That means tmux is running.
If it says "duplicate session: recovery" that means tmux is already running. Instead type:
tmux attach -t recovery and press Enter.Step 2: Figure Out What is Wrong
Now you are connected to the server. Let us check what is happening.
1
Run the health check
This checks EVERYTHING at once. Copy this command (click the green box), paste it into your terminal, and press Enter:
bash /opt/forge/scripts/recovery/health-check.sh
A report will print. Green items are OK. Red items are broken. Read through the report to see what is wrong.
2
Check which services are running
Copy and run this command:
systemctl list-units --type=service --state=running | grep forge
You will see a list of running Forge services. If one you expect is missing from the list, it is down.
3
Check recent error logs
This shows what happened recently with the Telegram bot:
journalctl -u forge-commander --since "1 hour ago" --no-pager | tail -30
Look for words like ERROR, FAIL, crash, or "exit code". These tell you what went wrong.
4
Check DEFCON level
DEFCON controls what automation is allowed. If it is locked down, things will not work:
bash /opt/forge/scripts/sentinel/defcon.sh status
Shows the current level (1-5). Level 5 = normal. Level 2 or 1 = locked down (see DEFCON section below to unlock).
5
Check disk space
df -h /opt/forge
Look at the "Use%" column. If it is above 90%, you need to free disk space.
Now go to the matching fix section below based on what you found.
Fix: A Single Service is Down
If only ONE service is broken, restart it. Here is how:
1
Find the service name in the table below, then run its restart command
| Service | What It Does | Restart Command (copy and paste) |
|---|---|---|
| forge-commander | Telegram bot | sudo systemctl restart forge-commander |
| forge-dashboard-observe | Dashboard API | sudo systemctl restart forge-dashboard-observe |
| forge-privileged-exec | Root command runner | sudo systemctl restart forge-privileged-exec |
| forge-knowledge | Knowledge graph | sudo systemctl restart forge-knowledge |
| forge-comms | Email service | sudo systemctl restart forge-comms |
| forge-gatekeeper | Task quality scoring | sudo systemctl restart forge-gatekeeper |
For example, if the Telegram bot is down, copy and paste this:
sudo systemctl restart forge-commander
2
Check if it started successfully
systemctl is-active forge-commander
If it says active it is working. You are done!
If it says inactive or failed, check the logs to see why:
journalctl -u forge-commander --since "2 minutes ago" --no-pager
Fix: Bad Code Was Deployed
If Ralph or Claude pushed broken code, roll it back to the previous version.
1
Turn on safe mode (stops all automation so nothing gets worse)
sudo bash /opt/forge/scripts/safe-mode.sh
You should see messages like "Stopping forge-..." for each service.
2
See what changed recently
cd /opt/forge && git log --oneline -10
A list of recent changes. The newest is at the top. The bad one is probably near the top.
3
See what files were changed in the most recent commit
cd /opt/forge && git diff HEAD~1 --stat
4
Roll back to the previous version
sudo bash /opt/forge/scripts/recovery/rollback.sh
It will show what it is rolling back to and ask for confirmation.
5
Bring everything back online
sudo bash /opt/forge/scripts/recovery/resume-all.sh
6
Verify everything is working
bash /opt/forge/scripts/recovery/health-check.sh
Everything should be green now.
Fix: Everything is Broken (Nuclear Option)
LAST RESORT ONLY. This resets Forge to a clean state. It preserves your API keys, configs, and backups. Only use when nothing else works.
1
Run the nuclear restore script
sudo bash /opt/forge/scripts/recovery/nuke-and-restore.sh
2
Complete the 3-step verification (only Jason can do this)
Step 1: The script asks you to type NUKE FORGE (all caps, with a space in the middle). Type it exactly and press Enter.
Step 2: Open the Google Authenticator app on your phone. Find the entry labeled "Forge DEFCON". Type the 6-digit code you see and press Enter.
If you have not set up the authenticator yet: it will ask for your VPS root password instead. Type it and press Enter.
Step 3: Type DO IT (all caps, with a space) and press Enter.
No agent, script, or AI can bypass these steps. Only you.
Step 2: Open the Google Authenticator app on your phone. Find the entry labeled "Forge DEFCON". Type the 6-digit code you see and press Enter.
If you have not set up the authenticator yet: it will ask for your VPS root password instead. Type it and press Enter.
Step 3: Type DO IT (all caps, with a space) and press Enter.
No agent, script, or AI can bypass these steps. Only you.
3
Wait for the restore (about 2-3 minutes)
It will stop everything, back up configs, reset the code, restore your API keys, reinstall services, and restart everything.
4
Run the health check to confirm
bash /opt/forge/scripts/recovery/health-check.sh
Fix: Database (Supabase) Issues
Supabase is cloud-hosted and almost never the problem. But if data looks wrong:
1
Check if Supabase is reachable from the server
bash /opt/forge/scripts/recovery/health-check.sh | grep -i supa
If it shows green/OK, Supabase is fine. The problem is elsewhere.
2
If you need to restore data
a. Open your browser and go to: https://supabase.com/dashboard
b. Log in with your Supabase account.
c. Click on your project.
d. In the left sidebar, click Settings, then Database, then Backups.
e. You will see automatic daily backups. Pick one from before the problem started and click Restore.
b. Log in with your Supabase account.
c. Click on your project.
d. In the left sidebar, click Settings, then Database, then Backups.
e. You will see automatic daily backups. Pick one from before the problem started and click Restore.
Fix: Dashboard is Down
The dashboard is hosted on Vercel (a cloud service). It usually fixes itself.
1
Check if it is a Vercel outage (not our fault)
Open your browser and go to: https://www.vercel-status.com
If Vercel is having problems, just wait. There is nothing to fix on our side.
If Vercel is having problems, just wait. There is nothing to fix on our side.
2
Force a fresh deploy (if Vercel status is fine)
Run this command on the server to trigger a new deployment:
curl -X POST "https://api.vercel.com/v1/integrations/deploy/prj_hhxYKrguyqv70HSVTdb7cA1uvPHb/9oxmvP6EeY"
You should see a JSON response. Wait 1-2 minutes for the new version to build.
DEFCON System
DEFCON controls how much automation is allowed to run.
| Level | Color | What Happens |
|---|---|---|
| 5 | GREEN | Normal. Everything runs. |
| 4 | BLUE | Elevated. Commands need approval. |
| 3 | YELLOW | High threat. Extra approvals required. |
| 2 | ORANGE | ALL automation paused. LOCKED. |
| 1 | RED | Active breach. Everything stopped. LOCKED. |
Check current DEFCON level
bash /opt/forge/scripts/sentinel/defcon.sh status
Unlock from DEFCON 2 or 1 (two commands, run them in order)
1
Create the unlock file
echo unlocked > /tmp/forge-defcon-unlock
2
Run the unlock command
bash /opt/forge/scripts/sentinel/defcon.sh unlock
Reset DEFCON to normal (level 5)
bash /opt/forge/scripts/sentinel/defcon.sh set 5 "Manual reset by Jason"
Unpause Ralph
Ralph (the build agent) might be paused separately:
rm -f /tmp/ralph-pause
Important Links and Files
Web Links (open in your browser)
| What | Address |
|---|---|
| Forge Dashboard | forge-dashboard-kbce.vercel.app |
| Hetzner (server control panel) | console.hetzner.cloud |
| Supabase (database) | supabase.com/dashboard |
| Vercel (dashboard hosting) | vercel.com/dashboard |
| Vercel Status (is Vercel down?) | vercel-status.com |
Server Connection
ssh root@178.156.253.142
Important Files on the Server
| File | What It Is For |
|---|---|
| /opt/forge/.env.system | All API keys and secrets (DO NOT share with anyone) |
| /opt/forge/.env.llm | LLM API keys (DO NOT share) |
| /opt/forge/config/defcon.json | Current DEFCON level |
| /opt/forge/.claude/memory.md | Agent memory |
| /opt/forge/context/SESSION-QUEUE.md | Work queue between sessions |
All Recovery Commands (quick reference)
| What To Do | Command |
|---|---|
| Stop all automation | sudo bash /opt/forge/scripts/safe-mode.sh |
| Stop EVERYTHING | sudo bash /opt/forge/scripts/safe-mode.sh --full |
| Resume from safe mode | sudo bash /opt/forge/scripts/recovery/resume-all.sh |
| Health check | bash /opt/forge/scripts/recovery/health-check.sh |
| Rollback last deploy | sudo bash /opt/forge/scripts/recovery/rollback.sh |
| Nuclear restore | sudo bash /opt/forge/scripts/recovery/nuke-and-restore.sh |
| Check DEFCON | bash /opt/forge/scripts/sentinel/defcon.sh status |
Last updated: March 2026 | Forge Recovery Playbook v2.0
For the authenticated version, use the Dashboard Recovery Page
For the authenticated version, use the Dashboard Recovery Page