Railway security incident: what to rotate, in what order, and how to check for misuse
On 21 October 2026, Railway asked all its users to roll every environment variable stored on the platform, including API keys and private registry credentials. If your Lovable, Bolt or Cursor app runs on Railway, here's how to do it without breaking anything, then check for misuse.
Railway's preliminary report says at most 1.85% of workspaces, about 100,000, had some variables potentially compromised. It still wants everyone who signed up before 21 October to rotate.
Last updated: Railway information checked up to its 7:43 am AEDT 24 October 2026 status update (20:43 UTC 23 October).
What Railway announced
Railway posted a security disclosure on 21 October (8:59 am AEDT, 22 October on its status page).
Railway then published a preliminary report at about 3:40 am AEDT on Saturday 24 October. Its investigation is still going, with outside incident response experts and the authorities involved.
The attacker got into some of Railway's internal systems and pulled customer environment variables in encrypted form. Using credentials taken from Railway's CI environment, they then decrypted a subset of them. Railway can't tell exactly which variables in an affected project were decrypted. Put simply, someone may have read some of the secrets your app keeps on Railway.
Railway believes the attack ran from 13:58 UTC on 20 October to 03:32 UTC on 22 October (12:58 am AEDT 21 October to 2:32 pm AEDT 22 October). The end time is when it confirmed the attacker no longer had access to its internal systems.
At most 1.85% of workspaces, about 100,000, had some variables potentially compromised. Railway still recommends that "all customers who signed up prior to October 21 rotate their keys stored on Railway."
The attacker didn't log in to user accounts. Railway hasn't ruled out access to source code and says it will confirm that in its final report.
Railway says the attacker was likely after AI and platform API keys, "for distilling model weights". It sees two higher-risk groups: workspaces with a very large number of environment variables, and environments holding a lot of AI keys.
Railway emailed customers with definite impact plus those it judged higher risk, a group bigger than the 1.85%. It finished sending those emails at 5:58 am AEDT on 24 October. Workspaces it didn't email "are not confirmed safe from compromise."
On 21 October, Railway also sent keys to providers including GitHub, GitLab, Anthropic, OpenAI, OpenRouter and Stripe, and asked them to revoke them. For some providers it sent every key on the platform. A revocation email from a provider "does not imply that their credentials were impacted", but a revoked key still has to be replaced.
The instruction is still to "immediately roll all environment variables, including any API keys, stored in Railway, as well as any private registry credentials, and review logs for those services for unusual activity." There's no deadline beyond "immediately". The report adds four steps for everyone: remove public network access to databases, rotate sensitive variables and keys, check your GitHub audit logs and app access, and check audit logs in registries such as Docker Hub.
Railway says keys you rotated after the attacker was locked out (2:32 pm AEDT 22 October) don't need rotating again. If you rotated a key before then, the attacker still had access to Railway's systems, so we'd rotate it again.
A full report will follow when the investigation ends. The incident is now at "Monitoring" on the status page, so check it before you start.
What Railway already did for you
Each of these still needs a small job from you.
- API and Project Tokens were invalidated. If you use the Railway CLI or Public API, including in CI, create new tokens.
- Storage bucket credentials were rotated. Audit logs show this as Railway Staff "Faraz Patankar", which is expected. Redeploy services that use a bucket and update local tools and CI.
- GitHub tokens Railway held were invalidated, and GitHub tokens saved as registry credentials were removed. Reconnect GitHub in Railway. Those registry tokens "still work on GitHub until you revoke them", so delete them in GitHub too.
- Users were signed out. Sign in again. Railway also reset its GitHub App installations, so if you see nothing after signing in and your GitHub email differs from your Railway email, use "Log in using email".
- Builds and deployments are back on all plans, though they may take longer than usual while Railway works through redeploys.
- Commits pushed during the pause weren't queued. Your service keeps running its last successful deploy. Reconnect GitHub if the build can't pull your repo, then use Deploy Latest Commit. For
railway up, runrailway logoutandrailway loginfirst. - New signups are still paused, and Railway's community site, station.railway.com, is still down. Email support@railway.com instead.
None of this rotates your own keys. Database passwords, Stripe keys, AI keys and anything else you put into Railway are still yours to change.
Before you start: list every secret
- Write down every variable holding a secret in each Railway service.
- Do it for every environment: production, staging and PR environments.
- Check Shared Variables in Project Settings.
- Note every other place each secret lives: your local
.env, CI, a Vercel or Netlify frontend, cron jobs elsewhere.
For each secret: create the new value, update the Railway variable, deploy, check the app works, then revoke the old value. During an incident you can revoke straight away if a short outage is fine. Railway's Postgres guide says: "If you suspect a leak, cutting off the old credential matters more than the short outage."
Rotate in this order
Start with the secrets that could do the most damage. Railway and each vendor have step-by-step guides (see Sources).
| Order | Secret | How to rotate it |
|---|---|---|
| 1 | Railway Postgres, MySQL, Redis, MongoDB | Database service > Database tab > Config > Connection > Regenerate. Then redeploy every service that uses it |
| 1 | External database (Supabase, Neon) | Reset the password with that provider and update the Railway variable. In Supabase: Database > Settings. Pooler connections may fail briefly; retry with the new password |
| 2 | Supabase secret key (sb_secret_) or legacy service_role | Settings > API Keys: create a new secret key, swap it in everywhere, then delete the old one (no undo). Deactivate legacy keys |
| 2 | Supabase legacy JWT secret | Move to publishable and secret keys first, because revoking it also stops the legacy anon and service_role keys. Then JWT Signing Keys: Migrate > Rotate > Revoke legacy |
| 3 | Stripe secret or restricted keys | API keys > ⋯ > Rotate key. "Now" deletes the old key at once; otherwise both work for up to 7 days. For a leak, Railway says choose "Now" |
| 3 | Stripe webhook secret (whsec_) | Workbench > Webhooks: roll the secret per endpoint, update the variable, redeploy |
| 4 | OpenAI | Delete the key on the API keys page and create a new one. Delete any exposed Admin keys. Then check the Usage page and set spend limits |
| 4 | Anthropic (Claude) | Claude Console API keys: Delete API key, then Create key. Delete any exposed Admin key |
| 4 | OpenRouter, Grok (xAI) | Follow Railway's guide for each |
| 5 | Resend, SendGrid | Create a new key, swap it in, delete the old one. SendGrid shows a key only once |
| 6 | Google OAuth client secret | Add a new secret, switch to it, then disable the old one |
| 6 | GitHub OAuth app secret | Settings > Developer settings > OAuth Apps > your app > Generate a new client secret. Update your app, then delete the old secret |
| 7 | Session and JWT secrets (JWT_SECRET, SESSION_SECRET, AUTH_SECRET, NEXTAUTH_SECRET) | Generate a new long random value (Auth.js: npx auth secret), update, redeploy |
| 8 | GitHub tokens, registry credentials (Docker Hub, GHCR) | Reconnect GitHub in Railway. Delete any personal access token stored in a variable and create a fine-grained one with minimal access |
| 9 | AWS, GCP | Follow Railway's guides |
| 9 | Everything else | Other webhook secrets (Clerk, Lemon Squeezy), Sentry, Twilio, Cloudinary, Upstash, Pinecone: rotate in each dashboard |
A few notes:
- Rotating session secrets signs everyone out, which stops stolen sessions working. For sessions in Redis, Railway says "Flush the session database or rotate the signing secret".
- Your Supabase publishable or anon key is public by design, so rotating it is optional. Row level security is what protects your data.
- A wallet private key can't be rotated. Move the funds to a new wallet.
- If a database has Public Access (TCP proxy) on, Railway says the old password works "from anywhere on the internet" until you rotate. Its report now asks everyone to remove public network access to databases. Turn it off unless something outside your project needs it, and update anything using a
_PUBLIC_URLvariable. - With Auth.js, Clerk or Supabase Auth, an OAuth secret may live in that provider's dashboard. Rotate it there.
Redeploy so the new values are used
Railway's guides say: "Containers read variables only at startup, so every service that uses the credential must be redeployed."
- Editing a variable creates staged changes. Review and deploy them.
- Regenerate only restarts the database. Redeploy your app services too.
- A reference variable like
${{ Postgres.DATABASE_URL }}updates on the next deploy. A pasted connection string doesn't, so switch it to a reference. - Rebuild your frontend.
NEXT_PUBLIC_andVITE_values are baked into your JavaScript at build time.
railway redeploy --service <name>Then check deploy logs. password authentication failed, Access denied or WRONGPASS means a service still has an old value. To test the database (or MySQL, Redis):
railway connect PostgresCheck whether anyone used the old keys
Rotating stops future misuse but won't show what already happened. Check each service you use.
| Where | What to look at |
|---|---|
| Railway | Audit Logs (variable views and changes) and deploy logs. Export what you need, as history length depends on your plan |
| Postgres | Active sessions (pg_stat_activity), unfamiliar roles (SELECT rolname FROM pg_roles;), new extensions, failed logins |
| MySQL | mysql.user, SHOW GRANTS, UPDATE_TIME in information_schema.tables |
| Redis | CLIENT LIST, DBSIZE, INFO keyspace, WRONGPASS in logs |
| Supabase | Logs Explorer (bulk reads or deletes), Auth users you don't recognise, Security Advisor |
| AI providers | Usage by key, cost spikes, models you don't use |
| Stripe | Each key's ⋯ > View request logs (works for deleted keys too), plus payouts, refunds, bank accounts, webhook endpoints and team members |
| Resend, SendGrid | Per-key logs, sending volume, unfamiliar recipients, domain changes. Watch for phishing sent from your domain |
| GitHub | Security log, authorised apps, SSH and deploy keys, webhooks, collaborators, and any commit Railway deployed that you don't recognise |
| Docker Hub and other registries | The registry's audit logs, as Railway's report recommends |
If you find something, export the logs before cleaning up and contact that vendor. For a database, restore from a backup taken before the exposure. Railway support is support@railway.com.
Make sure nothing is still exposed in your frontend or repo
Anything named NEXT_PUBLIC_ or VITE_ ships in the JavaScript visitors download. Only publishable keys belong there, like a Supabase sb_publishable_ or anon key, or a Stripe pk_ key. If a secret key ever had one of those prefixes, it has been public since that deploy. Rotate it and move that call to a server route.
To check by hand, open your live site, open developer tools, go to Sources or Network and search the JavaScript for sk_live, sk-, sk-ant-, sb_secret_, service_role, eyJ (the start of a JWT), whsec_, re_ and SG.. Short ones like re_ give some harmless matches.
Check whether a real .env file was ever committed. GitHub secret scanning, gitleaks or trufflehog can search your git history, which keeps deleted files, so rotate anything you find.
Keep row level security on for every Supabase table your publishable key can reach. Our Supabase permission denied guide has a query to check.
Mistakes to avoid
- Editing database password variables by hand. Changing
POSTGRES_PASSWORDor runningALTER USERputs the database and its variables out of sync. Use Regenerate, "the supported way to change a database password". - Revoking an old key before the new one is live by accident. Your app is down until the redeploy finishes.
- Swapping
ENCRYPTION_KEYwithout a plan. If it encrypts stored data, a straight swap makes that data unreadable. Get help first. - Forgetting staging, PR environments and your local
.env. The old key still works anywhere you didn't change it. - Assuming you're safe because you didn't get an email. Railway says workspaces it didn't email "are not confirmed safe from compromise", and it wants everyone who signed up before 21 October to rotate.
- Leaving a key you rotated early. If you changed it before 2:32 pm AEDT 22 October, rotate it again.
Checklist
- List every secret in every Railway service, environment and Shared Variable, plus copies elsewhere.
- Rotate in order: databases, Supabase, Stripe, AI keys, email, OAuth, session secrets, GitHub and registries, then the rest.
- Deploy staged changes, redeploy every affected service and rebuild your frontend.
- Create new Railway tokens, reconnect GitHub, deploy your latest commit and redeploy anything using a bucket.
- Check vendor logs for use of the old keys, and export anything suspicious.
- Search your live JavaScript and git history for secrets.
- Turn off public network access to your databases and check your registry audit logs.
- Keep an eye on Railway's status page and its final report.
Final step: scan your live app with LaunchVet
Once you've rotated and redeployed, run a LaunchVet scan on your live URL. It checks:
- Keys in client-side code. It reads the page you enter and the scripts it loads from your own domain, and looks for known secret key formats such as Stripe secret keys, OpenAI and Anthropic keys, GitHub tokens, AWS keys and a Supabase
service_rolekey. It also flags source maps anyone can download. - Security headers and HTTPS, such as HSTS and Content Security Policy.
What it can't do:
- It reads only that page and its scripts, and only spots formats it knows. It doesn't yet look for Supabase
sb_secret_keys or Stripe webhook secrets, so do the manual search too. - It doesn't test your Supabase tables. If it sees your anon key it reminds you about row level security, but it never connects to Supabase.
- It can't see your Railway variables or tell whether your keys were exposed or used.
- It doesn't replace rotating keys or checking vendor logs.
Sources
- Railway: Security disclosure, preliminary report, status page, post on X, all recommendation guides
- Railway rotation guides: PostgreSQL, MySQL, Redis, MongoDB, buckets and GitHub, recover failing deploys
- Railway usage checks: OpenAI, Anthropic, OpenRouter, Grok, Stripe, GitHub, AWS, GCP
- Railway docs: Variables, Database View, Audit Logs, Logs, Public API
- Supabase: API keys, JWT signing keys, signing keys announcement, migrating to new API keys, resetting your database password, logs, advisors
- Stripe: rotate an API key, compromised keys, restricted keys, webhooks
- AI providers: OpenAI API key safety, Anthropic API key best practices, Anthropic Console usage
- Email: Resend API keys, SendGrid API keys
- OAuth and GitHub: Google: rotating client secrets, GitHub OAuth app best practices, personal access tokens, security log, secret scanning
- Frameworks and tools: Auth.js deployment, Next.js environment variables, Vite env variables, MDN: browser developer tools, gitleaks, trufflehog