Skip to content

Franz Franz

goFranz // personal journal all writing
Security · security

From a Default Password to Root on Every Box

A recent engagement: how a default login led to root on four servers, what I fixed, and what monitors them now.

A small company asked me to look at their stack: a handful of production apps, each on its own server, serving real users and holding data for third parties. They’d grown fast, set things up as they went, and wanted to know how bad it was.

It was bad. Not because of some unusual mistake, but because the defaults were still on; That’s usually how it goes. Here’s how I got in, how far it went, what I changed, and what monitors the servers now.

The way in

The platform underneath these apps ships with a seeded account for first-boot setup: a fixed username and the password 123456. You’re meant to change it. On the live systems it still worked, on every tenant, including the one holding the most sensitive data. One POST to the login endpoint and I had a session.

That account wasn’t an admin; it had an ordinary user role. On most platforms that’s where it stops. Not here.

From a login to code execution

The product has a feature where an authenticated user can define an “API process”: a snippet of server-side JavaScript that runs when the endpoint is called. That’s the kind of feature that belongs behind an approval step. It wasn’t. The form that creates these was marked public, with an insert policy that let any user save a new one, without review.

The snippet runs inside a function that shadows the dangerous names: eval, require, process, fs are all blocked. It missed one: globalThis. So globalThis.process.mainModule.require("child_process") goes around the block-list to the real Node process, and from there it’s a shell.

Here’s what this looks like:

const R = globalThis.process.mainModule.require;
return { out: R("child_process").execSync(params.cmd, { encoding: "utf8" }) };

The container ran as root. So I had uid=0 inside the app container, and printenv gave me the whole environment: the JWT signing secret (short and patterned), the database root passwords, the mail credentials, OAuth secrets, a license-signing key. Every secret the app needed to run was readable by any user who’d logged in with 123456.

With the JWT secret I could forge an admin token and sign in as the real super-admin. The environment turned out to be more useful though, because of where two of those secrets led.

From a container to the host

The apps were deployed through a self-hosted panel, the kind that builds your containers and manages the host for you. That panel runs a PHP process on the internal container network. On these servers it answered on a port reachable from every tenant container on the same flat network, and it executes a request body as a script. That got me a second shell, this time inside the panel’s own container.

That container has everything needed to take over the host. It mounts the host’s Docker socket, and it stores the SSH private key the panel uses to manage its own host, in a readable file. Copy the key, and:

ssh -i stolen_key root@host    # uid=0(root)

Root on the host. Then I did the same on the next server, and the next: four servers, same pattern, root on all of them. One was a snapshot clone of another and had inherited the same SSH key, so it didn’t even need its own break-in.

None of this used a CVE. The panel was on its current version. The exposure was the internal listener, plus a world-readable key, plus one flat network - all defaults, none of them alarming when you look at it alone.

The mailbox

The app sent mail through a Gmail account, and the app password for it was in the container environment. A Gmail app password works over IMAP too. So the same string that let the app send a receipt let me open the company’s mailbox and read 22,000 messages.

This is where a technical finding becomes a business one. The mailbox received everything: mail from the hosting accounts, the domain registrar, the cloud consoles, and password resets for half the SaaS the company used. It also held account-creation emails with users’ passwords in plaintext (the app sent those), including staff on the sensitive system. Whoever holds the mailbox can reset the password on every one of those accounts.

I didn’t go looking for the owner’s bank details or crypto balances, but they were in there. It took a few hours to get from a default password to someone’s personal finances, and none of the steps were clever.

The database everyone could reach

Separately from the break-in, there was a quieter problem. The databases - a MongoDB with millions of records, a Postgres next to it - were published directly on the servers’ public addresses. The cloud provider had been sending abuse notices about the exposed MongoDB for over a year. Nobody read them.

The developers connected as the root database user, over plain TCP, from wherever they happened to be: laptops, scripts, database GUIs. No TLS, so anyone on the path could read the credentials and the data. One shared superuser account for the whole team, reachable from the entire internet, unencrypted. It worked, so nobody had a reason to change it.

Reporting

I wrote it up the way I’d want to receive it: a short overview that says how bad and how far, then a detailed chain anyone technical can follow and reproduce, then a flat list of what to rotate and fix, ordered by how much it matters. Working credentials and the copied keys went into a separate confidential section, marked to rotate on handover. The evidence (database exports, proof-of-access transcripts) stayed on my disk, encrypted, and I told them exactly what I’d touched, so nothing I did would be mistaken for an attacker later.

The report has to be usable by someone who has a day job, on the first read.

Fixing the running services

Then the larger part of the work: closing it. Not a rewrite; I hardened the running systems in place, and the apps kept serving the whole time.

  • The default account is gone. Removed from every tenant, with strong admin passwords set.
  • The RCE is closed off at the host. The firewall now drops traffic to the panel’s internal PHP listener between containers, so a tenant container can’t reach it even on the shared bridge. (This needed a kernel module that wasn’t loaded; without it, same-bridge traffic skips the firewall’s forward chain entirely.)
  • The databases are off the public internet. The published ports are firewalled. Where outside access was really needed, it goes through the reverse proxy with TLS only (a Let’s Encrypt certificate, plain connections refused), and the port stays closed to everyone not on the allow-list.
  • No more shared root on the database. Each developer has their own login, scoped to read-write on the one database they work in. The app connects as its own least-privilege user. The root password the whole office knew is rotated and out of daily use.
  • Everything the break-in touched is rotated. Signing secrets, every database password, mail, OAuth, the license key, the panel’s own encryption key and the host SSH keys. All of it was readable by anyone who’d logged in, so I treated all of it as compromised.
  • SSH is locked down. Key-only, root password locked, and the cloud image’s habit of re-enabling password login at boot disabled. The keys I’d used in the test are retired; a couple of unexplained keys on one host went with them.
  • Patched and rebooted onto a current kernel, one host at a time, with containers coming back up on their own.

Guidelines for the ongoing part

Fixing the live servers is the easier half. The harder half is leaving them in a state where they don’t drift back. So the handover wasn’t only “here’s what I changed”, but also how to keep using the stack without reopening the holes:

  • Secrets belong in a store with least-privilege access, not in container environment variables any logged-in user can print.
  • The server-side JavaScript feature needs its create step behind an admin, and a real sandbox instead of a block-list.
  • Developers get named, scoped database logins over TLS. Not the shared superuser, not plaintext, not from the whole internet.
  • Stop mailing people their passwords. Send a set-a-password link; the current template already does.
  • Turn on 2FA on the panel and the cloud accounts, and put the admin panels behind an access layer instead of the open web.

The firewall, the SSH config and the host settings are now in version control as Ansible, with a read-only check that has to report zero changes. That check proves the repo still matches the live hosts; drift shows up as a diff.

What monitors the servers now

A pen test only shows the state on the days I tested. What the client really needs is to notice the next problem without me. So the last piece was monitoring, set up the same way on all hosts:

  • Logs ship off the host (host audit, the panel’s events, the proxy) to append-only object storage, so someone who gets in can’t edit the record.
  • Alerts in two tiers. Mail for the dull but important: a failed update, a disk filling up, integrity drift. Mail and a phone push for anything that means someone’s inside: an SSH login, an interactive shell opening inside a container, the panel issuing a new API token or key, a burst of failed logins, a runtime-security hit.
  • An hourly integrity check hashes what an intruder would change (authorized keys, sshd config, sudoers, cron, the listening ports, setuid binaries) against an accepted baseline, and pages on any change that wasn’t me.
  • Falco watches for container escapes, fail2ban covers SSH, and daily encrypted backups go to locked, versioned storage.

Updates now run as a deliberate, logged step when an advisory comes in, with a worklog of what changed and when.

To summarize

Nothing in this engagement was advanced. A default password, a block-list with a gap, a readable key, databases on the open internet, secrets in plaintext, and a mailbox that also worked as a password vault. Each one is a five-minute fix. Chained, they went from a public login form to root on every server and the owner’s financial mail in an afternoon.

The individual mistakes are small and ordinary, which is why they were still there. None of what I set up makes the stack unbreakable; it means the next mistake of this kind shows up as an alert or a diff, and not a year later in an abuse notice.