Self-hosted IAM: Keycloak, realm-as-code

The fastest way to configure Keycloak is to click through the admin console. It's also the fastest way to end up with an identity provider nobody can rebuild, whose current state exists only in a Postgres table and in the memory of whoever last touched it.

That's an uncomfortable place for the component that decides who gets into everything else.

The last two posts gave the platform somewhere to put secrets and somewhere to put data. This one adds the third thing every real workload needs: knowing who the user is. And it adds it under the same rule as everything else — if it isn't in git, it doesn't exist.

What you'll get from this post

A self-hosted OIDC provider serving single sign-on to every app on the platform, with the realm — clients, groups, password policy, MFA settings — declared as YAML and reconciled on every change. Plus the decision against the lighter, EU-origin alternative, which lost for an architectural reason rather than a feature-list one.

Why not the lighter option

The steelman for Zitadel is strong, and on paper it should have won. Apache-2.0. EU-origin, based in Switzerland — which matters when the whole pitch is jurisdictional sovereignty. A single Go binary running in a fraction of Keycloak's memory, against a JVM that wants the better part of a gigabyte. Modern, API-first design.

It lost on the seam.

Zitadel's configuration-as-code story runs through its Terraform provider. This project keeps a hard line between platform-infra (the OpenTofu that provisions the box and the cluster) and platform-gitops (the workloads running on it). Driving realm state from Terraform would mean the identity provider's runtime configuration — clients, groups, policies — lives in the infrastructure repo, reaching across that seam to configure an application.

That's not a style preference. The seam is what lets the substrate be replaced without touching workloads, and every crossing makes the swap harder. A tool that can only be configured from the wrong side of a boundary is, at this tier, the wrong tool — no matter how much less RAM it uses.

Keycloak's realm goes in the gitops repo as YAML, reconciled by keycloak-config-cli. Config changes are commits, on the correct side of the line. It also happens to be the reference IdP that every consumer on the roadmap — Argo CD, Nextcloud, Vaultwarden — documents as its first-class OIDC provider, so integration risk is lowest exactly where the integrating happens.

The honest cost, stated plainly: about 900 Mi of an 8 GB box now belongs to identity (768 Mi request for the server, 128 Mi for the operator). On a single node that is a real number, and it moved the resize trigger a whole ticket earlier. The return trigger is written into the decision record: revisit Zitadel if Keycloak's memory becomes the reason for a node resize the budget otherwise wouldn't need.

The realm is a file

realm: platform
enabled: true

registrationAllowed: false      # accounts are created by an admin, never self-service
resetPasswordAllowed: true
bruteForceProtected: true
passwordPolicy: "length(12) and notUsername(undefined) and passwordHistory(3)"

157 lines covering the login and password policy, MFA, the groups that every app maps its RBAC against, the SMTP settings for password resets, and one block per OIDC client. The database it all persists into is the shared CloudNativePG cluster from the last post — database keycloak, owner role keycloak, no superuser, backed up on the same schedule as everything else.

One deliberate omission is worth more than anything in the file: there are no users in it.

Users are runtime data, not configuration. They get created through the console or the admin API and they never appear in YAML — because a realm re-import must never delete or overwrite people. The line between "config that gets reconciled" and "data that gets backed up" is one you draw once, and drawing it wrong here means an import job quietly deleting your colleagues.

Adding a client is four steps, and the interesting one is that the app and the IdP read the same secret:

1. Seed the client secret into OpenBao at platform/<app>/oidc.
2. Add an entry to the import job's ExternalSecret.
3. Add a client block to the realm YAML, referencing that variable.
4. Merge.

Because the realm import and the consuming application both read platform/<app>/oidc, the two can never disagree about the secret. There's no copy-paste step where a value gets pasted into the console and then rotated on only one side.

Drift is invisible, so re-converging has to be cheap

Here's the thing that makes realm-as-code different from ordinary GitOps: a realm is not a Kubernetes resource. Argo CD reconciles the import Job. It cannot see inside Keycloak. If somebody clicks something in the admin console, nothing goes OutOfSync — the cluster is exactly as declared, and the realm has quietly diverged anyway.

The mitigation is that re-running the import is one command:

$ argocd app sync keycloak-realm

The Job carries Replace=true,Force=true, so a sync recreates and re-runs it even when nothing is out of sync. And a real change to the realm file changes its hash, which triggers the same replacement automatically. The escalation, if console drift ever becomes a habit rather than an accident, is a CronJob around the identical image — deliberately not shipped, because a scheduled job that silently reverts a colleague's emergency fix has its own failure mode.

What this really buys you

Realm-as-code isn't about avoiding the console. It's about the answer to "what is our identity configuration right now" being a file you can read, diff and review — rather than a screenshot somebody took in March.

The console stays useful for looking at things, and for the one category of state that genuinely belongs at runtime: the people.

Gotchas

Every one of these cost real time.

• Variable substitution runs on the raw file, comments included. The import performs $(env:…) substitution before parsing the YAML, so a literal dollar-paren example written inside a comment — documentation for the next person — gets substituted, fails to resolve, and crash-loops the import Job. Learned live. Don't write the syntax in a comment unless it resolves.
• Rotating the admin password is a two-place operation. The bootstrap credential is only read at Keycloak's first start, but the import Job reads it on every run. Change it in the console and not in OpenBao and the server keeps working while every subsequent realm import fails authentication — a failure that shows up nowhere near the change.
• Token requests need --data-urlencode. Base64-encoded passwords contain +, which is a space in form encoding. A plain --data login fails with an invalid-credentials error against a password that is completely correct.
• Diffing a realm export needs normalisation. Keycloak's partial export emits arrays in nondeterministic order and includes server-assigned id fields, so a naive diff against your YAML is pure noise. Strip id and containerId and sort arrays by their serialised form before comparing, or you'll chase phantom drift.
• The operator's real footprint was double its request. It ran a ~264 Mi working set against a 128 Mi request on day one. Requests written from documentation rather than measurement are guesses; on a single node, guesses are what the resize trigger trips over.
• Argo CD honours rootCA on login and ignores it on session refresh. A private CA configured in the OIDC block worked for the initial sign-in and then failed on every session verification, because the two code paths don't share the setting. The fix is additive trust at the container level (SSL_CERT_DIR), not the tempting skip.verify knob — one of those bugs where the honest fix and the quick fix look equally reasonable at 1am.
• Keep a break-glass account that doesn't depend on the IdP. Argo CD keeps its local admin account precisely because "log in with Keycloak" is not a recovery plan for "Keycloak is down."

Where this leaves us

The platform can now answer who you are, from a realm that lives in git, backed by a database it already knows how to restore. Secrets, data, identity — the three things every workload assumes somebody else has solved.

Which means it's finally time to run something real on it. Next: an EU office suite — Nextcloud and Vaultwarden, both behind this single sign-on, with one gotcha that turns out to be a feature.

---

I'm building this in the open, and I'll run it for EU teams who'd rather ship product than hire a platform engineer. If that's you, follow along via RSS — every build post lands there first — or start with what this project is about.