Vaultless Tokenization

Tokenization replaces a sensitive value with a surrogate that has the same shape but no meaning, so that systems which need to carry the value around — order records, logs, analytics, a customer service screen — never hold the real one. The classical implementation keeps a vault: a secured database mapping each token to the value it stands for. Vaultless tokenization removes that database and derives the token from the value itself, using a key.

How it works

The token is computed by format-preserving encryption (FPE): a cipher whose output is drawn from the same alphabet and length as its input, so a 16-digit card number encrypts to 16 digits and still passes the field validation downstream systems apply. NIST SP 800-38G specifies the FF1 and FF3-1 modes for this. Alongside the key, FPE takes a tweak — a non-secret context value such as a merchant or account identifier — which makes the same input produce different tokens in different contexts.

flowchart TD A[Sensitive value] --> B[Tokenization service] B --> C{Format-preserving<br/>encryption} C --> D[Encryption key] C --> E[Tweak / context] D --> F[Derive token] E --> F F --> G[Token, same format as input] G --> H[Used by downstream systems] H --> I[De-tokenization] I --> J[Original value] subgraph Inputs D E end subgraph NoStore["No token store required"] B C F end

Tokens are deterministic: the same input under the same key and tweak always produces the same token, which is what preserves referential integrity — two records that held the same card number still join on the token.

What it buys

• No token store to run, secure, back up, replicate or reconcile.
• No database lookup on the tokenization path, so lower latency and no lookup bottleneck.
• Scaling is a matter of adding compute, not of scaling a shared store.
• Disaster recovery is simpler, because the thing that has to survive is the key rather than a growing table.
• Tokenization can be performed anywhere the key can be used, including at the edge, without network dependence on a central service.

Where it is used

Values that have to keep their format while losing their meaning: payment card numbers, national insurance or social security numbers, account and customer identifiers, and other personal identifiers that flow through systems which have no business reading them.

What to be careful about

It is encryption, not a random surrogate. The value is recoverable from the token with the key, so the key is as sensitive as the whole dataset. A vault-based scheme can use an unrelated random token whose compromise reveals nothing without the vault; a vaultless token is ciphertext. That distinction matters when arguing scope under PCI DSS or similar regimes — key custody is part of the argument, so treat scope reduction as something to establish with your assessor rather than something the technique grants.

Determinism leaks. Because the same input always maps to the same token, an attacker who can submit chosen values and observe the resulting tokens can build a dictionary for a small domain. The tweak limits this by partitioning the mapping per context; choosing the tweak well matters as much as choosing the key.

Key rotation re-tokenizes everything. With a vault you can re-key the vault; here, changing the key changes every token derived from it, so rotation is a data migration. Plan for it before the first token is issued, not after.

Algorithm choice is the security boundary. Use a published, reviewed FPE mode rather than an ad-hoc format-preserving construction. FF3 as originally published was found to be weak and was revised to FF3-1; homegrown schemes have generally fared worse.