Classical ciphers in CTFs: Caesar, Vigenère, substitution, and frequency analysis

Shift ciphers

A Caesar cipher shifts every letter by the same amount. With only 25 possible shifts, you try all of them and look for readable text. ROT13 is the shift of 13. Atbash maps A to Z, B to Y, and so on.

Snippet

python3 - <<'EOF'
ct = 'Iodj lv: sudfwlfh'
for k in range(26):
    pt = ''.join(chr((ord(c) - base - k) % 26 + base) if c.isalpha() else c
                 for c in ct for base in [65 if c.isupper() else 97])
    print(k, pt)
EOF

Substitution and frequency

A monoalphabetic substitution maps each letter to another letter. Brute force is impossible (26 factorial keys), but English letter frequencies leak through: E, T, A, O, I, N are most common, and short words like the, and, of are recognizable. Count letters, map the most common to likely candidates, and refine with word patterns. Online solvers such as quipqiup automate this.

Lab: break a Vigenère cipher you made

Vigenère uses a keyword: each letter is shifted by the corresponding key letter. Encrypt a paragraph of English with a five-letter key you choose. Then estimate the key length with the index of coincidence for candidate lengths, split the ciphertext into that many columns, and solve each column as a Caesar cipher using frequency. Confirm you recover your key.

Snippet

import string
EN = 'etaoinshrdlcumwfgypbvkjxqz'
def ic(s):
    n = len(s); return sum(s.count(c) * (s.count(c) - 1) for c in string.ascii_lowercase) / (n * (n - 1)) if n > 1 else 0
ct = ''.join(c for c in open('vig.txt').read().lower() if c.isalpha())
for L in range(1, 13):
    print(L, round(sum(ic(ct[i::L]) for i in range(L)) / L, 4))   # English is about 0.066

def best_shift(col):
    freq = [col.count(chr(97 + i)) for i in range(26)]
    return max(range(26), key=lambda k: freq[(4 + k) % 26] + freq[(19 + k) % 26] + freq[(0 + k) % 26])
L = 5
print(''.join(chr(97 + best_shift(ct[i::L])) for i in range(L)))

Why this matters beyond CTFs

These ciphers are broken because they preserve statistical structure of the plaintext. Modern ciphers are designed so ciphertext looks random. Seeing exactly how structure leaks builds intuition for why ECB mode is unsafe and why randomness and diffusion matter.

Diagram

Classical ciphers and how to break them

Each preserves some structure.

Caesar / ROT13
Try all 25 shifts.
Atbash
Reverse the alphabet.
Substitution
Letter frequency and word patterns.
Vigenère
Find key length (index of coincidence), then solve columns as Caesar.

Check yourself

1. How many non-trivial shifts does a Caesar cipher have?

2. Which letter is most frequent in typical English text?

3. What is the first step in breaking a Vigenère cipher?