HackBlue

Tools · Ghidra · 13 / 22

Ghidra: compiler output and hand-written code

Ordinary compiler output has habits: a consistent calling convention, structured branches, and library calls you can name. Hand-written bytes often lack those habits. Optimization is still the compiler. Density alone is not a verdict.

16 min read · practice only with authorization

What compiler output tends to look like

A compiler is consistent because it is a program with a small set of patterns. Arguments move through the calling convention you will study in the next lesson, the same registers in the same order at every call. Branches bracket a condition. Loops have an induction variable and a test. Library routines appear as named externals when the binary is dynamically linked. An unoptimized function often begins by saving the frame pointer and ends by restoring it. None of these habits is required in every function of every build. Together they are the texture of compiled code, and harbor-check at -O0 has that texture on purpose.

A reference to __stack_chk_fail, or a slot in the frame that is saved near the return address and checked before the epilogue, is the stack protector. It means the compiler was asked to detect a smashed frame and abort. It is evidence about the build flags. It is also evidence that the function has a buffer the compiler thought was worth guarding. You can name that buffer from its size. You do not need to attack it to learn that it is there.

Optimized output is still the compiler

Turn on optimization and the texture changes without a change of author. The frame pointer disappears. Locals live in registers. load_config may be inlined into main, so the function you named in the unoptimized build is simply gone, its body merged into the caller. Tail calls become jumps. A switch may become a jump table. The decompiler often recovers the structure anyway, because the structure is still there. When it does not, the Listing shows a denser version of a story you already know from the -O0 build.

That comparison is the cleanest way to train your eye. Import both builds, and do not expect the addresses or the function boundaries to match. Expect the string, the fopen call, and the status words to exist in both, arranged differently. If a behavior is present at -O0 and you cannot find it at -O2, look for an inlined body before you conclude the compiler deleted the logic.

When the habits are absent

Some bytes were not produced by a compiler’s usual patterns. The signs are practical. A jump lands in the middle of an instruction the Listing already decoded, so the same bytes are two different streams. A call’s argument registers do not match the convention the rest of the program uses. A long stretch of undefined bytes follows a tiny stub, and the import table is almost empty. Any one sign can have an innocent cause. Several together mean you should stop treating the decompiler as a guide and read the Listing in small pieces, fixing the decode only where the bytes force it.

A stub in front of a large undecoded blob is also how a packed file often looks: a little code, then bytes that are not yet instructions. Recognizing that shape is the skill this lesson teaches. This course does not walk through unpacking or rewriting those bytes. In a lab where execution is not authorized, you record the shape, you name the few imports you can see, and you stop. The triage lesson later in the curriculum is the same discipline applied to network and persistence references.

Lab: the same source, two compiler settings

Build gcc -O0 -o harbor-check-o0 harbor-check.c and gcc -O2 -o harbor-check-o2 harbor-check.c. Import both into the harbor folder of the project. In each, find the function that references the default path. Write five lines comparing them: whether a frame pointer is saved, whether the file-opening logic is its own function or merged into the caller, whether a stack check helper appears, and whether the status words are still separate references. Finish with one sentence: both are compiler output, and here is the structural difference.

Do not rewrite either binary. The comparison is the lesson. If you want a third data point later, a stripped -O0 copy is the finding-main exercise, and it should still show compiler habits once you land in the right function.

Snippet
A note that separates optimization from a different author

-O0 harbor-check: frame pointer, load_config as its own function.
-O2 harbor-check: frame pointer may be gone, load_config may be inlined.
Both are compiler output. The string and the fopen call remain.

Hand-written or packed bytes are a separate observation:
overlapping decodes, or a tiny stub and a blob.
This course stops at recognizing that shape.

Diagram

Two shapes of bytes, read differently

  1. Prologue

    An unoptimized build often saves a frame pointer. An optimized build often does not. Both can be gcc.

  2. Calls

    A call to fopen, or a call that sets up rdi, rsi, and rdx the same way every time, is a compiler talking to a convention.

  3. Canary

    A reference to __stack_chk_fail is a compiler defense inserted around a buffer. It is a sign of compiled C, and it is evidence of a stack check.

  4. Pause

    Overlapping instructions, a stub followed by a large undecoded blob, or a handful of imports that only load libraries are reasons to slow down and reread the map.

What the shape should change about your reading

Compiler habits

Calls. Arguments leave in the convention’s registers, the same way at every call.

Control. Branches close over structured conditions, loops, and switches.

Reading. The decompiler is usually a help. Types and names pay off quickly.

No compiler habits

Calls. The convention changes mid-function, or a call target is computed and unnamed.

Control. Jumps land in the middle of instructions the Listing already decoded.

Reading. Trust the Listing. Slow down. The decompiler may paper over a bad decode.

A note that separates optimization from a different author
-O0 harbor-check: frame pointer, load_config as its own function.
-O2 harbor-check: frame pointer may be gone, load_config may be inlined.
Both are compiler output. The string and the fopen call remain.

Hand-written or packed bytes are a separate observation:
overlapping decodes, or a tiny stub and a blob.
This course stops at recognizing that shape.

Check yourself

1. Your -O2 build of harbor-check has no push rbp / mov rbp, rsp at the start of main, and the decompiler still shows the config path. What is the likely explanation?

2. You see a call to __stack_chk_fail in a function that holds a 128-byte line buffer. What does that call indicate?

3. The Listing can be decoded two ways from the same address, and a jump targets the second way. How do you read it?