Folks, it was only a matter of time before I would talk about this subject. I’ve covered many different types of data formats, including secondary storage, network communications, and representation in RAM. But what about executable files?
If you’ve evaluated BARfly, you’ll know that the software is merciless. It doesn’t know when it’s being pushed too far. If you want to represent machine code instruction sets with it, you’re perfectly welcome to do so. Machine language is just another data format.
I won’t go into all the ethics of (or reasons for) reverse engineering. I will say this, though: most of the time, license agreements you sign when you purchase software or hardware have a clause forbidding the reverse engineering of your newly acquired property. Please obey those.
Now that I’ve said that, let’s talk about reverse engineering with style.
When a program’s source code is compiled and linked, it becomes machine language, a series of bytes that, when interpreted, tell a processor what to do. A machine language “instruction,” as it is known, generally has the form of an operational code (often known as an “opcode“), followed by one or more optional “mod bytes” that influence the performance of the opcode.
Most programmers nowadays don’t know about machine language. This is because compilers and linkers are so good anymore, and optimization so reliable, that everyone just thinks, “I’ll just create the source code and have the compiler handle the tough translation tasks.” Unless you’re tasked with making or modifying the compiler itself, you normally don’t need to know about the target instruction set.
But if you are performing reverse engineering, you’ll need to know about every instruction set you want to work with. Each computer manufacturer uses its own instruction set and has its own interface for programming the hardware, which isn’t necessarily linked to the instruction set.
Why do you need to know this info? Because once machine language is generated, it can’t be immediately molded back into source code. Machine language instructions tend to be simple operations. The equation A = B + C * D / E would require at least 5 different instructions.
As you can imagine, reverse engineering requires both a significant knowledge base and a whole lot of patience. The quality of the tools used in the process can make a huge difference in overall effectiveness.
A short list of tools you have:
- DEBUG: A very elementary disassembler and debugger, present since the early days of DOS.
- IDA Pro: A sophisticated, platform-independent graphical disassembly suite.
- Various off-the-shelf tools: Varying functionality and usefulness.
- Emulators: Programs designed to run a simulated version of the actual machine in a “box,” whose input and output can be easily manipulated.
DEBUG and similar disassemblers operate on the principle that any portion of memory can be reverse-engineered, as long as you know (a) the starting address, and (b) the range over which the code needs to be reverse-engineered. This is the “forward only” principle.
IDA Pro and similar tools operate on a “follow the path only” principle. Instead of linearly (and blindly) stepping through instructions in a memory buffer, IDA Pro allows you to keep track of the stack frame, the call tree of the program, the references to local and global variables, jump labels, etc. Generally, output from applications like IDA Pro are more useful than the “forward and blind” applications like DEBUG.
Programmers have made custom disassemblers that try to get around some of the limitations of DEBUG and/or IDA Pro. These tools have varying strengths and weaknesses, which I will discuss shortly. Finding the right tool can make a world of difference.
Emulation is the last resort of reverse engineering. With an emulator, you can get a realistic state of the processor and its memory at all times. DosBox is a common emulator for 16-bit DOS applications. Emulators are also commonly used to play classic games built for antiquated hardware; MAME is the best-known example.
The biggest strength of the “forward-only” principle in reverse engineering is that you cover all bases–you reach all portions of code, regardless of how they are actually covered. The weaknesses include possibly disassembling non-code (resulting in gibberish), straddling valid opcodes (resulting in incorrect disassembly), and limited ability to interpret the results.
The core strength of the “follow the path only” principle is that only the paths actually discovered through natural flow from points of entry will be disassembled. This yields much more useful and valid output than forward-only, and allows applications like IDA Pro to display all sorts of useful diagrams and relationships. The weakness of path-following is that it is restricted to just known points of entry and deterministic paths (not all code is deterministic).
I’m going to provide a case study in reverse engineering, using my own experience. I’ve already done a ROM hack total conversion with Metroid Master, but I wanted to do the next ROM hack without relying so much on others’ tools. This time, I decided to reverse engineer the code by building my own disassembler.
(NOTE: I feel I can safely talk about reverse engineering 8-bit Metroid because it’s been 25 years since it was popular. If I were talking about hacking Wii games, Nintendo might not like what I’m doing.)
There are many quality 6502 disassemblers out there. I’m mostly used to Intel’s x86 family of instruction sets. But the 6502 processor was once widely used, so learning it was something I absolutely had to do.
The rationale behind creating my own disassembler is that I wanted to circumvent the weaknesses of the existing disassemblers. My disassembler performs limited emulation, and can generate multiple types of readable assembly from machine language.
While 6502 disassembly output usually looks like this…
0001FFB0: sei
0001FFB1: cld
0001FFB2: ldx #$00
0001FFB4: stx $2000
0001FFB7: stx $2001
0001FFBA: lda $2002
0001FFBD: bpl $0001FFBA
0001FFBF: lda $2002
0001FFC2: bpl $0001FFBF
0001FFC4: ora #$FF
0001FFC6: sta $8000
0001FFC9: sta $A000
0001FFCC: sta $C000
0001FFCF: sta $E000
0001FFD2: jmp $C01A
…my disassembler, NESDis, can display it like this:
label_07_FFB0:
07/FFB0: 78 P |= F_INTERRUPT_DISABLE;
07/FFB1: D8 P &= ~F_DECIMAL_MODE;
07/FFB2: A2 00 X = 0×00;
07/FFB4: 8E 00 20 m[0x2000] = X; // PPU Command
07/FFB7: 8E 01 20 m[0x2001] = X; // PPU Command
label_07_FFBA:
07/FFBA: AD 02 20 A = m[0x2002]; // PPU Read
07/FFBD: 10 FB if (!(P & F_SIGN)) goto label_FFBA;
label_07_FFBF:
07/FFBF: AD 02 20 A = m[0x2002]; // PPU Read
07/FFC2: 10 FB if (!(P & F_SIGN)) goto label_FFBF;
label_07_FFC4:
07/FFC4: 09 FF A |= 0xFF;
07/FFC6: 8D 00 80 m[0x8000] = A; // MMC1 Control
07/FFC9: 8D 00 A0 m[0xA000] = A; // MMC1 CHR-ROM 1
07/FFCC: 8D 00 C0 m[0xC000] = A; // MMC1 CHR-ROM 2
07/FFCF: 8D 00 E0 m[0xE000] = A; // MMC1 PRG-ROM
07/FFD2: 4C 1A C0 goto label_0xC01A
Same machine language, but the assembly output looks COMPLETELY different. I think opcodes are much more readable if put into C-language equivalents. Wouldn’t you agree?
Critical hardware port programming is revealed. Original machine language and readable assembly are displayed concurrently. Addresses and bank numbers are known. More importantly, paths are followed in a similar way as IDA Pro, meaning that they need not be dumped in a linear order (they can be dumped hierarchically).
BUT…determinism is always an issue. Most instruction sets, including both x86 and 6502, have indirect jump opcodes. This constitutes a serious problem because the path to which to jump or call is determined at run-time. Emulation can help find the valid addresses, but it’s almost impossible to know all the valid addresses unless you watch every path being followed at run-time, which is usually not worth it.
Fortunately, programmers rarely get fancy with such opcodes for just any reason. A programmer will generally implement an indirect jump or call for one of the following reasons:
- Callback or “hook” procedure. Equivalent to a C-language pointer to a function.
- Switch control block. C-language “switch” statements can use a “jump table” to quickly direct program control to a wide variety of locations based on a scalar input. This is most economical when there are many scalar input choices that are numerically close to each other (e.g. 5, 6, 7, 8, and 9).
These are very specific circumstances. Therefore, the number of indirect jump opcodes in a program tends to be very few. When confronted with an indirect jump opcode, I had to perform manual inspection to figure out what the processor was trying to do:
label_07_C510:
07/C510: 0A A <<= 1;
07/C511: A8 Y = A;
07/C512: B9 1F C5 A = m[0xC51F + Y];
07/C515: 85 0A m[0x0A] = A;
07/C517: B9 20 C5 A = m[0xC520 + Y];
07/C51A: 85 0B m[0x0B] = A;
07/C51C: 6C 0A 00 indirect_jump(ia(0x000A));
What does this mean? Well, roughly translated, it means “use the value of Y as an index into a jump table.” Fortunately, the jump table is located immediately after the indirect jump opcode. I’ve found that the jump addresses, also co-located, evaluate to the following:
{ 0xC531, 0xC552, 0xC583, 0xC590, 0xC5B6, 0xC5C3, 0xC45C, 0xC45C, 0xC45C }
NESDis stops when it encounters an indirect jump opcode, because of lack of determinism. But you can pick any point of entry with NESDis. The next time I ran NESDis, I chose the point of entry as the first entry in the jump table, 0xC531. And lo! An entirely new set of paths was revealed:
label_07_C531:
07/C531: A0 00 Y = 0×00;
07/C533: 84 31 m[0x31] = Y;
07/C535: C8 Y++;
07/C536: 84 1D m[0x1D] = Y;
07/C538: 20 5D C4 funccall_0xC45D();
label_07_C53B:
07/C53B: 20 3E A9 funccall_0xA93E();
label_07_C53E:
07/C53E: 20 58 C1 funccall_0xC158();
…
By combining the results of individual NESDis outputs, we can eventually build a complete picture of the Metroid source code.
Individuals who are interested in reverse engineering now have an idea of where to begin. Of course, there are many other advanced tricks, but they have to be discovered on one’s own time.
I strongly suggest you check out emulation central, http://www.zophar.net, for more information.