From Assembly to C: Building a Semi-Automatic Decompiler

Have you ever attempted to create a decompiler for a program, or even just a disassembler to reverse-engineer the code? While most of these tasks are now often handled by specialized software, my journey started in high school, where I embarked on an ambitious project to create a simplistic operating system called FLOpenOS (FOS). This OS, although limited, served as an early exploration into the world of assembly and reverse engineering.

Creating FLOpenOS: A Floppy OS

While FLOpenOS was certainly a fun and educational project, it was frustratingly straightforward in comparison to modern operating systems. However, it did come with some unique features, including an assembly language assembler and disassembler. The disassembler was essential for opening and analyzing files that had been stored on disk. As it was assembly rather than a higher-level language, the decompilation process was relatively simple.

The limitations of FLOpenOS were stark compared to contemporary systems, but it laid the foundation for my curiosity and passion for reverse engineering and decompilation. Over the years, the necessity for such tools has only increased, and the techniques used back then are still relevant today.

Manual Decompilation in the Mid-1990s

During the mid-1990s, I found myself frequently diving into manual decompilation tasks. This involved using an existing disassembler to analyze and understand code snippets. The process was laborious and repetitive, and often required resolving ambiguities between code and data. Here is an example of the manual steps involved:

Identifying Code vs. Data

Start with distinguishing between code and data segments. A function call might be marked as code, and a data access related to printf() might be marked as data. Validate the disassembler's analysis. If it makes a wrong determination, manually correct it. Investigate variables: initially name them as addresses, and after understanding their roles, assign meaningful names and types. Apply the names and types consistently across all instances of the same element.

These processes, while tedious, were necessary to fully comprehend and document the semantics of the code. The lack of automation made the work more cumbersome but also honed my skills in understanding low-level programming.

Creating a Semi-Automated Decompiler

Recognizing the tedium of manual decompilation, I conceived an idea to develop a semi-automatic decompiler. This tool could take the output of a disassembler and apply a series of logical steps based on user-provided markup:

Mark up the disassembled code with meaningful identifiers and types. Automatically apply these identifiers and types to subsequent occurrences.

While this approach did not fully automate the process, it significantly reduced the repetitive nature of manual decompilation. By doing so, I aimed to bridge the gap between raw assembly and higher-level human-readable code. This work was inspired by my earlier success with FLOpenOS and the need for tools that could analyze and document complex codebases more efficiently.

Although I did not pursue the idea of directly producing C code from this tool, the concept was a valuable step towards a fully-fledged decompiler. The project taught me the limitations and challenges of reverse engineering, and paved the way for further advancements in the field.

Conclusion

The journey from a simple assembly language assembler and disassembler to a semi-automatic decompiler highlights the evolution of tools used in reverse engineering. While modern tools handle these tasks more efficiently, the techniques and insights gained from early projects continue to be significant in the realm of software analysis and reverse engineering.

From my experiences, it is clear that the path to automation in reverse engineering is paved with incremental steps. Each tool developed brings us closer to fully automated decompilers, which could revolutionize the way we understand and manipulate code.