녹에서 디컴파일러 파이프라인 구축: 핵분열이 NIR과 HIR을 분리하는 이유

Building a Decompiler Pipeline in Rust: Why Fission Separates NIR and HIR

Decompiler output often looks simple from the outside.

A binary goes in. Pseudocode comes out.

But between those two points, a decompiler must recover several different kinds of information:

• instruction semantics
• register and memory effects
• control flow
• stack variables
• calling conventions
• data types
• expressions
• loops and conditionals
• readable source-like structure

Trying to represent all of this in one intermediate representation quickly becomes difficult.

While building Fission, a reverse-engineering and binary decompilation workspace written primarily in Rust, I decided to separate the decompiler pipeline into two main intermediate representations:

• NIR, a lower-level representation intended to preserve machine semantics
• HIR, a higher-level representation intended to express recovered, human-readable program structure

This article explains why that separation exists, what each representation owns, and why it makes decompiler development easier to reason about.

Correctness and readability want different things

A decompiler has at least two responsibilities.

First, it must preserve the behavior of the original machine code.

Second, it must produce output that a human can understand.

Those goals overlap, but they are not identical.

Consider a simplified fragment of machine-level behavior:

tmp0 = RAX
tmp1 = tmp0 + 1
RAX = tmp1
flags = update_flags(tmp0, 1, tmp1)

Enter fullscreen mode Exit fullscreen mode

A human reader may prefer to see:

rax++;

Enter fullscreen mode Exit fullscreen mode

The concise form is easier to read, but it omits details that may still matter elsewhere in the pipeline.

The flags update could affect a later conditional branch. The operation width may matter. The source and destination could alias. The operation may have originated from an instruction with additional side effects.

If the decompiler converts everything into source-like syntax too early, it becomes easy to discard evidence.

If it keeps everything at machine level until the final rendering stage, the output remains explicit but difficult to understand.

That tension is the main reason Fission separates NIR and HIR.

The Fission pipeline

At a high level, the pipeline looks like this:

Binary bytes
    ↓
Binary loader and static facts
    ↓
Sleigh decoding
    ↓
Raw P-Code
    ↓
Fission NIR
    ↓
Normalization and semantic recovery
    ↓
Fission HIR
    ↓
Control-flow structuring
    ↓
Pseudocode rendering

Enter fullscreen mode Exit fullscreen mode

Fission uses Sleigh language definitions for instruction decoding and P-Code semantics.

The later semantic pipeline, however, is owned by Fission itself.

That includes:

• NIR and HIR
• normalization passes
• type and calling-convention recovery
• control-flow analysis
• structural recovery
• pseudocode rendering
• telemetry
• regression testing

This ownership boundary is important.

Sleigh describes what an instruction does.

Fission decides how that behavior is represented, analyzed, transformed, structured, and presented.

What NIR is for

NIR is the representation closest to lifted machine semantics.

Its primary responsibility is not readability. Its responsibility is preserving enough information to support reliable analysis and transformation.

NIR should make it possible to reason about:

• explicit data flow
• register and memory operations
• address spaces
• operation widths
• branch targets
• side effects
• stack behavior
• control-flow edges
• lifted instruction provenance

A simplified NIR sequence might look like this:

v0 = load64(stack_pointer + 8)
v1 = int_add(v0, 1)
store64(stack_pointer + 8, v1)
branch_if(v1 < 10, block_4)

Enter fullscreen mode Exit fullscreen mode

This is already easier to analyze than raw P-Code, but it remains explicit.

The representation should not claim more than the analysis has proven.

For example:

store64(stack_pointer + 8, v1)

Enter fullscreen mode Exit fullscreen mode

should not immediately become:

counter = v1;

Enter fullscreen mode Exit fullscreen mode

unless the system has enough evidence that the stack location represents a stable local variable and that counter is an appropriate recovered identity.

NIR is therefore an evidence-preserving layer.

What HIR is for

HIR represents higher-level concepts that are useful for transformation and human-readable output.

It may contain constructs such as:

Assignment
Variable
Call
Return
If
While
Loop
Break
Continue
BinaryExpression
UnaryExpression
Cast

Enter fullscreen mode Exit fullscreen mode

An HIR fragment could look like this:

while (counter < 10) {
    counter = counter + 1;
}

Enter fullscreen mode Exit fullscreen mode

HIR is not merely prettier NIR.

It has a different responsibility.

NIR models behavior in a form suitable for analysis.

HIR models recovered intent and program structure in a form suitable for normalization, structuring, and rendering.

This distinction helps prevent the renderer from becoming responsible for decompiler semantics.

The renderer should consume an already meaningful structure. It should not be the component that discovers loops, invents variables, repairs malformed conditions, or guesses expression relationships.

Why a single IR becomes problematic

A single representation initially appears simpler.

There are fewer types, fewer conversion steps, and less code.

Over time, however, one IR often accumulates incompatible responsibilities.

A single node may be expected to represent both:

INT_ADD(register_space[0x20], constant(1))

Enter fullscreen mode Exit fullscreen mode

and:

counter++;

Enter fullscreen mode Exit fullscreen mode

These are related, but they do not have the same invariants.

The lower-level operation cares about:

• bit width
• signedness evidence
• address spaces
• overflow behavior
• exact operands
• provenance

The higher-level expression cares about:

• variable identity
• expression simplification
• source-like syntax
• type presentation
• readability

Combining both levels usually leads to one of two outcomes.

The IR remains too low-level

The renderer becomes full of special cases:

If this store follows this addition,
and both access the same stack location,
and the increment is one,
render it as x++.

Enter fullscreen mode Exit fullscreen mode

This is dangerous because the renderer begins to perform semantic analysis.

The IR becomes too high-level too early

Low-level facts disappear before the analysis is complete.

When output is incorrect, it becomes difficult to determine whether the problem came from:

• instruction lifting
• NIR normalization
• variable recovery
• type inference
• control-flow structuring
• rendering

Separating the layers gives each stage a clearer contract.

Ownership boundaries matter

One of the main design principles in Fission is that a bug should be fixed in the layer that owns the behavior.

For example:

• Incorrect binary metadata belongs in the loader or static-analysis layer.
• Incorrect instruction semantics belong in the lifting layer.
• Broken data flow belongs in NIR analysis or normalization.
• Incorrect variable recovery belongs in the semantic-recovery layer.
• Broken loop reconstruction belongs in the structuring layer.
• Formatting problems belong in the renderer.

This sounds straightforward, but decompiler development makes it very tempting to patch the final output.

Suppose the decompiler prints:

if (x) {
    goto label_1;
}

Enter fullscreen mode Exit fullscreen mode

when the expected structure is:

while (x) {
    ...
}

Enter fullscreen mode Exit fullscreen mode

It may be possible to add a renderer heuristic that recognizes the pattern.

But the actual problem may be that:

• the CFG was classified incorrectly
• a dominance relation became stale
• a back edge was not recognized
• a region was collapsed before its proof was complete
• a branch polarity was reversed
• normalization removed required evidence

A renderer-level patch may improve one binary while hiding a structural defect that affects many others.

The NIR/HIR separation makes these ownership mistakes easier to detect.

Structuring is not formatting

Control-flow structuring deserves its own stage.

A CFG containing basic blocks and edges does not automatically become an if, while, or switch.

The structurer must reason about:

• dominators
• post-dominators
• strongly connected components
• loop headers
• back edges
• exit regions
• irreducible control flow
• region legality

In Fission, the structuring path is intended to be deterministic and evidence-driven.

A region should only be collapsed when the system has enough evidence that the resulting high-level construct preserves the original behavior.

When the proof is incomplete, explicit fallback output is preferable to deceptively clean pseudocode.

For example:

label_1:
    ...

    if (condition) {
        goto label_1;
    }

Enter fullscreen mode Exit fullscreen mode

This may be less readable than a reconstructed loop.

But it is still better than emitting a clean-looking while statement whose semantics are wrong.

Readability matters.

Incorrect readability is worse than honest low-level output.

Why Rust helps

Rust is not automatically the best language for every decompiler.

It introduces real costs:

• longer compile times
• complex generic types
• ownership friction in graph-heavy transformations
• difficult mutation patterns
• a learning curve around lifetimes and traits

Still, Rust has been useful for Fission because the system contains many ownership boundaries.

Different parts of the workspace own:

• binary metadata
• static facts
• lifted semantics
• NIR transformations
• HIR normalization
• CFG structures
• analysis caches
• emulator state
• telemetry
• user-facing output

Rust makes it harder to blur those boundaries accidentally.

Immutable inputs and explicit transformation outputs make questions like these easier to answer:

• Which pass changed this function?
• Is this graph analysis still valid after mutation?
• Does the renderer own any semantic decisions?
• Can two analysis components mutate the same state?
• Is this result derived from canonical telemetry or from a parallel counter?

The Rust type system does not prove the decompiler correct.

It can, however, make architectural mistakes more visible.

Determinism is a correctness property

A decompiler should produce the same output for the same binary across separate runs.

This is more subtle than it appears.

An algorithm may accidentally rely on the iteration order of a hash map or hash set.

For example:

let candidate = nodes
    .iter()
    .find(|node| is_valid(node));

Enter fullscreen mode Exit fullscreen mode

If nodes is backed by a randomized hash collection, the selected candidate may differ between processes.

That can alter:

• region-collapse order
• variable naming
• expression ordering
• block layout
• final pseudocode

For a compiler or decompiler, this is not merely cosmetic.

Nondeterminism makes regressions difficult to reproduce and quality measurements unreliable.

Fission therefore treats deterministic output as an explicit correctness requirement.

Transformations that choose among multiple candidates must use stable ordering or an explicitly defined ranking strategy.

Measuring decompiler quality

A major unresolved question is how to measure whether decompiler output is actually improving.

Readable output is subjective.

A change may make one function prettier while making another less accurate.

Potential quality signals include:

• semantic equivalence on controlled fixtures
• structural similarity against known source
• reduction in unnecessary goto statements
• successful loop and conditional recovery
• stable variable identity
• type-recovery confidence
• deterministic output across runs
• regression rates across real binaries
• preservation of side effects and branch behavior

No single metric is enough.

A practical decompiler quality system probably needs to combine:

1. mechanical checks
2. real-binary regression corpora
3. telemetry
4. differential testing
5. human review

The important distinction is between output that merely looks better and output that is structurally and semantically better.

Current status

Fission is still a research-heavy, early-stage project.

Most current work is focused on:

• x86 and x86-64 correctness
• NIR and HIR normalization
• control-flow structuring
• deterministic output
• readable pseudocode
• regression testing against real binaries

The repository also contains experimental work on:

• P-Code emulation
• deterministic execution recording and replay
• taint tracking
• concolic path exploration
• symbolic reasoning in Rust

Some of these areas remain incomplete.

I prefer to state those limitations clearly rather than present them as production-ready features.

Conclusion

The separation between NIR and HIR is not only an implementation detail.

It is a way to preserve low-level evidence while still allowing the decompiler to recover readable, source-like structure.

NIR asks:

What does the machine appear to do?

HIR asks:

What higher-level structure is supported by the available evidence?

Keeping those questions separate helps make transformations easier to review, bugs easier to locate, and final output less dependent on renderer-specific heuristics.

Fission is open source, and feedback is welcome from people working on:

• compiler IR design
• reverse engineering
• decompilation
• control-flow structuring
• symbolic execution
• Rust architecture

Repository:

https://github.com/sjkim1127/Fission