What I Learned After Building a Programming Language From Scratch in Python

I didn’t just write a programming language. I built the pipeline that makes the language work.

🔗 NexPro on GitHub: https://github.com/probal2005/NexPro

There are thousands of programming languages in existence.

So building another one sounds unnecessary.

But my goal with NexPro was never to compete with Python, JavaScript, Rust, or C++.

I wanted to answer a much simpler question:

What actually happens between writing code and getting an output?

So instead of only learning the theory of lexers, parsers, ASTs, interpreters, and runtimes, I decided to implement them.

This post documents what I have actually built, what works today, what I learned, and what still needs to be done.

1. The Experiment

NexPro is an experimental programming language implemented in Python.

Its source files use the:

.pa

Enter fullscreen mode Exit fullscreen mode

extension.

A simple NexPro program looks like this:

name = "Probal"
city = "Kolkata"

say name
say city

Enter fullscreen mode Exit fullscreen mode

The important part isn’t that this syntax is simple.

The important part is what happens internally.

The source doesn’t go directly from:

.pa file

Enter fullscreen mode Exit fullscreen mode

to:

output

Enter fullscreen mode Exit fullscreen mode

Instead, it passes through multiple stages.

2. The Actual Language Pipeline

The core idea behind NexPro is:

┌───────────────────┐
│   NexPro Source   │
│      .pa          │
└─────────┬─────────┘
          │
          ▼
┌───────────────────┐
│       Lexer       │
│  Source → Tokens  │
└─────────┬─────────┘
          │
          ▼
┌───────────────────┐
│      Parser       │
│ Tokens → AST      │
└─────────┬─────────┘
          │
          ▼
┌───────────────────┐
│        AST        │
│ Program Structure │
└─────────┬─────────┘
          │
          ▼
┌───────────────────┐
│    Interpreter    │
│   Execute Nodes   │
└─────────┬─────────┘
          │
          ▼
┌───────────────────┐
│      Runtime      │
│ Values & State    │
└─────────┬─────────┘
          │
          ▼
┌───────────────────┐
│      Output       │
└───────────────────┘

Enter fullscreen mode Exit fullscreen mode

This pipeline isn’t just a diagram for documentation.

These are the actual conceptual components implemented in the repository.

3. Evidence #1 — The Repository Has a Language Implementation Structure

The project is organized around the language itself:

NexPro/
│
├── nexpro/
│   ├── cli.py
│   ├── lexer.py
│   ├── parser.py
│   ├── interpreter.py
│   ├── runtime.py
│   ├── tokens.py
│   ├── ast.py
│   ├── errors.py
│   └── __version__.py
│
├── examples/
│   ├── hello.pa
│   └── variables.pa
│
├── tests/
│
├── README.md
├── LICENSE
└── pyproject.toml

Enter fullscreen mode Exit fullscreen mode

Each part exists for a reason.

Component Responsibility lexer.py Converts source text into tokens tokens.py Defines token types parser.py Builds program structure ast.py Represents syntax as nodes interpreter.py Executes the AST runtime.py Handles runtime behavior/state errors.py Language-level error handling cli.py Provides the command-line interface tests/ Tests language behavior

This separation is important because language implementations become difficult to maintain when lexing, parsing, execution, and runtime logic are mixed together.

4. Evidence #2 — NexPro Actually Executes .pa Programs

A language project shouldn’t stop at syntax diagrams.

It needs to execute programs.

NexPro provides a CLI command:

nexpro run examples/hello.pa

Enter fullscreen mode Exit fullscreen mode

For a program such as:

say "Hello NexPro!"

Enter fullscreen mode Exit fullscreen mode

the interpreter produces:

Hello NexPro!

Enter fullscreen mode Exit fullscreen mode

Variables can also be used:

name = "Probal"
city = "Kolkata"

say name
say city

Enter fullscreen mode Exit fullscreen mode

which produces:

Probal
Kolkata

Enter fullscreen mode Exit fullscreen mode

That gives us a complete path:

hello.pa
   ↓
CLI
   ↓
Lexer
   ↓
Parser
   ↓
AST
   ↓
Interpreter
   ↓
Output

Enter fullscreen mode Exit fullscreen mode

5. Evidence #3 — Tokens Exist Before Execution

Consider:

name = 10

Enter fullscreen mode Exit fullscreen mode

A programming language implementation doesn’t need to treat this as one giant string.

The lexer can break it into meaningful pieces:

IDENTIFIER(name)
ASSIGN(=)
NUMBER(10)

Enter fullscreen mode Exit fullscreen mode

That transformation is fundamental.

The lexer answers:

“What are the pieces?”

The parser answers:

“How are those pieces related?”

The interpreter answers:

“What should those relationships do?”

This separation is one of the biggest things I understood while building NexPro.

6. Evidence #4 — The AST Changes Everything

Consider:

a = 10 + 20

Enter fullscreen mode Exit fullscreen mode

The parser doesn’t simply need to remember:

"10 + 20"

Enter fullscreen mode Exit fullscreen mode

It can represent the expression structurally:

        Assignment
        /        \
       a        Binary(+)
                /      \
              10        20

Enter fullscreen mode Exit fullscreen mode

This is the Abstract Syntax Tree.

The AST gives the interpreter a structured representation of what the programmer wrote.

Instead of asking:

“What characters are in this string?”

the interpreter can ask:

“What kind of node am I executing?”

That distinction is fundamental to language implementation.

7. Evidence #5 — Arithmetic Becomes a Tree

For:

a = 10
b = 20

say a + b

Enter fullscreen mode Exit fullscreen mode

the important expression is:

a + b

Enter fullscreen mode Exit fullscreen mode

Conceptually:

       +
      / \
     a   b

Enter fullscreen mode Exit fullscreen mode

The interpreter can then resolve:

a → 10
b → 20

Enter fullscreen mode Exit fullscreen mode

and evaluate:

10 + 20

Enter fullscreen mode Exit fullscreen mode

giving:

30

Enter fullscreen mode Exit fullscreen mode

This is where a simple syntax feature starts demonstrating the full language pipeline.

8. What Building the Lexer Taught Me

At first, tokenization looks easy.

You see:

a = 10

Enter fullscreen mode Exit fullscreen mode

and think:

Split the string.

But real language syntax quickly introduces problems.

What happens with:

a = 10 + 20

Enter fullscreen mode Exit fullscreen mode

What about:

name = "Probal"

Enter fullscreen mode Exit fullscreen mode

What about:

say "Hello World"

Enter fullscreen mode Exit fullscreen mode

Now the lexer needs to distinguish:

IDENTIFIER
NUMBER
STRING
ASSIGN
PLUS
SAY

Enter fullscreen mode Exit fullscreen mode

It also needs to deal with things such as:

Whitespace
Unknown characters
Strings
Numbers
Identifiers
Operators
End of file

Enter fullscreen mode Exit fullscreen mode

That was one of my first major lessons:

A programming language begins before parsing.

9. What Building the Parser Taught Me

The parser is where syntax becomes structure.

For example:

a = 10 + 20

Enter fullscreen mode Exit fullscreen mode

is not equivalent to:

a + 10 = 20

Enter fullscreen mode Exit fullscreen mode

The parser needs to understand relationships and precedence.

Even simple arithmetic starts raising questions:

10 + 20 * 5

Enter fullscreen mode Exit fullscreen mode

Should it mean:

(10 + 20) * 5

Enter fullscreen mode Exit fullscreen mode

or:

10 + (20 * 5)

Enter fullscreen mode Exit fullscreen mode

Language design quickly turns into a combination of:

Syntax
+
Grammar
+
Precedence
+
AST design

Enter fullscreen mode Exit fullscreen mode

That’s something I didn’t fully appreciate before implementing it.

10. What Building the Interpreter Taught Me

The interpreter is where the language becomes executable.

It has to understand things such as:

Number
String
Variable
Assignment
Binary Expression
Say

Enter fullscreen mode Exit fullscreen mode

For example:

x = 100
say x

Enter fullscreen mode Exit fullscreen mode

requires the runtime to maintain state:

Environment

x → 100

Enter fullscreen mode Exit fullscreen mode

Then:

say x

Enter fullscreen mode Exit fullscreen mode

requires the interpreter to retrieve:

x → 100

Enter fullscreen mode Exit fullscreen mode

and send the value to the output.

This is where concepts like scope, environments, values, evaluation, and runtime state become practical instead of theoretical.

11. Why I Chose Python

NexPro is currently implemented in Python.

That was intentional.

For an experimental interpreter, Python gives me:

• Fast iteration
• Easy testing
• Simple data structures
• Classes for AST nodes
• Dictionaries for environments
• Straightforward exception handling

I can focus on:

language design
+
lexing
+
parsing
+
AST
+
interpretation

Enter fullscreen mode Exit fullscreen mode

without initially having to solve low-level implementation problems.

That doesn’t mean Python is necessarily the final implementation language.

It means Python is a practical place to start.

12. Testing Is Not Optional

A language implementation can break very easily.

For example:

Lexer change
     ↓
Token change
     ↓
Parser breaks
     ↓
AST changes
     ↓
Interpreter breaks

Enter fullscreen mode Exit fullscreen mode

That’s why NexPro includes tests.

The project has a dedicated:

tests/

Enter fullscreen mode Exit fullscreen mode

directory for language behavior.

As NexPro grows, I want the test suite to cover:

Lexer
  ↓
Parser
  ↓
AST
  ↓
Interpreter
  ↓
Runtime
  ↓
CLI

Enter fullscreen mode Exit fullscreen mode

The goal is to make every new language feature measurable and reproducible.

13. What NexPro Can Do Today

The current implementation is intentionally small.

The project has already moved beyond a source-code mockup into an executable interpreter with components for:

✓ CLI execution
✓ Lexical analysis
✓ Tokens
✓ Parsing
✓ AST representation
✓ Variables
✓ Assignment
✓ Strings
✓ Numbers
✓ Basic expressions
✓ Interpretation
✓ Runtime structure
✓ Error handling structure
✓ Tests

Enter fullscreen mode Exit fullscreen mode

The exact supported syntax will continue changing as the language develops.

That distinction matters.

I’m documenting what exists rather than presenting planned features as completed features.

14. What NexPro Cannot Do Yet

This is equally important.

NexPro is not yet:

❌ A production compiler
❌ A Python replacement
❌ A high-performance language
❌ A mature standard library
❌ A complete ecosystem
❌ A stable 1.0 language

Enter fullscreen mode Exit fullscreen mode

There are still major areas to develop:

Functions
Loops
Conditionals
Collections
Modules
Type system
Standard library
REPL
Tooling
Debugger
Package management
Performance

Enter fullscreen mode Exit fullscreen mode

And that’s exactly why I consider it an interesting engineering project.

15. Roadmap

My roadmap is currently divided into three stages.

Stage 1 — Language Core

✓ Lexer
✓ Tokens
✓ Parser
✓ AST
✓ Interpreter
✓ Variables
✓ Basic expressions
✓ CLI

→ Conditionals
→ Loops
→ Functions
→ Collections

Enter fullscreen mode Exit fullscreen mode

Stage 2 — Developer Experience

→ REPL
→ Better diagnostics
→ Formatter
→ Documentation
→ VS Code syntax highlighting
→ Debugging tools
→ Improved testing

Enter fullscreen mode Exit fullscreen mode

Stage 3 — Advanced Runtime

→ Modules
→ Standard library
→ Package system
→ Bytecode
→ Performance improvements
→ Possible compilation

Enter fullscreen mode Exit fullscreen mode

These are goals, not completed features.

16. Why This Project Matters to Me

The biggest result of NexPro isn’t the syntax.

It’s the understanding.

Before building a language, I knew the words:

Lexer
Parser
AST
Interpreter
Runtime

Enter fullscreen mode Exit fullscreen mode

After implementing them, I started seeing programming languages as pipelines.

When I write:

say 10 + 20

Enter fullscreen mode Exit fullscreen mode

I can now mentally see:

SOURCE
  ↓
TOKENS
  ↓
SYNTAX
  ↓
AST
  ↓
EVALUATION
  ↓
RUNTIME
  ↓
30

Enter fullscreen mode Exit fullscreen mode

That shift in understanding is the real reason I started NexPro.

17. What I Want to Investigate Next

The next interesting question isn’t just:

“What syntax should NexPro have?”

It’s:

“How far can I take a language that started as a Python interpreter?”

Some questions I want to explore:

Can NexPro have a real type system?

For example:

age: number = 20
name: string = "Probal"

Enter fullscreen mode Exit fullscreen mode

Can it compile to bytecode?

Instead of:

Source → AST → Interpreter

Enter fullscreen mode Exit fullscreen mode

potentially:

Source
  ↓
AST
  ↓
Bytecode
  ↓
Virtual Machine

Enter fullscreen mode Exit fullscreen mode

Can the language have its own package system?

Something like:

import math

Enter fullscreen mode Exit fullscreen mode

Can it eventually have an IDE experience?

For example:

NexPro
 ├── Language Server
 ├── Formatter
 ├── Debugger
 └── VS Code Extension

Enter fullscreen mode Exit fullscreen mode

Those are future experiments.

18. The Repository Is the Evidence

Rather than asking readers to trust a description, I want the repository to be the evidence.

You can inspect:

lexer.py
parser.py
ast.py
interpreter.py
runtime.py
tokens.py
errors.py
cli.py
tests/
examples/

Enter fullscreen mode Exit fullscreen mode

You can run the examples.

You can inspect the implementation.

You can open issues.

You can suggest changes.

You can fork it.

You can try to break it.

That’s how I want NexPro to evolve.

19. Open Source Means Feedback

I’m especially interested in feedback from developers who have experience with:

Compilers
Interpreters
Programming Languages
Parser Design
ASTs
Python
Developer Tooling
Language Design

Enter fullscreen mode Exit fullscreen mode

Some questions I’d genuinely like feedback on:

Is the current architecture reasonable for an interpreter?

Which language feature should be implemented next?

Should NexPro remain dynamically typed or eventually introduce static typing?

Should the next major milestone be a REPL, functions, or a bytecode VM?

What architectural mistakes should I fix before the project becomes larger?

20. Final Result

NexPro started with a simple experiment:

Can I build a programming language?

Enter fullscreen mode Exit fullscreen mode

The answer isn’t:

“I built the next Python.”

That’s not what this project is.

The more accurate answer is:

I built a small, executable programming language implementation, and I’m using it to explore how languages are designed and executed.

And that’s already taught me more about programming-language internals than simply reading about them.

The journey currently looks like:

          NEXPRO

       Source Code
            │
            ▼
          Lexer
            │
            ▼
          Tokens
            │
            ▼
          Parser
            │
            ▼
           AST
            │
            ▼
       Interpreter
            │
            ▼
         Runtime
            │
            ▼
          Output

Enter fullscreen mode Exit fullscreen mode

The interesting part is that this is only the beginning.

🚀 Try NexPro

If you want to inspect the implementation, experiment with the syntax, or contribute:

🔗 GitHub: https://github.com/probal2005/NexPro

If you find something wrong, open an issue.

If you have an idea, tell me.

If you want to experiment, fork it.

And if you know programming-language implementation better than I do, please tell me what I’m doing wrong.

That’s exactly the kind of feedback I want.

NexPro is small today.

But every programming language starts somewhere.

And this is mine.

What would you build next?

Functions, a REPL, a type system, or a bytecode VM?

programming #python #programminglanguages #compiler #interpreter #opensource #softwareengineering #parsing #ast #developer