PySHDL Internals

PySHDL is the Python implementation of the SHDL toolchain. This page documents its internal architecture and module structure.

Package Structure

shdl/
├── __init__.py          # Public API exports
├── errors.py            # Shared exception classes
├── flattener/           # Expanded SHDL → Base SHDL
│   ├── lexer.py         # Tokenizer
│   ├── parser.py        # Parser
│   ├── ast.py           # AST node definitions
│   └── flattener.py     # Flattening logic
├── compiler/            # Base SHDL → C code
│   ├── lexer.py         # Base SHDL tokenizer
│   ├── parser.py        # Base SHDL parser
│   ├── ast.py           # Base SHDL AST
│   ├── analyzer.py      # Semantic analysis
│   ├── codegen.py       # C code generation
│   ├── compiler.py      # Compilation orchestration
│   └── cli.py           # Command-line interface
└── driver/              # Python ↔ C interface
    ├── circuit.py       # Circuit class
    └── exceptions.py    # Driver-specific errors

Module Overview

Flattener (shdl.flattener)

The flattener transforms Expanded SHDL (with generators, expanders, hierarchy) into Base SHDL (flat, primitive-only).

Key Components

Module	Purpose
lexer.py	Tokenizes Expanded SHDL source code
parser.py	Parses tokens into an AST
ast.py	Defines AST node classes for Expanded SHDL
flattener.py	Implements the 5-phase flattening algorithm

Public API

from SHDL import parse_file, Flattener, flatten_file

# Parse SHDL file into AST
module = parse_file("adder.shdl")

# Create flattener and flatten a component
flattener = Flattener(include_paths=["./components"])
base_shdl = flattener.flatten(module, "Adder16")

# Or use the convenience function
base_shdl = flatten_file("adder.shdl", component="Adder16")

Compiler (shdl.compiler)

The compiler transforms Base SHDL into optimized C code.

Key Components

Module	Purpose
lexer.py	Tokenizes Base SHDL
parser.py	Parses Base SHDL into AST
ast.py	Base SHDL AST node definitions
analyzer.py	Semantic analysis, connection graph building
codegen.py	C code generation with bit-packing
compiler.py	Orchestrates the compilation pipeline

Public API

from SHDL import compile_base_shdl, compile_shdl_file

# Compile Base SHDL string to C
c_code = compile_base_shdl(base_shdl_source)

# Compile SHDL file (handles flattening automatically)
c_code = compile_shdl_file("adder.shdl", component="Adder16")

Driver (shdl.driver)

The driver provides a high-level Python interface for compiled circuits.

Key Components

Module	Purpose
circuit.py	Circuit class for circuit simulation
exceptions.py	Driver-specific exception classes

Public API

from SHDL import Circuit

# Create and use a circuit
circuit = Circuit("adder16.shdl")
circuit.poke("A", 100)
circuit.poke("B", 50)
circuit.step()
result = circuit.peek("Sum")

# Use as context manager
with Circuit("adder16.shdl") as circuit:
    circuit["A"] = 100
    circuit["B"] = 50
    circuit.step()
    print(circuit["Sum"])

Error Handling

All SHDL errors inherit from a base SHDLError:

from SHDL import SHDLError, LexerError, ParseError, FlattenerError

try:
    circuit = Circuit("broken.shdl")
except LexerError as e:
    print(f"Lexer error: {e}")
except ParseError as e:
    print(f"Parse error: {e}")
except FlattenerError as e:
    print(f"Flattener error: {e}")
except SHDLError as e:
    print(f"SHDL error: {e}")

Error Types

Error	Source	When
LexerError	Lexer	Invalid tokens, unexpected characters
ParseError	Parser	Invalid syntax
FlattenerError	Flattener	Undefined components, invalid ranges
CompilationError	Driver	C compilation failures
SimulationError	Driver	Runtime simulation errors
SignalNotFoundError	Driver	Invalid signal name in poke/peek

AST Node Types

Expanded SHDL AST

from SHDL import (
    Module, Component, Port, Instance, Constant, 
    Connection, Signal, IndexExpr, Generator, Import
)

# Module contains components
module = parse_file("adder.shdl")
for component in module.components:
    print(f"Component: {component.name}")
    
    # Access ports
    for port in component.input_ports:
        print(f"  Input: {port.name}[{port.width}]")
    
    # Access instances
    for instance in component.instances:
        print(f"  Instance: {instance.name}: {instance.component_type}")

Base SHDL AST

The compiler uses a simpler AST for Base SHDL:

from SHDL.compiler import BaseSHDLParser

parser = BaseSHDLParser()
module = parser.parse(base_shdl_source)

for component in module.components:
    print(f"Primitive instances: {len(component.instances)}")

Semantic Analysis

The SemanticAnalyzer validates Base SHDL and builds a connection graph:

from SHDL.compiler import analyze

result = analyze(component)

if result.has_errors:
    for error in result.errors:
        print(f"Error: {error}")
else:
    print(f"Valid circuit with {len(result.gates)} gates")

Analysis Result

@dataclass
class AnalysisResult:
    component: Component
    gates: dict[str, GateInfo]      # Instance name → gate info
    connections: list[Connection]    # All connections
    input_ports: list[PortInfo]     # Input port metadata
    output_ports: list[PortInfo]    # Output port metadata
    errors: list[Error]             # Semantic errors
    warnings: list[Warning]         # Warnings

Code Generation

The CodeGenerator produces optimized C code:

from SHDL.compiler import generate

c_code = generate(analysis_result)

Generated Code Features

1. Bit-packed state - Gates grouped by type in 64-bit words
2. Lane assignment - Each gate instance gets a unique bit position
3. SIMD-style evaluation - Single operations evaluate many gates
4. Output extraction - Functions to extract multi-bit outputs

Extending PySHDL

Adding a New Primitive

To add a new primitive gate type:

1. Add to PRIMITIVE_GATES in flattener/flattener.py
2. Add to grammar in compiler/parser.py
3. Add code generation in compiler/codegen.py

Custom Include Paths

from SHDL import Circuit

circuit = Circuit(
    "main.shdl",
    include_paths=["./components", "./lib"]
)

Keeping Generated Libraries

By default, compiled libraries are deleted. To keep them:

circuit = Circuit(
    "adder.shdl",
    keep_library=True,
    library_dir="./build"
)
print(f"Library at: {circuit.library_path}")

Performance Considerations

Compilation Caching

Currently, PySHDL recompiles circuits on each instantiation. For repeated use, consider:

# Compile once, reuse
circuit = Circuit("adder.shdl", keep_library=True, library_dir="./cache")

# Later, load the cached library
# (Not yet implemented - future feature)

Optimization Levels

Control C compiler optimization:

circuit = Circuit(
    "adder.shdl",
    optimize=3,  # -O3 (default)
    cc="clang"   # Compiler to use
)

Large Circuits

For circuits with many gates:

• Bit-packing handles up to 64 gates per chunk efficiently
• Beyond 64 gates of one type, multiple chunks are used
• Memory usage scales linearly with gate count