Heavyweight setup
A Python runtime can exceed 1 GB before model weights enter the picture; CUDA and cuDNN add more dependencies to the setup.
RUNTIME OVERHEADINDEPENDENT SYSTEMS RESEARCH
Exploring a Windows-native inference runtime that streams model weights from Safetensors and executes transformer layers on a consumer GPU.
01 / THE PROBLEM
Existing inference paths often assume generous hardware or a heavyweight software stack. That leaves a gap between a model that exists and a model that can actually run locally.
A Python runtime can exceed 1 GB before model weights enter the picture; CUDA and cuDNN add more dependencies to the setup.
RUNTIME OVERHEADONNX export and conversion can add steps between model weights and execution, complicating a direct, inspectable path.
TOOLING FRICTIONModel weights, intermediate activations and a growing KV cache compete for the same limited GPU memory.
MEMORY BUDGETA Windows Surface Book with a 6 GB NVIDIA GPU makes those constraints concrete—and measurable.
THE TEST ENVIRONMENTThe research question: Can a model execute directly from Safetensors on Windows without requiring every weight to occupy VRAM at once?
02 / THE RESEARCH MISSION
Existing inference solutions assumed that entire models had to fit into GPU memory. On a Windows Surface Book equipped with a 6 GB NVIDIA GPU, even small language models required heavyweight software stacks, conversion pipelines, or cloud-based execution.
This project explores a different approach: running transformer models directly from Safetensors files on Windows using C# and Direct3D compute shaders, while loading only the tensors required for the current layer.
03 / THE APPROACH
The runtime treats GPU memory as a working window, not a destination for the entire model.
Simple execution path, but peak residency can exceed the available GPU memory.
Trade persistent residency for explicit weight movement and per-layer scheduling.
Potential upsideLower resident weight memory; direct control over when tensors enter and leave the GPU.
Tradeoffs to measureTransfer overhead, synchronization, buffer reuse and end-to-end token throughput.
DESIGN PRINCIPLES
04 / THE PROJECT
The work spans file formats, GPU execution and token generation—not a new model architecture.
Inspect the file header, resolve tensor metadata, and read the data needed by each execution step.
Allocate GPU buffers and execute the operations required for transformer inference.
Connect model components into a generation path with a memory-aware cache strategy.
05 / FIELD NOTES
Engineering notes on the decisions, constraints and measurements behind the runtime.
Notes marked “Planned” are research outlines—not completed results or published claims.
06 / MEASUREMENT
Small Qwen3-8B and Qwen2.5 precision experiments compare answers across INT4 group sizes against BF16 and FP8. The separate streaming memory and throughput comparison below is still to be measured.
07 / ABOUT THE RESEARCHER
My work focuses on the practical execution of large language models on local hardware.
I hold a Ph.D. in Physics and have worked in software engineering, system architecture and technology leadership for more than three decades. Throughout my career I have been involved in building complex software systems, distributed platforms and data-driven applications.
My interest in neural networks goes back to the late 1980s, when I attended a lecture series on them in the physics department. Computing power was far more limited than it is today, so my exam for the course focused on implementing neural networks on VLSI chips—including learning directly on the chip.
Neural networks are surprisingly physical 🙂 Although inspired by biological neurons, the theoretical understanding of many neural networks owes a great deal to statistical physics and spin-glass theory.
My interest in artificial intelligence predates the current wave of large language models. Over the years I have implemented and experimented with various forms of neural networks and machine learning approaches, with a particular interest in understanding the mathematical foundations behind their behavior and limitations.
Once computers became powerful enough, I started implementing neural networks myself, including feedforward computation and backpropagation. I even derived the partial derivatives by hand—back then, you couldn't simply Google them 🙂. I kept the paper: about 30 A4 pages filled with equations. While working through the derivation, I realized that tensor algebra, including Kronecker products, could simplify many of the expressions. Read my original backpropagation derivation (PDF).
When ready-made tensor backends such as TensorFlow and PyTorch became available, I set those ideas aside and worked with TensorFlow. People often say, “It's all Python, so it must be slow.” In practice, the computationally intensive work runs in highly optimized C++ and CUDA code.
My view is that major gains from backend optimizations alone are becoming harder to achieve. Many of the obvious improvements have already been realized, leaving perhaps only a few percent. When training modern models takes months on tens of thousands of GPUs, a small efficiency gain may not fundamentally change the picture. Inference—the feedforward pass—faces a similar challenge.
More recently, my research has focused on transformer-based language models, inference runtimes and efficient execution strategies for resource-constrained hardware. A central motivation is to reduce unnecessary dependencies on large cloud infrastructures and heavyweight software stacks. I am particularly interested in approaches that enable modern AI models to run efficiently on local, consumer-grade systems.
This research led to the development of a Windows-native inference runtime written in C#, capable of loading Safetensors models directly and executing transformer workloads through Direct3D compute shaders. The runtime employs layer streaming and memory-aware KV-cache management to operate within the limitations of consumer GPUs.
I believe that the best way to understand complex AI systems is to build them from first principles. Consequently, much of my work involves implementing core components directly, including tensor loading, memory management, inference execution and optimization strategies.
AREAS OF FOCUS
My long-term goal is to make advanced AI systems more accessible, transparent and independent of heavyweight cloud environments by exploring efficient local inference architectures.