INDEPENDENT SYSTEMS RESEARCH

What if an LLM could run within the memory you have?

Exploring a Windows-native inference runtime that streams model weights from Safetensors and executes transformer layers on a consumer GPU.

WINDOWS · C# · DIRECT3DRESEARCH PROTOTYPE
MEMORY WINDOWCONSTRAINED / 6 GB
GPU MEMORYFINITE RESOURCE
6GB
MODEL WEIGHTSWORKING SET
LOAD · COMPUTE · RELEASE
01THE MEMORY PROBLEMSCROLL TO INVESTIGATE ↓

01 / THE PROBLEM

Inference has a memory problem.

Existing inference paths often assume generous hardware or a heavyweight software stack. That leaves a gap between a model that exists and a model that can actually run locally.

01

Heavyweight setup

A Python runtime can exceed 1 GB before model weights enter the picture; CUDA and cuDNN add more dependencies to the setup.

RUNTIME OVERHEAD
02

Conversion friction

ONNX export and conversion can add steps between model weights and execution, complicating a direct, inspectable path.

TOOLING FRICTION
03

Peak memory pressure

Model weights, intermediate activations and a growing KV cache compete for the same limited GPU memory.

MEMORY BUDGET
04

Consumer hardware

A Windows Surface Book with a 6 GB NVIDIA GPU makes those constraints concrete—and measurable.

THE TEST ENVIRONMENT
↳

The research question: Can a model execute directly from Safetensors on Windows without requiring every weight to occupy VRAM at once?

02 / THE RESEARCH MISSION

Start with the constraint.
Not the framework.

Existing inference solutions assumed that entire models had to fit into GPU memory. On a Windows Surface Book equipped with a 6 GB NVIDIA GPU, even small language models required heavyweight software stacks, conversion pipelines, or cloud-based execution.

This project explores a different approach: running transformer models directly from Safetensors files on Windows using C# and Direct3D compute shaders, while loading only the tensors required for the current layer.

THE HYPOTHESIS UNDER INVESTIGATION
01Load only what is needed.
02Execute layer by layer.
03Keep memory consumption under control.

03 / THE APPROACH

Make memory a managed resource.

The runtime treats GPU memory as a working window, not a destination for the entire model.

ATRADITIONAL LOADMODEL-WIDE
Load full model
→
Occupy VRAM
→
Inference

Simple execution path, but peak residency can exceed the available GPU memory.

BLAYER STREAMINGWORKING-SET
Load layer N
→
Compute
→
Release
→
Load N+1

Trade persistent residency for explicit weight movement and per-layer scheduling.

+

Potential upsideLower resident weight memory; direct control over when tensors enter and leave the GPU.

±

Tradeoffs to measureTransfer overhead, synchronization, buffer reuse and end-to-end token throughput.

DESIGN PRINCIPLES

Close to the hardware.

01Native WindowsBuild for the device in front of you.
02C# runtimeA managed application, explicit GPU work.
03Direct3D computeCompute shaders for tensor operations.
04Direct SafetensorsRead model weights without a conversion step.
05Layer streamingMove only the current working set.
06Dynamic KV cacheTrack context growth against a memory budget.

04 / THE PROJECT

A small runtime, end to end.

The work spans file formats, GPU execution and token generation—not a new model architecture.

COMPONENT 0101

Safetensors reader

Inspect the file header, resolve tensor metadata, and read the data needed by each execution step.

  • Header parsing
  • Tensor metadata & offsets
  • Targeted tensor access
  • Streaming reads
Research note ↗
COMPONENT 0202

Direct3D runtime

Allocate GPU buffers and execute the operations required for transformer inference.

  • GPU buffer lifecycle
  • Compute shader dispatch
  • Matrix multiplication
  • Tensor operations
Research note ↗
COMPONENT 0303

Token generator

Connect model components into a generation path with a memory-aware cache strategy.

  • Embedding & attention
  • Feed-forward network
  • Sampling
  • KV cache management
Research note ↗

05 / FIELD NOTES

Research, in the open.

Engineering notes on the decisions, constraints and measurements behind the runtime.

Notes marked “Planned” are research outlines—not completed results or published claims.

06 / MEASUREMENT

Evidence before claims.

MEASUREMENTS IN PROGRESS

Small Qwen3-8B and Qwen2.5 precision experiments compare answers across INT4 group sizes against BF16 and FP8. The separate streaming memory and throughput comparison below is still to be measured.

TEST DIMENSIONBASELINESTREAMING RUNTIMESTATUS
Target hardwareSurface Book · Intel CPUWindows · NVIDIA GPU · 6 GBConfigured
ModelQwen 1.5BQwen 1.5BTo measure
MemoryPeak GPU memoryResident weights + KV cacheTo measure
ThroughputPrompt processing · tokens/sGeneration · tokens/sTo measure
01 / HONEST BASELINES02 / REPRODUCIBLE RUNS03 / PUBLISHED METHODOLOGY

07 / ABOUT THE RESEARCHER

Building AI systems from first principles.

My work focuses on the practical execution of large language models on local hardware.

I hold a Ph.D. in Physics and have worked in software engineering, system architecture and technology leadership for more than three decades. Throughout my career I have been involved in building complex software systems, distributed platforms and data-driven applications.

My interest in neural networks goes back to the late 1980s, when I attended a lecture series on them in the physics department. Computing power was far more limited than it is today, so my exam for the course focused on implementing neural networks on VLSI chips—including learning directly on the chip.

Neural networks are surprisingly physical 🙂 Although inspired by biological neurons, the theoretical understanding of many neural networks owes a great deal to statistical physics and spin-glass theory.

My interest in artificial intelligence predates the current wave of large language models. Over the years I have implemented and experimented with various forms of neural networks and machine learning approaches, with a particular interest in understanding the mathematical foundations behind their behavior and limitations.

Once computers became powerful enough, I started implementing neural networks myself, including feedforward computation and backpropagation. I even derived the partial derivatives by hand—back then, you couldn't simply Google them 🙂. I kept the paper: about 30 A4 pages filled with equations. While working through the derivation, I realized that tensor algebra, including Kronecker products, could simplify many of the expressions. Read my original backpropagation derivation (PDF).

When ready-made tensor backends such as TensorFlow and PyTorch became available, I set those ideas aside and worked with TensorFlow. People often say, “It's all Python, so it must be slow.” In practice, the computationally intensive work runs in highly optimized C++ and CUDA code.

My view is that major gains from backend optimizations alone are becoming harder to achieve. Many of the obvious improvements have already been realized, leaving perhaps only a few percent. When training modern models takes months on tens of thousands of GPUs, a small efficiency gain may not fundamentally change the picture. Inference—the feedforward pass—faces a similar challenge.

More recently, my research has focused on transformer-based language models, inference runtimes and efficient execution strategies for resource-constrained hardware. A central motivation is to reduce unnecessary dependencies on large cloud infrastructures and heavyweight software stacks. I am particularly interested in approaches that enable modern AI models to run efficiently on local, consumer-grade systems.

This research led to the development of a Windows-native inference runtime written in C#, capable of loading Safetensors models directly and executing transformer workloads through Direct3D compute shaders. The runtime employs layer streaming and memory-aware KV-cache management to operate within the limitations of consumer GPUs.

I believe that the best way to understand complex AI systems is to build them from first principles. Consequently, much of my work involves implementing core components directly, including tensor loading, memory management, inference execution and optimization strategies.

AREAS OF FOCUS

Research Interests

  • Large Language Model Inference
  • Transformer Architectures
  • Memory-Constrained AI Systems
  • Local AI Execution
  • Safetensors Internals
  • Direct3D GPU Computing
  • State Space Models (Mamba)
  • AI Systems Architecture
  • Distributed Systems
  • Applied Machine Learning

My long-term goal is to make advanced AI systems more accessible, transparent and independent of heavyweight cloud environments by exploring efficient local inference architectures.

Content provided by the researcher; layout and textual and grammatical editing assisted by AI.