Lumai backs specialised hardware for AI prefill workloads

Lumai has set out its case for splitting AI inference workloads and using specialised hardware for the prefill stage. The Oxford company argues that growing context loads are shifting pressure to the processing that happens before a model generates its first token.

It contends that AI infrastructure has focused too heavily on token generation, even as longer context windows, retrieval-augmented generation, multimodal inputs and agentic workflows increase the amount of computation required earlier in the process.

That argument centres on the distinction between prefill and decode, two stages of AI inference with different technical demands, according to Lumai. Prefill handles the input context before generation starts and is dominated by parallel matrix multiplication. Decode generates output tokens and depends more on memory bandwidth and data movement.

Lumai says the industry is already moving towards separating those workloads into different infrastructure pools. It argues that this shift should extend to the underlying compute architecture used for each stage.

"If the workloads are fundamentally different, it makes sense to stop asking the same hardware to do both jobs," said Phil Burr, head of product at Lumai.

"Disaggregating prefill and decode is an important step. But the bigger opportunity is to match the compute architecture to the workload," Burr said.

Context pressure

Lumai argues that the economics of inference are changing as applications process larger volumes of information before any output appears. In systems that rely on long prompts, retrieved knowledge, codebases, document collections or multimodal data, that initial processing can become a larger share of overall compute demand.

The issue can intensify in agentic workflows, where a single user request may trigger several model calls and each stage carries forward accumulated context. In that model, more of a data centre's available power may be consumed by prefill rather than token generation.

The company frames this as both a technical and economic constraint. If prefill consumes more of the power budget, operators may get less useful inference capacity from the same infrastructure footprint, while GPUs spend more time processing context instead of generating tokens.

Lumai identifies three main challenges for operators: power becomes the limiting factor, GPU capacity is diverted from work for which it is better suited, and the cost of long-context or agentic applications becomes harder to sustain.

"The real issue is what happens at scale," Burr said.

"Power is fixed, GPU capacity is finite, and every interaction adds more context to process. We have gotten to the point where the economics of inference comes down to the efficiency of the hardware running it," he said.

Optical approach

Lumai is using that argument to support its optical computing design, Iris Nova. The system uses light rather than electricity for the matrix multiplications associated with prefill workloads, which it says allows each vector-matrix multiplication to be completed in a single optical cycle.

The point, Lumai says, is not simply that optical computing can carry out matrix multiplication, but that it could allow prefill to be treated as a dedicated infrastructure task rather than one handled by general-purpose silicon.

In testing cited by Lumai, Iris Nova ran billion-parameter large language models in real time and showed about 10 times more compute per watt than GPU-based equivalents on prefill workloads. The company says a more efficient prefill tier could make longer context windows and more complex agentic workflows more practical without a matching rise in power and compute requirements.

It also says moving prefill work onto dedicated hardware could free GPU resources for the decode stage, improving the productivity of existing infrastructure without requiring operators to replace entire fleets.

Lumai was spun out of optics research at the University of Oxford in 2021. It is part of a broader group of firms trying to apply optical or photonic methods to AI computing as data centres face tighter power limits and rising demand for inference.

"Our view is not that every accelerator needs to be replaced - it is that we should stop asking every accelerator to solve every problem," Burr said.

"The future AI data center will be increasingly heterogeneous, with different technologies optimized for different stages of inference," he said.