Hardware-in-the-Loop: The Architecture Search Behind LFM2

The last two years have produced a wave of efficient hybrid architectures for language models.^{1 1}Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models ¹Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models^{2 2}IBM Granite 4.0: hyper-efficient, high performance hybrid models for enterprise ²IBM Granite 4.0: hyper-efficient, high performance hybrid models for enterprise^{3 3}Kimi Linear: An Expressive, Efficient Attention Architecture ³Kimi Linear: An Expressive, Efficient Attention Architecture^{4 4}Mamba-3: Improved Sequence Modeling using State Space Principles ⁴Mamba-3: Improved Sequence Modeling using State Space Principles They combine convolutions, state-space models, and attention in varying proportions, and each combination arrives with an argument about what each component should contribute. What the papers rarely document is the design process itself. It is usually unclear whether an architecture was searched systematically or assembled from intuition, and ablations of the individual submodules are largely absent. The question hanging over the genre is: how much does each component actually contribute?

From liquid networks to LFMs

Liquid AI is a Cambridge, Massachusetts-based company founded in 2023. Its founders debuted with Liquid Neural Networks: compact, adaptive networks inspired by the neural architecture of a microscopic worm (C. elegans), designed to adjust their behavior to inputs over time, much as biological neurons do.

That efficiency-first orientation carried into their language-model work. The first Liquid Foundation Models, released in 2024, departed from the Transformer architecture that still dominates the field. The LFM2 family followed in July 2025: a hybrid design combining gated short convolutions for local sequence mixing with a minority of Grouped Query Attention blocks for long-range interaction. We are interested here in what sets LFM2 apart from the other hybrids: the systematic search that produced that combination, detailed in the paper published in December 2025.

Why proxy signals fail

Neural Architecture Search is not new. The idea is straightforward: define a space of possible architectures, then let an optimization algorithm find the best one according to some performance metric. The catch has always been cost. Training every candidate to convergence is unaffordable, so NAS traditionally leans on proxy signals — perplexity as a stand-in for quality, cache size as a stand-in for efficiency — to rank candidates cheaply.

Liquid AI had already been down that road with STAR, their earlier synthesis framework. The conclusion they drew from that experience shapes everything in the LFM2 paper: proxy signals are not sufficient. They fail worst exactly where an edge-deployment lab cares most, namely predicting how an architecture behaves on real hardware. Theoretical FLOPs and cache-size arithmetic do not capture what a phone’s memory hierarchy, its runtime, and its scheduler will do to a particular composition of blocks.

For LFM2 they inverted the approach: optimize directly on an internal suite of more than fifty evaluations (spanning knowledge recall, reasoning, multilingual robustness, tool use, math, and long-context tasks), together with latency and peak memory measured on the target devices themselves.

The search space

In optimization terms, a search space is every assignment of the decision variables that respects the problem’s constraints: all 0–1 vectors of length 100, all valid schedules, all graphs with n nodes. In NAS the assignments are neural network architectures. For LFM2, that means decoder-only stacks built from several block families: local-context and subquadratic blocks,^{5 5}Gated short convolutions, sliding-window attention, linear attention variants (GLA, HGRN2), state-space models (S4, Liquid-S4, S5, RTF, Mamba, Mamba2), and Liquid Time-Constant networks (CfC). ⁵Gated short convolutions, sliding-window attention, linear attention variants (GLA, HGRN2), state-space models (S4, Liquid-S4, S5, RTF, Mamba, Mamba2), and Liquid Time-Constant networks (CfC). global-context blocks,^{6 6}Grouped-Query Attention (GQA) with varying group counts and head dimensions, stabilized with QK-Norm. ⁶Grouped-Query Attention (GQA) with varying group counts and head dimensions, stabilized with QK-Norm. and position-wise blocks,^{7 7}SwiGLU FFNs whose expansion ratio is itself a search variable rather than fixed by hand. ⁷SwiGLU FFNs whose expansion ratio is itself a search variable rather than fixed by hand. plus layout^{8 8}Interleaving patterns of block types, total block counts under fixed parameter budgets, plus options for weight sharing and cache reuse across layers. ⁸Interleaving patterns of block types, total block counts under fixed parameter budgets, plus options for weight sharing and cache reuse across layers. and MoE options.^{9 9}Per-layer sparse FFNs with varying width and expert granularity. ⁹Per-layer sparse FFNs with varying width and expert granularity.

We want to pause on how broad this space is. It contains pure Transformers, pure state-space models, and essentially every hybrid pattern the recent literature has proposed. Nothing about the setup favors the architecture that eventually won.

Hardware in the loop

The paper’s central move is what happens to each candidate before quality is ever measured. Every architecture is exported and profiled directly on a Samsung Galaxy S24 Ultra (Qualcomm Snapdragon 8 Gen 3 SoC) and an AMD Ryzen HX 370 laptop CPU, on the release runtimes, recording four quantities:

• Time to first token (TTFT). How long the device takes to produce the first output token.
• Prefill throughput. Prompt-processing speed, in tokens per second.
• Decode latency. Milliseconds per generated token, at p50 and p95.^{10 10}p50: median decode latency — the "typical" speed a user experiences. p95: the slowest 5% of tokens fall above this threshold, capturing occasional spikes from cache pressure or scheduling jitter. ¹⁰p50: median decode latency — the "typical" speed a user experiences. p95: the slowest 5% of tokens fall above this threshold, capturing occasional spikes from cache pressure or scheduling jitter.
• Peak memory. Measured at 4K and 32K context windows.

Candidates that violate any device-side budget (TTFT, decode latency, or peak memory) are discarded outright, and never reach the quality evaluations. The expensive, noisy question of whether an architecture is smart is asked only about architectures already proven deployable.

Picking winners on the Pareto frontier

We are left with the surviving candidates, and the question becomes which one is best. There is no single answer, because quality, latency, and memory are competing objectives; an architecture that scores better on evaluations usually pays for it in latency or memory. This is the setting Pareto optimization was designed for. A candidate is Pareto optimal if no other candidate improves on it in every objective simultaneously. The set of all such candidates forms the Pareto frontier, and any point on the frontier is a defensible choice. The figure below illustrates the selection procedure:

In practice the paper ranks surviving candidates by hypervolume improvement — how much a new candidate expands the volume of objective space dominated by the current frontier. This collapses the three-way trade-off into a single scalar, making candidates directly comparable. Candidates that advance the frontier are carried forward; the rest are dropped.

What the search found

We come now to what all of this machinery converged on, and it is the paper’s most striking result. Across every size target, the search repeatedly selected the same minimal hybrid: a majority of inexpensive gated short convolution blocks, interleaved with a small minority of GQA blocks, and nothing else. State-space models, linear attention variants, Mamba, and the rest of the subquadratic menagerie were all valid candidates. Under the targeted on-device CPU budgets and runtimes, none of them improved the quality–latency–memory trade-off over the simplest possible combination.

Closing thoughts

We read the LFM2 search as, in effect, the broad ablation of the hybrid design space that the literature has been missing. The field has spent considerable effort on increasingly intricate hybrids, motivated by theoretical arguments about what each component should contribute. Liquid AI bypasses the arguments: rather than engineering a design from intuition and justifying it afterward, they let the hardware and the downstream tasks decide.

What the hardware and the tasks said is that once a handful of global attention layers handle long-range retrieval, inexpensive gated short convolutions appear to be sufficient for everything else. The verdict is specific to the regime in the loop (on-device CPU budgets and release runtimes) and may not transfer elsewhere. Within that regime, however, complexity was not a necessity. We would have no principled way of knowing that without a systematic search, which is, perhaps, the paper’s real contribution: less the architecture than the receipt.

@misc{desantis2026lfm2architecturesearch,
  author = {Marco De Santis},
  title = {Hardware-in-the-Loop: The Architecture Search Behind LFM2},
  year = {2026},
  month = mar,
  url = {https://marcodsn.me/blog/lfm2-architecture-search}
}