IFM K2 Horizon Deep Dive: Inside Mixture-of-Value Attention (MoVA), 2,880 Value Matrices, and the 36B-A4B Sparse Inference Frontier

Traditional FFN Mixture-of-Experts (Mixtral / DeepSeek):
[ Input Token ] ──> [ Dense Self-Attention (Q, K, V) ] ──> [ Router ] ──> [ Top-2 FFN Experts ] ──> [ Output ]
                     (All 100% compute active)                     (Sparse compute)

K2 Horizon Mixture-of-Value Attention (MoVA):
                    ┌──> Dense Query (Q) ────────────────────────┐
[ Input Token ] ────┼──> Dense Key (K) ──────────────────────────┼──> [ Attention Dot Product ]
                    └──> [ Value Router ] ──> [ Top-4 of 64      │                   │
                                              Value Matrices ] ──┘                   ▼
                                              (Ultra-Sparse V)          [ Scaled Weighted Sum ]
                                                                                     │
                                                                                     ▼
                                                                        [ Compact Shared FFN ]
                                                                                     │
                                                                                     ▼
                                                                                 [ Output ]

Section 4: The KV-Cache Dilemma & Memory Bandwidth Analysis

While MoVA delivers unprecedented compute efficiency during prefill and generation, it introduces a critical systems engineering trade-off: How does one cache Key-Value states when the Value projection is conditioned on dynamic routing?

In standard Multi-Query Attention (MQA) or Grouped-Query Attention (GQA), the KV cache stores static vectors:

$\mathbf{K}_{\text{cached}} = \mathbf{X} \mathbf{W}k, \quad \mathbf{V}{\text{cached}} = \mathbf{X} \mathbf{W}_v$

The Two Architectural Caching Strategies for MoVA

1. Option A: Post-Routing Projected Value Cache (Default vLLM Integration)

   • The inference engine computes the dynamic value projection $\mathbf{v}_t$ at token generation time and stores the resulting projected vector in standard PagedAttention memory pools.
   • Advantage: Preserves exact compatibility with FlashAttention-2/3 kernels; attention dot products execute with zero routing overhead on cached tokens.
   • Disadvantage: Memory footprint remains identical to standard GQA; no KV cache size reduction is realized from parameter sparsity.

2. Option B: Dual-State Compact Representation Cache (Experimental SGLang Primitives)

   • Rather than storing the expanded $d_{\text{model}}$ Value vector, the engine stores the unprojected activation $\mathbf{x}_t$ alongside a 16-bit bitmask encoding the Top-4 expert indices and their 4 routing weights (fp16).
   • At attention time, the Value vector is materialized on-the-fly inside SRAM via fused tensor-core kernels before the dot product.
   • Advantage: Reduces KV cache storage requirements by up to 64% for ultra-long context sequences (up to 512k tokens).
   • Disadvantage: Requires ~8% additional arithmetic intensity on memory-bound decoding steps.

KV-CACHE OPERATIONAL TRADEOFF MATRIX (512K CONTEXT):

Strategy                   | Memory Footprint | Decoding Throughput | Kernel Complexity
---------------------------|------------------|---------------------|------------------
Standard Dense GQA         | 32.0 GB / seq    | Baseline (1.0x)     | Low (PagedAttention)
MoVA Post-Projected (vLLM) | 32.0 GB / seq    | 3.8x (vs 36B Dense) | Low (Standard Kernels)
MoVA Dual-State (SGLang)   | 11.5 GB / seq    | 3.2x (vs 36B Dense) | High (Fused SRAM Decode)

Section 5: Comparative Benchmarks & Empirical Efficiency

To quantify the real-world operational benefits of Mixture-of-Value Attention, the table and interactive chart below compare K2-Horizon-MoVA-36B-A4B against established frontier models: Mixtral-8x22B, Llama-3.1-70B, and Qwen-2.5-32B.

Tests evaluated Terminal-Bench 2.1 (measuring complex CLI agent execution and multi-step tool reasoning), Active TFLOPs per Token, Time-To-First-Token (TTFT) at 32k prompt length, and Decoding Tokens/sec on an 8x NVIDIA H100 SXM5 node.

Key Architectural Takeaways

1. Agentic Mastery with 4B Active Cost: On Terminal-Bench 2.1, K2-Horizon-MoVA scores 68.4%, edging out the 70B dense baseline (67.2%) while requiring only 8.2 GFLOPs per token compared to Llama-3.1-70B's 140 GFLOPs—an astonishing 17x reduction in active compute.
2. Throughput Dominance: In production vLLM serving, K2-Horizon-MoVA sustains 185 tokens/second per stream, outperforming Qwen-2.5-32B (82 tokens/sec) by 2.25x and Llama-3.1-70B (38 tokens/sec) by 4.8x.
3. Low Prompt Latency: Because Query and Key representations are processed via unified dense projections while Value computations scale down via Top-4 gating, prefill TTFT drops to just 120 ms on 32k sequence prompts.

Section 6: Production Serving with vLLM & SGLang

Deploying K2-Horizon-MoVA in production environments is streamlined thanks to IFM's Day-0 integration pull requests.

Deploying via vLLM

Ensure you are running vllm >= 0.7.2 with native MoVA kernel support enabled:

# Launch K2-Horizon-MoVA-36B-A4B with Tensor Parallelism = 2
python3 -m vllm.entrypoints.openai.api_server \
    --model IFM/K2-Horizon-MoVA-36B-A4B \
    --tensor-parallel-size 2 \
    --max-model-len 131072 \
    --gpu-memory-utilization 0.92 \
    --enable-chunked-prefill \
    --trust-remote-code \
    --port 8000

Querying the Engine via OpenAI-Compatible Client

import openai

client = openai.OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="IFM/K2-Horizon-MoVA-36B-A4B",
    messages=[
        {
            "role": "system",
            "content": "You are an autonomous systems engineering agent. Optimize the following CUDA kernel for coalesced global memory access."
        },
        {
            "role": "user",
            "content": "__global__ void matrix_transpose(float* out, float* in, int width, int height) { ... }"
        }
    ],
    temperature=0.2,
    max_tokens=2048
)

print(response.choices[0].message.content)

Section 7: The Future of Sparse Attention

The launch of the IFM K2 Horizon model suite signals an inevitable shift in foundation model architecture. For years, the AI community treated Multi-Head Attention as an untouchable dense core, relegating all conditional routing experiments to post-attention MLPs.

By demonstrating that Mixture-of-Value Attention (MoVA) achieves state-of-the-art agentic reasoning and code synthesis while activating a mere 4 billion parameters out of 36 billion, IFM has proven that the attention pathway itself can be sparsely partitioned without compromising spatial coordinate integrity.

As context windows push towards 1M tokens and autonomous agents demand millisecond-level responsiveness, sparse value routing will likely become a cornerstone of next-generation transformer designs.

The full K2 Horizon family (0.9B, 3.7B, 7B, 32B, 36B-A4B, and 375B) is accessible on Hugging Face under the Apache 2.0 license via huggingface.co/IFM. :::