ZenteiQ boosts inference throughput with Google Gateway

ZenteiQ has adopted Google Kubernetes Engine Inference Gateway for its synthetic scientific data-generation pipeline, lifting aggregate inference throughput by 4.07 times across its production clusters.

The startup, spun out of the Indian Institute of Science and selected under the IndiaAI Mission, is building scientific foundation models designed to work with physical laws, governing equations and numerical solvers.

Its pre-training work depends on very large volumes of reasoning data drawn from scientific literature, simulation datasets and synthetic examples generated in-house. The pipeline creates prompt batches, generates candidate reasoning traces, scores responses and runs guardrail checks before verified datasets are passed into MaxText pre-training jobs.

This process had become a bottleneck because ZenteiQ operates with a fixed compute allocation. Time spent generating and checking data reduced the time available for pre-training, while delays in data creation left accelerator clusters waiting for datasets.

Routing problem

ZenteiQ had been running model pools on Kubernetes with standard ClusterIP round-robin load balancing across inference endpoints. As it scaled work tied to its 9B- and 35B-parameter model schedules, two main issues slowed dataset creation.

The first was uneven engine queues. Worker pods produced prompts with widely differing complexity and response length, but the round-robin approach could not see which vLLM serving pods were already occupied with long reasoning traces.

The second was repeated prefix computation. Prompts within a generation batch shared about 70% to 80% of their instructional context and task specifications, but round-robin routing spread similar requests across different serving instances, forcing repeated work on the same shared prompt prefix.

As a result, throughput could vary by as much as half within a single run, stretching dataset creation times and delaying scheduled pre-training runs.

Unified gateway

To address this, ZenteiQ deployed GKE Inference Gateway, powered by the open-source llm-d distributed inference framework, in front of its vLLM accelerator pools. The gateway now manages traffic across the company's model families and serving nodes.

Worker pods pull scientific seed documents and domain taxonomies from Cloud Storage before expanding them into multi-step prompts. Those requests, along with evaluation queries and guardrail checks, now go through a single gateway deployment rather than separate routing setups for each model tier.

The routing layer uses live serving signals from vLLM instances instead of static rotation. It monitors queue depth and key-value cache use, then directs incoming requests to serving nodes with available capacity and, where possible, to nodes already holding matching prompt prefixes in memory.

Measured gains

ZenteiQ reported that aggregate throughput rose from 16,964 to 69,114 tokens per second per 1,000 accelerators. Daily output also increased from 1.47 billion to 5.97 billion generated tokens for every 1,000 accelerators.

Individual model pools recorded throughput gains of between 3.5 times and 5.5 times. Prefix-cache hit rate also increased by 14 percentage points, reducing redundant prefill work before requests diverged into different derivation paths.

For the business, the more immediate effect was operational rather than purely technical. Verified datasets were available on time more consistently, allowing multi-host pre-training runs to begin as scheduled instead of waiting for data generation to catch up.

Fisayo Feyisetan, Product Manager, Google Cloud, said: "GKE and GKE Inference Gateway gave ZenteiQ the inference-routing layer within its pipeline to scale aggregate inference throughput 4.07x and grow daily output from 1.47 billion to 5.97 billion generated tokens for every 1,000 accelerators."

ZenteiQ is developing scientific foundation models that aim to interpret simulation outputs, evaluate equation formulations and work through mathematical derivations that follow physical laws across areas including thermodynamics, computational fluid dynamics and electromagnetics.

The next stage of work will involve scaling those models further and applying the same routing arrangement to post-training reinforcement learning and automated numerical solver evaluation loops, where maintaining context across multiple turns places more weight on queue balancing and warm prompt caches.