Neural Nova Brings Multi-GPU Inference Optimization to the Nova AI Platform
Neural Nova supports large-model inference optimization on 8× NVIDIA H100 and 8× NVIDIA A100 configurations, bringing serving configuration experiments, performance benchmarking, and accuracy evaluation into one platform.
For teams running large models, this means a single workflow for evaluating how a multi-GPU deployment performs, comparing optimization iterations, and selecting a configuration based on measured results.
An example run demonstrates the impact. Using Qwen3-Next-80B-A3B-Instruct with vLLM on eight A100 GPUs, the selected optimized configuration delivered 2.27× output throughput relative to the run’s baseline configuration, alongside 70.8% lower time to first token.
The GPU count stayed the same. The deployment produced substantially more output while reducing response latency.
Multi-GPU inference performance depends on several interacting decisions: how computation is distributed, how GPU memory is allocated, how requests are batched, and how prefill and token generation are scheduled. Evaluating those decisions requires measuring both aggregate capacity and the latency experienced by individual requests.
Neural Nova makes that process inspectable. The platform exposes baseline and optimized serving commands, records results for each optimization iteration, and compares performance across concurrency levels. Teams can examine the configuration behind a result and how it behaves as request load changes.
In this A100 run, the model was configured with an 8,192-token maximum model length, and the benchmark covered concurrency levels of 16, 32, 64, and 128. The platform’s selected-iteration comparison reported:
These measurements capture improvements in both serving capacity and responsiveness. Throughput increased, responses began sooner, and the interval between generated tokens decreased. The figures are specific to this run’s reported comparison; performance varies with workload and concurrency.
The selected configuration combined eight-way tensor parallelism with expert parallelism, alongside settings for memory utilization, batching, caching, attention execution, and scheduling. The measured improvement reflects the configuration as a whole. Individual settings were not evaluated in isolation.
Accuracy was evaluated alongside performance. On the configured 50-sample GSM8K test, both baseline and optimized configurations achieved 96% accuracy, with no observed regression on that evaluation.
With support for 8× H100 and 8× A100, Neural Nova gives teams a structured way to optimize large-model deployments across eight GPUs. The A100 example provides a concrete proof point: 2.27× baseline throughput with lower latency on the same hardware, backed by visible configuration comparisons and a defined accuracy check.
