One Platform Across Model Architectures: Faster Fine-Tuning for Mistral, Phi, Gemma, Qwen, and Llama

Fine-tuning performance is often approached as a scaling problem: add more GPUs, reduce precision, shorten sequence lengths, or change the training configuration. But significant performance headroom can still remain inside the execution of the training workload itself.

We used Nova AI Platform to optimize end-to-end full weights fine-tuning workloads across Mistral, Phi, Gemma, Qwen, and Llama on a single NVIDIA H100. Across the five workloads, training throughput improved by 26.9% to 72.6% while keeping the model, dataset, precision, training configuration, and hardware fixed.

Nova AI Platform profiles the complete training workload and optimizes performance-critical execution paths across both forward and backward computation. The goal is not to make one isolated kernel benchmark faster. The goal is to make the complete fine-tuning workload finish faster.

Results at a Glance

*Equivalent reduction in training time for a fixed amount of training work. Runtime is inversely proportional to throughput.

The key result is broader than the 72.6% peak improvement. Nova AI Platform found meaningful performance headroom across five different model families with different architectures and execution characteristics. The size of the gain varies because the bottlenecks vary. That is exactly why the optimization process needs to be workload-aware rather than applying one fixed optimization recipe to every model.

Optimizing Where the Training Workload Actually Spends Time

Fine-tuning performance is determined by the complete training step, not only the forward pass. A training workload includes forward computation, activation handling, backward computation, gradient calculation, memory movement, and optimizer-related execution. Improving one isolated operation does not necessarily translate into a meaningful improvement in end-to-end training throughput.

Nova AI Platform profiles the actual training workload, identifies performance-critical bottlenecks, and optimizes the relevant execution paths across both forward and backward computation. The optimization strategy is workload-dependent. Different architectures expose different bottleneck distributions, tensor shapes, memory behavior, and execution characteristics. An optimization that has high impact on one model may contribute far less to another.

The platform is designed to answer a practical question: Where is this workload actually losing performance, and what changes will make the complete job faster? After optimization, the full workload is benchmarked again and evaluated for correctness and potential model-quality regressions.

Benchmark Setup

All five workloads were benchmarked using the same general configuration:

Hardware: 1× NVIDIA H100
CUDA: 13.0
Precision: BF16
Fine-tuning mode: Full fine-tuning
Batch size: 1
Maximum sequence length: 4,096
Epochs: 1
Learning rate: 2e-5
Training samples used: 1,000
Warmup: 20 steps
Measured steps: ≥200

The fine-tuning workload used the math split of:

neural-nova/Nemotron-Post-Training-Dataset-v2-math-200

The original and optimized runs used the same model and training workload. The performance gains came from changing how the workload executes. We did not replace the model with a smaller one, reduce precision, shorten the sequence length, reduce the amount of training work, or add more GPUs.

We intentionally evaluate optimization inside an end-to-end fine-tuning workload using actual training data rather than relying only on isolated operator microbenchmarks.

Different Architectures, Different Performance Headroom

The largest throughput improvement in the current benchmark set came from:

mistralai/Mistral-7B-Instruct-v0.3

Training throughput increased from: 4,787.90 → 8,263.83 train tokens/sec representing a: 72.60% throughput improvement For a fixed amount of training work, that corresponds to approximately: 42.1% less training time. The gains extended across the rest of the benchmark set.

For Phi-4 Mini, throughput increased from 4,921.59 to 7,860.89 train tokens/sec, a 59.72% improvement.

For Gemma 4 E4B, throughput increased from 2,859.94 to 4,174.22 train tokens/sec, a 45.95% improvement.

For Qwen3 4B, throughput increased from 5,173.04 to 7,357.61 train tokens/sec, a 42.23% improvement.

For Llama 3.1 8B, throughput increased from 4,437.10 to 5,628.76 train tokens/sec, a 26.86% improvement.

The range of results matters. The objective of a workload optimization platform is not to reproduce the same speedup percentage on every architecture. Different models expose different execution bottlenecks and different amounts of recoverable performance headroom. The objective is to identify that headroom and recover it where it exists.

Performance With Post-Optimization Validation

Performance optimization is useful only when the resulting workload remains correct. Nova AI Platform therefore evaluates model behavior alongside throughput after optimization.

In the reported GSM8K validation runs on a 500-sample evaluation:

• Gemma remained at 0.84 → 0.84

• Phi moved from 0.92 → 0.94

• Llama moved from 0.80 → 0.78

• Mistral moved from 0.46 → 0.45

For Llama, the reported difference corresponds to one question in the evaluation sample. For Mistral, the larger movement requires further validation before making any claim about equivalent model quality. We report the result transparently rather than presenting throughput in isolation.

Small evaluation sets are useful as an initial regression signal, but they are not sufficient for strong statistical claims about small quality changes. Larger validation runs are necessary before drawing broader conclusions. The role of validation in our optimization workflow is straightforward: Performance gains should be measured together with correctness and model-quality checks, not separately from them.

Why End-to-End Throughput Matters

Higher throughput is not only a larger benchmark number. Fine-tuning workloads are repeated across datasets, experiments, hyperparameter configurations, model versions, and post-training pipelines.

Higher training throughput can translate into:

• shorter model iteration cycles;

• more experiments within the same GPU budget;

• faster hyperparameter exploration;

• lower infrastructure cost for fixed-token workloads;

• greater training capacity from existing hardware.

Throughput improvement and training-time reduction are not numerically identical. For example, Mistral's 72.60% throughput improvement corresponds to approximately 42.1% less training time for the same amount of training work. Similarly, Phi-4 Mini's 59.72% throughput improvement corresponds to approximately 37.4% less training time.

Across repeated experiments and model iterations, those reductions compound.

Beyond Individual Kernel Speedups

A production training workload does not execute one operator in isolation. The end-to-end impact of an optimization depends on where an operation appears in the workload, how frequently it executes, tensor shapes, memory behavior, launch overhead, backward execution, and integration cost.

An isolated kernel can show a large microbenchmark speedup while having limited impact on total training time. Conversely, targeted improvements across several important execution paths can combine into a significant end-to-end gain.

This is why Nova AI Platform focuses on the complete workload.

The workflow is designed around the full optimization loop:

Profile → Identify bottlenecks → Optimize → Integrate → Benchmark → Validate

The result that matters is not: “This kernel is faster.” It is: “This model completes the same training work faster on the same hardware.”

One Platform Across Model Architectures

Mistral, Phi, Gemma, Qwen, and Llama have different architectures and execution profiles. Yet across the five workloads in this benchmark set, Nova AI Platform found measurable performance headroom: +26.86% to +72.60% higher training throughput on the same single H100 GPU. The broader objective is simple Profile the real workload, identify where performance is being lost, optimize the relevant forward and backward execution paths, validate the result, and improve end-to-end performance.

Nova AI Platform is not built around one model family, one kernel, or one architecture.

It optimizes the execution layer across AI workloads.

What Comes Next

We are continuing to expand the benchmark set across additional models, architectures, and workload configurations while developing Nova AI Platform across:

• fine-tuning optimization;

• inference optimization;

• forward and backward execution optimization;

• automated profiling and bottleneck identification;

• correctness and regression validation;

• single-GPU and multi-GPU workloads.

Our goal is straightforward: Make AI workloads run faster on the infrastructure teams already have, without requiring every team to become experts in low-level GPU performance optimization.

See the Difference Yourself

The benchmark results are not just numbers on a chart. We have released optimized versions of several models from this benchmark so you can test them directly in your own workflows.

Mistral 7B: https://huggingface.co/neural-nova/Mistral-7B-Instruct-v0.3-optimized

Gemma 4 E4B: https://huggingface.co/neural-nova/gemma-4-E4B-it-optimized

Qwen3 4B: https://huggingface.co/neural-nova/Qwen3-4B-Instruct-2507-optimized

Llama 3.1 8B: https://huggingface.co/neural-nova/Llama-3.1-8B-Instruct-optimized

More optimized models are coming as Nova AI Platform expands across additional architectures, workloads, and deployment environments.