I am optimizing Breeze TTS 2 inference on an NVIDIA Tesla T4...

Prompt

I am optimizing Breeze TTS 2 inference on an NVIDIA Tesla T4. The model is BreezeBlue/Breeze-TTS-2, using the official breeze-tts inference stack. Act as a senior GPU inference-performance engineer specializing in PyTorch, CUDA, autoregressive/streaming audio models, and production TTS. I need a diagnosis and a prioritized optimization strategy, not speculative code yet. I have a remote English TTS inference pipeline running on an NVIDIA Tesla T4: - Compute capability: 7.5 - VRAM: ~14.6 GiB - PyTorch/CUDA stack is healthy - Model loads and generates correctly - Current model dtype: BF16 - Attention implementation: eager - FlashAttention is not installed - Peak reserved VRAM during normal generation: ~8.0–8.1 GiB - Generated audio quality is currently acceptable and must be preserved Each request may have its own instruction, CFG and seed. These settings are intentional and must NOT be normalized merely for speed. The reference speaker and exact reference transcript are also part of the request. Measured baseline on the same three chunks: - total generated audio: ~25.9 seconds - generation wall time: ~227 seconds - RTF: ~8.77 - model/runtime initialization: ~75 seconds Per-chunk inference is roughly: - 13.76 s audio → ~112 s streaming inference - 7.36 s audio → ~59 s - 4.80 s audio → ~39 s After the first chunk, input preparation is almost zero (~0.04 s), so preparation is not the main bottleneck. Experiments already performed: 1. Runtime reuse + cached reference encoding - worked correctly - reference encoding became effectively free after the first chunk - no meaningful throughput improvement - RTF remained around 8.7–9.0 2. Full-model FP16 on T4 - conversion succeeded - inference immediately produced invalid sampling probabilities (NaN/Inf, negative or zero probability mass) - safe fallback to original BF16 worked - therefore naive full-model FP16 is not usable 3. CUDA Graph for backbone decode only - graph captured successfully - no fallback - BF16 preserved - RTF: 8.75 versus 8.77 baseline - essentially no meaningful improvement 4. CUDA Graph for depth decoder only - graph captured successfully - no fallback - BF16 preserved - RTF: 8.71 - only ~0.7% improvement - therefore also not a meaningful solution So the dominant cost remains inside the actual streaming inference path. There is also a separate orchestration problem: After the remote GPU job is already complete, retrieving the generated results can take roughly 2.5–3.5 additional minutes even for only three WAV files. I want you to determine the highest-value next engineering direction. Please do NOT assume that the answer must be another fast_* flag. Analyze: - what the measurements imply about the true bottleneck - whether BF16 execution on T4 is likely the fundamental limitation - which part of the inference stack should be profiled next - whether selective/mixed precision could be safe and useful, rather than converting the whole model - whether attention, codec, text encoder, prefill, sampling, synchronization, CPU↔GPU transfers, kernel-launch overhead, or another component is more likely to dominate - whether a persistent long-lived GPU worker/model-loaded service would provide a larger real-world gain than micro-optimizing one inference pass - why result retrieval takes several minutes after GPU completion and how that should be investigated separately - whether this workload is simply poorly matched to T4 and another commonly available GPU class would change the economics substantially Constraints: - preserve speaker identity and audible quality - preserve per-request instruction / CFG / seed - do not change text or synthesis semantics for benchmarking - changes must fail safely - avoid experiments that have low expected value - do not recommend full FP16 again unless you have a materially different numerically safe method - VRAM headroom exists, but do not assume the full fast path will fit safely - prefer measurements over assumptions Give me: 1. Your diagnosis of the main bottleneck. 2. The 3 highest-value next experiments, ranked. 3. What each experiment would prove or disprove. 4. Exact metrics/logging needed to evaluate each one. 5. A stop condition: when should I conclude that further T4 optimization is not worth the engineering effort? 6. Separately, your diagnosis and optimization plan for the multi-minute post-completion result retrieval delay. Do not write implementation code yet. I want the architecture/performance decision first.