Improve Pascal/Turing GPU compatibility and short-audio latency

.gitignore

@@ -7,3 +7,8 @@ __pycache__/
 core.*
 client_logs/
 .venv/
+.tools/
+benchmark-results/
+models/
+results/
+*.gguf

README.md

@@ -1,23 +1,131 @@
 # NeMo-Speech.cpp
 
-A lightweight native C++ runtime for running the NVIDIA Nemotron Speech model family locally, with broad hardware support. It supports multilingual speech recognition, speaker diarization, translation, and speech synthesis in real-time and batch modes.
+> [!NOTE]
+> This is an unofficial community fork of NeMo-Speech.cpp. It is not affiliated with,
+> maintained by, or officially supported by NVIDIA.
 
-NeMo-Speech.cpp is NVIDIA's official local speech inference solution, with day-0 support for our latest speech models. It builds on models from [NVIDIA NeMo Speech](https://github.com/NVIDIA-NeMo/Speech), with native inference powered by [ggml](https://github.com/ggml-org/ggml).
+The original code belongs to the [NeMo-Speech.cpp](https://github.com/NVIDIA/NeMo-Speech.cpp)
+project. This fork preserves the original notices, credits, and licenses; its fork-specific
+credits apply only to the modifications, tests, scripts, and documentation added here.
 
-## Models and applications
+## About this fork
 
-| Application | Supported models |
-|---|---|
-| Speech recognition | [Nemotron 3.5 ASR Streaming 0.6B](https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b), [Nemotron Speech Streaming 0.6B](https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b), [Parakeet TDT 0.6B v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3), and [Parakeet CTC 1.1B](https://huggingface.co/nvidia/parakeet-ctc-1.1b) |
-| Speaker diarization | [Streaming Sortformer 4-speaker v2](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2), standalone or combined with ASR |
-| Text and speech translation | [Riva Translate 4B Instruct v2](https://huggingface.co/nvidia/Riva-Translate-4B-Instruct-v2), with composed ASR-to-NMT-to-TTS speech translation |
-| Speech synthesis | [MagpieTTS Multilingual 357M](https://huggingface.co/nvidia/magpie_tts_multilingual_357m) with [NeMo NanoCodec](https://huggingface.co/nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps) |
-| Speech processing | [Silero VAD](https://github.com/snakers4/silero-vad), punctuation and capitalization, endpointing, text normalization, and subtitles |
+This fork was started to improve compatibility, configuration, and day-to-day usability of
+NeMo-Speech.cpp on NVIDIA Pascal GPUs, especially the GeForce GTX 10 series. Initial development
+and validation used an NVIDIA GeForce GTX 1060 6 GB (Compute Capability 6.1).
+
+Its initial target family includes GTX 1050, GTX 1050 Ti, GTX 1060, GTX 1070, GTX 1080, and
+GTX 1080 Ti. So far, practical testing has been performed only on the GTX 1060 6 GB; community
+validation is required before claiming support for the other Pascal GPUs.
+
+It adds safe runtime controls and clearer diagnostics. It does **not** yet include a new kernel
+optimized specifically for Pascal.
+
+### Fork-specific author and maintenance
+
+The fork-specific changes, tests, and documentation were made by:
+
+- **GitHub:** [UNDER192103](https://github.com/UNDER192103)
+- **Name/project:** Under Nouzen
+
+This attribution does not apply to the original NeMo-Speech.cpp codebase.
+
+### Initial fork changes
+
+- `--skinny-q8 auto|on|off` runtime control.
+- Automatic CUDA Compute Capability detection.
+- Safe fallback for GPUs below SM 8.0, plus a controlled error if an incompatible Skinny Q8
+  mode is forced.
+- `--suppress-cuda-graph-log` to selectively hide the repeated CUDA Graph architecture message.
+- Windows build and execution scripts, plus Pascal/GTX 1060 documentation.
+
+`--skinny-q8 auto` does not add a Pascal kernel: it disables the incompatible Skinny Q8 path and
+uses the existing CUDA fallback. `--suppress-cuda-graph-log` does not enable CUDA Graphs and does
+not make inference faster. Neither change alters model precision, model contents, or transcription
+math.
+
+### Tested environment
+
+- OS: Windows 11
+- GPU: NVIDIA GeForce GTX 1060 6 GB (Pascal, Compute Capability 6.1)
+- CPU: Intel Xeon E5-2660 v2; RAM: 32 GB; CUDA Toolkit: 12.6
+- Model: Nemotron 3.5 ASR Streaming 0.6B Q8 GGUF
+- Mode: persistent HTTP server
+
+## Pascal performance observations
+
+On the tested GTX 1060 6 GB, the custom runtime showed performance similar to the default runtime
+for the included 11-second JFK sample.
+
+For manually recorded short requests around two to three seconds, four of the custom observations
+were around 72–80 ms, while one first custom observation was 620.39 ms. The default observations
+were generally above 130 ms and also included a large latency spike.
+
+These short-request results are preliminary. A fully reproducible benchmark using the same short
+English WAV is being prepared, pending a redistributable fixture.
+
+| Test | Default median | Custom median | Observation |
+| --- | ---: | ---: | --- |
+| 11-second JFK WAV | 181.41 ms | 178.26 ms | Similar performance |
+| Short local speech | 179.16 ms | 74.29 ms | Large preliminary median difference |
+
+See [Pascal performance observations](docs/pascal-performance-observations.md) for the complete
+methodology, raw values, limitations, and reproduction instructions.
+
+### Current status
+
+Validated on the GTX 1060 6 GB: CUDA SM 6.1 build, file transcription, persistent HTTP server,
+`/ready`, `/v1/audio/transcriptions`, CPU execution, CUDA execution, automatic Skinny Q8 fallback,
+the controlled `--skinny-q8 on` error, and selective CUDA Graph log suppression.
+
+Not yet implemented: a Pascal-specific Q8 kernel, DP4A optimization, CUDA Graphs on Pascal,
+testing on other GTX 10 GPUs, or a controlled reproducible upstream-versus-fork benchmark.
+
+Example server command (paths are intentionally generic):
+
+```powershell
+.\build\bin\nemo-speech.exe serve `
+  --asr-model "C:\Models\nemotron-3.5-asr-streaming-0.6b.q8_0.gguf" `
+  --gpu 0 `
+  --host 127.0.0.1 `
+  --port 8081 `
+  --skinny-q8 auto `
+  --suppress-cuda-graph-log
+```
+
+Community testing on Pascal GPUs is welcome. Please report GPU model, Compute Capability, operating
+system, CUDA Toolkit, build command, GGUF model, audio duration, latency, logs, and transcription
+result. Do not publish licensed models or protected audio.
+
+## Reproducible test audio
+
+The fork test fixture is intended to live at `test_files/fork/asr/teste-en.wav`, with the expected
+transcript in [`test_files/fork/asr/teste-en.txt`](test_files/fork/asr/teste-en.txt):
+
+```text
+Ask not what your country can do for you. Ask what you can do for your country.
+```
+
+The WAV itself is currently **not included** because its redistribution license still needs manual
+review. Do not publish it until that review is complete. Once a reviewed copy is present, the three
+main commands are:
+
+```powershell
+.\scripts\windows\test-pascal-wav.ps1 -Model "C:\Models\nemotron-3.5-asr-streaming-0.6b.q8_0.gguf"
+.\scripts\windows\run-pascal-server.ps1 -Model "C:\Models\nemotron-3.5-asr-streaming-0.6b.q8_0.gguf"
+.\scripts\windows\test-http-wav.ps1
+```
+
+For the microphone client, setup, WAV metadata, license-review checklist, and complete testing
+workflow, see [Testing with audio and microphone](docs/testing-with-audio-and-microphone.md).
+
+A lightweight native C++ runtime for NVIDIA Nemotron Speech models built on ggml. Runs speech models in realtime and in batch mode across platforms/backends.
 
 ## Contents
 
 - [Models and applications](#models-and-applications)
 - [Installation](#installation)
+- [About this fork](#about-this-fork)
 - [Quick start](#quick-start)
 - [Command line](#command-line)
 - [Local server and playground](#local-server-and-playground)
@@ -148,6 +256,7 @@ Windows, and container instructions are in
 | [Client integration](docs/clients.md) | OpenAI SDKs, curl, and Riva gRPC clients |
 | [Troubleshooting](docs/troubleshooting.md) | `doctor` output and common runtime failures |
 | [Build from source](docs/build.md) | Presets, optional components, dependencies, containers, and artifacts |
+| [Pascal performance](docs/pascal-performance-observations.md) | GTX 1060 short/long latency observations and reproducible benchmark instructions |
 | [All documentation](docs/README.md) | ASR, TTS, NMT, configuration, and developer references |
 
 ## License

app/bench.cpp

@@ -344,6 +344,9 @@ print_bench_help(const char* program) {
         "  --mode offline|stream   Recognition mode (default: offline)\n"
         "  --device, --backend DEVICE\n"
         "                          auto, cpu, cuda[:N], metal, or vulkan[:N]\n"
+        "  --skinny-q8 auto|on|off CUDA control: auto uses the safe Pascal fallback; off\n"
+        "                          replaces GGML_SKINNY_Q8=0; on requires SM 8.0+\n"
+        "  --suppress-cuda-graph-log Suppress only the repeated CUDA-graph architecture log\n"
         "  -l, --language CODE     Prompt language code\n"
         "  -r, --recursive         Recurse into input directories\n"
         "  --json                   Emit machine-readable results\n"

app/main.cpp

@@ -3,7 +3,9 @@
 
 #include <csignal>
 #include <cstdio>
+#include <cstdlib>
 #include <cstring>
+#include <stdexcept>
 #include <string>
 #include <utility>
 #include <vector>
@@ -109,10 +111,29 @@ print_help(const char* program) {
         "  --version        Show version\n"
         "  --json           Emit machine-readable results and errors\n"
         "  --quiet          Suppress non-result progress messages\n"
-        "  --verbose        Emit additional diagnostics on stderr\n",
+        "  --verbose        Emit additional diagnostics on stderr\n"
+        "  --suppress-cuda-graph-log\n"
+        "                   Suppress only the repeated CUDA-graph architecture message\n"
+        "  --skinny-q8 MODE Skinny Q8: auto, on, or off (CLI overrides GGML_SKINNY_Q8)\n",
         NEMO_SPEECH_VERSION_STR, program);
 }
 
+void
+set_process_environment(const char* name, const char* value) {
+#if defined(_WIN32)
+    if (_putenv_s(name, value) != 0)
+        throw std::runtime_error(std::string("failed to set environment variable ") + name);
+#else
+    if (setenv(name, value, 1) != 0)
+        throw std::runtime_error(std::string("failed to set environment variable ") + name);
+#endif
+}
+
+bool
+parse_skinny_q8_mode(const std::string& value) {
+    return value == "auto" || value == "on" || value == "off";
+}
+
 }  // namespace
 
 int
@@ -125,6 +146,8 @@ main(int argc, char** argv) {
     bool json = false;
     bool quiet = false;
     bool verbose = false;
+    bool suppress_cuda_graph_log = false;
+    std::string skinny_q8_mode;
     std::vector<char*> filtered;
     filtered.reserve(static_cast<size_t>(argc));
     filtered.push_back(argv[0]);
@@ -136,13 +159,30 @@ main(int argc, char** argv) {
             quiet = true;
         else if (arg == "--verbose")
             verbose = true;
-        else
+        else if (arg == "--suppress-cuda-graph-log")
+            suppress_cuda_graph_log = true;
+        else if (arg.rfind("--skinny-q8=", 0) == 0)
+            skinny_q8_mode = arg.substr(std::strlen("--skinny-q8="));
+        else if (arg == "--skinny-q8") {
+            if (++i >= argc)
+                return print_cli_error(
+                    "", "--skinny-q8 requires auto, on, or off", 2, "invalid_argument");
+            skinny_q8_mode = argv[i];
+        } else
             filtered.push_back(argv[i]);
     }
     configure_cli_output(json, quiet, verbose);
     if (quiet && verbose)
         return print_cli_error(
             "", "--quiet and --verbose cannot be used together", 2, "invalid_argument");
+    if (!skinny_q8_mode.empty() && !parse_skinny_q8_mode(skinny_q8_mode))
+        return print_cli_error("", "--skinny-q8 must be auto, on, or off", 2, "invalid_argument");
+    if (suppress_cuda_graph_log)
+        set_process_environment("NEMO_SPEECH_SUPPRESS_CUDA_GRAPH_LOG", "1");
+    if (!skinny_q8_mode.empty()) {
+        set_process_environment("NEMO_SPEECH_SKINNY_Q8_MODE", skinny_q8_mode.c_str());
+        set_process_environment("NEMO_SPEECH_SKINNY_Q8_SOURCE", "cli");
+    }
     argc = static_cast<int>(filtered.size());
     argv = filtered.data();
 

app/serve.cpp

@@ -648,6 +648,9 @@ print_serve_help(const char* program) {
 #endif
         "  --device, --backend DEVICE\n"
         "                          auto, cpu, cuda[:N], metal, or vulkan[:N]\n"
+        "  --skinny-q8 auto|on|off CUDA control: auto uses the safe Pascal fallback; off\n"
+        "                          replaces GGML_SKINNY_Q8=0; on requires SM 8.0+\n"
+        "  --suppress-cuda-graph-log Suppress only the repeated CUDA-graph architecture log\n"
         "  --config FILE           Apply YAML configuration\n"
 #if defined(NEMO_SPEECH_CLI_ASR)
         "  --asr.* VALUE           Override ASR engine configuration\n"

app/transcribe.cpp

@@ -497,6 +497,9 @@ print_transcribe_help(const char* program) {
         "  -l, --language CODE       Language code or prompt\n"
         "  --device, --backend DEVICE\n"
         "                            auto, cpu, cuda[:N], metal, or vulkan[:N]\n"
+        "  --skinny-q8 auto|on|off  CUDA control: auto uses the safe Pascal fallback; off\n"
+        "                            replaces GGML_SKINNY_Q8=0; on requires SM 8.0+\n"
+        "  --suppress-cuda-graph-log Suppress only the repeated CUDA-graph architecture log\n"
         "  -c, --concurrency N       Concurrent utterances; one shared model\n"
         "  -f, --format FORMAT       text, json, srt, or vtt (default: text)\n"
         "  -o, --output PATH         Output path for one input\n"

docs/README.md

@@ -16,6 +16,8 @@ Start with:
 - [Native SDK integration](sdk.md)
 - [Troubleshooting](troubleshooting.md)
 - [Build from source](build.md)
+- [Pascal performance observations](pascal-performance-observations.md) - GTX 1060 short/long
+  latency observations and the planned reproducible short-WAV benchmark.
 
 ## ASR