Summary of Changes:

README.md:

CPU Stress: Updated the description to clarify it now uses dedicated threads for spin loops (not Parallel.For), which ensures the dashboard remains responsive even under 100% load.
API: Updated the CPU trigger example to show the new targetPercentage parameter.
azure-monitoring-guide.md & azure-monitoring-guide.html:

CPU Code Pattern: Updated to show the new Thread() implementation and explained the reasoning (avoiding dashboard starvation).
Slow Request Code Pattern: Updated Scenario 1 to explicitly show Task.Delay().Wait(), reflecting the actual "Sync Over Async" anti-pattern in the code.
New Section - Queue Time: Added a generic explanation of the "gap" mechanism you discovered. It explains that a request taking 47s total but only 25s in logs means it spent 22s in the queue due to Thread Pool starvation.
Architecture: Added a note about the ETW Heartbeat mechanism (logging every 500ms) which prevents long idle requests from being lost in traces.
documentation.html:

Warnings: Updated to mention dedicated threads for CPU stress.
Tables: Updated the Slow Request scenario table to accurately describe Task.Delay().Wait().
Notes: Added a note about Queue Time vs Execution Time to the Recommended Settings section.

Author: rhamlett_microsoft

README.md

@@ -15,7 +15,7 @@ This application is designed to help developers and DevOps engineers:
 
 **This application intentionally creates performance problems!**
 
-- 🔥 **CPU stress** - Creates parallel spin loops that consume all CPU cores
+- 🔥 **CPU stress** - Creates dedicated threads running spin loops to consume all CPU cores
 - 📊 **Memory pressure** - Allocates and pins memory blocks to increase working set
 - 🧵 **Thread pool starvation** - Uses sync-over-async anti-patterns to block thread pool threads
 - � **Slow requests** - Generates long-running requests with sync-over-async patterns for CLR Profiler analysis
@@ -99,7 +99,8 @@ The CPU and Memory metric tiles use dynamic color coding based on utilization pe
 **Request body:**
 ```json
 {
-  "durationSeconds": 30
+  "durationSeconds": 30,
+  "targetPercentage": 100
 }
 ```
 

docs/azure-monitoring-guide.md

@@ -60,15 +60,22 @@ In the profiler trace, you'll see:
 ### Code Pattern (What's Wrong)
 
 ```csharp
-// This is the intentional "bad" code in CpuStressService:
-Parallel.For(0, Environment.ProcessorCount, _ =>
+// The service uses dedicated threads to avoid starving the Thread Pool,
+// ensuring the dashboard remains responsive even under 100% load.
+var threads = new Thread[Environment.ProcessorCount];
+
+for (int i = 0; i < threads.Length; i++)
 {
-    while (!cancellationToken.IsCancellationRequested)
+    threads[i] = new Thread(() =>
     {
-        // Intentionally consuming CPU with no useful work
-        Thread.SpinWait(10000);
-    }
-});
+        // Tight spin loop to consume 100% of a core
+        while (Stopwatch.GetTimestamp() < endTime)
+        {
+             // Spin
+        }
+    });
+    threads[i].Start();
+}
 ```
 
 ---
@@ -259,28 +266,37 @@ Simulated database and HTTP blocking calls:
 ### Code Pattern (What the Simulation Does)
 
 ```csharp
-// SCENARIO 1: Direct blocking - most obvious in traces
+// SCENARIO 1: Sync-Over-Async (Simple)
 public void FetchDataSync_BLOCKING_HERE(int delayMs)
 {
-    // Intentional blocking - will show in profiler as Thread.Sleep
-    Thread.Sleep(delayMs);
+    // FATAL FLAW: Calling .Wait() on a Task blocks the thread!
+    // This is the classic "Sync Over Async" anti-pattern.
+    Task.Delay(delayMs * 1000).Wait();
 }
 
 // SCENARIO 2: Nested blocking inside sync methods
 public void ValidateOrderSync_BLOCKS_INTERNALLY(int delayMs)
 {
-    // Each method internally blocks using Thread.Sleep
-    Thread.Sleep(delayMs);
-}
-
-// SCENARIO 3: Simulated database/HTTP pattern
-public void GetCustomerFromDatabaseSync_SYNC_BLOCK(int delayMs)
-{
-    // Simulates a blocking database call
-    Thread.Sleep(delayMs);
+    // Each method internally blocks using Task.Delay().Wait()
+    CheckInventorySync_BLOCKS_INTERNALLY(delayMs);
 }
 ```
 
+### Analyzing the Trace: The "Gap" (Queue Time vs Execution Time)
+
+When diagnosing slow requests, you may see a large gap where "No events emitted" appears before the request actually starts.
+
+1.  **Queue Time**: If the Thread Pool is starved (by other slow requests), a new request will sit in the queue waiting for a thread. PROFILER will show a gap here.
+2.  **Execution Time**: Once a thread is assigned, the method runs.
+
+**Example from a Trace:**
+*   Total Time seen by user: **47 seconds**
+*   Request Duration in log: **25 seconds**
+*   **Conclusion**: The request sat in the queue for **22 seconds** before it could even start!
+
+### ETW Flushing (Heartbeats)
+The simulator runs a background thread that logs "Generating trace noise..." every 500ms. This ensures that ETW buffers are flushed to the `.diagsession` file, preventing data loss during long idle periods.
+
 ### Real-World Equivalents
 
 The Thread.Sleep patterns simulate what you'd see with:

src/PerfProblemSimulator/wwwroot/azure-monitoring-guide.html

@@ -358,22 +358,29 @@ <h4>3. CPU Profiling (Advanced)</h4>
             <h3>What to Look For</h3>
             <div class="code-block">
 <span class="comment">In the profiler trace, you'll see:</span>
-- Parallel.For loops
-- SpinWait operations
-- High self-time in CpuStressService.RunStressLoop
+- Dedicated threads running a while loop
+- High self-time in CpuStressService
+- Thread names like CpuStress-*
             </div>
 
             <h3>Code Pattern (What's Wrong)</h3>
             <div class="code-block">
-<span class="comment">// This is the intentional "bad" code in CpuStressService:</span>
-Parallel.For(<span class="keyword">0</span>, Environment.ProcessorCount, _ =>
+<span class="comment">// The service uses dedicated threads to avoid starving the Thread Pool,</span>
+<span class="comment">// ensuring the dashboard remains responsive even under 100% load.</span>
+<span class="keyword">var</span> threads = <span class="keyword">new</span> Thread[Environment.ProcessorCount];
+
+<span class="keyword">for</span> (<span class="keyword">int</span> i = <span class="keyword">0</span>; i &lt; threads.Length; i++)
 {
-    <span class="keyword">while</span> (!cancellationToken.IsCancellationRequested)
+    threads[i] = <span class="keyword">new</span> Thread(() =>
     {
-        <span class="comment">// Intentionally consuming CPU with no useful work</span>
-        Thread.SpinWait(<span class="keyword">10000</span>);
-    }
-});
+        <span class="comment">// Tight spin loop to consume 100% of a core</span>
+        <span class="keyword">while</span> (Stopwatch.GetTimestamp() &lt; endTime)
+        {
+             <span class="comment">// Spin</span>
+        }
+    });
+    threads[i].Start();
+}
             </div>
         </section>
 
@@ -589,36 +596,35 @@ <h3>Recommended Settings for CLR Profiler (60s trace)</h3>
 
             <h3>Code Pattern (What's Wrong)</h3>
             <div class="code-block">
-<span class="comment">// SCENARIO 1: Direct blocking - most obvious in traces</span>
+<span class="comment">// SCENARIO 1: Sync-Over-Async (Simple)</span>
 <span class="keyword">public void</span> ProcessRequest()
 {
-    <span class="comment">// BAD: Blocking on async - thread is blocked for 25 seconds!</span>
-    FetchDataAsync().Wait();
+    <span class="comment">// FATAL FLAW: Calling .Wait() on a Task blocks the thread!</span>
+    <span class="comment">// This is the classic "Sync Over Async" anti-pattern.</span>
+    Task.Delay(<span class="keyword">25000</span>).Wait();
 }
 
 <span class="comment">// SCENARIO 2: Hidden blocking inside sync methods</span>
 <span class="keyword">public void</span> ValidateOrderSync_BLOCKS_INTERNALLY()
 {
-    <span class="comment">// BAD: Looks like a sync method but blocks internally</span>
-    ValidateOrderAsync().Wait();
-}
-
-<span class="comment">// SCENARIO 3: Common legacy migration pattern</span>
-<span class="keyword">public</span> Customer GetCustomer(<span class="keyword">int</span> id)
-{
-    <span class="comment">// BAD: GetAwaiter().GetResult() still blocks!</span>
-    <span class="keyword">return</span> GetCustomerAsync(id).GetAwaiter().GetResult();
+    <span class="comment">// Each method internally blocks using Task.Delay().Wait()</span>
+    ValidateOrderAsync_BLOCKS().Wait();
 }
             </div>
 
             <div class="info-box">
-                <h4>💡 Key Insight</h4>
-                <p>The slow request simulator makes sync-over-async <strong>visible in profiler call stacks</strong> because:</p>
+                <h4>💡 Key Insight: Queue Time vs Execution Time</h4>
+                <p>If you see a large gap where "No events emitted" appears before the request starts:</p>
                 <ul>
-                    <li>Requests are long enough (25s) to be captured during any reasonable trace period</li>
-                    <li>Method names explicitly indicate where blocking occurs</li>
-                    <li>Multiple scenarios show different blocking patterns commonly found in production code</li>
+                    <li><strong>Queue Time:</strong> The Thread Pool is starved, so your request sat in the queue waiting for a thread (e.g., 20+ seconds).</li>
+                    <li><strong>Execution Time:</strong> Once a thread was assigned, the method finally ran.</li>
                 </ul>
+                <p><strong>Example:</strong> User waits 47s, but log shows only 25s duration. The missing 22s was spent in the queue!</p>
+            </div>
+
+            <div class="info-box">
+                <h4>💡 ETW Heartbeats</h4>
+                <p>The simulator logs "Generating trace noise..." every 500ms to flush ETW buffers, ensuring long requests are not lost in traces.</p>
             </div>
         </section>
 

src/PerfProblemSimulator/wwwroot/documentation.html

@@ -339,7 +339,7 @@ <h3>Purpose</h3>
             <div class="warning-box">
                 <h4>⚠️ Warning - This application intentionally creates performance problems!</h4>
                 <ul>
-                    <li>🔥 <strong>CPU stress</strong> - Creates parallel spin loops that consume all CPU cores</li>
+                    <li>🔥 <strong>CPU stress</strong> - Creates dedicated threads running spin loops to consume all CPU cores</li>
                     <li>📊 <strong>Memory pressure</strong> - Allocates and pins memory blocks to increase working set</li>
                     <li>🧵 <strong>Thread pool starvation</strong> - Uses sync-over-async anti-patterns to block thread pool threads</li>
                     <li>� <strong>Slow requests</strong> - Generates long-running requests with sync-over-async patterns for CLR Profiler analysis</li>
@@ -499,7 +499,7 @@ <h3>Three Sync-Over-Async Scenarios</h3>
                     <tr>
                         <td><strong>Simple</strong></td>
                         <td><code>_BLOCKING_HERE</code></td>
-                        <td>Direct <code>.Wait()</code> blocking</td>
+                        <td><code>Task.Delay().Wait()</code> causing blocking</td>
                     </tr>
                     <tr>
                         <td><strong>Nested</strong></td>
@@ -522,6 +522,8 @@ <h3>Recommended Settings</h3>
                 <li><strong>Max Requests:</strong> 6 (doesn't overwhelm)</li>
             </ul>
 
+            <p><strong>Note on Queue Time:</strong> If the thread pool is starved, you may see a large gap in the trace before the request starts. This is "Queue Time" (waiting for a thread) vs "Execution Time" (running the code).</p>
+
             <h3>Using with CLR Profiler</h3>
             <ol>
                 <li>Start a CPU Profiler trace (60-120 seconds)</li>