Skip to content

Commit 14cbe8a

Browse files
rhamlettrhamlett
committed
updates to documentation and failed request sim
1 parent 78ac1da commit 14cbe8a

8 files changed

Lines changed: 405 additions & 7 deletions

File tree

‎README.md‎

Lines changed: 22 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,9 @@ This application is designed to help developers and DevOps engineers:
1818
- 🔥 **CPU stress** - Creates dedicated threads running spin loops to consume all CPU cores
1919
- 📊 **Memory pressure** - Allocates and pins memory blocks to increase working set
2020
- 🧵 **Thread pool starvation** - Uses sync-over-async anti-patterns to block thread pool threads
21-
- � **Slow requests** - Generates long-running requests with sync-over-async patterns for CLR Profiler analysis
22-
- �💥 **Application crashes** - Triggers fatal crashes for testing Azure Crash Monitoring and memory dumps
21+
- 🐢 **Slow requests** - Generates long-running requests with sync-over-async patterns for CLR Profiler analysis
22+
- 💥 **Application crashes** - Triggers fatal crashes for testing Azure Crash Monitoring and memory dumps
23+
- ❌ **Failed requests** - Generates HTTP 5xx errors visible in AppLens and Application Insights
2324

2425
**Only deploy this application in isolated, non-production environments.**
2526

@@ -160,6 +161,25 @@ The slow request simulator generates requests using three different sync-over-as
160161
- **NestedSyncOverAsync**: Sync methods that block internally - look for `*_BLOCKS_INTERNALLY` methods
161162
- **DatabasePattern**: Simulated database/HTTP blocking - look for `*Sync_SYNC_BLOCK` methods
162163

164+
### Failed Request Simulation
165+
166+
| Endpoint | Method | Description |
167+
|----------|--------|-------------|
168+
| `/api/failedrequest/start` | POST | Start generating HTTP 500 errors |
169+
| `/api/failedrequest/stop` | POST | Stop the simulation |
170+
| `/api/failedrequest/status` | GET | Get current simulation status |
171+
172+
**Request body (start):**
173+
```json
174+
{
175+
"requestCount": 10
176+
}
177+
```
178+
179+
- `requestCount`: Number of HTTP 500 errors to generate (default: 10)
180+
181+
Each failed request throws a random exception type (NullReferenceException, TimeoutException, InvalidOperationException, etc.) visible in AppLens and Application Insights.
182+
163183
### Crash Simulation
164184

165185
| Endpoint | Method | Description |

‎specs/001-perf-problem-simulator/contracts/openapi.yaml‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -404,6 +404,78 @@ paths:
404404
schema:
405405
$ref: '#/components/schemas/LoadTestStats'
406406

407+
/api/failedrequest/start:
408+
post:
409+
tags:
410+
- FailedRequest
411+
summary: Start generating failed requests (HTTP 500 errors)
412+
description: |
413+
Starts a background process that generates HTTP 5xx errors by calling the load test
414+
endpoint with 100% error probability. Each request throws a random exception type
415+
(NullReferenceException, TimeoutException, InvalidOperationException, etc.).
416+
417+
**Educational Note**: These failures will appear in AppLens and Application Insights,
418+
allowing you to practice diagnosing application errors using Azure diagnostic tools.
419+
420+
**Where to observe:**
421+
- Azure Portal → App Service → Diagnose and Solve Problems → AppLens
422+
- Application Insights → Failures blade
423+
- Dashboard Event Log (hot pink entries with exception type)
424+
operationId: startFailedRequest
425+
requestBody:
426+
required: false
427+
content:
428+
application/json:
429+
schema:
430+
$ref: '#/components/schemas/FailedRequestRequest'
431+
examples:
432+
default:
433+
summary: Generate 10 failed requests
434+
value:
435+
requestCount: 10
436+
stress:
437+
summary: Generate 50 failed requests
438+
value:
439+
requestCount: 50
440+
responses:
441+
'200':
442+
description: Failed request simulation started
443+
content:
444+
application/json:
445+
schema:
446+
$ref: '#/components/schemas/SimulationResult'
447+
448+
/api/failedrequest/stop:
449+
post:
450+
tags:
451+
- FailedRequest
452+
summary: Stop the failed request simulation
453+
description: |
454+
Stops generating new failed requests. Requests already in progress will complete.
455+
operationId: stopFailedRequest
456+
responses:
457+
'200':
458+
description: Simulation stopped
459+
content:
460+
application/json:
461+
schema:
462+
$ref: '#/components/schemas/SimulationResult'
463+
464+
/api/failedrequest/status:
465+
get:
466+
tags:
467+
- FailedRequest
468+
summary: Get failed request simulation status
469+
description: Returns the current status including requests sent, completed, and in progress.
470+
operationId: getFailedRequestStatus
471+
responses:
472+
'200':
473+
description: Current simulation status
474+
content:
475+
application/json:
476+
schema:
477+
$ref: '#/components/schemas/FailedRequestStatus'
478+
407479
components:
408480
schemas:
409481
CpuStressRequest:

‎specs/001-perf-problem-simulator/data-model.md‎

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
│ Request Models (DTOs) │
1212
├─────────────────────────────────────────────────────────────────┤
1313
│ CpuStressRequest MemoryAllocationRequest │
14-
│ ThreadBlockRequest ResetRequest │
14+
│ ThreadBlockRequest FailedRequestRequest ResetRequest │
1515
└─────────────────────────────────────────────────────────────────┘
1616
│
1717
▼
@@ -79,6 +79,37 @@ Represents a request to trigger sync-over-async thread blocking.
7979

8080
---
8181

82+
### FailedRequestRequest
83+
84+
Represents a request to generate HTTP 5xx errors.
85+
86+
| Field | Type | Required | Default | Constraints | Description |
87+
|-------|------|----------|---------|-------------|-------------|
88+
| `RequestCount` | `int` | No | 10 | Min: 1 | Number of HTTP 500 errors to generate |
89+
90+
**Behavior:**
91+
- Each request calls `/api/loadtest` with 100% error probability
92+
- Requests take ~1.5 seconds each for latency visibility
93+
- Random exception types are thrown (TimeoutException, NullReferenceException, etc.)
94+
- Exception type is displayed in dashboard Event Log
95+
96+
---
97+
98+
### FailedRequestStatus
99+
100+
Status information for the failed request simulation.
101+
102+
| Field | Type | Description |
103+
|-------|------|-------------|
104+
| `IsRunning` | `bool` | Whether the simulation is currently running |
105+
| `RequestsSent` | `int` | Number of requests initiated |
106+
| `RequestsCompleted` | `int` | Number of requests completed (with expected 5xx) |
107+
| `RequestsInProgress` | `int` | Number of requests currently in flight |
108+
| `TargetCount` | `int` | Total number of failures to generate |
109+
| `StartedAt` | `DateTimeOffset?` | When the simulation started |
110+
111+
---
112+
82113
## Domain Models
83114

84115
### AllocatedMemoryBlock
@@ -231,6 +262,7 @@ Standard error response format.
231262
| MemoryAllocationRequest | SizeMegabytes | 10 ≤ value ≤ 1024 |
232263
| ThreadBlockRequest | DelayMilliseconds | 100 ≤ value ≤ 30000 |
233264
| ThreadBlockRequest | ConcurrentRequests | 1 ≤ value ≤ 200 |
265+
| FailedRequestRequest | RequestCount | value ≥ 1 |
234266

235267
All validation failures return HTTP 400 with `ErrorResponse` body.
236268

‎specs/001-perf-problem-simulator/quickstart.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,30 @@ curl -X POST https://localhost:5001/api/trigger-sync-over-async `
109109
- Response times for ALL endpoints increase
110110
- CPU and memory appear normal (the key diagnostic clue!)
111111

112+
### Trigger Failed Requests (HTTP 500 Errors)
113+
114+
```powershell
115+
# Generate 10 failed requests (default)
116+
curl -X POST https://localhost:5001/api/failedrequest/start
117+
118+
# Generate 50 failed requests
119+
curl -X POST https://localhost:5001/api/failedrequest/start `
120+
-H "Content-Type: application/json" `
121+
-d '{"requestCount": 50}'
122+
123+
# Check status
124+
curl https://localhost:5001/api/failedrequest/status
125+
126+
# Stop the simulation
127+
curl -X POST https://localhost:5001/api/failedrequest/stop
128+
```
129+
130+
**What to observe:**
131+
- HTTP 500 errors appear in AppLens (Azure Portal → Diagnose and Solve Problems)
132+
- Application Insights → Failures blade shows error spikes
133+
- Dashboard Event Log shows specific exception types (NullReferenceException, TimeoutException, etc.) in hot pink
134+
- Each error takes ~1.5 seconds, making them visible in latency monitoring
135+
112136
---
113137

114138
## Environment Configuration

‎specs/001-perf-problem-simulator/spec.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,26 @@ As a learner, I want a simple web dashboard showing the current state of the app
8989

9090
---
9191

92+
### User Story 5 - Generate Failed Requests for AppLens Analysis (Priority: P2)
93+
94+
As a developer or support engineer, I want to generate HTTP 5xx errors so that I can observe how failed requests appear in AppLens and Application Insights, and practice diagnosing application failures.
95+
96+
**Why this priority**: HTTP 500 errors are one of the most common issues investigated through Azure diagnostics. Understanding how failures appear in AppLens and Application Insights is essential for effective production troubleshooting.
97+
98+
**Independent Test**: Can be tested by invoking the failed request endpoint and observing errors appear in AppLens (Azure Portal → Diagnose and Solve Problems) and the Application Insights Failures blade.
99+
100+
**Acceptance Scenarios**:
101+
102+
1. **Given** the application is running normally, **When** I invoke the failed request endpoint with a count of 10, **Then** 10 HTTP 500 errors are generated and logged.
103+
104+
2. **Given** I am monitoring Application Insights, **When** I trigger failed requests, **Then** the errors appear in the Failures blade within 2-3 minutes with detailed exception information.
105+
106+
3. **Given** I am viewing the dashboard, **When** failed requests are generated, **Then** each failure appears in the Event Log with the specific exception type (e.g., NullReferenceException, TimeoutException) displayed in hot pink.
107+
108+
4. **Given** I have deployed to Azure App Service, **When** I generate failed requests, **Then** the errors are visible in AppLens diagnostics for incident analysis practice.
109+
110+
---
111+
92112
### Edge Cases
93113

94114
- What happens when a user requests more memory than available? The application should cap allocation at a safe maximum and report the limitation rather than crashing.
@@ -107,6 +127,7 @@ As a learner, I want a simple web dashboard showing the current state of the app
107127
- **FR-003**: System MUST provide an endpoint to release previously allocated memory
108128
- **FR-004**: System MUST provide an endpoint that demonstrates sync-over-async thread blocking with configurable concurrency and delay parameters
109129
- **FR-005**: System MUST allow multiple problem types to be active simultaneously
130+
- **FR-018**: System MUST provide an endpoint to generate HTTP 5xx errors with configurable count, using random exception types visible in AppLens and Application Insights
110131

111132
**Observability & Feedback**
112133

‎src/PerfProblemSimulator/Controllers/LoadTestController.cs‎

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -269,8 +269,26 @@ public async Task<IActionResult> ExecuteLoadTest(
269269
request.ErrorAfterSeconds,
270270
request.ErrorPercent);
271271

272-
var result = await _loadTestService.ExecuteWorkAsync(request, cancellationToken);
273-
return Ok(result);
272+
try
273+
{
274+
var result = await _loadTestService.ExecuteWorkAsync(request, cancellationToken);
275+
return Ok(result);
276+
}
277+
catch (Exception ex)
278+
{
279+
// Return structured error response with exception type for diagnostics
280+
// This allows callers (like FailedRequestService) to extract the exception type
281+
_logger.LogInformation(
282+
"Load test threw expected exception: {ExceptionType} - {Message}",
283+
ex.GetType().Name, ex.Message);
284+
285+
return StatusCode(500, new
286+
{
287+
error = ex.GetType().Name,
288+
message = ex.Message,
289+
type = ex.GetType().FullName
290+
});
291+
}
274292
}
275293

276294
/*

‎src/PerfProblemSimulator/wwwroot/azure-monitoring-guide.html‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -968,6 +968,97 @@ <h4>Downloading the Dump File</h4>
968968
</ol>
969969
</section>
970970

971+
<!-- Diagnosing Failed Requests Section -->
972+
<section id="failedrequests" class="doc-section">
973+
<h2>❌ Diagnosing Failed Requests (HTTP 5xx)</h2>
974+
<p>HTTP 500 errors are one of the most common issues that impact application availability. This simulation generates random exceptions to practice diagnosing failed requests in Azure.</p>
975+
976+
<h3>What to Simulate</h3>
977+
<ol>
978+
<li>In the dashboard, click <strong>❌ Start Failed Requests</strong></li>
979+
<li>Wait for requests to complete (each takes ~1.5 seconds)</li>
980+
<li>Observe exception types in the Event Log (hot pink entries)</li>
981+
</ol>
982+
983+
<h3>Diagnosing with AppLens</h3>
984+
<ol>
985+
<li>In Azure Portal, go to your App Service</li>
986+
<li>Navigate to <strong>Diagnose and solve problems</strong></li>
987+
<li>Search for <strong>"Application Errors"</strong> or <strong>"HTTP 500"</strong></li>
988+
<li>AppLens will show:
989+
<ul>
990+
<li>Error rate over time</li>
991+
<li>Most common exception types</li>
992+
<li>Stack traces for each failure</li>
993+
<li>Correlation with other symptoms</li>
994+
</ul>
995+
</li>
996+
</ol>
997+
998+
<h3>Diagnosing with Application Insights</h3>
999+
<ol>
1000+
<li>In Azure Portal, go to your Application Insights resource</li>
1001+
<li>Click <strong>Failures</strong> in the left menu</li>
1002+
<li>Review the failures timeline and exception breakdown</li>
1003+
<li>Click on individual exceptions to see:
1004+
<ul>
1005+
<li>Full exception stack trace</li>
1006+
<li>Request details (URL, duration, response code)</li>
1007+
<li>Custom properties and telemetry</li>
1008+
</ul>
1009+
</li>
1010+
</ol>
1011+
1012+
<h3>Common Exception Types You'll See</h3>
1013+
<table class="latency-table">
1014+
<thead>
1015+
<tr>
1016+
<th>Exception</th>
1017+
<th>Real-World Cause</th>
1018+
<th>How to Fix</th>
1019+
</tr>
1020+
</thead>
1021+
<tbody>
1022+
<tr>
1023+
<td><code>NullReferenceException</code></td>
1024+
<td>Accessing property of null object</td>
1025+
<td>Add null checks, use nullable reference types</td>
1026+
</tr>
1027+
<tr>
1028+
<td><code>TimeoutException</code></td>
1029+
<td>External dependency too slow</td>
1030+
<td>Increase timeout, add circuit breaker</td>
1031+
</tr>
1032+
<tr>
1033+
<td><code>InvalidOperationException</code></td>
1034+
<td>Object in wrong state for operation</td>
1035+
<td>Check preconditions, validate state</td>
1036+
</tr>
1037+
<tr>
1038+
<td><code>HttpRequestException</code></td>
1039+
<td>External API call failed</td>
1040+
<td>Add retry logic, check connectivity</td>
1041+
</tr>
1042+
<tr>
1043+
<td><code>KeyNotFoundException</code></td>
1044+
<td>Missing configuration or cache key</td>
1045+
<td>Use TryGetValue, check existence first</td>
1046+
</tr>
1047+
</tbody>
1048+
</table>
1049+
1050+
<div class="info-box">
1051+
<h4>💡 Pro Tip: Creating Error Rate Alerts</h4>
1052+
<p>Set up an Azure Monitor alert to notify you when the HTTP 5xx error rate exceeds a threshold:</p>
1053+
<ol>
1054+
<li>Go to your App Service → <strong>Alerts</strong> → <strong>Create alert rule</strong></li>
1055+
<li>Signal: <strong>Http Server Errors</strong></li>
1056+
<li>Threshold: Dynamic or Static (e.g., > 5 per minute)</li>
1057+
<li>Action: Email, SMS, or webhook notification</li>
1058+
</ol>
1059+
</div>
1060+
</section>
1061+
9711062
<!-- Using the Request Latency Monitor Section -->
9721063
<section id="latency" class="doc-section">
9731064
<h2>⏱️ Using the Request Latency Monitor</h2>
@@ -1405,6 +1496,9 @@ <h2>🔗 Additional Resources</h2>
14051496
<li class="doc-toc-item">
14061497
<a href="#crash"><span class="toc-icon">💥</span>Crashes</a>
14071498
</li>
1499+
<li class="doc-toc-item">
1500+
<a href="#failedrequests"><span class="toc-icon">❌</span>Failed Requests</a>
1501+
</li>
14081502
<li class="doc-toc-divider-item" aria-hidden="true"></li>
14091503
<li class="doc-toc-section-item">Tools</li>
14101504
<li class="doc-toc-item">

0 commit comments

Comments
 (0)