From 1d593038c4f9afc8ef387b3b72b438243b751ab2 Mon Sep 17 00:00:00 2001 From: Yogesh Rao Date: Thu, 14 May 2026 12:35:43 +0530 Subject: [PATCH] =?UTF-8?q?feat:=20improve=20writing-autests=20skill=20sco?= =?UTF-8?q?re=20(38%=20=E2=86=92=2092%)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Hey @bneradt 👋 I ran your skills through `tessl skill review` at work and found some targeted improvements for the `writing-autests` skill. Here's the full before/after: | Skill | Before | After | Change | |-------|--------|-------|--------| | writing-autests | 38% | 92% | **+54%** | | build-pv | 52% | — | — | | plan-issue | 53% | — | — | | commit | 65% | — | — | | run-autests | 65% | — | — | | review-pr | 67% | — | — | | apply-pr-comments | 68% | — | — | | create-pr | 76% | — | — | | run-unit-tests | 77% | — | — | I picked `writing-autests` because it had the most room for improvement and it's a core skill for the project — AuTests are the backbone of your e2e testing.
Changes made to writing-autests - **Expanded description** with a "Use when..." clause listing concrete trigger scenarios (writing e2e tests, adding replay YAML, updating .test.py files, configuring gold files) - **Added directory structure reference** showing the standard `/` layout with `.test.py`, `replay_files/`, and `gold/` subdirectories - **Added a numbered workflow** (6 steps: create dir → write .test.py → create replay YAML → add gold files → run → verify) - **Added a minimal test example** based on real patterns from the repo (using `AddClientProcess`, `AddServerProcess`, `AddProxyProcess`) - **Added replay YAML structure example** showing the `meta`/`sessions`/`transactions` format with `esc_json` encoding - **Documented key extension APIs** (`AddClientProcess`, `AddServerProcess`, `AddProxyProcess`, `Testers.ContainsExpression`, `Testers.ExcludesExpression`) - **Fixed path typo** (`test/autests` → `tests/autests`) - **Cross-referenced** the `run-autests` sibling skill for running tests
I also stress-tested your `run-autests` skill against a few real-world task evals and it held up really well on parallel test execution with `autest.sh -j` and selective test filtering with `-f`. Kudos for that. Honest disclosure — I work at @tesslio where we build tooling around skills like these. Not a pitch — just saw room for improvement and wanted to contribute. Want to self-improve your skills? Just point your agent (Claude Code, Codex, etc.) at [this Tessl guide](https://docs.tessl.io/evaluate/optimize-a-skill-using-best-practices) and ask it to optimize your skill. Ping me — [@yogesh-tessl](https://github.com/yogesh-tessl) — if you hit any snags. Thanks in advance 🙏 --- .agents/skills/writing-autests/SKILL.md | 111 ++++++++++++++++++++++-- 1 file changed, 103 insertions(+), 8 deletions(-) diff --git a/.agents/skills/writing-autests/SKILL.md b/.agents/skills/writing-autests/SKILL.md index a59a0b16..641c7bbe 100644 --- a/.agents/skills/writing-autests/SKILL.md +++ b/.agents/skills/writing-autests/SKILL.md @@ -1,14 +1,109 @@ --- name: writing-autests -description: Guidance for adding Proxy Verifier autests, the end to end tests. +description: "Create and modify Proxy Verifier AuTests (end-to-end replay tests). Use when writing new e2e test cases, adding replay YAML scenarios for HTTP/HTTPS/HTTP2/HTTP3 verification, updating .test.py files, or configuring gold file comparisons." --- -# Write ATS AuTests +# Write Proxy Verifier AuTests -AuTest is an end to end testing framework documented here: -https://autestsuite.bitbucket.io/ +AuTest is the end-to-end testing framework for Proxy Verifier. Documentation: https://autestsuite.bitbucket.io/ -- Tests exis in the `test/autests/gold_tests` directory. -- The extensions to AuTest specific to Proxy Verifier are in the `test/autests/gold_tests/autest-site` directory. -- Tests end with the `.test.py` extension. -- When writing new tests, view the existing `test/autests/gold_tests` `.test.py` files for inspiration. +## Directory Structure + +Each test lives in its own subdirectory under `tests/autests/gold_tests/`: + +``` +tests/autests/gold_tests// +├── .test.py +├── replay_files/ +│ └── single_transaction.yaml +└── gold/ + ├── _client.gold + ├── _server.gold + └── _proxy.gold +``` + +Proxy Verifier extensions are in `tests/autests/gold_tests/autest-site/`. + +## Workflow + +1. Create a subdirectory under `tests/autests/gold_tests/` named for the test. +2. Write a `.test.py` file with the required file prologue (see example below). +3. Create replay YAML files under `replay_files/`. +4. Add gold files under `gold/` by running the test once to capture output, or write expected output manually. +5. Run the test with `./build/dev-external/autest.sh -f ` (see `../run-autests/SKILL.md`). +6. Verify output matches gold files and no `Violation:` errors appear. + +## Minimal Test Example + +```python +''' +Verify . +''' +# @file +# +# Copyright 2025, Yahoo Inc. +# SPDX-License-Identifier: Apache-2.0 +# + +Test.Summary = ''' +Verify . +''' + +r = Test.AddTestRun("Description of this test run") +client = r.AddClientProcess("client1", "replay_files/my_replay.yaml") +server = r.AddServerProcess("server1", "replay_files/my_replay.yaml") +proxy = r.AddProxyProcess("proxy1", + listen_port=client.Variables.http_port, + server_port=server.Variables.http_port) + +client.Streams.stdout = "gold/my_replay_client.gold" +server.Streams.stdout = "gold/my_replay_server.gold" + +client.Streams.stdout += Testers.ExcludesExpression( + "Violation:", "There should be no verification errors.") +server.Streams.stdout += Testers.ExcludesExpression( + "Violation:", "There should be no verification errors.") +``` + +For HTTPS, pass `use_ssl=True` to `AddProxyProcess` and use `Variables.https_port`. + +## Replay YAML Structure + +```yaml +--- +meta: + version: '1.0' + +sessions: +- protocol: + stack: http + transactions: + - client-request: + version: '1.1' + method: GET + url: /path + headers: + encoding: esc_json + fields: + - [ Host, example.com ] + - [ uuid, 1 ] + server-response: + status: 200 + reason: OK + headers: + encoding: esc_json + fields: + - [ Content-Length, '0' ] +``` + +For HTTPS sessions, use `stack: https` with a `tls: { sni: }` block. + +## Key Extension APIs + +Available in `.test.py` via extensions in `autest-site/`: + +- `r.AddClientProcess(name, replay_dir, configure_http=True, configure_https=True, use_ipv6=False, other_args='')` — verifier-client process +- `r.AddServerProcess(name, replay_dir, configure_http=True, configure_https=True, other_args='')` — verifier-server process +- `r.AddProxyProcess(name, listen_port, server_port, use_ssl=False)` — test proxy process +- `Testers.ContainsExpression(expr, reason)` — assert output contains a pattern +- `Testers.ExcludesExpression(expr, reason)` — assert output does not contain a pattern