Repository navigation
release 1.30.0: per-check states, --pick --verify, --diagnose - #341
Merged
Merged
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| try: | ||
| host, port = parts.hostname, parts.port | ||
| except ValueError: | ||
| host = port = None |
| break | ||
| writer.write(data) | ||
| await writer.drain() | ||
| except (ConnectionError, OSError): |
| await reader.readuntil(b"\r\n\r\n") | ||
| writer.write(_reply(self._seen(writer).encode() + b"\n")) | ||
| await writer.drain() | ||
| except (ConnectionError, OSError, asyncio.IncompleteReadError): |
| ok = self.https_status == b"200 OK" | ||
| writer.write(_reply(self._seen(writer).encode() + b"\n" if ok else b"busy", self.https_status)) | ||
| await writer.drain() | ||
| except (ConnectionError, OSError, asyncio.IncompleteReadError): |
| else: | ||
| writer.write(_reply(b"", b"404 Not Found")) | ||
| await writer.drain() | ||
| except (ConnectionError, OSError, asyncio.IncompleteReadError): |
| answer = _reply(rewrite(body), content_type=b"text/html") if answer_head else answer | ||
| writer.write(answer) | ||
| await writer.drain() | ||
| except (ConnectionError, OSError, asyncio.IncompleteReadError, ValueError): |
| during = len(checking()) | ||
| task.cancel() | ||
| with pytest.raises(asyncio.CancelledError): | ||
| await task |
A check that didn't run was reported like one that passed: without a reachable confirmation target confirm() returned True, and tampers() returned False without a reference or on any error. And a confirmation target that went down during a run turned every proxy after it into a "fake" that the history then learned as dead. Each check now leaves a Check on the result (state, reason code, target, time, measurements, limits). A failed second request or HTTPS test only counts against a proxy while the target itself still answers directly (asked at most every 10 s). proxies.json gains checked_at and checks, proxies.csv a column per check; every existing field keeps its name and type. Each run also writes run.json with its denominators.
…p what each check found --pick N --verify checks the candidates of the hourly list from this machine and prints only those that pass the handshake, the second request and the content check now (with --https-only the HTTPS test too). It tries 5 candidates per proxy asked for (10 to 60), at most 8 at a time, for 90 s at most; exit code 3 when fewer than N passed. --diagnose PROXY runs every check for one proxy and shows what each found, as text or JSON, with passwords masked. --pick --json adds when the list was published and whether it is overdue; plain --pick prints the same as before.
A proxy is listed only if its basic check, second request and content check passed; https.txt also needs the HTTPS test. "Unavailable" is not enough, so a run whose confirmation target was down publishes nothing and the old list stays online. stats.json says what each list requires and how many hits were held back; run.json and history.json carry the run's denominators (candidates, checked, not checked, each check by state and reason). MCP results gain checked_at, checks and list_stale.
…s why local addresses are skipped
…validated From two independent reviews of the check states and --pick --verify: - The direct question to a check target now has to be newer than the failing check; an older "fine" could predate an outage. Failures in between wait for the next answer (asked every 3 s at most). - A check target with a bad certificate of its own, or one that answers with an error over TLS, no longer fails the HTTPS test of every proxy. - --diagnose with a malformed URL printed the URL, password included. - Rows of the downloaded list are only used if they are ip:port of a public address: nothing else is printed or connected to. - Deeply nested JSON from a proxy crashed the run on Python 3.9. - Out of sockets is no longer learned as a dead proxy. - --pick: Ctrl+C during the download exits 130, --json is JSON also when nothing matches, a list without a readable time is stale, a slow candidate that is no longer needed isn't waited for.
"Honeypot check" becomes the second request it is, "injects scripts" the changed test page it is, and the HTTPS test a TLS test to one host. The "typical free list" column on the website is gone (no other list was measured), the animated run is labelled as an illustration and counts the 150,000 candidates a run really checks.
…at produce them docs/analysis reproduces the funnel from the workflow logs (kept as a CSV, since GitHub deletes the logs after 90 days) and the list churn from seen.json, per entry and per host. The article now says what each number counts and what it can't tell: 3.6 million are checks, not proxies; the page differed for 5.2 % of the proxies that got that far, for reasons nobody recorded; 85 % is a share of ended stretches, and about a quarter of a list is on the list again a day later. docs/check-states.md is the reference for the new fields.
…aims the data carries The top is now one sentence, a terminal recording, a quickstart, the three ways to get proxies side by side, what you get and what it can't do. The recording is made against simulated proxies on localhost and says so; docs/demo has the lab and the tape to make it again. The generated screenshots are labelled as illustrations with sample data. The comparison is by task (get a few checked proxies, check your own list, understand a failure, run a pool, use it from Python), includes proxy_pool and proxybroker2, and says where each is the better choice. Corrected in it: monosans does verify TLS by default, clarketm does list countries, and nobody ran the other lists through our checks.
…ge, by their names
Also: the last rows of "Why not just download a list?" still had the removed competitor column.
…dn't An independent pass over every public statement found 41 places. Fixed in code where the statement was worth making true: - The hourly list's CSV keeps its old columns in place (the new ones moved four of them). - --diagnose masks a login written as host:port:user:pass too. - --diagnose and --pick --verify ask the first check target directly after a failed first request, like the later checks; a scan keeps JudgeWatch. A run still calls check(), so a replaced check() counts. - run.json counts changed test pages with and without a <script tag; before, those measurements were dropped with the proxy. - Rows of the downloaded list are validated where the list is loaded, so the MCP tools and the rotating fetcher get them too. Fixed in the text where it said more than there is: the churn table counted only each entry's last stretch (now both counts, from the script), 5.2 % is a lower bound, the control is asked every 3 s, a shorter list is not "no list", "every check" does not include HTTPS, and several cells of the comparison that went beyond their source.
… and 'nothing may be injected'
1.29.0 shipped without this work, so every 'from 1.29' in the docs becomes 'from 1.30'.
…ort test ids for the 200k-bracket bodies
… the 15.6 ms loop clock
…C end them mid-answer
start_server keeps no reference to its handler task before 3.11, so a lab proxy could be destroyed
while forwarding ('Task was destroyed but it is pending!') and the checker read an empty reply as
bad_status. The rotating server already holds its handlers in _handlers.
…mands; social preview shows a --diagnose result instead of '700+ sources, five checks'
maximilianfeix
force-pushed
the
release/1.30.0
branch
from
October 16, 2026 20:50
b8cd66c to
1b72b87
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
1.29.0 went out without the check-states work that had been sitting on a local branch. This PR rebases it onto
main(after #340) and bumps to 1.30.0 in the same PR, so the hourly workflow never advertises commands PyPI doesn't have yet.What users get
checks(basic,confirm,content,https), eachpassed/failed/unavailable/skippedwith a reason code, the target and the time. They show up inproxies.json, as new columns at the end ofproxies.csv, in the Python API and in the MCP results. Seedocs/check-states.md.--pick N --verifychecks candidates from your own network and prints only the ones that pass right now (bounded: 5 per proxy asked for, 8 at once, 90 s).--diagnose PROXYshows what each check found and why.docs/demo/(localhost lab, labelled as such).What the docs no longer say
The numbers article (
docs/free-proxies-in-numbers.md) was redone with its denominators and the scripts indocs/analysis/. "Honeypots", "1 in 40 inject scripts" and "85 % gone in an hour" are gone from the README, the translations, the website, the metadata and the Discord bot's/about.Rebase notes
README.ru.md: the native proofread from docs(ru): native-speaker proofread of README.ru.md #339 (@mrvi0) is kept everywhere except the 7 paragraphs this PR rewrites for content. Those paragraphs are machine-translated again and need another look.--serve-open(serve: no open proxy by accident – --serve-host needs a password or --serve-open #328) and the new--verify/--diagnose/--jsonrows are both in the option tables.Found by the CI matrix (only macOS ran locally before)
timeout("accepted but did not answer"), and with a control set it could become "target unreachable". On Windows, which retries a refused local port for ~2 s, this was the normal case. It is nowconnect_timeout, with a test.time.perf_counter().start_serverhandler tasks, and the GC ended lab proxies mid-answer. That caused the intermittentbad_statusfailures. The lab now holds its handlers (the rotating server already did).Also
docs/social-preview.pngnow shows a--diagnoseresult instead of "700+ sources, five checks".After merge
Tag
v1.30.0, then watch the first three hourly runs (held_backinstats.json,run.jsonon the list branch).Local: 1199 passed, 2 skipped; ruff clean.