Skip to content

Write live proxies to output file incrementally as they are found#19

Open
AlisaLC wants to merge 1 commit into
mheidari98:mainfrom
AlisaLC:feature/write-live-proxies-incrementally
Open

Write live proxies to output file incrementally as they are found#19
AlisaLC wants to merge 1 commit into
mheidari98:mainfrom
AlisaLC:feature/write-live-proxies-incrementally

Conversation

@AlisaLC
Copy link
Copy Markdown

@AlisaLC AlisaLC commented Mar 27, 2026

Previously, the output file was only written after all threads finished. Now a shared list and lock are used across threads so the file is written (sorted by ping) each time a live proxy is discovered, making results available immediately during a long-running check.

Previously, the output file was only written after all threads finished.
Now a shared list and lock are used across threads so the file is
written (sorted by ping) each time a live proxy is discovered, making
results available immediately during a long-running check.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@mheidari98
Copy link
Copy Markdown
Owner

Code review

Found 2 issues:

  1. --reuse data loss: when --reuse is set, prior contents of the output file are loaded as candidates to be re-checked. The new write path opens outputFile in 'w' mode (truncating) and rewrites only sharedProxy, which starts empty. As soon as the first live proxy is found, the file is truncated and replaced with that single entry — wiping every previously-saved proxy that has not yet been re-tested. On Ctrl+C mid-run this leaves the user with a strict subset of their reused list. Previously the file was rewritten once at the end after results were merged.

if args.reuse and os.path.isfile(args.output):
with open(args.output, 'r', encoding='UTF-8') as f:
lines.update( parseContent(f.read().strip()) )

liveProxy.append((url, ping))
if sharedProxy is not None and lock is not None and outputFile is not None:
with lock:
sharedProxy.append((url, ping))
sharedProxy.sort(key=lambda x: x[1])
with open(outputFile, 'w', encoding="utf-8") as f:
for ss_url in sharedProxy:
f.write(f"{ss_url[0]}\n")
else:

  1. Full file rewrite + sort under the lock on every live hit: each successful check sorts the entire sharedProxy and rewrites the whole output file while holding the global lock, so file I/O serializes all worker threads. Cost grows O(k²) in live count and contention scales with thread count. Move the I/O outside the lock (e.g. snapshot the list under the lock, write outside), or write incrementally without re-sorting+rewriting from scratch (bisect.insort + atomic temp-file rename).

liveProxy.append((url, ping))
if sharedProxy is not None and lock is not None and outputFile is not None:
with lock:
sharedProxy.append((url, ping))
sharedProxy.sort(key=lambda x: x[1])
with open(outputFile, 'w', encoding="utf-8") as f:
for ss_url in sharedProxy:
f.write(f"{ss_url[0]}\n")
else:

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants