Skip to content

rayforce: wait for the old server to exit before restarting; client timeouts cover a long query - #2244

Merged
alexey-milovidov merged 4 commits into
ClickHouse:mainfrom
ser-vasilich:rayforce-harness-restart
Sep 27, 2026
Merged

alexey-milovidov merged 4 commits into
ClickHouse:mainfrom
ser-vasilich:rayforce-harness-restart

Conversation

@ser-vasilich

Copy link
Copy Markdown
Contributor

Harness scripts only; no results.

stop / start. After ./stop the rayforce server keeps its listening socket open while it unmaps the table, and its command line is already cleared. An immediate ./start therefore neither found the process nor could bind the port and failed with Address already in use; the try was recorded as null. Both scripts now wait until the process and the port are gone.

check.rfl / query.rfl. The server answers one IPC request at a time. With a 2 s handshake timeout the watchdog's probe saw a busy server as dead during the concurrent phase and restarted it mid-query; with a 60 s connect timeout a client waiting behind a long query gave up. Both timeouts now cover the 600 s concurrent window.

@CLAassistant

CLAassistant commented Sep 26, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

…longer client timeouts

After `./stop` the server keeps its listening socket while it unmaps the
table, and its command line is already cleared, so an immediate `./start`
could neither bind the port nor see the process and failed with "Address
already in use". Both scripts now wait until the process and the port are
gone.

The client timeouts (2 s in check.rfl, 60 s in query.rfl) were shorter than
a long query: the server answers one request at a time, so during the
concurrent phase a waiting client, or the watchdog's probe, timed out and
the server was restarted while busy. They now cover the 600 s window.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@ser-vasilich
ser-vasilich force-pushed the rayforce-harness-restart branch from 2a865af to 98d7670 Compare September 26, 2026 09:42
@alexey-milovidov alexey-milovidov added the machine:all PR benchmark on every machine type label Sep 26, 2026
@alexey-milovidov
alexey-milovidov deployed to benchmark-approval September 26, 2026 20:37 — with GitHub Actions Active
@github-actions

Copy link
Copy Markdown
Contributor

The run of rayforce on c6a.2xlarge did not produce results.
The run of rayforce on c6a.xlarge did not produce results.

Logs:

@github-actions

Copy link
Copy Markdown
Contributor

Results for rayforce are ready for: c6a.metal, c7a.metal-48xl, c8g.metal-48xl.
The result files are committed as d881363.
The run of rayforce on c6a.large did not produce results.

Logs:

@github-actions

Copy link
Copy Markdown
Contributor

Results for rayforce are ready for: c8g.4xlarge.
The result files are committed as 06192f1.

Logs:

@github-actions

Copy link
Copy Markdown
Contributor

Results for rayforce are ready for: c6a.4xlarge.
The result files are committed as 0ab7c1a.

Logs:

@github-actions

Copy link
Copy Markdown
Contributor

The run of rayforce on t3a.small did not produce results.

Logs:

@alexey-milovidov
alexey-milovidov merged commit 273d3a3 into ClickHouse:main Sep 27, 2026
1 of 2 checks passed
@alexey-milovidov alexey-milovidov self-assigned this Sep 27, 2026
@ser-vasilich
ser-vasilich deleted the rayforce-harness-restart branch September 27, 2026 06:16

This branch is waiting to be deployed

1 waiting deployment
benchmark-approval — 0ab7c1a3 Waiting Sep 26, 2026 by robot-clickhouse via launch #540
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

machine:all PR benchmark on every machine type

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants