feat(firmware): OTA tooling that verifies the node actually took the image - #1836
Open
clonea1 wants to merge 1 commit into
Open
feat(firmware): OTA tooling that verifies the node actually took the image#1836clonea1 wants to merge 1 commit into
clonea1 wants to merge 1 commit into
Conversation
…image Two scripts for pushing firmware over the air to a fleet, both built around one idea: "the OTA endpoint returned 200" is not evidence that anything changed. ota_push.py pushes to one node and then polls /ota/status until the node reports the EXPECTED version from a DIFFERENT partition than it started on. A version string that did not move means the update did not take, however healthy the response looked -- and a node that reboots the instant it finishes writing can drop the HTTP response entirely, so the upload result alone cannot be trusted either way. It also refuses to push a file that does not start with 0xE9 rather than discovering that after the write. fleet_ota.sh rolls an image across the fleet ONE node at a time and stops at the first node that fails to verify. Sequential and fail-fast is deliberate: a parallel push puts every node at risk simultaneously, and a node that does not come back needs physical access. On failure the nodes it has not reached are left alone. --first lets you start with a board you can recover, so a bad image lands where it is cheapest to fix. Node addresses are discovered from /api/v1/nodes rather than hard-coded, so this works on any fleet with no edits; --nodes overrides for a fleet the server has not seen. The version defaults to the contents of version.txt so the check cannot drift from what was built. The PSK is taken from a FILE named by RUVIEW_OTA_PSK_FILE, never from argv or an environment variable holding the key itself -- both are visible in a process listing, and that key can replace firmware. Both scripts refuse to run without it rather than falling back to something permissive. Co-Authored-By: claude-flow <ruv@ruv.net>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two scripts for pushing firmware over the air to a fleet, both built around one
idea: "the OTA endpoint returned 200" is not evidence that anything changed.
ota_push.py pushes to one node and then polls /ota/status until the node
reports the EXPECTED version from a DIFFERENT partition than it started on. A
version string that did not move means the update did not take, however healthy
the response looked -- and a node that reboots the instant it finishes writing
can drop the HTTP response entirely, so the upload result alone cannot be
trusted either way. It also refuses to push a file that does not start with
0xE9 rather than discovering that after the write.
fleet_ota.sh rolls an image across the fleet ONE node at a time and stops at
the first node that fails to verify. Sequential and fail-fast is deliberate: a
parallel push puts every node at risk simultaneously, and a node that does not
come back needs physical access. On failure the nodes it has not reached are
left alone. --first lets you start with a board you can recover, so a bad image
lands where it is cheapest to fix.
Node addresses are discovered from /api/v1/nodes rather than hard-coded, so
this works on any fleet with no edits; --nodes overrides for a fleet the server
has not seen. The version defaults to the contents of version.txt so the check
cannot drift from what was built.
The PSK is taken from a FILE named by RUVIEW_OTA_PSK_FILE, never from argv or
an environment variable holding the key itself -- both are visible in a process
listing, and that key can replace firmware. Both scripts refuse to run without
it rather than falling back to something permissive.
Verified against a live nine-node fleet: discovery returns all nine from
/api/v1/nodeswith nothing hard-coded, both guards fire (missingRUVIEW_OTA_PSK_FILE, missing image), andbash -nis clean. The roll itselfwas used to move nine boards through two firmware versions today, one node at a
time, with the verify step catching each before the next was touched.
No hard-coded addresses, paths, or versions remain in either file.