Repository navigation
fix: name the status of a refused download, hide URL credentials in messages, update via manifestUrl - #747
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #747 +/- ##
==========================================
+ Coverage 51.87% 54.83% +2.96%
==========================================
Files 111 112 +1
Lines 8276 8368 +92
==========================================
+ Hits 4293 4589 +296
+ Misses 3561 3326 -235
- Partials 422 453 +31 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
linghengqian
left a comment
There was a problem hiding this comment.
Thanks for the PR. Rejecting anything but 200 before the file is opened is the right fix, and the new tests do what the description says: on main the 403, 429 and 500 cases fail with error = nil, on this branch they pass. With the repro from #746, those three now stop at the download and name the status and the URL (403 also without a checksum), and 404 is unchanged.
I'm requesting changes. Everything below should go into this PR instead of a follow-up.
-
The new error prints the URL with its password.
sdk.go:1087formatsuwith%s. IfPreInstallreturnshttp://user:s3cret@127.0.0.1:18080/403/repro.zip?token=abc123, the message ends with:server returned 403 Forbidden for http://user:s3cret@127.0.0.1:18080/403/repro.zip?token=abc123When the connection fails instead, the
net/httperror for the same URL masks the password (http://user:***@127.0.0.1:18080/...). Useu.Redacted()here, and add a test with userinfo in the URL that checks the password is not in the message. If you want the query gone too,sanitizeSourceURLinreceipt.godoes that. -
The status assertion in
download_test.go:50is too loose:if !strings.Contains(err.Error(), http.StatusText(status)) && !strings.Contains(err.Error(), string(rune('0'+status/100))) {
The second operand is the character
4or5, and the random port in the test server's URL often contains one. With the message changed tofmt.Errorf("download failed for %s", u), which has no status in it, the test still passed in 6 of 30 runs. Compare against the whole status line,fmt.Sprintf("%d %s", status, http.StatusText(status)), and drop the fallback. -
The same bug is in four more places. With a 403 from an
httptestserver, each of these returns no error and leaves the page of HTML in the file:Manager.downloadPlugin,internal/manager.go:455(plugins added or updated from a URL or the registry)http.download_file,internal/plugin/luai/module/http/http.go:228(returnsnilto the plugin)downloadFileincmd/commands/upgrade.go:98(vfox upgrade; it checks no status at all)Downloader.Download,internal/shared/util/downloader.go:47(nothing calls it yet, same hole)
Add the same check to each (the existing 404 messages can stay) and a test for each.
TestDownloadFileinhttp_test.gogoes to the live plugin site, so use anhttptestserver there, asTestUserAgentDefaultdoes. A small helper ininternal/shared/utilwould keep the message and the redaction identical in all five places. -
Fixes #746closes the issue on merge, but #746 also reports that the download goes out asGo-http-client/1.1while plugin requests sendvfox/<version> vfox-<plugin>/<version>. This PR doesn't change that: with the repro, the server log still showsGo-http-client/1.1forrepro.zip. Either fix it here (ComputeUserAgentininternal/plugin/luai/context.gobuilds the string, and aUser-Agentreturned fromPreInstallheaders should still win) or changeFixestoRefsso the issue stays open. -
Smaller things:
download_test.gohas no Apache 2.0 header. The other test files ininternal/sdkhave one, exceptsdk_test.go, andAGENTS.mdasks for it in new Go files.failed to download ... file, err:download failed: server returned ...says "download failed" twice. #746 asked forunexpected HTTP status 403 Forbidden;unexpected HTTP status 403 Forbidden for <url>reads better aftermoveRemoteFilehas wrapped it.
I'll take another look once these are in.
|
All points are addressed in
New tests are |
|
The macOS leg of the CI run failed with a Go module A re-run of that job should clear it. |
linghengqian
left a comment
There was a problem hiding this comment.
Thanks, the points from the first review are all in and check out on c040d87.
One thing is left, in two parts. Both were already like this before the PR, but the PR now redacts the status error in downloadPlugin, so a plugin URL with credentials is shown in clear text in some places and redacted in others.
First, the 404 message prints the URL as given (internal/manager.go:457):
return "", fmt.Errorf("plugin not found at %s", downloadUrl)$ vfox add --source http://user:s3cret@127.0.0.1:18080/404/plugin.zip
download wrapper error: plugin not found at http://user:s3cret@127.0.0.1:18080/404/plugin.zip
A 403 for the same kind of URL prints user:xxxxx@, so whether the password shows up depends on the status code. Please redact the URL in the 404 message with the same code the status error uses (redactSourceURL is unexported, so it needs exporting), and add a test next to TestDownloadPluginKeepsNotFoundMessage with userinfo in the URL that checks the password is not in the error.
Second, the --debug output. vfox --debug add --source http://user:s3cret@127.0.0.1:18080/403/plugin.zip prints the password in five debug lines before the request is made: Adding plugin: name=..., url=... (manager.go:494), Installing plugin to temp from (527 and 584), Plugin from (594) and Downloading plugin from (432). On a 404 there is also Plugin not found (404) (456). The lines that print a manifest or update URL (167, 330, 394 and 420) print it as given too. Debug output tends to end up in issues and CI logs, so please run the URL through the same function in all of these. The SDK install path doesn't need anything: vfox --debug install with credentials in the PreInstall URL doesn't print the password.
|
Thanks for the careful re-review. Both points are addressed in 0d9e155:
Verification:
|
linghengqian
left a comment
There was a problem hiding this comment.
Thanks, the 404 message and the debug lines listed in the last review check out on 0d9e155.
Two debug lines in Manager.Update still print their URL as given. They were not on that list. With registry.address: http://user:s3cret@127.0.0.1:18747/registry in config.yaml, a plugin with manifestUrl = "http://user:s3cret@127.0.0.1:18747/manifest/demo.json", and the registry answering 404 for the plugin (so Update falls back to manifestUrl), vfox --debug update demo prints:
Fetching plugin demo from http://user:s3cret@127.0.0.1:18747/registry/demo.json...
Fetching plugin manifest from: http://user:xxxxx@127.0.0.1:18747/registry/demo.json
Manifest not found (404): http://user:xxxxx@127.0.0.1:18747/registry/demo.json
Fetching plugin demo from http://user:s3cret@127.0.0.1:18747/manifest/demo.json...
Fetching plugin manifest from: http://user:xxxxx@127.0.0.1:18747/manifest/demo.json
The first and the fourth line come from manager.go:302 (address) and manager.go:307 (pluginMetadata.ManifestUrl). Please pass both through util.RedactSourceURL, like the line at 330 in the same function. With 403 and 404 responses, the --debug output of add --source, add <name> from the registry, the automatic plugin install in vfox install, update through updateUrl, and install of an SDK with credentials in the PreInstall URL contains no password.
The description still says "Existing 404 messages are unchanged" and mentions the redaction only for the status error. The plugin 404 message and the debug lines are redacted now too, so please update the description to say so.
- Manager.Update printed the registry address and the fallback manifestUrl as given; a registry/manifest URL with credentials therefore leaked the password into vfox --debug update output. Pass both through util.RedactSourceURL, like the download line in the same function. - go build ./... and go test ./internal ./internal/shared/util ./internal/sdk ./cmd/commands pass.
|
Both are fixed in
Thanks for the careful reproduction; the description no longer overstates the untouched messages. |
linghengqian
left a comment
There was a problem hiding this comment.
Thanks, the two Manager.Update debug lines and the description check out on 2601d98.
Eight things are left. They go into this PR too.
-
Manager.Updatenever installs an update that it finds throughmanifestUrl. When the registry answers 404 and the manifest atmanifestUrlis newer,downloadUrlis set atinternal/manager.go:316, and thereturn errat line 321 then returns the registry'sManifestNotFound(theerrofdu, err :=at line 308 lives only in the inner block):Manifest found, name: demo, version: 9.9.9, downloadUrl: http://user:xxxxx@127.0.0.1:18747/403/demo.zip manifest not foundThe block has had this shape since 654aa18 (#151). It is the block where this PR edits the debug lines, and while it stays like this the
manifestUrlflow never reaches the download that the rest of the PR is about. Fetching the fallback manifest into the variable that holds the registry result, and checking the version once for both, removes it. Please add a test: anhttptestserver whose registry path answers 404 and whose manifest offers a newer version with adownloadUrlto a plugin zip, an installed plugin whosemanifestUrlpoints to that manifest, andUpdatereturning nil with the new version in the plugin directory. On this branch it fails withUpdate() error = manifest not found. -
RedactSourceURLhides the password ofuser:password@and nothing else. A 403 is the answer to a rejected credential, so the new message is where one gets echoed. With aPreInstallURL ofhttp://tok3n@127.0.0.1:18747/403/demo-1.0.0.zipand ofhttp://127.0.0.1:18747/403/demo-1.0.0.zip?token=abc123:failed to download demo@1.0.0 file, err:unexpected HTTP status 403 Forbidden for http://tok3n@127.0.0.1:18747/403/demo-1.0.0.zip failed to download demo@1.0.0 file, err:unexpected HTTP status 403 Forbidden for http://127.0.0.1:18747/403/demo-1.0.0.zip?token=abc123sanitizeSourceURLininternal/sdk/receipt.goalready names "credentials, tokens in the query" as what must not be stored, pip printsaccesstoken@example.comas****@example.com, and git strips the user name and password from the URLs it shows (transport_anonymize_url). Please mask inRedactSourceURLa user name that has no password (a user name with a password stays visible, asTestSdk_Download_ErrorStatusRedactsPasswordexpects) and the values of the query (the keys stay). A string that starts withscheme://but thaturl.Parserejects, such as a password with a bare%or/, is returned as given today, password included. Hide everything between://and its last@and drop its query instead. -
The errors that
net/httpandnet/urlbuild print the URL themselves.net/httpmasks only the password, andnet/urlquotes the input that it rejects. With the same URLs and nothing listening on the port:failed to download demo@1.0.0 file, err:Get "http://user:***@127.0.0.1:1/demo-1.0.0.zip?token=abc123": dial tcp 127.0.0.1:1: connect: connection refused failed to download demo@1.0.0 file, err:Get "http://tok3n@127.0.0.1:1/demo-1.0.0.zip": dial tcp 127.0.0.1:1: connect: connection refused parse "http://user:Zq9%ss@127.0.0.1:1/demo-1.0.0.zip": invalid URL escape "%ss" parse "http://user:A/b+c==@127.0.0.1:1/demo-1.0.0.zip": invalid port ":A" after hostThe same lines show up for
add --source, for a registry address with credentials inconfig.yaml(add,update,available) and in the text that a plugin gets fromhttp.get,http.headandhttp.download_file. Please add a helper next toRedactSourceURLthat returns a copy of a*url.ErrorwithURLpassed throughRedactSourceURLand, for aparseerror whose URL had something to hide, a reason without the quoted input. Use it where these errors are returned or logged:internal/sdk/sdk.go:368and:1077,internal/manager.go:398-399,:436-437,:449-450and:656,internal/plugin/luai/module/http/http.go:97,:114,:151,:167,:211and:225, andinternal/shared/util/downloader.go:38and:42. -
Five places format a URL, or a name taken from it, themselves:
installPluginToTemptakes the extension of the whole URL (internal/manager.go:586), sovfox add --source 'http://host/demo.zip?token=abc123'fails withunsupported .zip?token=abc123 type wrapper to install, only support .lua or .zip(lines 588 and 589). Print the extension ofutil.RedactSourceURL(path).http.download_fileusesfilepath.Base(urlStr.String())as the label of the progress bar (internal/plugin/luai/module/http/http.go:245-246), so a download of.../file.zip?token=abc123that succeeds printsfile.zip?token=abc123 100% [...]. Take it fromreq.URL.Path.- A
PreInstallURL with a scheme other than http or https is handled as a local path, andmoveLocalFile(internal/sdk/sdk.go:357-359) prints it, in the progress line and in the error:Moving ftp://user:s3cret@127.0.0.1:1/demo-1.0.0.zip?token=abc123 to ...andfailed to move file, err:stat ftp://user:s3cret@127.0.0.1:1/demo-1.0.0.zip?token=abc123: no such file or directory. Pass the path throughRedactSourceURLin both; the error is an*fs.PathError, so itsPathfield is the string. - The offline transport of
vfox plugin run --offlineformatsreq.URLitself (internal/plugin/development_http.go:45), so the password is not masked:Get "http://user:***@127.0.0.1:1/releases?token=abc123": HTTP is disabled offline: GET http://user:s3cret@127.0.0.1:1/releases?token=abc123. Useutil.RedactSourceURL(req.URL.String()). downloadPluginsaves the plugin underfilepath.Base(downloadUrl)(internal/manager.go:465), so a URL that passes the extension check, such ashttp://host/dl?token=abc123&file=plugin.zip, prints the query inSaving plugin to(466),Downloading dl?token=abc123&file=plugin.zip...(473),Plugin downloaded successfully(479),Unpacking wrapper(624) andValidating(632). Take the file name from the path of the URL, and usepluginplus the extension when the path has none.
-
The branch of
RedactSourceURLthat returns its input (internal/shared/util/status.go:43) is not run by any test. It is the only block of that file thatgo test ./... -coverprofilereports with a count of 0, and the Codecov comment lists the same file. Droppingu.Scheme == "" || u.Host == ""from the condition leavesgo test ./...green, although that check keeps a Windows path intact (url.Parse("C:\\plugins\\plugin.zip")returns schemec, andRedacted()would printc:\plugins\plugin.zip), andAddandinstallPluginToTempcall the function for every local--source. Please add a table test next toTestUnexpectedStatusError: a password, a user name without a password, a query, a password with a query, a URL with nothing to hide, a URL thaturl.Parserejects, and/tmp/plugin.zip,./plugin.zip,/tmp/100%/plugin.zip,C:\plugins\plugin.zip,C:/plugins/plugin.zipand the empty string. The helper from 3 needs a test with a connection error and one with aparseerror. Each place in 3 and 4 needs a test that puts a user name token, a password and a query token in the URL and checks that none of them is in the error, the log line or the label. -
fetchLatestVersion(cmd/commands/upgrade.go:43) is the only request in this file that still accepts any status. With a transport that answers 403, 429 or 500 with an HTML page, it returnsFailed to fetch the version.because the tag regexp finds nothing in the error page.
vfox upgradethen fails without naming the status, which is what #746 reports for the download. Please add the sameutil.UnexpectedStatusErrorcheck after thedefer resp.Body.Close()at line 48, with a test (aRoundTripperthat answers 429 is enough, the URL is fixed), and mention it in the description. -
AGENTS.mdcalls itself the single project-wide guide, and #744 updated it together with its change. This PR adds rules for later changes, and the URL lines were found one round at a time, so the next change can easily print a URL as given. Please add them under "Shared utilities": a download checks its status withutil.UnexpectedStatusErrorbefore the file is created, a URL that goes into an error, a message or a--debugline goes throughutil.RedactSourceURL, and the error of a request or ofurl.Parsegoes through the helper from 3. -
The release notes are built from commit titles (
.goreleaser.yaml), and the squash commit of a PR with several commits takes the PR title. With the points above in, the title should say what users will notice, for examplefix: name the status of a refused download, hide URL credentials in messages, update via manifestUrl.
|
All eight points are in.
|
linghengqian
left a comment
There was a problem hiding this comment.
Thanks, the eight points of the last review are in and check out on bf59e05, apart from what is listed below.
Seven things are left. They go into this PR too.
-
vfox add --sourcepanics for a URL whose.zipis only in the query.installPluginToTempaccepts a URL by the extension of the whole URL (internal/manager.go:606), sohttp://host/dl?token=abc123&file=plugin.zippasses.pluginFileName(internal/manager.go:428) then saves the file under the last element of the URL path, which isdl.util.NewDecompressorpicks the unpacker from the file name and returns nil for a name it does not know (internal/shared/util/decompressor.go:729), andinternal/manager.go:648callsDecompresson that nil:$ vfox add --source 'http://127.0.0.1:18748/dl?token=abc123&file=plugin.zip' Downloading dl... panic: runtime error: invalid memory address or nil pointer dereference ... github.com/version-fox/vfox/internal.(*Manager).installPluginToTemp(...) internal/manager.go:648On main the same command installs the plugin, because the file is saved as
dl?token=abc123&file=plugin.zip, which ends in.zip. With a server that answers every path with a plugin zip,/dl/?file=plugin.zip,/dl.php?file=plugin.zipand/download.php?id=7&name=plugin.zipinstall on main and panic on this branch, and/plugin.zipinstalls on both. No test runsinstallPluginToTempon such a URL:TestDownloadPluginNamesFileFromURLPathcalls onlydownloadPluginand asserts the namedl(internal/manager_test.go:464).The saved name has to end with the extension that
installPluginToTempaccepted. Please use the last element of the URL path only when it does, andpluginplus that extension otherwise:ext := filepath.Ext(downloadUrl) if u, err := url.Parse(downloadUrl); err == nil { if name := path.Base(u.Path); name != "." && name != "/" && strings.HasSuffix(name, ext) { return name } } return "plugin" + ext
With that, all five paths above install, and
TestDownloadPluginNamesFileFromURLPathhas to expectplugin.zipfor its first URL. Please add a test that goes throughinstallPluginToTempwith anhttptestserver that answers with a plugin zip (updateTestPlugininmanager_update_test.gobuilds one) for the five paths. On this branch it fails withinvalid memory address or nil pointer dereferencefor four of them. -
The rewrite of the fallback in
Manager.Updatechanged two things that the fix does not need. When the registry and themanifestUrlboth answer 404, the command now ends withdemo plugin not support update(internal/manager.go:312). On main it ends withmanifest not found, andnot support updatewas only for a plugin without amanifestUrl:registry 404, manifestUrl 404 main: manifest not found branch: demo plugin not support updatePlease keep the check for the missing
manifestUrlinside the first 404 branch:registryManifest, err := m.fetchPluginManifest(address) if errors.Is(err, ManifestNotFound) { if pluginMetadata.ManifestUrl == "" { return fmt.Errorf("%s plugin not support update", pluginName) } logger.Debugf("Fetching plugin %s from %s...\n", pluginName, util.RedactSourceURL(pluginMetadata.ManifestUrl)) registryManifest, err = m.fetchPluginManifest(pluginMetadata.ManifestUrl) } if err != nil { return err }
The second change is the color of
is already the latest versionon the registry path:pterm.LightBlueon main,pterm.Blueatinternal/manager.go:318. Please keepLightBluethere.Neither the
not support updatereturn (line 312) nor thealready the latestreturn (lines 318-319) is run by any test;go test ./... -coverprofilereports both with a count of 0. Please add a test for each outcome of the block: registry 404 without amanifestUrl, registry 404 with amanifestUrlthat answers 404, and a manifest that offers the installed version. -
Sdk.Downloadreturns the error ofhttp.NewRequestas it is (internal/sdk/sdk.go:1080-1082). The same call inDownloader.Download(internal/shared/util/downloader.go:46-48) and indownloadPlugin(internal/manager.go:451-455) goes through the redaction. With the URL built the wayTestDownloaderRedactsRequestErrorsbuilds it (Host: "in valid", a user name and password, a query token):parse "http://user:s3cret@in%20valid/repro.zip?token=abc123": invalid URL escape "%20"moveRemoteFileis the only caller and has parsed the string by then, so the command line does not reach this.Downloader.Download, which nothing calls, is handled the same way in this PR. Please redact it here too and add a case next toTestSdk_Download_RedactsConnectionError. -
RedactURLErrortakes a*url.Error, so every caller needs anerrors.Asaround it, and the same wrapper was written four times:redactURLErrorininternal/manager.go:439andinternal/sdk/sdk.go:375,redactErrorininternal/shared/util/downloader.go:37and ininternal/plugin/luai/module/http/http.go:58. The fallback for other errors in each copy is run by no test (internal/manager.go:444,internal/sdk/sdk.go:380,internal/shared/util/downloader.go:42,internal/plugin/luai/module/http/http.go:63), and the rule added toAGENTS.mdsays the error of a request goes throughutil.RedactURLError, which no call site does literally. Please add one function next to it, delete the four copies and call it from the sites:// RedactError passes a *url.Error found in err through RedactURLError and // returns any other error unchanged. func RedactError(err error) error { var urlErr *url.Error if errors.As(err, &urlErr) { return RedactURLError(urlErr) } return err }
The
AGENTS.mdrule then namesutil.RedactError, and the function needs a test with a*url.Errorand one with another error, next toTestRedactURLError. -
A query part without
=is printed as it is, becauseredactQueryValuesmasks only what follows an=(internal/shared/util/status.go:81). A token passed that way stays visible:RedactSourceURL("http://example.com/file.zip?abc123") = http://example.com/file.zip?abc123 RedactSourceURL("http://example.com/file.zip?abc123&b=2") = http://example.com/file.zip?abc123&b=xxxxxsanitizeSourceURLininternal/sdk/receipt.godrops the whole query. Please mask a part without=as a whole (?xxxxx&b=xxxxx) and add the two inputs toTestRedactSourceURL. -
go test ./... -coverprofilereports three blocks ofinternal/shared/util/status.gowith a count of 0: the case ofisURLSchemefor a digit,+,-or.(lines 117-119), itsdefaultreturn (line 121) andreturn sinRedactSourceURL(line 75).isURLSchemeis what keeps a local path that contains://from being rewritten, as its comment says, and no row ofTestRedactSourceURLreaches those branches. Please add/tmp/100%/dir://x@yand1http://user:Zq9%ss@host/file.zipto the table; both come back unchanged. Thereturn scannot be reached: a URL with a scheme and a host prints asscheme://host, andParselowercases the scheme, so://starts atlen(u.Scheme). Please drop the search and the fallback:s := u.String() n := len(u.Scheme) + len("://") return s[:n] + userinfo + s[n:]
-
archiver.decompresshas the same nil unpacker (internal/plugin/luai/module/archiver/archiver.go:33). Of the callers ofNewDecompressor,internal/sdk/sdk.go:405checks for nil andcmd/commands/upgrade.go:185always passesvfox_latest.ziporvfox_latest.tar.gz, but this one passes a name that the plugin chooses. For a name that no unpacker is registered for, the call fails with a nil pointer dereference instead of returning an error string like its other failures:local archiver = require("vfox.archiver") archiver.decompress("/tmp/not-an-archive.dat", "/tmp/out") -- runtime error: invalid memory address or nil pointer dereference
Please check for nil there, push an error such as
unsupported archive type: .dat, and add a test.
The description needs to follow points 1, 4 and 5: the rule for the saved file name (it says the name falls back to plugin plus the extension when the path has none), the helper name, and the query rule.
- pluginFileName falls back to "plugin" plus the accepted extension unless the URL path base already ends with it, so installPluginToTemp never panics on a nil decompressor for URLs whose extension lives in the query. - Manager.Update keeps "not support update" for a missing manifestUrl and returns "manifest not found" when the fallback 404s as well, and keeps the LightBlue "already the latest" line. - New util.RedactError replaces the four private redact wrappers and also covers the request build error of Sdk.Download. - redactQueryValues masks query parts without "=" as a whole. - RedactSourceURL drops the unreachable scheme search fallback. - archiver.decompress reports an unsupported archive type instead of panicking on a nil decompressor.
|
Thanks — all seven points are in
The description follows points 1, 4 and 5: the saved-name rule, |
linghengqian
left a comment
There was a problem hiding this comment.
Thanks, the seven points of the last review are in and check out on 7882fa4.
One thing is left. It goes into this PR too.
-
pluginFileNametakes the last element of the decoded URL path withpath.Base(internal/manager.go:433), which does not split at a backslash. On Windowsfilepath.Joindoes (internal/manager.go:478), so a backslash in the URL path, literal or as%5C, moves the download out of the plugin directory.installPluginToTempremoves the downloaded file when it returns (internal/manager.go:616-618), so a file that already sits at that place is gone afterwards. With a Windows build of the tests run under Wine, the plugin directory at<root>\a\b\pluginsand a server that answers every path,downloadPluginsaves the file here:URL path main this branch /..\..\evil.zip <root>\a\b\plugins\evil.zip <root>\a\evil.zip /..%5C..%5Cevil.zip <root>\a\b\plugins\..%5C..%5Cevil.zip <root>\a\evil.zip /sub/..%5Cevil.zip <root>\a\b\plugins\..%5Cevil.zip <root>\a\b\evil.zipWith a
victim.zipplaced next to the plugin directory and<server>/..\victim.zipas the URL,installPluginToTempreturns nil on both,victim.zipis untouched on main and gone on this branch.installPluginToTempalso receives thedownloadUrlof a registry manifest and theupdateUrlof a plugin, so a registry can supply such a URL as well asvfox add --sourcecan.Main used
filepath.Baseon the raw URL string, and the SDK path uses it on the decoded path (internal/sdk/sdk.go:1104,internal/shared/util/downloader.go:55). Please do the same here and drop thepathimport:if name := filepath.Base(u.Path); name != "." && name != "/" && strings.HasSuffix(name, ext) {
With that, the three URLs above are saved in the plugin directory as
evil.zip, andTestDownloadPluginNamesFileFromURLPathandTestInstallPluginToTempNamesFileWithAcceptedExtensionpass on the same Windows build.Linux and macOS do not show this, because a backslash is an ordinary file name character there, and no test uses one, so CI is green. Please add a test next to
TestDownloadPluginNamesFileFromURLPaththat downloads from anhttptestserver for/..\evil.zip,/..%5Cevil.zipand/sub/..%5C..%5Cevil.zipand checks that the saved file is directly in the plugin directory:if rel, err := filepath.Rel(plugins, saved); err != nil || rel != filepath.Base(saved) { t.Fatalf("downloadPlugin() saved %q, want a file directly in %q", saved, plugins) }
It passes on Linux either way. On the Windows build it fails for all three paths on this branch and passes with
filepath.Base.
…escape the plugin dir on Windows
|
Done in
Added Local verification: |
|
CI note on |
Refs #746.
Downloadininternal/sdk/sdk.goonly treated 404 as an error and saved the response body as the archive for every other status, so a mirror answering 403/429/500 ended the install later withChecksum errororzip: not a valid zip filewhile never saying the server refused the request.The status check lives in one helper,
util.UnexpectedStatusError(internal/shared/util/status.go), used by every path that downloads a file:internal/sdk/sdk.go—Download(SDK archives);internal/manager.go—downloadPlugin(plugins from a URL or the registry);internal/plugin/luai/module/http/http.go—http.download_file(plugins);cmd/commands/upgrade.go—downloadFile, and nowfetchLatestVersionas well, so a 403/429/500 from the tags page names the status instead of failing withFailed to fetch the version.;internal/shared/util/downloader.go—Downloader.Download.The error reads
unexpected HTTP status 403 Forbidden for <url>.URL credentials in messages and logs.
util.RedactSourceURLnow masks a user name without a password (****@host, the way pip prints it), keepsuser:xxxxx@for a user name with a password, masks query values while keeping the keys (a query part without=, which carries no key, is masked as a whole), and for ascheme://string thaturl.Parserejects (e.g. a password with a bare%) hides everything between://and the last@and drops the query. Local paths and Windows paths pass through unchanged.The errors that
net/httpandnet/urlbuild print the URL themselves —net/httpmasks only the password andnet/urlquotes the input it rejects — sointernal/shared/util/status.gogets two helpers:util.RedactURLErrorreturns a copy of a*url.Errorwith the URL redacted and, for aparseerror whose URL had something to hide, a reason that does not quote the rejected input, andutil.RedactErrorapplies it to any error and passes other errors through. Every call site usesutil.RedactError:Manager.fetchPluginManifest,Manager.downloadPlugin,Manager.Available,Sdk.Download(including its request build error),moveRemoteFile, the Luahttp.get/http.head/http.download_file, andDownloader.Download.The places that format a URL, or a name taken from it, no longer print secrets either:
installPluginToTempprints the extension of the redacted path,http.download_filelabels the progress bar fromreq.URL.Path,moveLocalFileredacts the progress line and the*fs.PathErrorpath, the offline development transport redacts the request URL, anddownloadPluginnames the saved file from the URL path — the last path element only when it already ends with the accepted extension,pluginplus that extension otherwise. That rule also fixes a panic:installPluginToTempaccepts a URL by the extension at the end of the whole URL, so a URL whose.ziplives only in the query (/dl?token=...&file=plugin.zip) used to save the file asdland crash on a nil decompressor inutil.NewDecompressor(...).Decompress(...).archiver.decompressnow returnsunsupported archive type: <ext>for a name no unpacker is registered for instead of panicking the same way.Update via manifestUrl.
Manager.Updatefell back to the manifestUrl manifest when the registry answered 404 but then returned the registry'sManifestNotFoundanyway (thedu, err :=shadowederr), so a plugin installed from outside the registry could never update. The fallback manifest now goes into the variable that holds the registry result and the version check runs once for both; the error wording stays as on main —not support updateonly for a plugin without amanifestUrl,manifest not foundwhen the fallback 404s too. Covered byTestUpdateInstallsManifestUrlUpdate(registry 404, manifest offers 9.9.9, installed demo@0.0.1 ends with 9.9.9) plus tests for each other outcome of the block: nomanifestUrl, fallback 404, already the latest.AGENTS.md "Shared utilities" records the rules for later changes: a download checks the status with
util.UnexpectedStatusErrorbefore the file is created, a URL that goes into an error, a message or a--debugline goes throughutil.RedactSourceURL, and the error of a request or ofurl.Parsegoes throughutil.RedactError.Tests, all local (
httptest, an in-memoryRoundTripper, or a refused local port):internal/shared/util/status_test.go:TestRedactSourceURLtable (password, user name without password, query, query parts without=, password with query, nothing to hide, URLsurl.Parserejects including a leading-digit scheme,/tmp/plugin.zip,./plugin.zip,/tmp/100%/plugin.zip, a local path containing://,C:\plugins\plugin.zip,C:/plugins/plugin.zip, empty),TestRedactURLError(connection error, parse error, parse error without secrets keeps the reason) andTestRedactError(plain, wrapped and non-URL errors);downloader_test.goadds redacted request/parse errors;internal/manager_test.go:fetchPluginManifest,downloadPlugin(request and connection errors),AvailableandinstallPluginToTempeach with a user name, password and query token in the URL;downloadPluginfile naming from the URL path and thepluginfallback;manager_update_test.goadds the Update-via-manifestUrl test above plusTestInstallPluginToTempNamesFileWithAcceptedExtension, which installs a plugin zip throughinstallPluginToTempfor/plugin.zip,/dl/?file=plugin.zip,/dl.php?file=plugin.zip,/download.php?id=7&name=plugin.zipand/dl?token=abc123&file=plugin.zip(four of them panic without the fix);internal/plugin/luai/module/archiver/archiver_test.go:decompressof an unknown extension returnsunsupported archive type;internal/sdk/download_test.go:Downloadrequest build error and connection error,moveRemoteFileparse error,moveLocalFileprogress line and*fs.PathErrorredaction (captured viapterm.SetDefaultOutput);internal/plugin/luai/module/http/http_test.go:get/head/download_filerequest and connection errors redacted, and the progress-bar label taken from the URL path;internal/plugin/development_test.go: the offline transport redacts the URL;cmd/commands/upgrade_download_test.go:fetchLatestVersionnames a 429.Verification on this machine:
Full
go test ./...still hits two pre-existing, environment-dependent groups:internal/plugin/luai/module/httpTestGetRequest/TestHeadRequestfetch the live plugin site and time out from this network, andinternal/shared/printerwaits on a keyboard listener in a non-interactive shell.New Go files carry the Apache 2.0 header.
Not addressed here: #746 also reports that SDK downloads use
Go-http-client/1.1while plugin requests sendvfox/<version> <plugin>/<version>. That is a separate behavior change, so this PR usesRefs #746and leaves the issue open for it.