chore(ci): indexnow doc updates#176
Open
coryrylan wants to merge 1 commit into
Open
Conversation
github-actions
Bot
added
scope(ci)
scope(internals)
github_actions
coryrylan
force-pushed
the
topic-index-now
branch
2 times, most recently
from
66d997a to
f0af6ec
Compare
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Enterprise Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughAdds an IndexNow CLI module for manifest generation, diffing, key verification, and submission. Integrates ChangesIndexNow manifest, diff, and submission
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant CIJob as ci job
participant DeployJob as deploy-pages job
participant NotifyJob as notify-indexnow job
participant IndexNowJS as indexnow.js
participant IndexNowAPI as IndexNow API
CIJob->>IndexNowJS: run prepare
CIJob->>DeployJob: pass indexnow-state artifact
DeployJob->>IndexNowJS: run diff
IndexNowJS->>IndexNowAPI: fetch previous manifest
IndexNowAPI-->>IndexNowJS: manifest or 404
DeployJob->>NotifyJob: pass indexnow-payload artifact
NotifyJob->>IndexNowJS: run submit
IndexNowJS->>IndexNowAPI: verify key and submit URL batches
IndexNowAPI-->>IndexNowJS: batch statuses
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
coryrylan
force-pushed
the
topic-index-now
branch
from
f0af6ec to
f402fdb
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 10
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/ci.yml:
- Around line 235-267: Add a descriptive job name to notify-indexnow for clearer
Actions UI output, pin the reusable actions in this job to immutable SHAs
instead of the v6/v8 tags, and set persist-credentials: false on the
actions/checkout step just like deploy-pages. Also add a brief comment
explaining why the permissions block only needs actions: read and contents: read
so the intent is obvious when reviewing the workflow.
- Around line 74-87: The workflow now uses unpinned third-party actions, which
violates the repository policy. Update the Upload pages artifact and Upload
IndexNow state steps in the CI workflow to reference immutable commit SHAs
instead of the current version tags, matching the pinning style already used by
jdx/mise-action. Ensure the action references for actions/upload-pages-artifact
and actions/upload-artifact are replaced without changing their inputs or
behavior.
- Around line 190-233: The reworked deploy-pages job needs hardening: replace
the tag-based references in the checkout, download-artifact, upload-artifact,
and deploy-pages steps with pinned commit SHAs, and update the checkout step to
set persist-credentials to false since the job is read-only. Also add a brief
explanatory comment for the permissions block in this job, following the
existing pattern elsewhere in the workflow, so the required actions: read,
pages: write, and id-token: write scopes are documented.
- Around line 199-215: The CI workflow sparse checkout is missing the local
package metadata needed by the IndexNow script. Update the checkout
configuration used before running `projects/internals/ci/indexnow.js` to include
`projects/internals/ci/package.json` alongside
`projects/internals/ci/indexnow.js` in both sparse-checkout blocks so Node
resolves the file as an ES module. Keep the change in the workflow job that uses
`actions/checkout` and the `indexnow` step so the script runs with the correct
module type.
- Around line 65-87: The IndexNow preparation step is currently part of the
required ci job flow, so failures in the IndexNow-specific logic can block the
Pages deployment. In the ci workflow, move the Prepare IndexNow artifacts logic
out of the critical path used by actions/upload-pages-artifact and deploy-pages,
or make the IndexNow step non-blocking so a missing or malformed INDEXNOW_KEY
does not fail ci. Use the existing Prepare IndexNow artifacts and Upload
IndexNow state steps as the place to isolate this behavior.
In `@projects/internals/BUILD.md`:
- Around line 202-205: The IndexNow entry in the Repo Configuration section
breaks the existing bullet-list format and misses the required project
reference. Update the documentation item to match sibling entries like GitHub
Pages and Scheduled Workflows by using a bullet with a linked title, and include
the IndexNow project link to https://www.indexnow.org/ in the description or
title. Keep the existing workflow/secret details, but make the entry consistent
with the surrounding list structure.
In `@projects/internals/ci/indexnow.js`:
- Around line 104-111: The baseline manifest fetch in createDiffPayload
currently makes a one-shot request, unlike the other network paths in this
module. Update createDiffPayload to reuse the existing retry/backoff helper (for
example the same wrapper used by fetchWithRetry or waitForKey) when calling
request for previousUrl, so transient 5xx or network errors are retried before
failing the diff step.
- Around line 176-178: Return early from the IndexNow submission flow when
`urls` is empty, before building `keyLocation` or calling `waitForKey`, so the
idle/baseline path skips remote key verification entirely. Update the logic in
the IndexNow handler around the `waitForKey` check to move the existing `{
submitted: 0, batches: 0, statuses: [] }` return ahead of any key validation,
keeping the submission path unchanged when URLs are present.
- Around line 166-175: Validate the IndexNow payload shape before reading
payload.urls in indexnow.js; the current logic in the payload handling block
builds urls with new Set(payload?.urls ?? []) before checking
Array.isArray(payload?.urls), which can throw a generic TypeError for
non-iterable values. Move the version/array validation ahead of the urls
construction in the same flow around validateKey, normalizeSiteUrl, and the URL
loop so invalid payloads consistently throw the intended Invalid IndexNow
payload error.
- Around line 82-91: The manifest hashing in generateManifest currently uses
Promise.all over every sitemap URL, which can exhaust file handles on large
sites. Update generateManifest to process URLs in bounded batches or with a
small concurrency limit, reusing the existing chunkUrls helper if available, so
readFile and createHash run without fan-out across all entries at once. Keep the
returned { version, urls } shape unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 90b2570d-797a-4a32-8132-28395c86d7aa
📒 Files selected for processing (3)
.github/workflows/ci.ymlprojects/internals/BUILD.mdprojects/internals/ci/indexnow.js
Comment on lines
+199
to
+215
| - uses: actions/checkout@v6 | ||
| with: | ||
| sparse-checkout: | | ||
| .nvmrc | ||
| mise.toml | ||
| projects/internals/ci/indexnow.js | ||
| sparse-checkout-cone-mode: false | ||
| - uses: jdx/mise-action@e6a8b3978addb5a52f2b4cd9d91eafa7f0ab959d | ||
| with: | ||
| version: 2026.6.14 | ||
| install: true | ||
| cache: true | ||
| - name: Download IndexNow state | ||
| uses: actions/download-artifact@v8 | ||
| with: | ||
| name: indexnow-state | ||
| path: indexnow/current |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Confirm module type declaration reachable from indexnow.js's directory tree
fd package.json --max-depth 6
for f in package.json projects/internals/package.json projects/internals/ci/package.json; do
if [ -f "$f" ]; then
echo "== $f =="
grep -n '"type"' "$f"
fi
done
rg -n '^export ' projects/internals/ci/indexnow.js | head -5Repository: NVIDIA/elements
Length of output: 1876
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the workflow sections that use the sparse checkout and indexnow.js
sed -n '190,220p' .github/workflows/ci.yml
printf '\n---\n'
sed -n '238,265p' .github/workflows/ci.yml
# Show the package.json that governs projects/internals/ci/indexnow.js
printf '\n---\n'
cat -n projects/internals/ci/package.jsonRepository: NVIDIA/elements
Length of output: 2261
Include projects/internals/ci/package.json in both sparse checkouts
node ./projects/internals/ci/indexnow.js relies on the local package.json with "type": "module"; without it, Node will treat indexnow.js as CommonJS and fail on the first export.
🧰 Tools
🪛 zizmor (1.26.1)
[warning] 199-205: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
[error] 199-199: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 212-212: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/ci.yml around lines 199 - 215, The CI workflow sparse
checkout is missing the local package metadata needed by the IndexNow script.
Update the checkout configuration used before running
`projects/internals/ci/indexnow.js` to include
`projects/internals/ci/package.json` alongside
`projects/internals/ci/indexnow.js` in both sparse-checkout blocks so Node
resolves the file as an ES module. Keep the change in the workflow job that uses
`actions/checkout` and the `indexnow` step so the script runs with the correct
module type.
Comment on lines
+202
to
+205
| ### IndexNow | ||
|
|
||
| The Pages deployment workflow notifies IndexNow after it publishes documentation updates. Before merging changes that enable this workflow, add an `INDEXNOW_KEY` repository Actions secret containing 8 to 128 letters, numbers, or dashes. The workflow publishes the verification file under the Pages project path, submits only changed sitemap URLs, and retains each notification payload for 30 days so maintainers can rerun a failed notification job. | ||
|
|
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
Subsection breaks the established bullet-list pattern and omits the IndexNow project link.
Every other item in "## Repo Configuration" is a - [Title](link) bullet with a short description (e.g. GitHub Pages, Scheduled Workflows), but this entry introduces a ### heading instead. The PR description also calls out referencing the IndexNow project at https://www.indexnow.org/, which isn't linked anywhere in this section.
Proposed fix — match sibling bullet format and add the project link
-### IndexNow
-
-The Pages deployment workflow notifies IndexNow after it publishes documentation updates. Before merging changes that enable this workflow, add an `INDEXNOW_KEY` repository Actions secret containing 8 to 128 letters, numbers, or dashes. The workflow publishes the verification file under the Pages project path, submits only changed sitemap URLs, and retains each notification payload for 30 days so maintainers can rerun a failed notification job.
+- [IndexNow](https://www.indexnow.org/)
+
+ The Pages deployment workflow notifies IndexNow after it publishes documentation updates. Before merging changes that enable this workflow, add an `INDEXNOW_KEY` repository Actions secret containing 8 to 128 letters, numbers, or dashes. The workflow publishes the verification file under the Pages project path, submits only changed sitemap URLs, and retains each notification payload for 30 days so maintainers can rerun a failed notification job.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| ### IndexNow | |
| The Pages deployment workflow notifies IndexNow after it publishes documentation updates. Before merging changes that enable this workflow, add an `INDEXNOW_KEY` repository Actions secret containing 8 to 128 letters, numbers, or dashes. The workflow publishes the verification file under the Pages project path, submits only changed sitemap URLs, and retains each notification payload for 30 days so maintainers can rerun a failed notification job. | |
| - [IndexNow](https://www.indexnow.org/) | |
| The Pages deployment workflow notifies IndexNow after it publishes documentation updates. Before merging changes that enable this workflow, add an `INDEXNOW_KEY` repository Actions secret containing 8 to 128 letters, numbers, or dashes. The workflow publishes the verification file under the Pages project path, submits only changed sitemap URLs, and retains each notification payload for 30 days so maintainers can rerun a failed notification job. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@projects/internals/BUILD.md` around lines 202 - 205, The IndexNow entry in
the Repo Configuration section breaks the existing bullet-list format and misses
the required project reference. Update the documentation item to match sibling
entries like GitHub Pages and Scheduled Workflows by using a bullet with a
linked title, and include the IndexNow project link to https://www.indexnow.org/
in the description or title. Keep the existing workflow/secret details, but make
the entry consistent with the surrounding list structure.
Comment on lines
+82
to
+91
| export async function generateManifest({ siteDir, siteUrl, sitemapXml }) { | ||
| const urls = extractSitemapUrls(sitemapXml); | ||
| const entries = await Promise.all( | ||
| urls.map(async url => { | ||
| const content = await readFile(urlToHtmlPath(siteDir, siteUrl, url)); | ||
| return [url, createHash('sha256').update(content).digest('hex')]; | ||
| }) | ||
| ); | ||
| return { version: INDEXNOW_MANIFEST_VERSION, urls: Object.fromEntries(entries) }; | ||
| } |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win
Bound the concurrency of manifest hashing.
Promise.all fans out one readFile per sitemap URL simultaneously. For a large docs site this can open thousands of file handles at once and hit EMFILE ("too many open files"), failing the prepare step. Consider processing in batches (you already have chunkUrls) or using a small concurrency limit.
♻️ Batched hashing
export async function generateManifest({ siteDir, siteUrl, sitemapXml }) {
const urls = extractSitemapUrls(sitemapXml);
- const entries = await Promise.all(
- urls.map(async url => {
- const content = await readFile(urlToHtmlPath(siteDir, siteUrl, url));
- return [url, createHash('sha256').update(content).digest('hex')];
- })
- );
+ const entries = [];
+ for (const batch of chunkUrls(urls, 64)) {
+ entries.push(
+ ...(await Promise.all(
+ batch.map(async url => {
+ const content = await readFile(urlToHtmlPath(siteDir, siteUrl, url));
+ return [url, createHash('sha256').update(content).digest('hex')];
+ })
+ ))
+ );
+ }
return { version: INDEXNOW_MANIFEST_VERSION, urls: Object.fromEntries(entries) };
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| export async function generateManifest({ siteDir, siteUrl, sitemapXml }) { | |
| const urls = extractSitemapUrls(sitemapXml); | |
| const entries = await Promise.all( | |
| urls.map(async url => { | |
| const content = await readFile(urlToHtmlPath(siteDir, siteUrl, url)); | |
| return [url, createHash('sha256').update(content).digest('hex')]; | |
| }) | |
| ); | |
| return { version: INDEXNOW_MANIFEST_VERSION, urls: Object.fromEntries(entries) }; | |
| } | |
| export async function generateManifest({ siteDir, siteUrl, sitemapXml }) { | |
| const urls = extractSitemapUrls(sitemapXml); | |
| const entries = []; | |
| for (const batch of chunkUrls(urls, 64)) { | |
| entries.push( | |
| ...(await Promise.all( | |
| batch.map(async url => { | |
| const content = await readFile(urlToHtmlPath(siteDir, siteUrl, url)); | |
| return [url, createHash('sha256').update(content).digest('hex')]; | |
| }) | |
| )) | |
| ); | |
| } | |
| return { version: INDEXNOW_MANIFEST_VERSION, urls: Object.fromEntries(entries) }; | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@projects/internals/ci/indexnow.js` around lines 82 - 91, The manifest hashing
in generateManifest currently uses Promise.all over every sitemap URL, which can
exhaust file handles on large sites. Update generateManifest to process URLs in
bounded batches or with a small concurrency limit, reusing the existing
chunkUrls helper if available, so readFile and createHash run without fan-out
across all entries at once. Keep the returned { version, urls } shape unchanged.
- Added steps to prepare and upload IndexNow artifacts during the CI process. - Implemented a new script for managing IndexNow manifests and payloads. - Updated documentation to include instructions for setting up IndexNow integration. Signed-off-by: Cory Rylan <crylan@nvidia.com>
coryrylan
force-pushed
the
topic-index-now
branch
from
f402fdb to
7e74caf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://www.indexnow.org/
Summary by CodeRabbit
New Features
Documentation