nginx intermittently 404s wp-json/* routes (tries to serve as static files)

[JohnRDOrazio]

Summary

In production, nginx intermittently fails to route /wp-json/*
requests to PHP-FPM and instead tries to serve them as static files
from disk, returning 404. Same endpoint, same client, same minute —
some requests reach PHP and succeed, others 404 at the filesystem
layer.

Evidence

From logs/cms.catholicdigitalcommons.org/proxy_error_log on 2026-04-29
during a workflow_dispatch deploy:

Successful PHP-routed request (16:25:23):

[error] FastCGI sent in stderr: "PHP message: cdcf_process_translation:
  Translation complete for post 858 (es)" while reading response header
  from upstream, ... request: "POST /wp-json/cdcf/v1/process-queue
  HTTP/2.0", upstream:
  "fastcgi://unix:/var/www/vhosts/system/cms.catholicdigitalcommons.org/php-fpm.sock"

404 from same client minutes later (16:28:32):

[error] openat() "/var/www/vhosts/catholicdigitalcommons.org/cms.catholicdigitalcommons.org/wp-json/cdcf/v1/process-queue"
  failed (2: No such file or directory),
  client: 2001:41d0:52:bff::14c, server: cms.catholicdigitalcommons.org,
  request: "POST /wp-json/cdcf/v1/process-queue HTTP/2.0"

The 404 case shows nginx never reached the FastCGI upstream — it tried
openat() on the URL path as a literal file. That only happens when
the WordPress URL-rewrite rules don't match the request, which on a
correctly-configured Plesk + nginx + WP setup should never happen for
/wp-json/*.

Affected endpoints I've seen 404 in the logs include:

• POST /wp-json/cdcf/v1/process-queue (the redis-queue cron-like
  worker trigger)
• GET /wp-json/wp/v2/pages/<id> (verify step's polling reads)

PHP-routed (successful) and filesystem-404 responses are interleaved
seconds apart. The 404 case appears to be ~30-50% of requests during
heavy traffic windows.

Hypotheses

1. Plesk's nginx config has a try_files order that races. A
   common Plesk pattern is something like
   try_files $uri $uri/ /index.php?$args, which checks for a real
   file first. If the request is somehow routed to a stale or
   uninitialized worker, try_files may fail without falling through
   to PHP.
2. Caching or worker-lifecycle issue. During the burst of cron
   POSTs from the GitHub Actions runner, nginx may be hitting a
   per-worker code path that doesn't have the WP rewrite rules loaded.
3. Plesk's "Apache + nginx" hybrid mode interaction. The site uses
   Plesk's nginx-in-front-of-Apache setup; each layer has its own URL
   rewrite handling and they can fall out of sync after a config
   reload.

Workarounds in place

The clients we control already retry — the GitHub Actions verify step
(foundation-docs/.github/workflows/deploy-docs.yml) now uses
curl --retry 3 --retry-all-errors --retry-delay 5 plus an outer
30-minute polling window, which is enough to ride out the 404 wave.
But this masks the underlying issue.

What we don't know

• How exactly the cron driver is set up (it hits
  /cdcf/v1/process-queue from 2001:41d0:52:bff::14c, which is a
  GitHub Actions IPv6 — but no scheduled workflow in our repos calls
  this endpoint, so it must be configured outside the repo, possibly
  via the Plesk panel or a third party).
• Whether the 404 rate correlates with any Plesk maintenance event
  or nginx config reload.
• Plesk's actual nginx + Apache vhost config for cms.catholicdigitalcommons.org.

Suggested next steps

1. Pull the actual nginx config off the production VPS:
   /etc/nginx/plesk.conf.d/vhosts/cms.catholicdigitalcommons.org*.conf
   and /var/www/vhosts/system/cms.catholicdigitalcommons.org/conf/
2. Check the WordPress permalink rewrite rules are present in the
   additional nginx directives slot in the Plesk panel
3. If using Plesk's "static files served by nginx" optimization,
   verify it doesn't intercept /wp-json/*
4. Identify the cron driver hitting /process-queue (Plesk
   scheduler? Cloudflare Workers? UptimeRobot?) and confirm where it's
   configured

Severity

Low for now — clients retry, recoveries complete, no end-user impact.
But it's noise in the logs and an availability landmine if/when the
404 rate spikes further.

Related

• Surfaced during recovery deploys for the foundation-docs translation
  pipeline (see feat(deploy): verify translations completed before declaring success foundation-docs#22, chore(deps): Bump react from 19.2.4 to 19.2.5 #23).

[JohnRDOrazio]

Investigation: cron driver identified, hypothesis for the 404 race

The cron driver is our own infrastructure

The IPv6 2001:41d0:52:bff::14c is OVH (Roubaix, France) per RIPE — not GitHub Actions as the issue body originally hypothesised. Cross-checked against the repo's own VPS_HOST_KEY variable, which pins 92.222.13.25 (also OVH). Same provider, same locality — almost certainly the same VPS.

The driver is scripts/cdcf_queue_worker.sh, installed as cdcf-queue-worker.service on the production VPS per docs/redis-queue-worker.md. It loops every POLL_INTERVAL=15s and (per the documented example config with CONCURRENCY=5) fires 5 parallel POST requests every cycle to https://cms.catholicdigitalcommons.org/wp-json/cdcf/v1/process-queue.

So the requests aren't external traffic — they're loopback POSTs from the same VPS to itself, going out over the public internet via Plesk's nginx → PHP-FPM stack.

Why the 404s correlate with deploys

Both timestamps quoted in the issue (success at 16:25:23, 404 at 16:28:32) fall inside the same workflow_dispatch deploy window. Looking at .github/workflows/deploy.yml:

• Lines 311-315: tar -xzf extracts new theme + plugin bundles over the live WP install
• Line 325: "Activate plugins" — POST /wp/v2/plugins/<slug> with {"status":"active"} for every deploy, even when the plugin is already active
• The deploy does not pause the queue worker first

Each plugin activation in WordPress triggers flush_rewrite_rules(), which rewrites .htaccess. Plesk's nginx mirrors .htaccess into its own internal rewrite-rule list — but with a small lag during regeneration. During that lag window, requests to /wp-json/* fall through nginx's try_files ladder, fail to match anything in the filesystem, and 404 from the openat() filesystem layer — exactly what the issue's evidence shows. Meanwhile the queue worker keeps firing 5 concurrent requests every 15 seconds throughout the deploy, hitting that race window every cycle.

This fully explains the "30-50% of requests during heavy traffic windows" observation: the heavy traffic is the deploy.

Mitigations

1. Idempotent plugin activation — feasible from the repo. PR forthcoming. The deploy will check current plugin status via GET /wp/v2/plugins/<slug> and only call POST when status isn't already active. This eliminates the redundant flush_rewrite_rules() calls that cause the rewrite-rule lag, which should kill most of the 404 burst on subsequent deploys (since most deploys don't change which plugins are active).
2. Plesk-side nginx tweak — independent investigation underway to verify whether the vhost's nginx config has a location /wp-json/ block that would isolate API routes from the try_files race entirely. Will follow up in this issue or a separate one.

Status update for the issue body

The "GitHub Actions IPv6" assumption in the original report is incorrect. Updating the title and severity is probably warranted once mitigation #1 lands and we measure the residual 404 rate.

[JohnRDOrazio]

Update: investigation invalidates the rewrite-rule-lag theory

Pulled some operational telemetry from the production environment. The data substantially changes the picture.

Worker-side error distribution

Across the journal retention window (4+ days), the queue worker has logged:

• 317 HTTP 504 Gateway Timeout
• 4 HTTP 502 Bad Gateway
• 0 HTTP 404 responses

So the cron driver (confirmed: cdcf-queue-worker.service on the VPS, source IPv6 = 2001:41d0:52:bff::14c) experiences timeouts during deploys, not routing failures.

504 rate by day

Apr 27: 10 504s
Apr 28: 10 504s
Apr 29: 262 504s   ← deploy day
Apr 30: 25 504s

The Apr 29 spike is concentrated 16:00–19:00:

• 16:00 hour: 66 warnings
• 17:00 hour: 133 warnings
• 18:00 hour: 68 warnings

So during the deploy, the worker saw a ~25× spike in 504s and zero 404s.

.htaccess stability

Vhost .htaccess was last modified 2026-02-18 — over two months before the deploy referenced in the issue. Plugin activation via POST /wp/v2/plugins/X does NOT trigger a .htaccess rewrite (probably because the activation hooks don't call flush_rewrite_rules(true)). My PR #61 was built on a faulty premise.

Live routing check

From within the same hosting environment, with the queue worker idle:

• GET /wp-json/ → 200 (~300ms)
• POST /wp-json/cdcf/v1/process-queue (no auth) → 401 (~440ms)

Routing is healthy. nginx → PHP-FPM works correctly when load is normal.

Re-interpretation of the original proxy_error_log evidence

[error] openat() ".../wp-json/cdcf/v1/process-queue" failed (2: No such file or directory)

These openat() errors are likely routine try_files probing, not actual 404 responses. Plesk's nginx vhost almost certainly has try_files $uri $uri/ /index.php?$args (or similar), which probes the filesystem before falling through to PHP. Each failed probe logs at level [error] but the request still proceeds to the index.php fallback and succeeds. That would explain why the queue worker's logs show zero 404s — the requests actually completed with 200/200/200 even when proxy_error_log showed the openat() noise.

The wp/v2/pages/<id> 404s mentioned in the issue body remain a separate question — those are from the foundation-docs deploy verify step (GitHub Actions IPv4/IPv6 from Microsoft Azure), not from the queue worker.

Revised root-cause hypothesis: PHP-FPM saturation during deploys

The 504s during Apr 29 16:00–19:00 are most consistent with PHP-FPM pool exhaustion during deploys:

1. The deploy extracts theme + plugin tarballs → OPcache invalidation across all FPM workers → each request triggers a fresh PHP compile, multiplying CPU and memory pressure
2. The plugin activation REST POSTs run synchronous activation hooks (Polylang, ACF, custom theme code)
3. Meanwhile the queue worker keeps firing CONCURRENCY=5 parallel POSTs every 15s, processing translations that themselves can be slow (OpenAI calls)
4. Plesk's default FPM pool size is 5–10 workers — easily saturated by the combination
5. New requests time out at nginx's fastcgi_read_timeout → 504

If this is right, the real levers are:

a. Pause/throttle the worker during deploys via a maintenance flag — deploy POSTs to a /cdcf/v1/maintenance endpoint that sets a transient flag; the worker checks the flag and skips its loop. Coding work that stays inside the existing automation surface.
b. Increase Plesk's PHP-FPM pool size for cms.catholicdigitalcommons.org (panel-side change).
c. Set opcache.validate_timestamps=0 + explicit reset post-deploy. Removes per-request opcache stat overhead and avoids mass-recompile thundering herd. Per-vhost PHP setting in Plesk panel.

What this means for PR #61

#61 (idempotent plugin activation) is still useful as code hygiene — eliminates redundant work and at least removes one variable. But it probably won't move the 504 needle much, since the dominant load source is FPM saturation, not rewrite-rule lag. Updating the PR description accordingly.

Closing the .htaccess-race theory as disproven; opening the FPM-saturation theory for further investigation. Suggesting we either (a) implement the maintenance-flag throttle in code, or (b) bump FPM pool size in Plesk, before further code changes.