fix: remediate Sentry errors — scanner blocking, daily stats key, contact form validation, mail failure handling - #9
Open
bluebamus wants to merge 12 commits into
Open
Conversation
The 400/403/404/500 handlers built an HttpResponse, set status_code on it, then discarded it and returned render()'s new response, which defaults to 200. Every error page answered HTTP 200. Scanners saw every path as valid, and search engines and monitoring could not tell an error from a normal page. The existing test only checked for the "404 Error" string in the body, so it did not catch it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
get_or_create(created_at__date=today) could not write the lookup value on insert, so the same day accumulated multiple rows. Once a duplicate existed, every later request raised MultipleObjectsReturned (Sentry PYTHON-DJANGO-7H, 7G). Add stat_date as the aggregation key with a unique constraint, and replace the lookup with UPDATE -> INSERT -> (on race) UPDATE. A single UPDATE statement is atomic, so select_for_update is no longer needed, and the unique constraint is what finally prevents duplicates. stat_date is nullable because migrations are gitignored here: adding a not-null unique column to a populated table would give every existing row the same default and violate the constraint. The new dedupe_connection_stats command merges duplicates by summing their counters and backfills stat_date afterwards. Also: - exclude paths by prefix instead of testing for the "admin" substring, which skipped legitimate URLs that merely contained it - keep statistics DB errors from failing the request - fix the admin changelist, which matched created_at__day (day of month) and so mixed in rows from other months Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PHP/WordPress probes such as /wp.php and /bbs/board.php reached Django, and because the statistics middleware runs before URL resolution, each probe also took a DB write path and produced Sentry events. BlockSuspiciousPathMiddleware sits right after SecurityMiddleware and ahead of the statistics middlewares, returning an empty-bodied 404 so blocked requests touch neither the statistics tables nor the resolver. It is a fallback for the nginx rules in docs/ai-docs/security/02-nginx-php-scan-blocking.md, not a replacement: Django cannot drop a connection the way nginx's `return 444` does. This commit also adds the tunables the remediation introduces to base.py: BLOCK_SUSPICIOUS_PATHS, SUSPICIOUS_PATH_RESPONSE_STATUS, SUSPICIOUS_PATH_PATTERNS, STATS_EXCLUDED_PATH_PREFIXES, EMAIL_DNS_VALIDATION and EMAIL_DNS_VALIDATION_TIMEOUT. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… delivery The view read request.POST directly and saved without validating, so a phone number longer than the varchar(16) column reached the INSERT (Sentry PYTHON-DJANGO-7B). GetInTouchForm now holds the input policy in one place, matching the model column constraints. Mail handling is reworked so an outage cannot be mistaken for success: - store the inquiry first, then send, so a vendor failure never loses it - send_mail_sync() returns whether delivery succeeded; the thread-based send_mail() is kept for verify_email_mixins and now shares its body - GetInTouchLog.status (received/queued/sent/failed) records the mail result, while state keeps its meaning as "the inquiry was received" - on failure the user is told the message was received but the mail is delayed, instead of being shown a success message - recipients are masked in logs and Sentry Other fixes in the same path: - remove six unreachable duplicate save blocks after the first return - move DNS validation into clean_emailfrom(), which also fixes the uninitialised is_valid returned from the old exception path - treat DNS timeout / nameserver failures as "cannot verify" and let the address through, instead of rejecting the user as before; missing MX records are still a rejection - pin the recipient to settings.DEFAULT_FROM_EMAIL rather than the emailto hidden input - replace the inline phone regex, whose [0|1|6|7|8|9] character class literally allowed "|", with a shared validator - mirror the server-side limits as maxlength/required on the template Tests use the locmem mail backend and skip DNS lookups. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Separate real failures from probe traffic so a recurring event is worth investigating. before_send discards Http404, DisallowedHost and SuspiciousOperation, and any event whose request URL is a .php, wp-admin/wp-content/wp-includes or dotfile probe. The URL pattern accepts ?, # and / as path terminators because it matches a full URL here, unlike the middleware which matches path_info. Wired into sentry_sdk.init() in both prod and stage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add reports/06-implementation-report.md and update the existing document set to reflect what was implemented, what was decided, and what is left as operations work. Records three things the planning documents could not confirm: - statistics migrations are generated under home/, not custom_middlewares/, because of Meta.app_label - with no migrations/ directory in the repo, a bare `makemigrations` reports "No changes detected" and creates nothing; the app labels must be named explicitly the first time - config/settings/prod.py fails `manage.py check` with staticfiles.E002, a pre-existing problem that matters when running the cleanup command Also documents the migration rehearsal used to verify that the schema change applies cleanly to a table that already holds duplicate rows. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
prod listed STATIC_ROOT inside STATICFILES_DIRS, which raises staticfiles.E002. Because system checks run ahead of every management command, prod settings could not run migrate or the new dedupe_connection_stats command at all. stage.py already commented STATICFILES_DIRS out for this reason, noting that it is unnecessary when static files are served from one place. prod now matches: static is served from STATIC_ROOT (ROOT_DIR/static), which already holds the collectstatic output. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Sentry DataError came from an automated submission, and form validation alone only stops malformed input, not the volume. Gate the form behind reCAPTCHA, which is already installed and configured. CONTACT_FORM_CAPTCHA controls it and defaults to False, so environments without real keys keep working. prod and stage enable it. The field is built in __init__ rather than declared as a class attribute: ReCaptchaField reads the keys when it is constructed, so a class attribute would demand them at import time. Verification calls Google, so tests keep it off and mock captcha.fields.client.submit for the enabled paths. PortfolioView puts the form into the context after the redis cache is written; caching a form instance would share it across every visitor. The template renders the widget only when the field exists. Note: RECAPTCHA_PUBLIC_KEY/PRIVATE_KEY still default to Google's test keys, and django-recaptcha's own system check fails on them, so a deployment cannot silently ship with a decorative captcha. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bluebamus
temporarily deployed
to
deply-oci-single-service
August 10, 2026 04:17 — with
GitHub Actions
Inactive
Coverage on the modules this branch touches was 83%, with two gaps that mattered: - both statistics changelist_view methods were at 0%, even though "the existing admin statistics screen still works" is an acceptance criterion of the remediation plan - only the 404 error handler was exercised, so the 400/403/500 fixes were unverified Also covers EmailThread, which this branch rewrote and which the signup verification mail still uses, plus the remaining branches in the statistics middleware and the get-in-touch form. Every module changed on this branch is now at 100% line coverage. Separately, clear the redis cache around each test. The cache outlives a test run, so a leftover key flipped cached/uncached branches and made coverage of PortfolioView swing between 62% and 78% depending on what had run before. Two consecutive runs now report identical coverage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bluebamus
temporarily deployed
to
deply-oci-single-service
August 10, 2026 04:27 — with
GitHub Actions
Inactive
Running the app end-to-end against PostgreSQL surfaced two problems that the test suite could not see. Masking the recipient moved the address and the failure reason into extra=, but the project's log formatter does not render extra fields. File logs were left with a bare "Error sending email" — less useful than before this branch, since the original line at least named the recipient. Put the masked value and the error back in the message and keep extra= for Sentry. The same applied to access_guard, statistics and the get-in-touch form. The blocked-path line never appeared at all: blocking is normal behaviour so it logs at INFO, but COMMON_LOGGER is set to WARNING and filtered it out. The implementation report claimed blocked volume could be read from this log, which was untrue. SUSPICIOUS_PATH_LOG_LEVEL now controls it, and the report says what the default actually does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
docs/error의 Sentry 이슈 5건을docs/ai-docs의 기획·설계·작업 계획서에 따라 구현했다.계획서 Phase 1~5의 코드 작업이 끝났다. Phase 0(nginx)과 SendGrid 계정 점검은 이 저장소 밖의 운영 작업이다.
테스트: 123 passed / 1 skipped (기준선 9 passed / 1 skipped). skip 1건은 이 작업과 무관한 기존 항목이다.
dev·test·stage·prod네 설정 모두manage.py check오류가 없다.Sentry 이슈별 처리
ConnectionHardwareStats/MethodStats.MultipleObjectsReturnedget_or_create(created_at__date=…). 생성 시 조회 조건을 컬럼에 반영할 수 없어 중복 row가 쌓이고, 한 번 생기면 이후 모든 요청이 실패stat_date유니크 집계키 + UPDATE→INSERT→UPDATEDataError value too long for varchar(16)request.POST를 직접 읽고 저장 전 검증 없음GetInTouchForm이 저장 전에 거부HTTP 401 / Maximum credits exceededstate)와 발송(status) 분리. 401 자체의 원인은 운영 점검 필요주요 변경
BlockSuspiciousPathMiddleware—SecurityMiddleware직후, 통계 미들웨어 앞에서.php/wp-*/.env경로를 본문 없는 404로 조기 차단. nginx의 대체물이 아니라 fallback이다.select_for_update없이 동작하고, DB 유니크 제약이 중복을 최종적으로 막는다.dedupe_connection_stats— 날짜별 합산 병합 +stat_datebackfill.--dry-run지원, 멱등.GetInTouchForm— 입력 정책을 한곳에 모아 모델 컬럼 제약과 일치시켰다. 도달 불가능한 저장 코드 6블록 제거. DNS 타임아웃은 '검증 불가'로 보고 통과시키고 MX 부재만 거부한다.send_mail_sync()+GetInTouchLog.status— 문의를 먼저 저장한 뒤 발송하므로 벤더 장애로 문의가 유실되지 않는다. 실패 시 "접수 완료, 발송 지연"으로 안내. 로그의 수신자 주소는 마스킹.CONTACT_FORM_CAPTCHA로 제어, 기본False, prod/stage만 활성.before_send—Http404/DisallowedHost/SuspiciousOperation과 스캐너 URL 이벤트를 버린다.계획 외 발견 사항
커스텀 에러 페이지가 전부 HTTP 200을 반환하고 있었다. 400/403/404/500 핸들러가
status_code를 설정한 응답을 버리고render()의 새 응답(기본 200)을 반환했다. 존재하지 않는 URL이 200으로 응답하면 스캐너에게는 모든 경로가 유효해 보인다. 기존 테스트는 본문 문자열만 검사해 놓치고 있었다.prod설정이manage.py check에서 실패하고 있었다.staticfiles.E002. 체크는 모든 관리 명령 앞에서 실행되므로 prod 설정으로는migrate도 정리 명령도 돌릴 수 없었다.stage.py가 이미 같은 이유로 처리해 둔 형태에 맞췄다.마이그레이션 동작 2건 확인. 통계 마이그레이션은
Meta.app_label때문에home/migrations/에 생성된다. 그리고migrations/디렉터리가 없는 상태에서 인자 없는makemigrations는 프로젝트 앱에 아무것도 만들지 않으므로 최초 1회는 앱 이름을 명시해야 한다.검증
실제 운영 업그레이드 순서를 재현했다 — 변경 전 모델로 레거시 스키마 생성 → 같은 날짜 중복 row 심기 →
migrate→dedupe_connection_stats.2026-08-05: win 10+7 = 17)머지 후 운영 작업
순서를 지켜야 한다. 상세 절차는
docs/ai-docs/reports/06-implementation-report.md에 있다.docs/ai-docs/security/02-nginx-php-scan-blocking.md)makemigrations home portfolio→migratededupe_connection_stats --dry-run→dedupe_connection_statsRECAPTCHA_PUBLIC_KEY/RECAPTCHA_PRIVATE_KEY에 실제 키 설정 (테스트 키면check가 실패한다)알려진 트레이드오프
문의 폼 메일이 동기 발송이라 발송 결과를 정확히 알 수 있는 대신, 벤더 응답이 느리면 요청이 길어진다. SendGrid 백엔드는
EMAIL_TIMEOUT을 따르지 않는다.status에queued를 미리 정의해 두었으므로 Celery 이관은 바로 가능하다.🤖 Generated with Claude Code