develop#94
Open
ariedotcodotnz wants to merge 22 commits into
Open
Conversation
…side using `logger.exception`; return user-friendly error messages.
…logs from admin panel
…dd `check` command for verifying Flask-Migrate installation and setup
…ng, downloading, and clearing admin logs
…ng for consistency with generated text
… and adjust scaling logic for consistency with generated text layout
…e target height, improve consistency with generated text, and add debug logs for preprocessing and overrides
…simplify stroke generation, and dynamically render overrides in SVG gaps.
…es, ensure clean SVG insertion, and improve debug logs for exclusion zones and rendering steps.
…s for precise override positioning, implement model-level cutting, and refine rendering logic for transitions.
…reserve RNN context, refine chunking, and improve line stitching with attention-based character indices.
…ezier curve support, simplify exclusion logic, and enhance debug logs.
… override position tracking, and refine stroke generation logic.
…efine character index handling, and enhance debug logs for stroke generation with styles.
…ogic, implement auto-detection from data, and refine override position calculations with enhanced debug logs.
…n-based shifting for gap creation, and simplify rendering logic by removing exclusion zones.
…ld-based character searches, refine override positioning, and improve debug logging for gap and shift calculations.
… long words Improvements to text chunking (operations/chunking.py) -- the step that splits text before generation and drives how lines fill. No model retraining. - Fix 'balanced' strategy: its punctuation branch sat behind an if/elif on overlapping sets and was unreachable, so the DEFAULT strategy never broke at commas/semicolons. 'balanced' now considers sentence + punctuation breaks. - Honor target_chars_per_chunk: it was accepted everywhere but never used. The character budget now bounds the break-search window so breaks land on real punctuation that also fits the target, giving even line filling. - Hard-split pathological space-less tokens (URLs, long ids) so they cannot exceed the model's sequence limit or run off the page (lossless reassembly). - Guarantee loop progress (>=1 word/iter) under degenerate params. Adds model-free tests (tests/test_operations.py) covering all strategies, the character budget, long-word splitting, and whitespace handling. 8/8 passing. Note: the original local work also touched _draw.py/Hand.py (override spacing, per-line auto-size). Those were dropped during rebase because origin/develop independently rewrote that override/layout subsystem (char_indices, effective_target_h) and supersedes them; this commit keeps only the upstream-untouched chunking improvements. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…hance writing size handling
…ting size estimation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| flash('Invalid log file path.', 'error') | ||
| return redirect(url_for('admin.error_logs')) | ||
|
|
||
| if os.path.exists(filepath): |
|
|
||
| if os.path.exists(filepath): | ||
| try: | ||
| with open(filepath, 'r', encoding='utf-8', errors='replace') as f: |
| flash('Invalid log file path.', 'error') | ||
| return redirect(url_for('admin.error_logs')) | ||
|
|
||
| if not os.path.exists(filepath): |
|
|
||
| log_activity('admin_action', f'Downloaded log file: {filename}') | ||
|
|
||
| return send_file(filepath, as_attachment=True, download_name=filename) |
| flash('Invalid log file path.', 'error') | ||
| return redirect(url_for('admin.error_logs')) | ||
|
|
||
| if not os.path.exists(filepath): |
|
|
||
| try: | ||
| # Truncate the file | ||
| with open(filepath, 'w') as f: |
| if not file_path.startswith(base_path + os.sep) and file_path != base_path: | ||
| # Check if the file exists before attempting to serve | ||
| file_path = os.path.join(base_directory, safe_filename) | ||
| if not os.path.isfile(file_path): |
There was a problem hiding this comment.
Pull request overview
This PR makes broad changes across the Flask webapp and the handwriting engine to improve “natural” sizing/layout (writing-size in mm, page-fill auto sizing, balanced chunk line breaking), add optional text reflow, expand character-override rendering using attention-derived character indices, and introduce an admin error-log viewer while tightening error responses/logging.
Changes:
- Add “natural” handwriting sizing controls (
writing_size_mm) and optional input reflow (reflow) through UI → API → generation pipeline. - Rework chunking/wrapping behavior (balanced DP line breaks, char-budgeted chunking, hard-splitting pathological tokens) and add model-free tests for operations + sizing.
- Add admin log viewing/downloading/clearing UI and routes; reduce user-facing exception detail across several endpoints.
Reviewed changes
Copilot reviewed 27 out of 28 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| webapp/utils/text_utils.py | Adds reflow_paragraphs() to join soft-wrapped paragraphs before wrapping/generation. |
| webapp/utils/generation_utils.py | Plumbs writing_size_mm + reflow through param parsing and generation dispatch. |
| webapp/templates/index.html | Updates sizing UI copy and adds “Writing Size (mm)” + “Reflow text” controls. |
| webapp/templates/admin/logs.html | New admin UI for browsing/filtering/downloading/clearing log files. |
| webapp/templates/admin/character_overrides/view.html | Tweaks override preview stroke-width input defaults/labeling. |
| webapp/templates/admin/base.html | Adds “Error Logs” to admin navigation. |
| webapp/static/js/modules/alpine-app.js | Adds frontend state + request payload fields for writing_size_mm and reflow. |
| webapp/routes/style_routes.py | Switches to send_from_directory and logs exceptions server-side with generic client error. |
| webapp/routes/presets_routes.py | Replaces detailed exception strings with server-side logging + generic error responses. |
| webapp/routes/job_routes.py | Removes internal exception details from API responses; logs stack traces server-side. |
| webapp/routes/generation_routes.py | Splits ValueError vs generic exceptions and logs exceptions; changes error payloads. |
| webapp/routes/character_override_routes.py | Replaces print() exception logging with current_app.logger.exception() and generic error text. |
| webapp/routes/batch_routes.py | Replaces print() debugging/errors with structured logging + generic client error. |
| webapp/routes/admin_routes.py | Adds admin log management endpoints and supporting parsing/sanitization helpers. |
| webapp/init_db.py | Adds schema reconciliation for missing columns + stamps Alembic head; seeds default page sizes. |
| tests/test_sizing.py | Adds model-free dimensional tests for the new natural sizing behavior in _draw. |
| tests/test_operations.py | Adds model-free tests for the updated chunking + balanced line-break behavior. |
| handwriting_synthesis/rnn/RNN.py | Extends sampling to return attention-derived char_indices per timestep. |
| handwriting_synthesis/hand/operations/sampling.py | Adds optional return of attention char_indices alongside strokes. |
| handwriting_synthesis/hand/operations/chunking.py | Adds balanced DP line breaking + char-budgeted chunking + hard split of long tokens. |
| handwriting_synthesis/hand/operations/init.py | Exports balanced_line_breaks. |
| handwriting_synthesis/hand/Hand.py | Plumbs writing_size_mm; overhaul override handling to use space placeholders + attention indices. |
| handwriting_synthesis/hand/character_override_utils.py | Improves SVG path coordinate extraction to include curve commands for bounding boxes. |
| handwriting_synthesis/hand/_draw.py | Major sizing/layout rework (natural x-height, width clamp, vertical fit) + override insertion logic. |
| deploy/db-migrate.sh | Ensures FLASK_APP=webapp.app:app is set for Flask-Migrate commands; adds a check command. |
| CLAUDE.md | Adds repository guidance (commands, pipeline overview, model constraints, gotchas). |
| .claude/settings.local.json | Expands allowed bash commands for Claude tooling (wc/grep). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+842
to
+845
| # Security check: ensure the file is within logs_dir | ||
| real_logs_dir = os.path.realpath(logs_dir) | ||
| real_filepath = os.path.realpath(filepath) | ||
| if not real_filepath.startswith(real_logs_dir): |
Comment on lines
+916
to
+920
| # Security check: ensure the file is within logs_dir | ||
| real_logs_dir = os.path.realpath(logs_dir) | ||
| real_filepath = os.path.realpath(filepath) | ||
| if not real_filepath.startswith(real_logs_dir): | ||
| flash('Invalid log file path.', 'error') |
Comment on lines
+945
to
+949
| # Security check: ensure the file is within logs_dir | ||
| real_logs_dir = os.path.realpath(logs_dir) | ||
| real_filepath = os.path.realpath(filepath) | ||
| if not real_filepath.startswith(real_logs_dir): | ||
| flash('Invalid log file path.', 'error') |
Comment on lines
+869
to
+880
| # Apply filters | ||
| if level_filter != 'all' and entry['level'] != level_filter: | ||
| continue | ||
|
|
||
| if search_query and search_query.lower() not in entry['clean'].lower(): | ||
| continue | ||
|
|
||
| log_entries.append(entry) | ||
|
|
||
| # Limit entries | ||
| if len(log_entries) >= lines_limit: | ||
| break |
Comment on lines
+84
to
+90
| conn.execute(text( | ||
| f'ALTER TABLE "{table.name}" ADD COLUMN "{column.name}" {col_type}')) | ||
| if not column.nullable: | ||
| conn.execute( | ||
| text(f'UPDATE "{table.name}" SET "{column.name}" = :val ' | ||
| f'WHERE "{column.name}" IS NULL'), | ||
| {"val": _placeholder_for(column)}) |
Comment on lines
+226
to
+239
| # Convert to line_segments format (single segment per line, like non-override) | ||
| line_segments = [] | ||
| for line_idx, (original_line, strokes, char_indices) in enumerate( | ||
| zip(lines, generated_strokes, char_indices_list) | ||
| ): | ||
| line_segments.append([{ | ||
| 'type': 'generated', | ||
| 'text': original_line, # Keep original text for reference | ||
| 'modified_text': modified_lines[line_idx], # Text that was actually generated | ||
| 'strokes': strokes, | ||
| 'char_indices': char_indices, # NEW: Character index per stroke from attention | ||
| 'line_idx': line_idx, | ||
| 'override_positions': override_positions[line_idx] # [(char_idx, char), ...] | ||
| }]) |
Comment on lines
+212
to
+214
| print(f"DEBUG: Original lines: {lines}") | ||
| print(f"DEBUG: Modified lines (with placeholders): {modified_lines}") | ||
| print(f"DEBUG: Override positions: {override_positions}") |
Comment on lines
+86
to
+90
| A run of consecutive non-blank lines is treated as a single paragraph and | ||
| joined with spaces (so the generator re-wraps it to the page width); blank | ||
| lines are preserved as paragraph separators. This prevents short, hard-wrapped | ||
| input (e.g. a letter pasted with a line break every few words) from rendering | ||
| as a column of half-empty lines. Turn it off to keep your exact line breaks. |
Comment on lines
+114
to
+116
| except Exception as e: | ||
| current_app.logger.exception('Generation error') | ||
| return jsonify({"error": "Failed to generate handwriting. Please check your parameters."}), 400 |
Comment on lines
+462
to
+467
| <label class="bx--checkbox-wrapper m-0"> | ||
| <input id="reflowText" type="checkbox" class="bx--checkbox" x-model="reflowText" /> | ||
| <label for="reflowText" class="bx--checkbox-label"> | ||
| <span class="bx--checkbox-label-text">Reflow text to fill width (Recommended)</span> | ||
| </label> | ||
| </label> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.