Repository navigation
chore: promote main to staging (progressive visual ingestion) - #472
Merged
Merged
Conversation
- Added `ingestion_mode` to the `DocumentIngestionCommand` and `DocumentIngestionService` to support both "full" and "progressive" ingestion modes. - Implemented validation to ensure "progressive" mode is only used with page-memory formats. - Updated job metadata to include `ingestion_mode` and adjusted related tests to validate the new functionality. - Enhanced page memory processing to accommodate the new ingestion mode logic.
- Updated the granularity decision logic to reflect a maximum of 5 pages for whole-document chunks instead of 6. - Removed unused asset allocation logic from the phase A processing and related functions. - Cleaned up references to asset max pages across various modules, ensuring consistency in asset extraction handling. - Updated tests to align with the new granularity and asset handling changes.
…pages handling - Eliminated the `derive_excluded_toc_pages` function from the utility module and its references across various services and scripts. - Updated toc_pages processing to directly sort and filter pages, enhancing clarity and reducing complexity. - Adjusted related tests to ensure they align with the new toc_pages handling logic.
…val tools - Revised the description of body text handling in the agent exploration prompt and configuration files for improved clarity. - Adjusted documentation in the corpus overview and enrichment state files to better explain tool usage and state signals. - Updated tests to reflect changes in the body text description, ensuring consistency across documentation and functionality.
- Deleted the entire `deprecated/mapnav` directory, including all associated files and configurations, as it is no longer in use. - Updated references in documentation to reflect the removal of map-nav, ensuring clarity in the current retrieval architecture. - Adjusted agent retrieval descriptions in `AGENTS.md` and `CONTEXT.md` to emphasize the transition to `agent_tools` and `agent_explore`. - Cleaned up related test cases to remove dependencies on the deprecated map-nav functionality.
- Updated the `pgvector` dependency to allow for versions greater than 0.2.4, ensuring compatibility with new features. - Introduced a new test for scanned progressive M0 documents that skips OCR processing and builds visual page chunks. - Implemented logic to handle visual embeddings for scanned documents, enabling hybrid visual and lexical retrieval. - Added new configuration options for embedding models and visual retrieval settings to support enhanced functionality. - Updated relevant services and database models to accommodate the new visual embedding structure.
- Introduced new functions to manage structure claims and expiration, improving the handling of job metadata during enrichment tasks. - Updated the `execute_structure_enrichment` function to ensure proper cleanup and error handling. - Enhanced the `grep` tool to provide informative notes for scanned documents pending OCR, improving user feedback during retrieval operations. - Adjusted embedding model configurations across various services to support the new multimodal embedding approach. - Updated tests to validate the new structure enrichment logic and retrieval enhancements, ensuring robustness in functionality.
- Updated the `visual_page_discovery` function to include support for excluding specific sections during retrieval, improving query flexibility. - Refactored the embedding replacement logic to streamline the rebinding of existing page embeddings to the current revision's page chunks. - Modified tests to validate the new retrieval logic and ensure accurate scoring and chunk identification. - Improved the clarity of docstrings and comments for better maintainability and understanding of the codebase.
- Added VERCEL_KEY to environment configurations for API access. - Updated embedding model to use Google Gemini Embedding 2, enhancing the quality of page-image and query embeddings. - Adjusted embedding dimensions to 3072 across relevant services and database models for consistency. - Refactored embedding client to support new Vercel API, ensuring compatibility with updated embedding logic. - Enhanced tests to validate the new embedding configurations and retrieval functionalities.
- Updated pgvector dependency to version 0.5.1 across multiple files for improved functionality. - Refactored embedding column resizing logic to support new column types and index operations. - Enhanced error handling in the embedding client to manage retry logic for HTTP requests. - Updated tests to validate new embedding behavior and ensure robustness against various HTTP responses.
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…al-ingestion feat: progressive visual ingestion and Gemini page embeddings
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| import sqlalchemy as sa | ||
| from alembic import op | ||
|
|
||
| revision: str = "6f7a8b9c0d1e" |
| from alembic import op | ||
|
|
||
| revision: str = "6f7a8b9c0d1e" | ||
| down_revision: str = "5e6f7a8b9c0d" |
|
|
||
| revision: str = "6f7a8b9c0d1e" | ||
| down_revision: str = "5e6f7a8b9c0d" | ||
| branch_labels: Sequence[str] | None = None |
| revision: str = "6f7a8b9c0d1e" | ||
| down_revision: str = "5e6f7a8b9c0d" | ||
| branch_labels: Sequence[str] | None = None | ||
| depends_on: Sequence[str] | None = None |
| from alembic import op | ||
| from pgvector.sqlalchemy import Vector | ||
|
|
||
| revision: str = "7a8b9c0d1e2f" |
| revision: str = "8b9c0d1e2f3a" | ||
| down_revision: str = "7a8b9c0d1e2f" | ||
| branch_labels: Sequence[str] | None = None | ||
| depends_on: Sequence[str] | None = None |
| ] | ||
| } | ||
|
|
||
| import app.services.page_memory.page_assets as page_assets_mod |
| } | ||
|
|
||
| import app.services.page_memory.page_assets as page_assets_mod | ||
| import app.services.page_memory.page_renderer as page_renderer_mod |
|
|
||
| import app.services.page_memory.page_assets as page_assets_mod | ||
| import app.services.page_memory.page_renderer as page_renderer_mod | ||
| import app.services.page_memory.page_tagger as page_tagger_mod |
| "_resume_profile_from_m0_workspace", | ||
| lambda **_kw: fake_profile, | ||
| ) | ||
| import app.services.page_memory.skeleton_extractor as skel_mod |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mainintostagingafter feat: progressive visual ingestion and Gemini page embeddings #471 (progressive visual ingestion and Gemini page embeddings).Includes
Test plan
stagingMade with Cursor