refactor: PROFILE coarse workflow and TOC hierarchy cleanup - #302
Merged
Conversation
…TOC hierarchy handling - Removed `toc_page_offset` from `ProfileCoordinator`, `PageAnatomyMap`, and `AgentBlackboard` classes. - Updated `build_anatomy_map` and related functions to eliminate reliance on `toc_page_offset`. - Introduced new functions for aggregating and resolving TOC hierarchies, improving the handling of TOC data during shard planning and anchoring processes. - Enhanced debug scripts to log pending TOC relationships and their statuses. - Adjusted tests to reflect the removal of `toc_page_offset` and ensure proper functionality of TOC anchoring and grafting logic.
- Updated `.env.example` files for both API and worker to set `PDF_PROFILE_TOC_ENABLED` to true. - Introduced `backfill_parent_offset_matches` function to improve TOC parent anchoring by reusing descendant offsets. - Added new helper functions for iterating and managing printed page parents in TOC structures. - Enhanced `layout_parser` and `body_boundary` to better handle TOC entries and their levels. - Updated tests to ensure proper functionality of new TOC handling logic and verify that profiling behavior aligns with the new defaults.
- Updated the document profile coordinator to enhance workflow clarity and efficiency. - Removed the ReActExecutor and its associated logic, replacing it with a more deterministic approach for shard planning and validation. - Introduced new methods for dispatching anatomy tools and finalizing shard plans, improving error handling and decision-making processes. - Refactored calibration agent components to align with the new workflow, including the removal of obsolete files and functions. - Enhanced documentation and comments to reflect changes in the calibration process and tool interactions.
- Replaced references to the `planner` with `coarse_profile` in the document profile workflow for clarity. - Updated the `ProfileCoordinator` to utilize `CoarseProfiler` instead of `ProfilePlanner`. - Adjusted budget handling to reflect the new coarse profile stage. - Removed obsolete planner files and functions to streamline the codebase. - Enhanced tests to ensure proper functionality of the new coarse profile integration.
- Updated the `ProfileCoordinator` to streamline shard planning and validation processes. - Removed obsolete methods and classes, including the `toc_link_enrichment` module and related tests, to clean up the codebase. - Improved documentation and comments for clarity on the new workflow. - Adjusted various components to ensure compatibility with the updated architecture and enhance overall functionality.
- Deleted the `page_links.py` module, consolidating its functionality into `probe_links.py` for better organization. - Updated `probe_links.py` to include necessary classes and functions previously in `page_links.py`, ensuring all page link handling is centralized. - Removed obsolete contract tests related to page link normalization, streamlining the test suite. - Enhanced documentation in `probe_links.py` to clarify the handling of internal page hyperlinks.
Move calibration out of agents/, rename Profile* types and inspect_vlm/locate_method labels, and align PARSE_PROFILE_* env keys with the fixed workflow. Co-authored-by: Cursor <cursoragent@cursor.com>
… profile handling - Eliminated H1BoundaryResult and H1Candidate classes from the document agent, simplifying the profile workflow. - Updated related components and validation logic to remove references to H1 results, enhancing clarity and maintainability. - Adjusted tests and debug scripts to reflect the removal of H1-related functionality, ensuring consistency across the codebase.
Co-authored-by: Cursor <cursoragent@cursor.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| # Canonical profile is at package root; drop nested duplicate. | ||
| try: | ||
| (out_dir / "_doc_agent" / "anatomy_map.json").unlink() | ||
| except FileNotFoundError: |
| write_debug_json(profile_path, anchored.to_dict()) | ||
| try: | ||
| (out_dir / "_doc_agent" / "anatomy_map.json").unlink() | ||
| except FileNotFoundError: |
| os.environ.setdefault("S3_SECRET_ACCESS_KEY", "test") | ||
| os.environ.setdefault("S3_TEMP_PATH", "/tmp") | ||
|
|
||
| import app.services.document_agent.tools as _tools # noqa: F401 |
- Introduced functionality to adopt PDF bookmarks as TOC anchors when validation passes, improving the accuracy of document profiling. - Added `run_outline_anchoring` method to handle anchoring based on the adopted outline, enhancing the overall TOC extraction pipeline. - Updated `TocResult` to include a new method for deriving the profile source based on the anchoring method. - Refactored related components to support the new outline anchoring logic and ensure consistency across the document agent. - Removed obsolete contract tests for `probe_outline` and updated tool registry to include `judge_toc_source` for better integration.
…oring - Consolidated the TOC extraction process by iterating over tools in a loop, improving code clarity and maintainability. - Removed the obsolete `run_outline_anchoring` method and integrated its logic into the main TOC anchoring flow. - Introduced a new mechanism to handle PDF outline roots, allowing for conditional anchoring based on outline validation. - Updated the `ProfileBlackboard` to include `pdf_outline_roots`, facilitating better management of outline data. - Enhanced error handling in `probe_outline` to ensure robustness during PDF processing. - Adjusted related tests to reflect changes in the TOC anchoring logic and validate the new outline handling approach.
…choring logic - Introduced `_first_sibling_null_parent_scan_start` function to determine the left edge for first-at-level null parents, improving the anchoring process. - Updated `locate_null_page_parent_overrides` to utilize the new scanning logic, enhancing accuracy in locating null-page parents with and without left siblings. - Refactored TOC anchoring logic to better handle cases with previous same-level siblings, ensuring robust parent locating strategies. - Enhanced documentation and comments for clarity on the new scanning approach and its implications for TOC structure handling. - Added tests to validate the new scanning behavior and ensure correct functionality across various scenarios.
- Removed budget limits for PROFILE token usage, allowing for record-only tracking via BudgetTracker. - Updated BudgetTracker to support a new mode where limits are disabled, enhancing flexibility in budget management. - Refactored TOC extraction logic to accommodate changes in budget handling, ensuring accurate token accounting during processing. - Enhanced tests to validate the new budget behavior and confirm correct functionality across various scenarios.
- Eliminated the BudgetTracker from various document agent services, simplifying the budget management process. - Removed budget-related parameters and logic from functions and classes, ensuring a cleaner codebase. - Updated environment variable examples to reflect the absence of budget tracking. - Adjusted tests and debug scripts to remove references to budget tracking, maintaining consistency across the codebase.
…rt after TOC end - Adjusted the logic in `run_calibration_phase1` to set the scan start page to the page immediately following the end of the TOC range. - Implemented error handling for cases where the TOC end is unknown or the scan start exceeds the total page count. - Updated related tests to reflect the new scan start behavior, ensuring accurate assertions for page calls in various scenarios.
- Introduced a new function `_anchor_one_pending_toc` to streamline the anchoring of individual TOCs, improving clarity and maintainability. - Updated `_anchor_pending_tocs` to utilize concurrency for processing multiple TOCs, enhancing performance. - Adjusted constants for concurrency limits in both TOC anchoring and extraction processes, ensuring better resource management. - Refactored related logic to improve error handling and logging during TOC calibration, providing clearer insights into processing outcomes.
Keep PR cleanup of calibration_inspect while adopting unlink(missing_ok=True) from main. Co-authored-by: Cursor <cursoragent@cursor.com>
- Updated `select_global_toc_hierarchies` to clarify the distinction between primary and pending TOC hierarchies. - Simplified the selection logic by removing unnecessary variables and conditions, enhancing readability and maintainability. - Introduced new functions to handle contiguous TOC prefixes and conditions for expanding TOC windows, improving extraction accuracy. - Added tests to validate the new TOC handling behavior, ensuring robustness in various scenarios.
- Removed `markitdown`, `pptx2md`, `magika`, and `defusedxml` from `pyproject.toml`, `requirements.txt`, and `uv.lock` as they are no longer needed. - Updated dependency paths in `requirements.txt` for consistency. - Simplified the `aggregate_stats.py`, `orchestrator.py`, and other files by removing unused variables and signals, improving code clarity. - Refactored the `parse_pptx` function to eliminate the now-removed `pptx2md` integration, streamlining the parsing process.
…rvices - Changed references from "agent" to "profile" in the document agent services for consistency. - Enhanced logging messages to clarify the context of document profile operations. - Removed unused parameters related to page texts in various functions, streamlining the code. - Updated comments to reflect the new terminology and improve clarity in the codebase.
- Eliminated the `confidence` attribute from various classes and functions in the document agent services to streamline data handling. - Updated related logic and tests to reflect the removal of confidence values, ensuring consistency across the codebase. - Simplified evidence handling in TOC and page classification processes by removing unnecessary confidence parameters.
…ling - Updated `_finalize_shard_plan` to abort on validation failure instead of falling back to a single-shard plan, improving clarity in error handling. - Refactored TOC extraction logic to raise exceptions on failure, ensuring that errors are properly propagated and handled. - Revised related tests to validate the new behavior of shard plan finalization and TOC extraction, ensuring robustness in error scenarios. - Improved logging messages for better traceability during TOC profiling and validation processes.
Keep lazy __getattr__ re-exports without listing undefined names in __all__. Co-authored-by: Cursor <cursoragent@cursor.com>
Importing layout_parser pulls shared settings; match other worker contracts. Co-authored-by: Cursor <cursoragent@cursor.com>
Contract fixtures clear app.* from sys.modules, leaving collection-time imports as zombies; patch verify by dotted path and rebind each test. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
toc_page_offsetin favor of hierarchy-based TOC handling and structure anchoring.body_boundary.extract_level1_titlesby skipping missing TOC levels beforeint().Test plan
make check(ruff + pyright; 6 existing calibration__all__lazy-export warnings remain)Made with Cursor