docs: add comprehensive documentation for orchestrator_mapreduce module - #115
Open
groupthinking wants to merge 1 commit into
Open
docs: add comprehensive documentation for orchestrator_mapreduce module#115groupthinking wants to merge 1 commit into
groupthinking wants to merge 1 commit into
Conversation
Includes: - Architecture diagram and lifecycle flow - Full API reference for all classes and methods - Customization guide (verification gates, reducers) - Protocol compatibility table (legacy vs new-style) - 5 usage examples (minimal, fan-out, heterogeneous, custom gate, step-by-step) - Integration reference with existing modules - Configuration tuning guidance - Error handling and escalation tiers
Contributor
Summary by CodeRabbit
WalkthroughAdds ChangesHierarchicalOrchestrator Documentation
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related PRs
Suggested labels
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
coderabbitai
Bot
added
the
documentation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new, comprehensive Markdown document describing the orchestrator_mapreduce.py module (Plan → Map → Reduce flow, public API/classes, customization points, protocol calling behavior, persistence, and examples) to help developers integrate and extend the orchestrator.
Changes:
- Introduces full module documentation for
orchestrator_mapreduce.pyindocs/orchestrator_mapreduce.md. - Adds architecture/lifecycle diagrams, API reference, customization contracts, and usage examples.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+277
to
+284
| The module uses `inspect.signature()` to determine how to call each protocol's `task()` function: | ||
|
|
||
| | Protocol Signature | Behavior | | ||
| | :--- | :--- | | ||
| | `def task():` | Called with no arguments (legacy compatibility) | | ||
| | `def task(**kwargs):` | Receives the full `inputs` dict as keyword arguments | | ||
| | `def task(endpoint, timeout=30):` | Receives matching keys from `inputs` as named arguments | | ||
|
|
Comment on lines
+285
to
+297
| **Example — New-style protocol (`protocols/api_health_checker.py`):** | ||
|
|
||
| ```python | ||
| import requests | ||
|
|
||
| def task(endpoint: str, timeout: int = 10) -> dict: | ||
| """Check if an API endpoint is healthy.""" | ||
| try: | ||
| resp = requests.get(f"https://api.example.com{endpoint}", timeout=timeout) | ||
| return {"success": resp.status_code == 200, "status_code": resp.status_code} | ||
| except Exception as e: | ||
| return {"success": False, "error": str(e)} | ||
| ``` |
| ], | ||
| ) | ||
| ``` | ||
|
|
| **Behavior:** | ||
| - Acquires a semaphore slot before executing each subtask | ||
| - Applies `subtask_timeout` via `asyncio.wait_for()` | ||
| - Tracks outcomes via `track_outcome()` (non-blocking, via `run_in_executor`) |
| | `verification_gate` | `Callable` | `_default_gate` | Async function that evaluates whether a job passes verification | | ||
| | `reducer` | `Callable` | `_default_reducer` | Async function that aggregates subtask results into a single output | | ||
| | `state_file` | `str` | `<module_dir>/STATE.md` | Path to the Markdown file where job state is persisted | | ||
| | `subtask_timeout` | `float` | `300.0` | Maximum seconds a single subtask may run before being killed | |
| | :--- | :--- | :--- | | ||
| | **Retry** | Subtask failed, `attempts < max_attempts` | Mutate protocol, re-execute | | ||
| | **Escalate** | All retries exhausted, verification still failing | Log escalation, notify human (via MCP/Slack when configured) | | ||
| | **Timeout** | Subtask exceeds `subtask_timeout` seconds | Kill subtask, mark as FAILED, enter retry tier | |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/orchestrator_mapreduce.md`:
- Line 21: The fenced code block at line 21 in the orchestrator_mapreduce.md
file is missing a language identifier, which violates the MD040 markdown linting
rule. Add the language identifier `text` immediately after the opening triple
backticks on the line containing the fenced code block start marker to specify
that this is a text/ASCII art block, ensuring compliance with markdown
standards.
- Line 520: The fenced code block at line 520 in the orchestrator_mapreduce.md
file is missing a language identifier, which violates the MD040 Markdown linting
rule. Add the language identifier `text` to the opening of the fenced code block
(the line with the opening triple backticks) since this contains an ASCII
diagram. Change the opening ``` to ```text to specify that the content is plain
text.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 1cc1a810-26b5-4476-a2fa-6f3e527537c5
📒 Files selected for processing (1)
docs/orchestrator_mapreduce.md
|
|
||
| ## Architecture | ||
|
|
||
| ``` |
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add language identifier to fenced code block (MD040).
Fenced code blocks should specify a language. Since this is ASCII art, use text:
📝 Fix MD040 violation
-```
+```text📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| ``` |
🧰 Tools
🪛 markdownlint-cli2 (0.22.1)
[warning] 21-21: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/orchestrator_mapreduce.md` at line 21, The fenced code block at line 21
in the orchestrator_mapreduce.md file is missing a language identifier, which
violates the MD040 markdown linting rule. Add the language identifier `text`
immediately after the opening triple backticks on the line containing the fenced
code block start marker to specify that this is a text/ASCII art block, ensuring
compliance with markdown standards.
Source: Linters/SAST tools
|
|
||
| ## Lifecycle Diagram | ||
|
|
||
| ``` |
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add language identifier to fenced code block (MD040).
Fenced code blocks should specify a language. Since this is an ASCII diagram, use text:
📝 Fix MD040 violation
-```
+```text🧰 Tools
🪛 markdownlint-cli2 (0.22.1)
[warning] 520-520: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/orchestrator_mapreduce.md` at line 520, The fenced code block at line
520 in the orchestrator_mapreduce.md file is missing a language identifier,
which violates the MD040 Markdown linting rule. Add the language identifier
`text` to the opening of the fenced code block (the line with the opening triple
backticks) since this contains an ASCII diagram. Change the opening ``` to
```text to specify that the content is plain text.
Source: Linters/SAST tools
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds full documentation for the
orchestrator_mapreduce.pymodule atdocs/orchestrator_mapreduce.md.Contents
TaskStatus,SubTask,OrchestratedJob,HierarchicalOrchestrator)