Skip to content
BranchesTags

Repository files navigation

ReasoningBank

Self-Evolving Agent with Reasoning Memory

Implementation of the ReasoningBank framework from the research paper "ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory" (Google Cloud AI Research + UIUC, September 2025).

Overview

ReasoningBank is a memory framework that enables AI agents to learn from both successful and failed experiences, extracting generalizable reasoning strategies that improve performance over time.

Key Features

  • Closed-Loop Learning: Complete cycle of Retrieve → Act → Judge → Extract → Consolidate
  • Dual-Prompt Extraction: Learn from both successes (strategies) and failures (preventative lessons)
  • Embedding-Based Retrieval: Semantic similarity search using OpenAI or Google embeddings
  • Test-Time Scaling (MaTTS): Parallel and sequential scaling for improved performance
  • ReAct Format: Reasoning + Acting loop for transparent agent execution
  • Persistent Memory: JSON-based storage with import/export capabilities

Implementation Status

Component Status Notes
Core Agent ✅ Complete Full closed-loop implementation
Gap 21 (Streaming) ✅ Validated Test-time constraint enforcement
Gap 22 (Memory Growth) Not Validated
Gap 24 (Dual Learning) ✅ Validated Learn from successes AND failures
MaTTS Parallel ✅ Complete k-trajectory sampling
MaTTS Sequential ✅ Complete Progressive refinement

Architecture

System Flow Diagram

┌─────────────────────────────────────────────────────────────────┐
│                     ReasoningBank Agent                        │
│                                                                 │
│  Query ──► 1.RETRIEVE ──► 2.ACT ──► 3.JUDGE                   │
│               ▲                          │                      │
│               │                          ▼                      │
│           5.CONSOLIDATE ◄── 4.EXTRACT ◄─┘                      │
│               │                                                 │
│               ▼                                                 │
│         Memory Bank (JSON)                                     │
└─────────────────────────────────────────────────────────────────┘

Closed-Loop Learning Cycle:
1. RETRIEVE: Find relevant past experiences using embeddings
2. ACT:      Execute task with memory-augmented ReAct prompts
3. JUDGE:    Determine success/failure (binary classification)
4. EXTRACT:  Mine strategies (success) or lessons (failure)
5. CONSOLIDATE: Add to persistent memory bank

MaTTS (Memory-Aware Test-Time Scaling)

PARALLEL (Breadth):          SEQUENTIAL (Depth):
    Query                         Query
   ╱  │  ╲                         │
  A1  A2  A3                      A1 ──► M1
   ╲  │  ╱                         │
   Best Result                    A2 ──► M1+M2
                                   │
                                  A3 ──► M1+M2+M3

📖 ARCHITECTURE.md - Comprehensive architecture documentation

Installation

Prerequisites

  • Python 3.9+ (recommended: 3.10 or 3.11)
  • At least one LLM API key (Anthropic, Google, or OpenAI)
  • 4GB+ RAM, 5GB+ disk space

Quick Install

# Clone the repository
git clone https://github.com/[YOUR-USERNAME]/ReasoningBank.git
cd ReasoningBank

# Create virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install package with dependencies
pip install -e .  # Basic installation
# OR
pip install -e ".[test]"  # Include testing tools
# OR
pip install -e ".[dev]"  # Include testing + development tools

# Create environment configuration
cp .env.example .env
# Edit .env and add your API key(s) - see Configuration section

Configuration

Environment Setup

Create a .env file with your API keys:

# Minimum requirement: ONE of these API keys
ANTHROPIC_API_KEY=sk-ant-api03-your-key-here
# OR
GOOGLE_API_KEY=AIza-your-key-here
# OR
OPENAI_API_KEY=sk-your-key-here

# Optional: Configure provider (defaults to anthropic)
LLM_PROVIDER=anthropic  # or google, openai
LLM_MODEL=claude-3-5-sonnet-20241022

# Optional: Memory storage location
MEMORY_BANK_PATH=./data/memory_bank.json

Configuration Options

Paper Replication (matches original research):

from reasoningbank import get_config_for_paper_replication
config = get_config_for_paper_replication()  # Uses Gemini

Claude Optimized (recommended):

from reasoningbank import get_config_for_claude
config = get_config_for_claude()  # Uses Claude-3.5

Custom Configuration:

from reasoningbank import ReasoningBankConfig

config = ReasoningBankConfig(
    llm_provider="anthropic",
    llm_model="claude-3-5-sonnet-20241022",
    llm_api_key="your-key",
    agent_temperature=0.7,
    judge_temperature=0.0,
    extractor_temperature=1.0,
    extract_from_failures=True  #Learn from failures too
)

Quick Start

Basic Usage

import os
from reasoningbank import ReasoningBankAgent, get_config_for_claude

# Set API key (or use .env file)
os.environ["ANTHROPIC_API_KEY"] = "your-key-here"

# Create agent
config = get_config_for_claude()
agent = ReasoningBankAgent(config)

# Execute task with learning
result = agent.run("What is 25 * 4 + 15?")

print(f"Success: {result.success}")
print(f"Output: {result.model_output}")
print(f"Memories extracted: {len(result.memory_items)}")

# Check memory bank
stats = agent.get_statistics()
print(f"Total memories: {stats['total_entries']}")
print(f"Success rate: {stats['success_rate']:.1%}")

MaTTS Parallel Scaling

from reasoningbank import run_matts_parallel, get_config_for_matts_parallel

config = get_config_for_matts_parallel(k=3)

result = run_matts_parallel(
    query="Design an efficient algorithm to find the median of two sorted arrays.",
    config=config,
    k=3  # Sample 3 trajectories in parallel
)

print(f"Best trajectory: {result.best_trajectory.success}")
print(f"Aggregated memories: {len(result.aggregated_memories)}")

MaTTS Sequential Refinement

from reasoningbank import run_matts_sequential, get_config_for_matts_sequential

config = get_config_for_matts_sequential(k=3)

result = run_matts_sequential(
    query="Write a function to reverse a linked list in-place.",
    config=config,
    k=3  # 3 refinement iterations
)

print(f"Refinements: {len(result.all_trajectories)}")
print(f"Final success: {result.best_trajectory.success}")

Testing

Running Tests

# Run all tests (uses mocked LLM responses, no API costs)
python tests/run_all_tests.py

# Run specific test categories
python -m pytest tests/unit/          # Unit tests only
python -m pytest tests/integration/   # Integration tests
python -m pytest tests/e2e/           # End-to-end tests

# Run with coverage report
python -m pytest --cov=reasoningbank --cov-report=html

# Open coverage report
open htmlcov/index.html  # Mac/Linux
# OR
start htmlcov/index.html  # Windows

Gap Validation Tests

These tests validate key paper claims:

# Gap 21: Streaming Constraint (no future information leakage)
python -m pytest tests/e2e/test_streaming_constraint.py -v

# Gap 22: Memory Growth (WARNING: costs ~$0.60 for quick test)
python -m pytest tests/stress/test_memory_growth_long_term.py::TestMemoryGrowthLongTerm::test_memory_bank_grows_linearly_quick

# Gap 24: Success+Failure Learning (core innovation)
python -m pytest tests/ablation/test_success_and_failure_extraction.py -v

Cost Warning ⚠️

Some tests make real API calls and cost money:

  • Gap 22 quick test: ~$0.60
  • Gap 22 full test: ~$6-8
  • Gap 22 stress test: ~$20-25

Always run quick validation first!

Project Structure

ReasoningBank/
├── reasoningbank/              # Main package
│   ├── __init__.py            # Package exports
│   ├── agent.py               # Core ReasoningBankAgent
│   ├── config.py              # Configuration management
│   ├── models.py              # Data models
│   ├── judge.py               # Success/failure evaluation
│   ├── extractor.py           # Memory extraction (dual-prompt)
│   ├── retriever.py           # Embedding-based retrieval
│   ├── retriever_optimized.py # Performance optimizations
│   ├── consolidator.py        # Memory bank management
│   └── matts/                 # Test-time scaling
│       ├── parallel.py        # Parallel sampling (breadth)
│       └── sequential.py      # Sequential refinement (depth)
│
├── tests/                      # Comprehensive test suite
│   ├── unit/                  # Component tests
│   ├── integration/           # Component interaction tests
│   ├── e2e/                   # End-to-end validation
│   ├── ablation/              # Core innovation tests
│   ├── stress/                # Performance and scale tests
│   └── fixtures/              # Test data and mocks
│
├── examples/                   # Usage examples
│   ├── basic_usage.py         # Simple agent usage
│   ├── matts_parallel_example.py    # Parallel scaling demo
│   └── matts_sequential_example.py  # Sequential refinement demo
│
├── data/                       # Runtime data (git-ignored)
│   ├── memory_bank.json       # Persistent memory storage
│   └── embeddings.json        # Cached embeddings
│
└── docs/                       # Documentation
    └── archive/                # Development docs

Troubleshooting

Common Issues

Import Error: No module named 'reasoningbank'

# Solution: Install in development mode
pip install -e .

API Key Not Found

# Solution: Create .env file
cp .env.example .env
# Edit .env and add your API key

Rate Limit Errors

# Solution: Use optimized retriever with retry logic
from reasoningbank.retriever_optimized import OptimizedMemoryRetriever

Memory Growth Concerns

  • The system uses "simple addition" (no deduplication) by design
  • This is intentional per the paper's approach
  • For production with >1000 memories, consider vector database

Performance Tips

  1. Enable Embedding Cache: Reduces API calls by 100x
  2. Use Batch Operations: Process multiple queries together
  3. Monitor Memory Size: Check stats regularly
  4. Use MaTTS Wisely: Parallel for exploration, Sequential for refinement

Documentation

📖 Core Documentation:

📊 Implementation Reports:

API Reference

Core Components

# Agent
agent = ReasoningBankAgent(config, environment=None)
result = agent.run(query, max_steps=30, enable_memory_injection=True)

# Memory Operations
success = judge_trajectory_success(query, trajectory, final_state, output)
memories = extract_memories(query, trajectory, final_state, output, success)
relevant = retrieve_memories(query, memory_bank, k=1)

# MaTTS
result = run_matts_parallel(query, config, k=3)
result = run_matts_sequential(query, config, k=3)

Performance

Paper Results

  • WebArena: 17% relative improvement (22.1% → 25.8%)
  • Mind2Web: 15% relative improvement (31.5% → 36.1%)
  • SWE-Bench: 43% relative improvement (7.0% → 10.0%)

MaTTS Improvements

  • Parallel (k=3): Up to 30% additional improvement
  • Sequential (k=3): Up to 50% improvement on complex tasks

Implementation Performance

  • Memory Retrieval: <5s for 1000+ memories
  • Embedding Cache: 100x reduction in API calls
  • Memory Growth: Linear (by design, no deduplication)

Contributing

Contributions welcome! Please ensure:

  1. All tests pass: python tests/run_all_tests.py
  2. Coverage maintained: python -m pytest --cov=reasoningbank
  3. Code formatted: black reasoningbank/
  4. Type hints included
  5. Docstrings for public APIs

License

MIT License - See LICENSE file

Citation

@article{reasoningbank2025,
  title={ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory},
  author={Google Cloud AI Research and UIUC},
  year={2025},
  month={September}
}

Acknowledgments

Based on the research paper by Google Cloud AI Research and UIUC. This implementation faithfully reproduces the paper's methodology while providing a production-ready codebase.

About

ReasoningBank Implementation based on Google Paper https://arxiv.org/abs/2509.25140

Resources

Readme
MIT license
Activity

Stars

17 stars

Watchers

0 watching

Forks

2 forks
Report repository

Releases

Packages

Contributors

Languages