Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 99 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# Contributing to READI

Thank you for your interest in contributing! This guide covers everything you need to get started.

---

## Development Setup

**Prerequisites:** Python 3.11+, [uv](https://docs.astral.sh/uv/), Git with [git-lfs](https://git-lfs.com/).

```bash
git clone https://github.com/IBM/READI.git
cd READI

# Create virtual environment and install all dependencies (including dev extras)
uv venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
uv pip install -e ".[dev]"

# Install pre-commit hooks
prek install
```

---

## Running Tests

```bash
# Run the full test suite
pytest

# Run a specific test file
pytest tests/readi/test_analyzer.py -v

# Run with coverage report
pytest --cov=risk_assessment --cov-report=html
```

---

## Linting and Formatting

The project uses [Ruff](https://docs.astral.sh/ruff/) for linting and formatting.

```bash
# Check for lint errors
ruff check src/ tests/

# Auto-fix lint errors
ruff check --fix src/ tests/

# Format code
ruff format src/ tests/
```

Pre-commit hooks run these checks automatically on every commit. To run them manually:

```bash
pre-commit run --all-files
```

---

## Code Style

- Follow existing patterns in the surrounding code.
- All public classes, methods, and module-level functions must have docstrings.
- Use Google-style docstrings with `Args:`, `Returns:`, and `Raises:` sections where applicable.
- Type hints are required on all function signatures.
- Line length limit is 120 characters (enforced by Ruff).

---

## Pull Request Process

1. **Branch** off `main` using a descriptive name, e.g. `feat/italian-fiscal-code` or `fix/email-regex`.
2. **Write tests** for any new functionality. PRs without tests for new features will not be merged.
3. **Ensure all checks pass** locally before opening a PR:
```bash
prek run --all-files
pytest
```
4. **Open a Pull Request** against `main`. Fill in the PR template, linking any related issues.
5. At least one maintainer review is required before merging.

---

## Reporting Issues

Use [GitHub Issues](https://github.com/IBM/READI/issues) to report bugs or request features. Please include:
- A minimal reproducible example for bugs.
- The Python version and OS you are using.
- The full error traceback if applicable.

---

## License

By contributing, you agree that your contributions will be licensed under the [Apache License 2.0](LICENSE).
30 changes: 14 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,12 @@ READI augments the functionalities provided by [IBM Data Privacy Toolkit](https:

### Installation

**Recommended: Using uv (10-100x faster)**
**Simplest: install from PyPI**
```bash
pip install readi-privacy
```

**Using uv (10-100x faster)**
```bash
# Install uv if you haven't already
curl -LsSf https://astral.sh/uv/install.sh | sh
Expand All @@ -46,15 +51,10 @@ uv venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate

# Install READI
uv pip install git+https://github.com/IBM/READI.git
```

**Standard Installation with pip:**
```bash
pip install git+https://github.com/IBM/READI.git
uv pip install readi-privacy
```

**Clone Repository:**
**Install from source:**
```bash
git clone https://github.com/IBM/READI.git
cd READI
Expand All @@ -74,19 +74,17 @@ For contributors and developers:

**Recommended: Using uv**
```bash
# Install in editable mode with development dependencies
uv pip install -e .
uv pip install -r requirements-dev.txt
# Install in editable mode with all development dependencies
uv pip install -e ".[dev]"

# Set up pre-commit hooks (recommended)
pre-commit install
```

**Alternative: Using pip**
```bash
# Install in editable mode with development dependencies
pip install -e .
pip install -r requirements-dev.txt
# Install in editable mode with all development dependencies
pip install -e ".[dev]"

# Set up pre-commit hooks (recommended)
pre-commit install
Expand Down Expand Up @@ -138,7 +136,7 @@ Explore our comprehensive Jupyter notebooks in the [`notebooks/`](./notebooks) d

## 📖 Documentation

For detailed documentation, API references, and advanced usage patterns, please visit our [documentation portal](https://github.com/IBM/READI/docs) *(coming soon)*.
For detailed documentation, API references, and advanced usage patterns, please refer to the [`notebooks/`](./notebooks) directory and the inline docstrings throughout the source code. A hosted documentation portal is planned for a future release.

---

Expand Down Expand Up @@ -167,7 +165,7 @@ If you use READI in academic work, please cite the most relevant publication fro
@software{readi_ibm,
title = {READI: Risk Evaluation and De-Identification},
author = {Stefano Braghin and Liubov Nedoshivina and Anisa Halimi and Naoise Holohan and Kieran Fraser},
year = {2026},
year = {2025},
url = {https://github.com/IBM/READI}
}
```
Expand Down
7 changes: 4 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[project]
name = "readi-privacy"
dynamic = ["version"]
description = "A collection of funcionality to perform data classification, data privacy risk assessment, and enforce mitigation"
description = "A collection of functionality to perform data classification, data privacy risk assessment, and enforce mitigation"
readme = "README.md"
requires-python = ">=3.11,<3.14"
authors = [
Expand All @@ -16,13 +16,14 @@ classifiers = [
"Development Status :: 4 - Beta",
# Indicate who your project is intended for
"Intended Audience :: Developers",
"Topic :: Software Development :: Build Tools",
"Intended Audience :: Science/Research",
"Topic :: Security",
"Topic :: Scientific/Engineering :: Artificial Intelligence",

# Specify the Python versions you support here.
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Programming Language :: Python :: 3.14",
]
dependencies = [
"datasets==5.0.0",
Expand Down
9 changes: 9 additions & 0 deletions src/risk_assessment/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
"""READI — Risk Evaluation and De-Identification.

Top-level package exposing the primary public API. For most use-cases,
import directly from the relevant sub-package::

from risk_assessment.readi.analyzer import READIAnalyzer
from risk_assessment.anonymization.mondrian import Mondrian, MondrianOptions
from risk_assessment.anonymization import KAnonymity
"""
112 changes: 112 additions & 0 deletions src/risk_assessment/anonymization/__init__.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,22 @@
"""Privacy constraints and base classes for data anonymization algorithms.

This module provides the foundational abstractions for dataset anonymization,
including the abstract base classes for anonymization algorithms and privacy
constraints, as well as concrete implementations of the most common privacy
models: k-Anonymity, l-Diversity (Distinct and Entropy variants), and
t-Closeness.

Typical usage::

from risk_assessment.anonymization import KAnonymity, TCloseness
from risk_assessment.anonymization.mondrian import Mondrian, MondrianOptions

constraints = [KAnonymity(k=5), TCloseness(t=0.2)]
options = MondrianOptions(privacy_constraints=constraints)
mondrian = Mondrian(options)
anonymized_df, report = mondrian.anonymize(df, column_information)
"""

import math
from abc import ABC, abstractmethod
from dataclasses import dataclass
Expand All @@ -12,35 +31,96 @@

@dataclass
class AnonymizationReport:
"""Result report produced by an anonymization algorithm.

Attributes:
anonymized: Whether the dataset was successfully anonymized.
suppression_rate: Fraction of rows suppressed (0.0–100.0). Defaults to 0.0.
generalization_levels: Per-column generalization levels applied, if available.
"""

anonymized: bool
suppression_rate: float = 0.0
generalization_levels: list[int] | None = None


class AnonymizationAlgorithm(ABC):
"""Abstract base class for anonymization algorithms.

Subclasses implement the ``anonymize`` method to transform a dataset so that
it satisfies the configured privacy constraints.
"""

@abstractmethod
def anonymize(
self, dataset: DataFrame, column_information: list[ColumnInformation]
) -> tuple[DataFrame, AnonymizationReport]:
"""Anonymize the dataset.

Args:
dataset: The input DataFrame to anonymize.
column_information: Metadata describing each column's type, class,
and associated hierarchy or range.

Returns:
A tuple of ``(anonymized_dataset, report)`` where ``report``
summarises whether anonymization succeeded and at what cost.
"""
pass


class PrivacyConstraint(ABC):
"""Abstract base class for privacy constraints.

A privacy constraint defines the condition that each equivalence class
(partition) in an anonymized dataset must satisfy.
"""

@abstractmethod
def check(self, dataset: DataFrame, column_information: list[ColumnInformation]) -> bool:
"""Check whether the partition satisfies this constraint.

Args:
dataset: A single equivalence-class partition of the full dataset.
column_information: Metadata describing each column.

Returns:
True if the constraint is satisfied, False otherwise.
"""
pass


@dataclass
class KAnonymity(PrivacyConstraint):
"""k-Anonymity privacy constraint.

A partition satisfies k-Anonymity when it contains at least *k* records,
ensuring that each individual is indistinguishable from at least k-1 others
with respect to the quasi-identifier attributes.

Attributes:
k: Minimum required equivalence-class size.
"""

k: int

def check(self, dataset: DataFrame, column_information: list[ColumnInformation]) -> bool:
"""Return True if the partition has at least k records."""
return len(dataset) >= self.k


@dataclass
class DistinctLDiversity(PrivacyConstraint):
"""Distinct l-Diversity privacy constraint.

A partition satisfies Distinct l-Diversity when every sensitive attribute
column contains at least *l* distinct values, limiting the ability to infer
a specific sensitive value for any individual.

Attributes:
l: Minimum number of distinct sensitive values required per partition.
"""

l: int # noqa: E741

def check(self, dataset: DataFrame, column_information: list[ColumnInformation]) -> bool:
Expand All @@ -59,6 +139,17 @@ def check(self, dataset: DataFrame, column_information: list[ColumnInformation])

@dataclass
class EntropyLDiversity(PrivacyConstraint):
"""Entropy l-Diversity privacy constraint.

A partition satisfies Entropy l-Diversity when, for every sensitive column,
the Shannon entropy of the value distribution is at least log(l). This is
a stronger guarantee than Distinct l-Diversity because it also requires the
values to be roughly evenly distributed.

Attributes:
l: Minimum entropy threshold expressed as log(l).
"""

l: int # noqa: E741

def check(self, dataset: DataFrame, column_information: list[ColumnInformation]) -> bool:
Expand Down Expand Up @@ -129,13 +220,34 @@ def _initialize_histograms_and_order(


class TCloseness(PrivacyConstraint):
"""t-Closeness privacy constraint.

A partition satisfies t-Closeness when the distribution of each sensitive
attribute within the partition is no further than *t* from the distribution
in the full dataset, measured using the Earth Mover's Distance (categorical)
or the ordered-distance metric (numerical).

Call :meth:`initialize` with the full dataset before using :meth:`check`.

Attributes:
t: Maximum allowed distance between the partition and global distributions.
"""

def __init__(self, t: float):
self.t = t
self.histograms: list[Series | None] | None = None
self.orders: list[Series | None] | None = None
self.total_count: int | None = None

def initialize(self, dataset: DataFrame, column_information: list[ColumnInformation]) -> None:
"""Pre-compute global histograms and value ordering from the full dataset.

Must be called once before :meth:`check` is used on individual partitions.

Args:
dataset: The complete (un-partitioned) dataset.
column_information: Metadata describing each column.
"""
(self.histograms, self.orders) = _initialize_histograms_and_order(dataset, column_information)
self.total_count = len(dataset)

Expand Down
Loading
Loading