diff --git a/README.md b/README.md index 7868afc..128a02f 100644 --- a/README.md +++ b/README.md @@ -131,6 +131,7 @@ Explore our comprehensive Jupyter notebooks in the [`notebooks/`](./notebooks) d | [**Unstructured Data Classification**](https://github.com/IBM/READI/blob/main/notebooks/example-unstructured-data-classification.ipynb) | General overview of READI API for free-text processing | | [**Structured Data Classification**](https://github.com/IBM/READI/blob/main/notebooks/example-structured-data-classification.ipynb) | Working with tabular and structured datasets | | [**Unstructured Data Masking**](https://github.com/IBM/READI/blob/main/notebooks/example-unstructured-masking.ipynb) | Applying masking actions (redaction, tagging, hash) after PII classification | +| [**Structured Data Masking**](https://github.com/IBM/READI/blob/main/notebooks/example-structured-data-masking.ipynb) | Column classification followed by per-type masking of tabular data | --- diff --git a/notebooks/example-structured-data-masking.ipynb b/notebooks/example-structured-data-masking.ipynb new file mode 100644 index 0000000..7f85e51 --- /dev/null +++ b/notebooks/example-structured-data-masking.ipynb @@ -0,0 +1,1311 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Masking of Structured Data\n", + "\n", + "This notebook demonstrates how to apply **masking actions** to columns of a tabular (structured) dataset.\n", + "\n", + "The workflow has two steps:\n", + "\n", + "1. **Classify** — automatically identify which columns contain sensitive data (PII/PHI) using `DatasetClassification`.\n", + "2. **Mask** — apply a per-type masking action to every value in the identified columns using functions from `risk_assessment.masking.actions`.\n", + "\n", + "The available masking actions all share the signature `(entity_type: str, entity_text: str) -> str`:\n", + "\n", + "| Action | Description |\n", + "|---|---|\n", + "| `tagging_factory()` | Stable sequential label per unique value: `EMAIL-1`, `EMAIL-2`, … |\n", + "| `tagging_with_hash` | Deterministic but non-reversible hash label: `EMAIL-a3f2c` |\n", + "| `redact_factory()` | Fixed-width placeholder: `XXX` |\n", + "| `redact_size_preserving` | Replaces every character with `X`, preserving length |\n", + "| `format_preserving_redact` | Replaces alphanumeric chars with `X`, keeps separators |\n", + "| `random_from_series_factory(pool)` | Draws a random replacement value from a `pandas.Series` |\n", + "| `no_action` | Leaves the value unchanged |" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1. Load the dataset\n", + "\n", + "We use the synthetic healthcare dataset included in this repository. It contains typical demographic and clinical columns." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Shape: (32440, 11)\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
patient_idnamesurnameemailyobzip_codegenderethnicityreligionmarital_statusicd_code
0P00001SophiaRosuRosSophi@hotmail.co.uk197733917MaleWhiteRoman CatholicNever-married401.9
1P00002KennyBognerKe@gmail.com196632526MaleWhiteBaptistMarried-civ-spouse244.9
2P00003LawrenceMoneypennyMoneypLawrence@live.com197832811MaleWhiteNaNDivorced530.81
3P00004KatelynMartsolfMartKa@gmail.com196334453MaleBlackUnknownMarried-civ-spouse250.00
4P00005KolbyMcglonMcKolby@gmail.com198833596FemaleBlackUnknownMarried-civ-spouse401.9
\n", + "
" + ], + "text/plain": [ + " patient_id name surname email yob zip_code \\\n", + "0 P00001 Sophia Rosu RosSophi@hotmail.co.uk 1977 33917 \n", + "1 P00002 Kenny Bogner Ke@gmail.com 1966 32526 \n", + "2 P00003 Lawrence Moneypenny MoneypLawrence@live.com 1978 32811 \n", + "3 P00004 Katelyn Martsolf MartKa@gmail.com 1963 34453 \n", + "4 P00005 Kolby Mcglon McKolby@gmail.com 1988 33596 \n", + "\n", + " gender ethnicity religion marital_status icd_code \n", + "0 Male White Roman Catholic Never-married 401.9 \n", + "1 Male White Baptist Married-civ-spouse 244.9 \n", + "2 Male White NaN Divorced 530.81 \n", + "3 Male Black Unknown Married-civ-spouse 250.00 \n", + "4 Female Black Unknown Married-civ-spouse 401.9 " + ] + }, + "execution_count": 1, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "import pandas as pd\n", + "\n", + "df = pd.read_csv(\n", + " \"./healthcare-dataset.csv\",\n", + " header=None,\n", + " names=[\n", + " \"patient_id\",\n", + " \"name\",\n", + " \"surname\",\n", + " \"email\",\n", + " \"yob\",\n", + " \"zip_code\",\n", + " \"gender\",\n", + " \"ethnicity\",\n", + " \"religion\",\n", + " \"marital_status\",\n", + " \"icd_code\",\n", + " ],\n", + ")\n", + "\n", + "print(f\"Shape: {df.shape}\")\n", + "df.head()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2. Classify columns\n", + "\n", + "Before masking we need to know *what type of sensitive data* each column holds.\n", + "`DatasetClassification` scans every value in every column against a list of identifiers and returns the best-matching type per column.\n", + "\n", + "We configure the classifier with the identifiers that are relevant to this dataset." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Best type per column:\n", + " patient_id -> UNKNOWN\n", + " name -> Name\n", + " surname -> Surname\n", + " email -> Email\n", + " yob -> YearOfBirth\n", + " zip_code -> ZipCode\n", + " gender -> Gender\n", + " ethnicity -> Etnicity\n", + " religion -> Name\n", + " marital_status -> MaritalStatus\n", + " icd_code -> ICDv9\n" + ] + } + ], + "source": [ + "from risk_assessment.classification import DatasetClassification, DatasetClassificationConfiguration\n", + "from risk_assessment.classification.identifiers import (\n", + " SSN,\n", + " DateTime,\n", + " Email,\n", + " Etnicity,\n", + " Gender,\n", + " ICDv9,\n", + " MaritalStatus,\n", + " Name,\n", + " Religion,\n", + " Surname,\n", + " YearOfBirth,\n", + " ZipCode,\n", + ")\n", + "\n", + "configuration = DatasetClassificationConfiguration(\n", + " identifiers=[\n", + " DateTime(),\n", + " Email(),\n", + " Etnicity(),\n", + " Gender(),\n", + " ICDv9(),\n", + " MaritalStatus(),\n", + " Name(),\n", + " Religion(),\n", + " SSN(),\n", + " Surname(),\n", + " YearOfBirth(),\n", + " ZipCode(),\n", + " ]\n", + ")\n", + "\n", + "report = DatasetClassification(configuration).classify(df)\n", + "\n", + "print(\"Best type per column:\")\n", + "for col, best_type in report.best_types.items():\n", + " print(f\" {col:>15s} -> {best_type}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `reports` dictionary contains the full detection frequencies (ratio of matching values) per column, which is useful for debugging ambiguous columns." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " name: Name (100%), Surname (57%), Religion (0%)\n", + " surname: Surname (100%), Name (3%)\n", + " email: Email (98%)\n", + " yob: YearOfBirth (100%)\n", + " zip_code: ZipCode (100%)\n", + " gender: Gender (100%), Name (67%)\n", + " ethnicity: Etnicity (100%), Surname (96%), Religion (1%)\n", + " religion: Name (75%), Religion (69%), Surname (39%)\n", + " marital_status: MaritalStatus (100%)\n", + " icd_code: ICDv9 (100%)\n" + ] + } + ], + "source": [ + "# Show the top-scoring types for each column (excluding UNKNOWN)\n", + "for col, freq_map in report.reports.items():\n", + " top = sorted(\n", + " [(t, f) for t, f in freq_map.items() if t != \"UNKNOWN\"],\n", + " key=lambda x: x[1],\n", + " reverse=True,\n", + " )[:3]\n", + " if top:\n", + " top_str = \", \".join(f\"{t} ({f:.0%})\" for t, f in top)\n", + " print(f\" {col:>15s}: {top_str}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 3. Define a masking policy\n", + "\n", + "A **masking policy** is a dictionary mapping a column type (as returned by the classifier) to a masking action callable.\n", + "\n", + "Here we choose an action appropriate for each sensitive type:\n", + "\n", + "| Type | Action | Rationale |\n", + "|---|---|---|\n", + "| `Name` / `Surname` | `random_from_series_factory` | Replace with a random name from a synthetic pool |\n", + "| `Email` | `tagging_factory()` | Consistent pseudonym per unique address |\n", + "| `YearOfBirth` | `no_action` | Retain for demographic analysis |\n", + "| `ZipCode` | `format_preserving_redact` | Preserve structure, redact digits |\n", + "| `Gender` / `Etnicity` / `Religion` / `MaritalStatus` | `no_action` | Quasi-identifiers kept for analysis |\n", + "| `ICDv9` | `no_action` | Sensitive but required for clinical use |\n", + "| Everything else | `redact_factory()` | Default: fixed-width redaction |" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Policy defined for types: ['Name', 'Surname', 'Email', 'YearOfBirth', 'ZipCode', 'Gender', 'Etnicity', 'Religion', 'MaritalStatus', 'ICDv9']\n" + ] + } + ], + "source": [ + "from risk_assessment.masking.actions import (\n", + " format_preserving_redact,\n", + " no_action,\n", + " random_from_series_factory,\n", + " redact_factory,\n", + " tagging_factory,\n", + " tagging_with_hash,\n", + ")\n", + "\n", + "# Synthetic name pool used for random name replacement\n", + "fake_first_names = pd.Series(\n", + " [\n", + " \"Alex\",\n", + " \"Jordan\",\n", + " \"Morgan\",\n", + " \"Taylor\",\n", + " \"Riley\",\n", + " \"Casey\",\n", + " \"Dana\",\n", + " \"Avery\",\n", + " \"Quinn\",\n", + " \"Sage\",\n", + " ]\n", + ")\n", + "fake_last_names = pd.Series(\n", + " [\n", + " \"Smith\",\n", + " \"Jones\",\n", + " \"Williams\",\n", + " \"Brown\",\n", + " \"Davis\",\n", + " \"Miller\",\n", + " \"Wilson\",\n", + " \"Moore\",\n", + " \"Taylor\",\n", + " \"Anderson\",\n", + " ]\n", + ")\n", + "\n", + "# Policy maps detected type -> masking action\n", + "POLICY = {\n", + " \"Name\": random_from_series_factory(fake_first_names),\n", + " \"Surname\": random_from_series_factory(fake_last_names),\n", + " \"Email\": tagging_factory(),\n", + " \"YearOfBirth\": no_action,\n", + " \"ZipCode\": format_preserving_redact,\n", + " \"Gender\": no_action,\n", + " \"Etnicity\": no_action,\n", + " \"Religion\": no_action,\n", + " \"MaritalStatus\": no_action,\n", + " \"ICDv9\": no_action,\n", + "}\n", + "\n", + "DEFAULT_ACTION = redact_factory()\n", + "\n", + "print(\"Policy defined for types:\", list(POLICY.keys()))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 4. Apply masking\n", + "\n", + "We apply the policy column by column. For each column we look up its detected type, find the matching action, and transform every value in the column." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
patient_idnamesurnameemailyobzip_codegenderethnicityreligionmarital_statusicd_code
0XXXRileyMillerROSSOPHI@HOTMAIL.CO.UK-11977XXXXXXXMaleWhiteSageNever-married401.9
1XXXSageAndersonKE@GMAIL.COM-11966XXXXXXXMaleWhiteAveryMarried-civ-spouse244.9
2XXXRileyMooreMONEYPLAWRENCE@LIVE.COM-11978XXXXXXXMaleWhiteRileyDivorced530.81
3XXXAlexWilsonMARTKA@GMAIL.COM-11963XXXXXXXMaleBlackSageMarried-civ-spouse250.00
4XXXMorganMillerMCKOLBY@GMAIL.COM-11988XXXXXXXFemaleBlackTaylorMarried-civ-spouse401.9
\n", + "
" + ], + "text/plain": [ + " patient_id name surname email yob zip_code \\\n", + "0 XXX Riley Miller ROSSOPHI@HOTMAIL.CO.UK-1 1977 XXXXXXX \n", + "1 XXX Sage Anderson KE@GMAIL.COM-1 1966 XXXXXXX \n", + "2 XXX Riley Moore MONEYPLAWRENCE@LIVE.COM-1 1978 XXXXXXX \n", + "3 XXX Alex Wilson MARTKA@GMAIL.COM-1 1963 XXXXXXX \n", + "4 XXX Morgan Miller MCKOLBY@GMAIL.COM-1 1988 XXXXXXX \n", + "\n", + " gender ethnicity religion marital_status icd_code \n", + "0 Male White Sage Never-married 401.9 \n", + "1 Male White Avery Married-civ-spouse 244.9 \n", + "2 Male White Riley Divorced 530.81 \n", + "3 Male Black Sage Married-civ-spouse 250.00 \n", + "4 Female Black Taylor Married-civ-spouse 401.9 " + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from collections.abc import Callable\n", + "\n", + "\n", + "def mask_dataframe(\n", + " data: pd.DataFrame,\n", + " best_types: dict[str, str],\n", + " policy: dict[str, Callable[[str, str], str]],\n", + " default: Callable[[str, str], str] = redact_factory(),\n", + ") -> pd.DataFrame:\n", + " \"\"\"Apply masking actions to every column of a DataFrame based on its detected type.\n", + "\n", + " Columns whose detected type is ``UNKNOWN`` or not listed in the policy\n", + " fall back to ``default``.\n", + "\n", + " Args:\n", + " data: The original DataFrame (not modified in-place).\n", + " best_types: Mapping of column name -> detected type, as returned by\n", + " ``DatasetClassificationReport.best_types``.\n", + " policy: Mapping of detected type -> masking action callable.\n", + " default: Fallback action for unrecognised or UNKNOWN-typed columns.\n", + "\n", + " Returns:\n", + " A new DataFrame with sensitive columns masked.\n", + " \"\"\"\n", + " masked = data.copy()\n", + " for col in masked.columns:\n", + " col_type = best_types.get(col, \"UNKNOWN\")\n", + " action = policy.get(col_type, default)\n", + " masked[col] = masked[col].apply(lambda val, a=action, t=col_type: a(str(val), t))\n", + " return masked\n", + "\n", + "\n", + "masked_df = mask_dataframe(df, report.best_types, POLICY, default=DEFAULT_ACTION)\n", + "masked_df.head()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 5. Inspect the results\n", + "\n", + "Let's compare the original and masked values side by side for the most sensitive columns." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
email_maskedemail_originalname_maskedname_originalsurname_maskedsurname_originalzip_code_maskedzip_code_original
0ROSSOPHI@HOTMAIL.CO.UK-1RosSophi@hotmail.co.ukRileySophiaMillerRosuXXXXXXX33917
1KE@GMAIL.COM-1Ke@gmail.comSageKennyAndersonBognerXXXXXXX32526
2MONEYPLAWRENCE@LIVE.COM-1MoneypLawrence@live.comRileyLawrenceMooreMoneypennyXXXXXXX32811
3MARTKA@GMAIL.COM-1MartKa@gmail.comAlexKatelynWilsonMartsolfXXXXXXX34453
4MCKOLBY@GMAIL.COM-1McKolby@gmail.comMorganKolbyMillerMcglonXXXXXXX33596
5MINIUK@GMAIL.COM-1Miniuk@gmail.comJordanKeishaSmithMiniukXXXXXXX33484
6ROCLYNE@HOTMAIL.CO.UK-1RocLyne@hotmail.co.ukAveryLynetteAndersonRockholdXXXXXXX34681
7DUDOME@GMAIL.COM-1DuDome@gmail.comAveryDomenicBrownDumaineXXXXXXX32223
\n", + "
" + ], + "text/plain": [ + " email_masked email_original name_masked \\\n", + "0 ROSSOPHI@HOTMAIL.CO.UK-1 RosSophi@hotmail.co.uk Riley \n", + "1 KE@GMAIL.COM-1 Ke@gmail.com Sage \n", + "2 MONEYPLAWRENCE@LIVE.COM-1 MoneypLawrence@live.com Riley \n", + "3 MARTKA@GMAIL.COM-1 MartKa@gmail.com Alex \n", + "4 MCKOLBY@GMAIL.COM-1 McKolby@gmail.com Morgan \n", + "5 MINIUK@GMAIL.COM-1 Miniuk@gmail.com Jordan \n", + "6 ROCLYNE@HOTMAIL.CO.UK-1 RocLyne@hotmail.co.uk Avery \n", + "7 DUDOME@GMAIL.COM-1 DuDome@gmail.com Avery \n", + "\n", + " name_original surname_masked surname_original zip_code_masked \\\n", + "0 Sophia Miller Rosu XXXXXXX \n", + "1 Kenny Anderson Bogner XXXXXXX \n", + "2 Lawrence Moore Moneypenny XXXXXXX \n", + "3 Katelyn Wilson Martsolf XXXXXXX \n", + "4 Kolby Miller Mcglon XXXXXXX \n", + "5 Keisha Smith Miniuk XXXXXXX \n", + "6 Lynette Anderson Rockhold XXXXXXX \n", + "7 Domenic Brown Dumaine XXXXXXX \n", + "\n", + " zip_code_original \n", + "0 33917 \n", + "1 32526 \n", + "2 32811 \n", + "3 34453 \n", + "4 33596 \n", + "5 33484 \n", + "6 34681 \n", + "7 32223 " + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "sensitive_cols = [\"name\", \"surname\", \"email\", \"zip_code\"]\n", + "\n", + "comparison = pd.concat(\n", + " [\n", + " df[sensitive_cols].add_suffix(\"_original\"),\n", + " masked_df[sensitive_cols].add_suffix(\"_masked\"),\n", + " ],\n", + " axis=1,\n", + ").sort_index(axis=1)\n", + "\n", + "comparison.head(8)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Email consistency check\n", + "\n", + "`tagging_factory()` assigns the *same* label every time it sees the same value. Let's verify that the same original email always maps to the same masked label." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Unique emails in original : 29993\n", + "Unique labels after masking: 29829\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
originalmasked
872@gmail.co.uk@GMAIL.CO.UK-1
152@gmail.com@GMAIL.COM-1
1218@hotmail.co.uk@HOTMAIL.CO.UK-1
105@hotmail.com@HOTMAIL.COM-1
24@hotmail.gov@HOTMAIL.GOV-1
310@live.co.uk@LIVE.CO.UK-1
140@live.com@LIVE.COM-1
111@live.gov@LIVE.GOV-1
\n", + "
" + ], + "text/plain": [ + " original masked\n", + "872 @gmail.co.uk @GMAIL.CO.UK-1\n", + "152 @gmail.com @GMAIL.COM-1\n", + "1218 @hotmail.co.uk @HOTMAIL.CO.UK-1\n", + "105 @hotmail.com @HOTMAIL.COM-1\n", + "24 @hotmail.gov @HOTMAIL.GOV-1\n", + "310 @live.co.uk @LIVE.CO.UK-1\n", + "140 @live.com @LIVE.COM-1\n", + "111 @live.gov @LIVE.GOV-1" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "email_map = (\n", + " pd.DataFrame({\"original\": df[\"email\"], \"masked\": masked_df[\"email\"]}).drop_duplicates().sort_values(\"masked\")\n", + ")\n", + "\n", + "# Each original email should map to exactly one masked label\n", + "assert email_map.groupby(\"original\")[\"masked\"].nunique().max() == 1, (\n", + " \"Inconsistent tagging: same email mapped to different labels!\"\n", + ")\n", + "\n", + "print(f\"Unique emails in original : {df['email'].nunique()}\")\n", + "print(f\"Unique labels after masking: {masked_df['email'].nunique()}\")\n", + "email_map.head(8)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 6. Selective masking — only the sensitive columns\n", + "\n", + "Sometimes you want to retain columns whose type is `UNKNOWN` (e.g. `patient_id`) unchanged and only mask the columns that were positively identified. We can achieve this by passing `no_action` as the default." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
patient_idnamesurnameemailyobzip_codegenderethnicityreligionmarital_statusicd_code
0P00001MorganSmithROSSOPHI@HOTMAIL.CO.UK-11977XXXXXXXMaleWhiteJordanNever-married401.9
1P00002CaseyDavisKE@GMAIL.COM-11966XXXXXXXMaleWhiteCaseyMarried-civ-spouse244.9
2P00003DanaSmithMONEYPLAWRENCE@LIVE.COM-11978XXXXXXXMaleWhiteSageDivorced530.81
3P00004DanaJonesMARTKA@GMAIL.COM-11963XXXXXXXMaleBlackSageMarried-civ-spouse250.00
4P00005QuinnMillerMCKOLBY@GMAIL.COM-11988XXXXXXXFemaleBlackAveryMarried-civ-spouse401.9
\n", + "
" + ], + "text/plain": [ + " patient_id name surname email yob zip_code \\\n", + "0 P00001 Morgan Smith ROSSOPHI@HOTMAIL.CO.UK-1 1977 XXXXXXX \n", + "1 P00002 Casey Davis KE@GMAIL.COM-1 1966 XXXXXXX \n", + "2 P00003 Dana Smith MONEYPLAWRENCE@LIVE.COM-1 1978 XXXXXXX \n", + "3 P00004 Dana Jones MARTKA@GMAIL.COM-1 1963 XXXXXXX \n", + "4 P00005 Quinn Miller MCKOLBY@GMAIL.COM-1 1988 XXXXXXX \n", + "\n", + " gender ethnicity religion marital_status icd_code \n", + "0 Male White Jordan Never-married 401.9 \n", + "1 Male White Casey Married-civ-spouse 244.9 \n", + "2 Male White Sage Divorced 530.81 \n", + "3 Male Black Sage Married-civ-spouse 250.00 \n", + "4 Female Black Avery Married-civ-spouse 401.9 " + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "sensitive_only_df = mask_dataframe(\n", + " df,\n", + " report.best_types,\n", + " POLICY,\n", + " default=no_action, # keep UNKNOWN-typed columns as-is\n", + ")\n", + "\n", + "# patient_id was classified as UNKNOWN, so it must be unchanged\n", + "assert (sensitive_only_df[\"patient_id\"] == df[\"patient_id\"]).all(), \"patient_id should not have been modified!\"\n", + "\n", + "sensitive_only_df.head()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 7. Custom identifier — Patient ID\n", + "\n", + "The classifier correctly left `patient_id` as `UNKNOWN` because no built-in identifier matches the local `P` + 5-digits format. We can register a custom `RegexIdentifier` to handle it and include it in the masking policy." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Best types (with custom PatientID identifier):\n", + " patient_id -> UNKNOWN\n", + " name -> Name\n", + " surname -> Surname\n", + " email -> Email\n", + " yob -> YearOfBirth\n", + " zip_code -> ZipCode\n", + " gender -> Gender\n", + " ethnicity -> Etnicity\n", + " religion -> Name\n", + " marital_status -> MaritalStatus\n", + " icd_code -> ICDv9\n" + ] + } + ], + "source": [ + "import re\n", + "\n", + "from risk_assessment.classification.identifiers import RegexIdentifier\n", + "\n", + "patient_id_identifier = RegexIdentifier(\n", + " \"PatientID\",\n", + " [re.compile(r\"^P\\d{5}$\")],\n", + ")\n", + "\n", + "configuration_with_patient_id = DatasetClassificationConfiguration(\n", + " identifiers=[\n", + " DateTime(),\n", + " Email(),\n", + " Etnicity(),\n", + " Gender(),\n", + " ICDv9(),\n", + " MaritalStatus(),\n", + " Name(),\n", + " Religion(),\n", + " SSN(),\n", + " Surname(),\n", + " YearOfBirth(),\n", + " ZipCode(),\n", + " patient_id_identifier, # <-- custom identifier\n", + " ]\n", + ")\n", + "\n", + "report_with_patient_id = DatasetClassification(configuration_with_patient_id).classify(df)\n", + "\n", + "print(\"Best types (with custom PatientID identifier):\")\n", + "for col, best_type in report_with_patient_id.best_types.items():\n", + " print(f\" {col:>15s} -> {best_type}\")" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
original_idmasked_idoriginal_emailmasked_email
0P00001P00001RosSophi@hotmail.co.ukROSSOPHI@HOTMAIL.CO.UK-1
1P00002P00002Ke@gmail.comKE@GMAIL.COM-1
2P00003P00003MoneypLawrence@live.comMONEYPLAWRENCE@LIVE.COM-1
3P00004P00004MartKa@gmail.comMARTKA@GMAIL.COM-1
4P00005P00005McKolby@gmail.comMCKOLBY@GMAIL.COM-1
5P00006P00006Miniuk@gmail.comMINIUK@GMAIL.COM-1
6P00007P00007RocLyne@hotmail.co.ukROCLYNE@HOTMAIL.CO.UK-1
7P00008P00008DuDome@gmail.comDUDOME@GMAIL.COM-1
\n", + "
" + ], + "text/plain": [ + " original_id masked_id original_email masked_email\n", + "0 P00001 P00001 RosSophi@hotmail.co.uk ROSSOPHI@HOTMAIL.CO.UK-1\n", + "1 P00002 P00002 Ke@gmail.com KE@GMAIL.COM-1\n", + "2 P00003 P00003 MoneypLawrence@live.com MONEYPLAWRENCE@LIVE.COM-1\n", + "3 P00004 P00004 MartKa@gmail.com MARTKA@GMAIL.COM-1\n", + "4 P00005 P00005 McKolby@gmail.com MCKOLBY@GMAIL.COM-1\n", + "5 P00006 P00006 Miniuk@gmail.com MINIUK@GMAIL.COM-1\n", + "6 P00007 P00007 RocLyne@hotmail.co.uk ROCLYNE@HOTMAIL.CO.UK-1\n", + "7 P00008 P00008 DuDome@gmail.com DUDOME@GMAIL.COM-1" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Extend the policy with a hash-based action for PatientID\n", + "policy_with_patient_id = dict(POLICY)\n", + "policy_with_patient_id[\"PatientID\"] = tagging_with_hash\n", + "\n", + "masked_full_df = mask_dataframe(\n", + " df,\n", + " report_with_patient_id.best_types,\n", + " policy_with_patient_id,\n", + " default=no_action,\n", + ")\n", + "\n", + "# The patient_id column is now masked\n", + "pd.DataFrame(\n", + " {\n", + " \"original_id\": df[\"patient_id\"],\n", + " \"masked_id\": masked_full_df[\"patient_id\"],\n", + " \"original_email\": df[\"email\"],\n", + " \"masked_email\": masked_full_df[\"email\"],\n", + " }\n", + ").head(8)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 8. Summary\n", + "\n", + "| Step | API | Key point |\n", + "|---|---|---|\n", + "| **Classify** | `DatasetClassification(config).classify(df)` | Automatically detects the type of each column |\n", + "| **Define policy** | `{type: action}` dict | One masking action per detected type |\n", + "| **Mask** | `mask_dataframe(df, best_types, policy)` | Applies actions column-by-column |\n", + "| **Extend** | `RegexIdentifier(name, patterns)` | Add domain-specific identifiers for custom column types |\n", + "\n", + "The classification + masking pipeline is fully composable:\n", + "- Swap any masking action to change the privacy strategy for a particular column type.\n", + "- Add custom identifiers to handle domain-specific ID formats.\n", + "- Use `no_action` as the default to mask only positively identified columns." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": ".venv", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.12" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/src/risk_assessment/masking/actions.py b/src/risk_assessment/masking/actions.py index 711fc58..f74e080 100644 --- a/src/risk_assessment/masking/actions.py +++ b/src/risk_assessment/masking/actions.py @@ -16,6 +16,8 @@ preserving the original length. - :func:`format_preserving_redact` — replaces alphanumeric characters with ``"X"`` while keeping punctuation and spaces, preserving the original format. +- :func:`random_from_series_factory` — replaces the entity with a random + value drawn from a provided ``pandas.Series``. - :func:`no_action` — returns the entity text unchanged (pass-through). """ @@ -23,6 +25,9 @@ from collections import defaultdict from collections.abc import Callable from hashlib import sha256 +from random import choice + +from pandas import Series class MappingStorage(ABC): @@ -148,6 +153,39 @@ def format_preserving_redact(_: str, enity_text: str) -> str: return "".join(["X" if c.isalnum() else c for c in enity_text]) +def random_from_series_factory(values: Series) -> Callable[[str, str], str]: + """Create a masking action that replaces an entity with a random value from *values*. + + A replacement is drawn uniformly at random from the provided + ``pandas.Series`` each time the action is called, so different occurrences + of the same entity text may receive different replacements. + + Args: + values: Series of candidate replacement values. Must be non-empty. + + Returns: + A ``(entity_type, entity_text) -> replacement`` callable. + + Raises: + IndexError: If *values* is empty. + + Example:: + + import pandas as pd + from risk_assessment.masking.actions import random_from_series_factory + + fake_names = pd.Series(["Alice", "Bob", "Carol", "Dave"]) + action = random_from_series_factory(fake_names) + print(action("Person", "John Doe")) # e.g. "Carol" + """ + pool = list(values) + + def _replace(_entity_type: str, _entity_text: str) -> str: + return str(choice(pool)) + + return _replace + + def no_action(value: str, _: str) -> str: """Pass-through transformation — returns the entity text unchanged. diff --git a/tests/masking/test_actions.py b/tests/masking/test_actions.py index a42759c..fd62495 100644 --- a/tests/masking/test_actions.py +++ b/tests/masking/test_actions.py @@ -1,36 +1,209 @@ +import re + +import pandas as pd +import pytest + from risk_assessment.masking.actions import ( + InMemoryMappingStorage, format_preserving_redact, + no_action, + random_from_series_factory, redact_factory, redact_size_preserving, tagging_factory, tagging_with_hash, ) +# --------------------------------------------------------------------------- +# tagging_with_hash +# --------------------------------------------------------------------------- + + +def test_tagging_with_hash_differs_from_input(): + assert tagging_with_hash("Person", "John Doe") != "John Doe" + + +def test_tagging_with_hash_format(): + result = tagging_with_hash("Email", "user@example.com") + # Format: TYPE-<5 hex chars> + assert re.fullmatch(r"EMAIL-[0-9a-f]{5}", result) + + +def test_tagging_with_hash_is_deterministic(): + assert tagging_with_hash("SSN", "123-45-6789") == tagging_with_hash("SSN", "123-45-6789") + -def test_tagging(): - assert tagging_with_hash("person", "FOO") != "FOO" +def test_tagging_with_hash_differs_for_different_inputs(): + assert tagging_with_hash("SSN", "123-45-6789") != tagging_with_hash("SSN", "999-00-0001") -def test_format_preserving_redact(): +def test_tagging_with_hash_uppercases_type(): + result = tagging_with_hash("email", "a@b.com") + assert result.startswith("EMAIL-") + + +# --------------------------------------------------------------------------- +# format_preserving_redact +# --------------------------------------------------------------------------- + + +def test_format_preserving_redact_differs_from_input(): assert format_preserving_redact("FOO", "BAR") != "BAR" + +def test_format_preserving_redact_preserves_length(): assert len(format_preserving_redact("FOO", "BAR")) == len("BAR") - assert format_preserving_redact("WHATEVER", "192.168.1.1") == "XXX.XXX.X.X" -def test_redact_size_preserving(): +def test_format_preserving_redact_ip_address(): + assert format_preserving_redact("IP", "192.168.1.1") == "XXX.XXX.X.X" + + +def test_format_preserving_redact_phone(): + assert format_preserving_redact("Phone", "+1 (800) 555-0100") == "+X (XXX) XXX-XXXX" + + +def test_format_preserving_redact_empty_string(): + assert format_preserving_redact("Any", "") == "" + + +def test_format_preserving_redact_only_punctuation(): + assert format_preserving_redact("Any", "---") == "---" + + +# --------------------------------------------------------------------------- +# redact_size_preserving +# --------------------------------------------------------------------------- + + +def test_redact_size_preserving_all_x(): assert redact_size_preserving("FOO", "THIS IS LONG") == "X" * len("THIS IS LONG") -def test_tagging_sequential(): - tagging = tagging_factory() +def test_redact_size_preserving_preserves_length(): + text = "hello world" + result = redact_size_preserving("Any", text) + assert len(result) == len(text) + assert result == "X" * len(text) + - assert tagging("foo", "BAR") == tagging("foo", "BAR") - assert tagging("fooooo", "BAR") == tagging("fooooo", "BAR") - assert tagging("foo", "BAR") != tagging("fooooo", "BAR") +def test_redact_size_preserving_empty(): + assert redact_size_preserving("Any", "") == "" -def test_redaction(): +# --------------------------------------------------------------------------- +# redact_factory +# --------------------------------------------------------------------------- + + +def test_redact_factory_default(): assert redact_factory()("", "VALUE") == "XXX" + + +def test_redact_factory_custom_size(): assert redact_factory(size=1)("", "VALUE") == "X" + assert redact_factory(size=10)("", "long text here") == "X" * 10 + + +def test_redact_factory_custom_symbol(): assert redact_factory(symbol="Y", size=5)("", "VALUE") == "YYYYY" + + +def test_redact_factory_ignores_input(): + redact = redact_factory(size=3) + assert redact("Email", "short") == redact("Person", "a much longer string") + + +# --------------------------------------------------------------------------- +# tagging_factory / InMemoryMappingStorage +# --------------------------------------------------------------------------- + + +def test_tagging_factory_same_value_same_label(): + tagging = tagging_factory(InMemoryMappingStorage()) + assert tagging("Person", "John") == tagging("Person", "John") + + +def test_tagging_factory_different_values_different_labels(): + tagging = tagging_factory(InMemoryMappingStorage()) + assert tagging("Person", "John") != tagging("Person", "Jane") + + +def test_tagging_factory_same_value_different_types_different_labels(): + tagging = tagging_factory(InMemoryMappingStorage()) + assert tagging("Person", "Smith") != tagging("Organization", "Smith") + + +def test_tagging_factory_sequential_labels(): + storage = InMemoryMappingStorage() + tagging = tagging_factory(storage) + first = tagging("Email", "a@example.com") + second = tagging("Email", "b@example.com") + assert first == "EMAIL-1" + assert second == "EMAIL-2" + + +def test_tagging_factory_label_uppercases_type(): + tagging = tagging_factory(InMemoryMappingStorage()) + label = tagging("email", "x@y.com") + assert label.startswith("EMAIL-") + + +# --------------------------------------------------------------------------- +# no_action +# --------------------------------------------------------------------------- + + +def test_no_action_returns_first_argument(): + assert no_action("Person", "ignored") == "Person" + + +def test_no_action_empty_string(): + assert no_action("", "anything") == "" + + +# --------------------------------------------------------------------------- +# random_from_series_factory +# --------------------------------------------------------------------------- + + +def test_random_from_series_returns_value_from_pool(): + pool = pd.Series(["Alice", "Bob", "Carol"]) + action = random_from_series_factory(pool) + for _ in range(50): + assert action("Person", "John Doe") in {"Alice", "Bob", "Carol"} + + +def test_random_from_series_single_value_always_returns_it(): + action = random_from_series_factory(pd.Series(["Only"])) + assert action("Person", "anyone") == "Only" + assert action("Email", "test@test.com") == "Only" + + +def test_random_from_series_returns_string(): + action = random_from_series_factory(pd.Series([1, 2, 3])) + result = action("Age", "25") + assert isinstance(result, str) + + +def test_random_from_series_ignores_entity_type_and_text(): + pool = pd.Series(["X", "Y"]) + action = random_from_series_factory(pool) + # Both type and text are ignored; only pool contents matter + r1 = action("TypeA", "foo") + r2 = action("TypeB", "bar") + assert r1 in {"X", "Y"} + assert r2 in {"X", "Y"} + + +def test_random_from_series_uses_full_pool(): + pool = pd.Series(["A", "B", "C", "D", "E"]) + action = random_from_series_factory(pool) + seen = {action("T", "v") for _ in range(500)} + assert seen == {"A", "B", "C", "D", "E"} + + +def test_random_from_series_raises_on_empty_pool(): + action = random_from_series_factory(pd.Series([], dtype=str)) + with pytest.raises(IndexError): + action("Person", "John")