qdat_bench

Bench marking Some of Tajweed Rules to help creating New AI models to assist Muslim to learn reciting the Holy Quran using AI

Usage

Evaluation

Run standard evaluation:

uv run qdat-bench-eval

Run evaluation with bootstrap analysis (10,000 iterations):

uv run qdat-bench-eval --bootstrap

Options:

• --transcription-file: Path to predictions file (default: ./assets/muaalem-transcripts/muaalem-model-v3_2_predictions.jsonl)
• --save-dir: Directory to save results (default: ./assets/results)
• --n-bootstrap: Number of bootstrap iterations (default: 10000)
• --seed: Random seed for reproducibility (default: 42)

Plotting

Generate violin plots from bootstrap samples:

uv run python plot_stats.py --bootstrap-samples assets/results/result_muaalem-model-v3_2_predictions_bootstrap_avg_samples.json --plot-type bootstrap_violin

Generate dataset statistics:

uv run python plot_stats.py --plot-type dataset_stats

Generate all plots:

uv run python plot_stats.py --plot-type all

Options:

• --bootstrap-samples: Path to bootstrap samples JSON file (required for violin plots)
• --save-dir: Directory to save plots (default: assets)
• --plot-type: Type of plot to generate: bootstrap_violin, dataset_stats, or all (default: all)

Python API

The eval_qdat_bench function provides a programmatic interface for evaluating transcription models against the qdat_bench dataset:

from datasets import Dataset
from quran_muaalem.modeling.multi_level_tokenizer import MultiLevelTokenizer
from qdat_bench.eval_results import eval_qdat_bench

pred_ds = Dataset.from_json("path/to/predictions.jsonl")
tokenizer = MultiLevelTokenizer("obadx/muaalem-model-v3_2")

metrics = eval_qdat_bench(
    pred_trans_ds=pred_ds,
    multi_level_tokenizer=tokenizer,
    qdat_bench_name_or_path="obadx/qdat_bench",
    bootstrap=True,
)

print(metrics["speech_metrics"])
print(metrics["qdat_metrics"])
print(metrics["qdat_avg_metrics"])

Note: MultiLevelTokenizer requires the muaalem optional dependency. Install it with: uv sync --group muaalem

Parameters:

• pred_trans_ds (Dataset) — Model predictions (see format below)
• multi_level_tokenizer (MultiLevelTokenizer) — A MultiLevelTokenizer instance from quran_muaalem (required)
• qdat_bench_name_or_path (str) — Name or path to the qdat_bench dataset (default: "obadx/qdat_bench")
• bootstrap (bool) — Whether to run bootstrap analysis (default: False)
• n_bootstrap (int) — Number of bootstrap iterations (default: 10000)
• seed (int) — Random seed (default: 42)

Returns: dict with keys speech_metrics, qdat_metrics, qdat_avg_metrics, and optionally bootstrapped_speech_metrics, bootstrapped_qdat_metrics, bootstrapped_avg_metrics, bootstrap_avg_samples.

Predictions Dataset Format

Each sample in pred_trans_ds must contain the following fields:

Field	Type	Description
id	str	Unique identifier used to align predictions with ground truth
levels_labels	dict[str, list[int]]	Predicted token IDs per level after CTC decoding
level_to_scripts	dict	String representations per level; "phonemes" must be a single concatenated string

levels_labels maps each level name to a list of integer token IDs. The expected level names come from the tokenizer vocabulary (e.g. phonemes, ghonna, hams_or_jahr, qalqla, etc.).

level_to_scripts maps each level name to a list of script labels, except for "phonemes" which is a single concatenated phonetic string. The Tajweed attribute extractors consume level_to_scripts["phonemes"].

Example structure:

{
  "id": "95dae92b",
  "levels_labels": {
    "phonemes": [21, 32, 29, 29, 23, ...],
    "ghonna": [2, 2, 2, 2, 2, ...],
    "qalqla": [2, 2, 2, 2, 2, ...]
  },
  "level_to_scripts": {
    "phonemes": "تُصْطَلَىْنَا...",
    "ghonna": ["[لا غنة]", "[لا غنة]", ...],
    "qalqla": ["[لا قلقلة]", "[لا قلقلة]", ...]
  }
}

Output Files

• result_*.json: Contains speech_metrics, qdat_metrics, qdat_avg_metrics, and their bootstrapped versions (*_mean, *_std)
• result_*_bootstrap_avg_samples.json: Bootstrap samples for violin plot generation
• bootstrap_violin_plots.png: Violin plots grouped by metric type (PER, RMSE, percentage)