GEMMA - Gerbang Evaluasi dan Monitoring Mitigasi Bencana

Multi-task NLP pipeline for Indonesian disaster tweets using IndoBERTTweet and ONNX Runtime.

Guide For Usage

1. Download the models folder Gemma-Models and add it in the repo
2. Folder Train and Test is in
3. Notebook of Train and Inference

Run the training on kaggle

1. Donwload the raw data zip
2. Download the gemma_train.ipynb notebook
3. Import the notebook and Upload the dataset
4. Change the input path accordingly
5. Add hugging face token to read repository

Architecture

GEMMA is built as a two-stage system:

1. Training pipeline (domain adaptation + multi-task fine-tuning + export)
2. Offline inference pipeline (privacy-first preprocessing + dual-head ONNX inference)

System Overview

Training flow
Raw token-level SRL labels
    -> dataset reconstruction (tweet-level seq labels + token-level NER)
    -> text_id-aware split (zero leakage)
    -> TAPT (masked language modeling on disaster corpus)
    -> multi-task fine-tuning (shared backbone + 2 heads)
    -> threshold optimization per label
    -> ONNX export + quantization + label_maps.json

Inference flow
Raw tweet
    -> PII censor (NIK/phone/email/bank)
    -> paper-aligned normalization
    -> strict filter gate (optional)
    -> tokenizer
    -> ONNX runtime (seq head + NER head)
    -> urgency scoring + entity decoding
    -> structured JSON output

Model Topology

                                        Shared Backbone
                 IndoBERTTweet (TAPT-adapted, ~110M params)
                                                         |
                                 ---------------------------
                                 |                         |
                Head 1: Sequence            Head 2: Token/Ner
             Classification (12)          Classification (17)
                                 |                         |
        5 event + 6 assistance +      BIO tags for place/street/
            1 is_urgent signal           time/org/event/argument

Model parameter budget is approximately 111M, which satisfies the <= 4B SLM constraint.

Training Architecture

Primary implementation: src/notebooks/gemma_training.py

1. Data reconstruction

• Rebuild tweet text by grouping token-level rows by text_id.
• Build Head 1 multi-label targets: 5 event labels + 6 assistance labels + is_urgent.
• Build Head 2 token labels (BIO, 17 tags) with argument-to-NER routing.

2. Split strategy and leakage control

• Stratified split is performed on tweet IDs (not per token).
• Train/test IDs are asserted disjoint to enforce zero leakage.
• Split CSVs are written to data/processed/train_data and data/processed/test_data.

3. TAPT (Task-Adaptive Pretraining)

• Corpus is built from disaster-domain preprocessed CSV files.
• MLM pretraining runs for one epoch on IndoBERTTweet to adapt to disaster language.
• TAPT artifacts are stored under models/tapt.

4. Multi-task optimization

• Shared encoder with two heads:
  • Sequence head: multi-label logits
  • NER head: token logits
• Combined objective:

$$ \mathcal{L}=\lambda_{seq},\mathrm{BCEWithLogits}(y_{seq}) + \lambda_{ner},\mathrm{CrossEntropy}(y_{ner}) $$

• Class imbalance is handled with positive class weights (sequence) and class weights (NER).
• Best checkpoint is selected by weighted combined F1 and saved to models/finetuned/best_model.pt.

5. Decision threshold calibration

• Per-label optimal thresholds are derived from precision-recall curves.
• These thresholds are persisted into models/onnx/label_maps.json and used at inference.

6. Export and verification

• ONNX graph exports two outputs: seq_logits and ner_logits.
• Quantization paths:
  • FP16 model for GPU-oriented deployment
• Numerical equivalence is checked against PyTorch outputs.

Inference Architecture (100% Offline)

Primary implementation: src/notebooks/gemma_inference.py

1. Runtime initialization

• Loads local tokenizer from models/tapt.
• Loads ONNX model and label_maps.json from models/onnx.
• Chooses provider order automatically:
  • CUDAExecutionProvider + CPUExecutionProvider when CUDA is available
  • CPUExecutionProvider otherwise

2. Privacy-first preprocessing

• Every tweet passes through PII censoring before normalization.
• PII types: NIK, phone, email, bank account.
• 16-digit values are context-disambiguated (NIK vs bank account) using nearby keywords.

3. Paper-aligned normalization and gating

• Steps: lowercase, remove URL/mention/hashtag/emoji/punctuation, slang normalization, whitespace cleanup.
• Strict filter can skip inference when:
  • text has fewer than 4 words
  • text is classified as non-Indonesia context (heuristic)
• Skip behavior is explicit in output via skipped and skip_reasons.

4. Dual-head ONNX inference

• Sequence logits -> sigmoid -> thresholding via persisted per-label thresholds.
• NER logits -> argmax -> BIO span decoding.

5. Weighted urgency scoring

• Urgency score combines label probabilities with domain weights and false-event penalty:

$$ \mathrm{score}=\mathrm{clip}_{[0,100]}\left(100\left(\frac{\sum_i w_i^+ p_i}{\sum_i w_i^+} - |w_{false}|,p_{false}\right)\right) $$

• Tier mapping from label_maps.json:
  • CRITICAL >= 70
  • HIGH >= 40
  • MEDIUM >= 15
  • LOW < 15

6. Output contract

• Structured response includes:
  • original/censored/preprocessed text
  • PII findings
  • predicted crisis types and assistance needs
  • binary urgency (is_urgent) and weighted urgency (urgency_score, urgency_tier)
  • model confidence
  • extracted entities
  • preprocessing metadata and skip reasons

Core Modules

Module	Responsibility
src/core/pii_filter.py	Offline regex-based PII detection/censoring and 16-digit context disambiguation
src/core/preprocessing.py	Privacy-first text normalization, slang mapping, strict filtering heuristics
src/utils/model.py	Canonical dual-head MTL model + combined loss definition
src/utils/data_utils.py	Dataset reconstruction, stratified splitting, TAPT corpus creation
src/utils/onnx_utils.py	ONNX export, quantization, verification, model-size reporting
src/notebooks/gemma_training.py	End-to-end training notebook/script (Kaggle-oriented)
src/notebooks/gemma_inference.py	End-to-end offline inference notebook/script

Artifact Contract

Artifact	Path	Description
TAPT checkpoint	models/tapt/	Domain-adapted tokenizer + backbone weights
Best fine-tuned checkpoint	models/finetuned/best_model.pt	Best validation model state dict
ONNX FP32	models/onnx/gemma_mtl.onnx	Reference ONNX graph with dual outputs
ONNX FP16	models/onnx/gemma_mtl_fp16.onnx	Reduced-precision model for GPU inference
Label metadata	models/onnx/label_maps.json	Seq labels, per-label thresholds, urgency weights/tiers, NER maps

Hackathon Constraints

Constraint	Status
Model <= 4B params	~111M params
100% Offline inference	ONNX Runtime on localhost
PII Filter (Privacy Brain)	Regex-based: NIK, phone, email, bank account
Data segregation	Disjoint split IDs with leakage checks + split artifacts

Quick Start

# 1) Install dependencies
uv sync

# 2) Build processed splits/manifests from raw dataset
python -c "from src.utils.data_utils import run_data_pipeline; run_data_pipeline()"

# 3) Training (GPU recommended)
# Open and run: src/notebooks/gemma_training.ipynb
# (or execute src/notebooks/gemma_training.py in a notebook-compatible flow)

# 4) Offline inference
# Open and run: src/notebooks/gemma_inference.ipynb
# (or execute src/notebooks/gemma_inference.py after model artifacts exist)

Testing

Run these commands from the project root (GEMMA/).

# Run all test targets (batch + stream + stress)
uv run python src/test/run_test_suite.py --target all --stress-mode sequential --stress-texts 16

# Run only stream test (local default input: data/processed/test_data/seq_test.csv)
uv run python src/test/test_ingest_stream.py

# Run stream self-check
uv run python src/test/test_ingest_stream.py --self-check

# Run only batch self-check
uv run python src/test/run_test_suite.py --target batch

# Run only stress tests
uv run python src/test/run_test_suite.py --target stress --stress-mode all

Project Structure

GEMMA/
├── data/
│   ├── raw/                         # Source SRL and preprocessed corpora
│   └── processed/                   # Split CSVs, manifests, and inference outputs
├── models/
│   ├── tapt/                        # TAPT tokenizer + checkpoint artifacts
│   ├── finetuned/                   # best_model.pt
│   └── onnx/                        # ONNX models + label_maps.json
├── src/
│   ├── core/
│   │   ├── pii_filter.py
│   │   └── preprocessing.py
│   ├── utils/
│   │   ├── data_utils.py
│   │   ├── model.py
│   │   └── onnx_utils.py
│   ├── notebooks/
│   │   ├── gemma_training.ipynb
│   │   ├── gemma_training.py
│   │   ├── gemma_inference.ipynb
│   │   └── gemma_inference.py
│   └── test/
└── pyproject.toml

Dataset

Based on Semantic Role Labeling Datasets for Crisis Event, with approximately 4,150 annotated Indonesian tweets covering floods, fires, earthquakes, and accidents.

License

Dataset: CC BY-NC 4.0