NetworkRKSha

Closed-Loop Network Intrusion Detection & Autonomous Response System

NetworkRKSha is a real-time, event-driven network intrusion detection system (NIDS) designed to autonomously detect and mitigate stealthy cyber threats like Command & Control (C2) beaconing and DNS-over-HTTPS (DoH) tunneling.

By closing the loop between detection and mitigation, NetworkRKSha not only alerts administrators to threats but immediately contains them by dynamically updating firewall rules and collecting forensic packet captures for further analysis.

---

Operational Telemetry & Architecture Proofs

[+] View

1. Dashboard Overview & Live Ingestion

NetworkRKSha's dashboard. The frontend connects to the Python FastAPI backend via dual WebSocket channels, with all state persisted in Redis and task routing handled by RabbitMQ.

Live threat indicator transitioning from baseline Normal Flow to active Threat Analysis.

Interactive architecture pipeline mapping the 7-stage detection flow

Dual Random Forest classifier rendered from validation metrics computed during training against the CTU-13 and CIRA-CIC-DoHBrw-2020 datasets.

2. SOC-Grade Threat Feed

SOC-grade Live Threat Feed aggregating real-time alerts from both the Scapy live traffic sniffer and the offline ingestor pipeline.

Simultaneous action states in the same feed, showing the dual-mode operation logic.

3. Forensic Drill-Down & ML Feature Extraction

Forensic dive into a DNS-over-HTTPS tunneling detection (MITRE T1071.004). The engine dynamically computes a forensic narrative from live alert data including byte asymmetry ratios and flow duration.

Zero-decryption ML feature array extracted from TLS-encrypted traffic. All 29 features are computed from header metadata, with raw packets preserved as a downloadable PCAP for independent forensic verification.

4. Detection Analytics

Detection Analytics dashboard rendering real-time Recharts visualizations.

Balanced dual-model detection distribution alongside the Response Actions donut separating iptables BLOCKED actions from LOGGED advisory entries.

5. Model Evaluation & Benchmarking

Live ML evaluation pipeline processing test rows through the RabbitMQ queue. The RabbitMQ-backed async worker processes the test CSV in 50K chunks.

Model Evaluation Engine comparing our production DoH Random Forest against the published UNB CIC baseline.

CTU-13 baseline comparison showing accuracy improvements over the published BClus behavioral clustering approach.

6. Active Defense & Engine Controls

NIDS Control Panel configured for Active Defense with a 5-layer safety architecture posture. Settings are persisted in Redis hash.

Operation mode setup for anomaly confidence threshold. The IP whitelist is stored in a Redis SET preventing gateway lockout. Live Sniffer Feed displaying real-time packet statistics from Scapy.

7. Forensic Evidence Audit & Artifact Generation

Forensic evidence audit showing the dual-artifact chain: each ML classification simultaneously writes a JSON incident report and a raw PCAP capture.

Evidence chain verification proving 1:1 correspondence between JSON incident reports and PCAP flow captures via deterministic SHA-256 flow ID generation.

---

Architecture & Workflow

NetworkRKSha operates using a modern, event-driven architecture combining low-level packet sniffing, machine learning, and an asynchronous message queue.

1. Traffic Ingestion: The system captures network traffic in real-time using Scapy, processing both live interfaces and offline .pcap, .csv files for forensic replay.
2. Feature Extraction & ML Classification: Extracted flow features are evaluated against pre-trained Random Forest machine learning models to identify anomalous behaviors characteristic of C2 beacons or DoH tunnels. For offline datasets, it leverages a distributed worker architecture via RabbitMQ and Redis.
3. Autonomous Mitigation: When a threat is confirmed, the Responder module autonomously updates local iptables to block the malicious source IP (with configurable dry-run and whitelisting safety layers) to instantly stop the attack.
4. Forensic Collection: The exact packets triggering the alert are saved locally as .pcap files, providing indisputable evidence of the intrusion.
5. Real-Time Dashboard: A Next.js frontend connects to a FastAPI backend via WebSockets to stream live detection metrics, active blocks, and system health via Redis pub/sub.

Core Technologies

• Backend & ML: Python, FastAPI, Scikit-Learn, Pandas, Scapy
• Event & State Management: RabbitMQ, Redis
• Frontend: Next.js (React), WebSockets
• System/Network: iptables (Linux Firewall), Bash

Key Capabilities

• DoH Tunnel Detection: Identifies malicious DNS-over-HTTPS tunneling inside encrypted HTTPS traffic by analyzing packet size and flow byte heuristics.
• C2 Beaconing Detection: Detects persistent, low-and-slow command and control beaconing using statistical variance of flow durations and inter-arrival times.
• Autonomous Active Defense: Instantly blocks confirmed threats using dynamically managed iptables chains, complete with auto-expiring blocks and SSH safety guards.
• Live Forensic Evidence: Automatically extracts and dumps the specific packets associated with an alert into a timestamped .pcap file for post-incident analysis.
• Academic Validation Mode: Supports replaying massive feature-extracted CSV datasets or raw PCAP files through the exact same ML pipeline for research benchmarking.

Installation & Usage

[+] View

Prerequisites

• Linux OS (for iptables and live network sniffing)
• Python 3.10+
• Node.js & npm (for the web dashboard)
• Redis & RabbitMQ servers

Setup

1. Install System Dependencies (Debian/Ubuntu example):

   sudo apt-get install redis-server rabbitmq-server

2. Setup Python Environment:

   python3 -m venv .venv
   source .venv/bin/activate
   pip install -r requirements.txt

3. Install Frontend Dependencies:

   cd web-dashboard
   npm install
   cd ..

Generating Machine Learning Models

To keep this repository lightweight, the pre-trained .pkl machine learning models (which can exceed 300MB) are not included in version control. You must generate them locally before starting the pipeline.

1. Acquire and Structure Raw Data

Download your preferred Network Intrusion datasets (e.g., CIRA-CIC-DoHBrw-2020 for DoH and CTU-13 for C2 Beacons) and place them in a new data/ directory at the root of the project. Ensure the file structure matches the expectations of the training scripts:

NetworkRKSha/
└── data/
    ├── raw_doh.csv                 # Raw DoH dataset for the live sniffer
    ├── raw_beacon.csv              # Raw CTU-13 dataset for the live sniffer
    └── NIDS-Data/
        ├── doh_evaluation.csv      # DoH dataset for the async worker
        └── ctu13_beacon_attack.csv # CTU-13 dataset for the async worker

2. Prepare the Datasets

Run the dataset preparation script. This cleans the raw data and maps it exactly to the 29-feature (DoH) and 25-feature (Beacon) extraction pipelines used by the engine, outputting clean_*_train.csv files.

python prepare_datasets.py

3. Train the Models

Run the two training scripts to generate the .pkl files.

First, train the models used by the asynchronous RabbitMQ worker (this also generates the required features.json mapping):

python train_models.py

Next, train the models used by the real-time Scapy sniffer (live_sniffer.py):

python train_live_models.py

Once completed, check the models/ directory. You should see four .pkl files and one features.json file. The system is now ready to run.

Running the Pipeline

NetworkRKSha includes a unified launch script that starts all necessary services (Backend API, ML Worker, Live Sniffer, and Next.js Frontend) in a single terminal.

# Start the full pipeline (Requires sudo for sniffing and iptables)
./start_pipeline.sh

Once running, the web dashboard will be available at http://localhost:3000 and the API at http://localhost:8000.

Configuration

System behavior, detection thresholds, and responder settings (e.g., dry-run mode, whitelists) can be modified in config.yaml. Changes are hot-reloaded dynamically without requiring a restart.