---

📊 NYC Motor Vehicle Collisions – Data Engineering & Visualization Dashboard

A complete workflow: Exploration → Cleaning → Integration → Dashboard → Deployment

---

📊 Research Questions Answered in the Dashboard

• "How do crash severity levels (no injury, minor injury, severe/fatal) vary across the most common contributing factors in NYC traffic collisions?"
• "How does the distribution of crash injury severity differ across age groups and genders?"
• "Which boroughs exhibit the strongest temporal patterns (seasonality & hourly peaks) in pedestrian-related injuries?"
• "How do crash frequencies and severity levels vary spatially across New York City boroughs, and which boroughs exhibit disproportionately high crash intensity?"
• "How do vehicle type and contributing factor interact to determine which road users (pedestrians, cyclists, motorists, passengers) are most affected?"
• "Which hours of the day show the greatest mismatch between crash frequency and crash severity (rare but deadly hours)?"
• "How do pedestrian, cyclist, and motorist injury trends evolve over the years, and which mode of transportation is becoming riskier over time?"
• "How do traffic crashes trend over the years in NYC?"
• "At what times (day of week × hour) do most crashes occur?"
• "Which body injury types are associated with the highest proportions of severe or fatal crash outcomes?"

---

📁 Project Structure

. ├── notebooks/
│ ├── 01_exploration.ipynb
│ ├── 02_cleaning.ipynb
│ └── 03_integration.ipynb
├── app.py
├── requirements.txt
├── Dockerfile
├── data/
│ ├── raw/
│ ├── cleaned/
│ └── final.parquet
└── README.md

---

⚙️ ETL Pipeline Overview

This project uses a 3-notebook data engineering pipeline.
Running them in order generates all cleaned CSVs and the final Parquet file consumed by the dashboard.

---

📘 1. 01_exploration.ipynb — Data Exploration

• Loads raw datasets
• Basic profiling
• Missing value analysis
• Outlier detection
• Preliminary visualizations
• Identifies transformations needed for cleaning

Output: None (analysis only)

---

🧽 2. 02_cleaning.ipynb — Cleaning Pipeline

• Handles missing values properly
• Normalizes formatting & column names
• Fixes inconsistent categories
• Parses date/time
• Removes duplicates
• Creates standardized severity + injury fields

Outputs saved to data/cleaned/:

• cleaned_crashes.csv
• cleaned_persons.csv

---

🔗 3. 03_integration.ipynb — Dataset Integration

• Performs inner merge between crashes + persons
• Creates analytical fields:
  • SEVERITY
  • SEVERITY_SIMPLE
  • INJURY FLAGS
  • AGE_GROUP
• Produces the final dataset used in the dashboard

Final outputs:

• data/cleaned/final.csv
• data/final.parquet

---

▶️ Running the Pipeline Locally

Option A — Using Jupyter Notebook

1. Run 01_exploration.ipynb
2. Run 02_cleaning.ipynb
3. Run 03_integration.ipynb

Option B — Programmatically

%run 01_exploration.ipynb
%run 02_cleaning.ipynb
%run 03_integration.ipynb

---

🚀 Running the Dashboard Locally

Install dependencies

pip install -r requirements.txt

Run the application

python app.py

Dashboard will be available at:

http://127.0.0.1:8050

---

🐳 Deployment — Hugging Face Spaces (Docker)

Dockerfile

FROM python:3.10-slim
WORKDIR /app
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
COPY . .
EXPOSE 7860
CMD ["gunicorn", "-b", "0.0.0.0:7860", "app:app"]

Push to your HF Space

git add .
git commit -m "Deploy dashboard"
git push

Your application will appear at:

https://-.hf.space

---

🧠 Team Contributions

Hala Sameh — Exploration Notebook
Malak El Akkad — Cleaning Notebook
Sara Ahmed — Integration & Post-Integration Notebook
Ammar Waleed — Plotly visualizations, Dash website
Malak Mostafa — Plotly visualizations, Dash website, deployment

---

📌 Additional Notes

• The dashboard automatically loads the optimized final.parquet for fast performance.
• The entire project follows a reproducible ETL structure:
  Run the notebooks → generate cleaned data → launch dashboard