Steam Player Engagement Analytics

An automated data pipeline that tracks player engagement across popular Steam games. It pulls live data daily, stores it in a cloud database, and surfaces it through an interactive dashboard.

Live dashboard: [add your Streamlit Community Cloud link here after deploying]

---

What This Project Does

Rather than analyzing a static, pre-cleaned dataset, this project builds and runs its own data pipeline end-to-end:

1. Pulls live game data (genres, pricing, current player counts) from the Steam Web API and SteamSpy
2. Stores it in a cloud MySQL database (Aiven), building a daily historical snapshot over time
3. Cleans and reshapes the raw data by splitting multi-genre strings, parsing dates, filtering out non-game software that occasionally shows up in the source data
4. Runs automatically every day via GitHub Actions, with no manual intervention
5. Visualizes the results in a live, interactive Streamlit dashboard

Why This Approach

Steam, Xbox, and Nintendo don't publish real sales figures, because that data is proprietary, and most "Steam sales" datasets found online are actually rough, unreliable estimates. Instead, this project uses live concurrent player counts as an engagement metric, which is real, verifiable data pulled directly from Steam's own public API, rather than guessing at unavailable sales numbers.

Architecture

┌──────────────┐     ┌──────────────┐     ┌──────────────────┐
│  Steam API   │     │  SteamSpy    │     │  (top-played     │
│ (game details│     │  API         │     │   game list)     │
│  + players)  │     │              │     │                  │
└──────┬───────┘     └──────┬───────┘     └──────────────────┘
       │                    │
       └────────┬───────────┘
                 ▼
       ┌────────────────────┐
       │  fetch_steam_      │   Runs daily at 00:00 UTC
       │  data.py           │   via GitHub Actions
       └─────────┬──────────┘
                 ▼
       ┌────────────────────┐
       │  Aiven MySQL       │
       │  games_raw table   │
       └─────────┬──────────┘
                 ▼
       ┌────────────────────┐
       │  clean_data.py     │   Splits genres, parses dates,
       │                    │   filters non-game entries
       └─────────┬──────────┘
                 ▼
       ┌────────────────────┐
       │  Aiven MySQL       │
       │  games_clean table │
       └─────────┬──────────┘
                 ▼
       ┌─────────────────────┐
       │  app.py             │
       │  Streamlit Dashboard│
       └─────────────────────┘

Tech Stack

• Python: requests, pandas
• MySQL (hosted on Aiven): cloud database, SSL-secured
• GitHub Actions: daily automated pipeline scheduling
• Streamlit + Plotly: interactive dashboard, deployed on Streamlit Community Cloud

Key Findings So Far

• Free-to-play games draw dramatically higher concurrent player counts than paid games, regardless of genre
• Within paid games specifically, price shows no strong relationship to player count
• Genre-level comparisons are shown alongside sample sizes, since some genres in the tracked dataset are represented by very few games, a deliberate choice to avoid presenting misleading averages

Data Limitations (Being Upfront)

• Player counts ≠ sales. This project deliberately avoids using sales estimates (which aren't publicly available and are unreliable when "estimated"), and instead uses live concurrent player counts as a proxy for engagement.
• Sample is popularity-biased. Tracked games come from SteamSpy's "top played" lists, so this isn't a random or complete sample of all Steam games, it is specifically the most popular ones.
• Time-series depth grows daily. Trend analysis (player counts over time) becomes more meaningful the longer the automated pipeline runs, since it started collecting data recently.

Project Structure

├── .github/workflows/
│   └── daily_pull.yml       # GitHub Actions automation
├── fetch_steam_data.py      # Ingestion: Steam API + SteamSpy → games_raw
├── clean_data.py            # Transformation: games_raw → games_clean
├── setup_db.py               # One-time schema setup script
├── view_data.py               # Quick CLI script to inspect table contents
├── app.py                    # Streamlit dashboard
├── schema.sql                 # Database schema
├── requirements.txt
├── .env.example               # Template for local environment variables
└── README.md

Running This Locally

1. Clone the repo and install dependencies:

   pip install -r requirements.txt

2. Copy .env.example to .env and fill in your own MySQL credentials (Aiven or local MySQL/XAMPP both supported, see comments in .env.example)
3. Set up the database schema:

   python setup_db.py

4. Pull data and clean it:

   python fetch_steam_data.py
   python clean_data.py

5. Launch the dashboard:

   streamlit run app.py

Automation

The pipeline runs automatically every day at 00:00 UTC via GitHub Actions (.github/workflows/daily_pull.yml), fetching fresh data and re-running the cleaning step, with no manual steps required. Credentials are stored securely as GitHub Actions repository secrets, never committed to the codebase.

Features

• Automated Steam data collection
• MySQL database pipeline
• Interactive Streamlit dashboard
• Data visualization with Plotly
• LLM-generated executive insights using the Google Gemini API
• Daily data refresh and caching

Future Improvements

• Expand tracked game list beyond current top-played titles for broader genre coverage
• Incremental cleaning (append-only) rather than full table rebuild, to support larger-scale data
• Deeper trend analysis once more days of historical data accumulate
• Player count anomaly detection (e.g. flagging unusual spikes around major game updates or sales events)

---

Built as a portfolio project to demonstrate end-to-end data pipeline design: ingestion, cloud storage, LLM utilization, automated orchestration, transformation, and deployment, not just static analysis of a downloaded file.