Taxi-Optimization

Project Overview

This project aims to analyze New York City taxi trip data to understand various factors influencing trip duration, identify urban mobility patterns, and build predictive models. The analysis involves data loading, cleaning, feature engineering, exploratory data analysis, clustering, classification, and regression modeling.

Data Sources

• train.csv: Contains individual taxi trip records, including pickup/dropoff times, locations, passenger counts, and trip durations.
• weather_data_nyc_centralpark_2016(1).csv: Provides daily weather information for NYC Central Park in 2016.

Key Steps and Analysis

1. Data Loading & Initial Exploration

• Loaded train.csv into a Pandas DataFrame.
• Initial inspection of data shape, head, and data types using trip.info() and trip.describe(). Discovered 266,870 records and 11 columns.
• Identified missing values in several columns, predominantly one missing record for passenger_count, dropoff_datetime, and coordinate data.
• Visualized trip_duration (log scale) and passenger_count distributions before cleaning, showing a wide range of durations and a peak for 1 passenger.

2. Data Cleaning & Preprocessing

• Dropped rows with critical missing IDs to ensure data integrity.
• Filled remaining null values for passenger_count (with 0), store_and_fwd_flag (with '0'), and trip_duration (with 0).
• Removed duplicate trip records based on id (0 duplicates found).
• Outlier Removal: Filtered trips based on:
  • passenger_count: Restricted to 1-6 passengers.
  • trip_duration: Filtered for durations between 100 seconds and 100,000 seconds (approx. 28 hours).
  • Geographic Bounding Box: Constrained pickup and dropoff coordinates to a defined NYC area (40.0-41.0 latitude, -74.5 to -73.0 longitude). This removed 3,584 outlier rows.

3. Feature Engineering

Created several new features to enrich the dataset:

• Temporal Features:
  • pickup_hour, pickup_day, pickup_month from pickup_datetime.
  • is_weekend: Binary flag (1 if weekend, 0 otherwise).
  • period_of_day: Categorized hours into 'Night', 'Morning', 'Afternoon', 'Evening'.
• Spatial Features:
  • haversine_dist: Haversine distance between pickup and dropoff points (km).
  • manhattan_dist: Manhattan distance approximation.
  • bearing: Initial bearing (direction of travel) in degrees.
• Speed Feature:
  • average_speed: Calculated as haversine_dist / trip_duration (km/h).
• Weather Data Integration:
  • Merged daily weather data (maximum, minimum, average temperature, precipitation, snowfall, snow depth) with trip data based on pickup_datetime.
  • Handled 'T' (trace) values in weather data by replacing them with a small float (0.001).

4. Exploratory Data Analysis (EDA)

• Trip Volume: Visualized trip counts by pickup_hour (peak during evening rush) and pickup_day (weekends show higher activity than weekdays, particularly Saturday).
• Distribution Analysis: Histograms for trip_duration (minutes), haversine_dist (km), and average_speed (km/h) provided insights into their distributions.
• Correlation Heatmap: Explored correlations between numeric features, revealing strong positive correlation between trip_duration and distance metrics (haversine_dist, manhattan_dist), and negative correlations between temperature and precipitation/snow features.
• Speed by Period of Day: Box plots showed how average speed varies across different periods, with night time generally having higher speeds.
• Trip Duration by Vendor: Box plots indicated slight differences in trip duration distributions between the two vendors.

5. Clustering Analysis

Applied clustering to a sample of 50,000 trips using pickup_latitude, pickup_longitude, and pickup_hour to identify urban mobility patterns.

• KMeans Clustering: Used the Elbow Method to determine an optimal K=15 clusters, visualizing distinct geographic zones of pickup activity.
• DBSCAN Clustering: Identified 4 density-based clusters and highlighted noise points (10.2% of data) as potential areas of congestion or unusual activity.
• Rush Hour Heatmap: A heatmap of trip counts by cluster and pickup_hour revealed which geographic zones experience peak activity at different hours.

6. Vendor ID Classification

Built a Random Forest Classifier to predict vendor_id based on trip characteristics.

• Features: pickup_latitude, pickup_longitude, dropoff_latitude, dropoff_longitude, trip_duration, passenger_count, haversine_dist, average_speed, pickup_hour, pickup_day, is_weekend.
• Model Performance: Achieved an accuracy of 0.5850. A classification report and confusion matrix provided detailed metrics, showing better recall for Vendor 1 but better precision for Vendor 2.
• Feature Importances: trip_duration, dropoff_longitude, and haversine_dist were the most important features for classifying the vendor.

7. Trip Duration Regression

Developed models to predict trip_duration (log-transformed for normality).

• Features: haversine_dist, manhattan_dist, bearing, passenger_count, pickup_hour, pickup_day, is_weekend, pickup_month, and weather features.
• Models Compared:
  • Linear Regression (Baseline): RMSE: 0.5453, MAE: 0.4117, R²: 0.4469
  • Random Forest Regressor: RMSE: 0.4082, MAE: 0.2816, R²: 0.6901
• Results: Random Forest Regressor significantly outperformed Linear Regression, capturing more variance in trip duration. Plots of actual vs. predicted values further illustrated the models' performance.
• Feature Importances: haversine_dist, manhattan_dist, and pickup_hour were identified as the most influential features for predicting trip duration.

8. Corridor Analysis

Identified and visualized the busiest transport corridors, focusing on trips with 1 or 2 passengers (potential for ride-sharing).

• Methodology: Gridded pickup and dropoff coordinates to a precision of 2 decimal places (~1.1 km cells) and grouped by pickup_hour.
• Top Corridors: Listed the top 10 busiest corridors across all hours.
• Visualization: Plotted the top 20 busiest corridors during peak hours (8 AM, 12 PM, 6 PM), illustrating popular routes for ride-sharing or public transport optimization.

Conclusion

This analysis provides valuable insights into NYC taxi trip data. We've successfully cleaned and enriched the dataset, identified key factors influencing trip duration and urban mobility, and built predictive models. The clustering and corridor analysis highlight areas and routes that could benefit from optimized services or public transport planning. The Random Forest Regressor proved to be a robust model for predicting trip duration, with distance metrics and pickup time being the most significant features.

Technologies Used

• Python
• Scikit-learn
• Gradient boosted decision trees
• Pandas (for data manipulation)
• NumPy (for numerical operations)
• Matplotlib & Seaborn (for data visualization)