Data Analysis in Python

A comprehensive collection of 13 data analysis and machine learning projects covering statistical methods, predictive modeling, and advanced algorithms. This repository serves as a complete course in practical data analysis using Python.

📊 Repository Overview

This repository contains 13 progressive data analysis projects that build skills from basic statistical analysis to advanced machine learning techniques. Each project includes detailed Jupyter notebooks with real-world datasets and practical implementations.

📂 Project Structure

data-analysis-in-python/
├── 01-clustering/               # Consumer segmentation using clustering
│   ├── ASSIGNMENT1.ipynb       # Clustering analysis notebook
│   ├── TR.txt                  # Task requirements
│   ├── Леденцы.dat            # Lollipop consumption dataset
│   └── Леденцы_описание_2.pdf  # Dataset description
├── 02-factor-analysis/         # Factor analysis of economic data
│   ├── ASSIGNMENT2.ipynb       # Factor analysis notebook
│   ├── Econom_Cities_data.csv  # Economic indicators dataset
│   └── TR.txt                  # Task requirements
├── 03-digit-clustering/        # Handwritten digit clustering
│   ├── ASSIGNMENT3.ipynb       # Digit clustering analysis
│   └── TR.txt                  # Task requirements
├── 04-hypothesis-testing/      # Statistical hypothesis testing
│   ├── ЛАБА 4.ipynb           # Hypothesis testing notebook
│   ├── Nursing Home Data/      # Healthcare dataset
│   └── Unions and State Labor Law/ # Labor economics dataset
├── 05-prediction/              # Predictive modeling
│   ├── Лаба 5.ipynb           # Prediction analysis
│   ├── Chicago_hotels.csv      # Hotel booking dataset
│   └── Лабораторная работа_прогнозирование2.txt
├── 06-grid-search/             # Hyperparameter optimization
│   ├── Лаба 6.ipynb           # Grid search implementation
│   └── default of credit card clients.csv
├── 07-knn/                     # K-Nearest Neighbors classification
│   ├── Лаба7.ipynb            # KNN implementation
│   ├── default of credit card clients.csv
│   └── lab_1_А_К.txt          # Lab assignments
├── 08-decision-trees/          # Decision tree classification
│   ├── Лаба8.ipynb            # Decision trees notebook
│   └── TR.txt                  # Task requirements
├── 09-random-forest/           # Random forest ensemble
│   ├── Лаба9.ipynb            # Random forest analysis
│   ├── default of credit card clients.csv
│   └── lab_1_А_К.txt          # Lab assignments
├── 10-monte-carlo/             # Monte Carlo simulations
│   └── Лаба10.ipynb           # Monte Carlo methods
├── 11-gradient-boosting/       # Gradient boosting
│   ├── Лаба11.ipynb           # Gradient boosting implementation
│   ├── default of credit card clients.csv
│   └── lab_1_А_К.txt          # Lab assignments
├── 12-xgboost/                 # XGBoost algorithm
│   ├── Лаба12.ipynb           # XGBoost analysis
│   ├── default of credit card clients.csv
│   └── lab_1_А_К.txt          # Lab assignments
├── 13-time-series/             # Time series analysis
│   ├── Лаба13.ipynb           # Time series forecasting
│   └── benzak_01.csv          # Time series dataset
├── README.md                   # This file
├── LICENSE                     # License information
└── .gitattributes              # Git attributes

🎯 Projects Overview

1. Consumer Clustering (01-clustering/)

Focus: Customer segmentation using clustering algorithms Dataset: Lollipop consumption survey data (11 variables) Key Concepts: Consumer behavior analysis, market segmentation Files: ASSIGNMENT1.ipynb, Леденцы.dat

2. Factor Analysis (02-factor-analysis/)

Focus: Dimensionality reduction and factor extraction Dataset: Economic indicators for cities Key Concepts: Factor analysis, principal components, data reduction Files: ASSIGNMENT2.ipynb, Econom_Cities_data.csv

3. Digit Clustering (03-digit-clustering/)

Focus: Unsupervised learning on image data Dataset: Handwritten digits Key Concepts: Image clustering, pattern recognition Files: ASSIGNMENT3.ipynb

4. Hypothesis Testing (04-hypothesis-testing/)

Focus: Statistical inference and hypothesis validation Datasets: Nursing home data, labor economics data Key Concepts: T-tests, ANOVA, statistical significance Files: ЛАБА 4.ipynb, healthcare and labor datasets

5. Prediction (05-prediction/)

Focus: Predictive modeling and forecasting Dataset: Chicago hotel bookings Key Concepts: Regression analysis, predictive analytics Files: Лаба 5.ipynb, Chicago_hotels.csv

6. Grid Search (06-grid-search/)

Focus: Hyperparameter optimization Dataset: Credit card default data Key Concepts: Cross-validation, parameter tuning, model selection Files: Лаба 6.ipynb, credit card dataset

7. K-Nearest Neighbors (07-knn/)

Focus: Instance-based learning and classification Dataset: Credit card default data Key Concepts: Distance metrics, k-parameter selection, classification Files: Лаба7.ipynb, credit card dataset

8. Decision Trees (08-decision-trees/)

Focus: Tree-based classification methods Key Concepts: Information gain, tree pruning, classification trees Files: Лаба8.ipynb

9. Random Forest (09-random-forest/)

Focus: Ensemble learning with random forests Dataset: Credit card default data Key Concepts: Bagging, ensemble methods, feature importance Files: Лаба9.ipynb, credit card dataset

10. Monte Carlo (10-monte-carlo/)

Focus: Stochastic simulation methods Key Concepts: Random sampling, probability simulation, risk analysis Files: Лаба10.ipynb

11. Gradient Boosting (11-gradient-boosting/)

Focus: Advanced ensemble learning Dataset: Credit card default data Key Concepts: Boosting, gradient descent, sequential learning Files: Лаба11.ipynb, credit card dataset

12. XGBoost (12-xgboost/)

Focus: Extreme Gradient Boosting Dataset: Credit card default data Key Concepts: Regularized boosting, parallel processing, scalability Files: Лаба12.ipynb, credit card dataset

13. Time Series (13-time-series/)

Focus: Temporal data analysis and forecasting Dataset: Time series data (benzak_01.csv) Key Concepts: Trend analysis, seasonality, forecasting models Files: Лаба13.ipynb, benzak_01.csv

🛠️ Installation & Setup

Prerequisites

• Python 3.8+
• Jupyter Notebook/JupyterLab
• Git

Installation Steps

1. Clone the repository:

   git clone <repository-url>
   cd data-analysis-in-python

2. Create a virtual environment (recommended):

   python -m venv data_analysis_env
   source data_analysis_env/bin/activate  # On Windows: data_analysis_env\Scripts\activate

3. Install required packages:

   pip install numpy pandas matplotlib seaborn scikit-learn statsmodels xgboost lightgbm

4. Launch Jupyter:

   jupyter notebook

📦 Key Dependencies

Core Libraries:

• numpy: Numerical computing
• pandas: Data manipulation and analysis
• matplotlib/seaborn: Data visualization
• scikit-learn: Machine learning algorithms
• statsmodels: Statistical modeling
• xgboost/lightgbm: Gradient boosting frameworks

Specialized Libraries:

• scipy: Scientific computing
• plotly: Interactive visualizations
• seaborn: Statistical visualization

🎓 Learning Objectives

By working through these projects, you'll learn:

1. Statistical Analysis: Hypothesis testing, factor analysis, correlation
2. Machine Learning: Classification, regression, clustering algorithms
3. Data Preprocessing: Cleaning, normalization, feature engineering
4. Model Evaluation: Cross-validation, performance metrics, validation
5. Advanced Techniques: Ensemble methods, boosting, time series analysis
6. Practical Application: Real-world datasets and business problems

📊 Analysis Techniques Covered

Statistical Methods

• Descriptive Statistics: Mean, median, variance, correlation
• Inferential Statistics: Hypothesis testing, confidence intervals
• Factor Analysis: Dimensionality reduction, principal components

Machine Learning Algorithms

• Unsupervised Learning: K-means clustering, hierarchical clustering
• Supervised Learning: KNN, Decision Trees, Random Forest
• Ensemble Methods: Bagging, Boosting, Gradient Boosting, XGBoost
• Time Series: Trend analysis, seasonal decomposition, forecasting

Data Science Workflow

• Data Exploration: EDA, visualization, pattern discovery
• Feature Engineering: Variable transformation, feature selection
• Model Development: Algorithm implementation, parameter tuning
• Model Validation: Performance evaluation, model comparison

🚀 Usage Examples

Running Individual Projects

Each project is self-contained. Navigate to any project directory and run its notebook:

cd 01-clustering
jupyter notebook ASSIGNMENT1.ipynb

Data Analysis Workflow

import pandas as pd
import numpy as np
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt

# Load and explore data
data = pd.read_csv('your_dataset.csv')
print(data.head())
print(data.describe())

# Perform clustering
kmeans = KMeans(n_clusters=3, random_state=42)
clusters = kmeans.fit_predict(data)

# Visualize results
plt.scatter(data.iloc[:, 0], data.iloc[:, 1], c=clusters)
plt.show()

📈 Project Complexity Progression

The projects are designed to build skills progressively:

1. Foundation (1-3): Basic statistical analysis and clustering
2. Intermediate (4-7): Hypothesis testing and classification algorithms
3. Advanced (8-13): Ensemble methods, boosting, and time series

🤝 Contributing

This educational repository welcomes contributions:

1. Bug Fixes: Correct implementation errors or improve accuracy
2. Documentation: Add explanations or improve code comments
3. New Analyses: Implement additional statistical methods
4. Visualizations: Create better plots and charts
5. Performance: Optimize computationally intensive operations

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Educational content based on statistical and machine learning courses
• Datasets from various public sources and research projects
• Open-source Python libraries that enable these analyses

📞 Support

For questions or issues:

1. Check the individual project notebooks for detailed implementations
2. Review the task requirements (TR.txt files) for project objectives
3. Examine the data documentation files for dataset descriptions
4. Open an issue for technical problems or suggestions

---

Happy Analyzing! 📊

This repository provides a complete journey through modern data analysis techniques, from statistical foundations to advanced machine learning algorithms.