Linear Regression from Scratch (2 Features, 1 Target, 3D visualization)

This project demonstrates a minimal linear regression trained with gradient descent using two intuitive features from the Ames Housing dataset: GrLivArea (living area) and OverallQual (overall quality).

We minimize the Mean Squared Error (MSE) evaluate performance with the Root Mean Squared Error (RMSE) and visualize the fitted regression plane along with the training dynamics over epochs.

---

Used maths

Linear model

$$\hat{Y} = XW + b$$

Standardization

$$X_{\text{std}} = \frac{X - \mu}{\sigma}$$

Loss functions

$$\text{MSE}(Y,\hat{Y}) = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$$ $$\text{RMSE}(Y,\hat{Y}) = \sqrt{\text{MSE}(Y,\hat{Y})}$$

Gradients

$$\frac{\partial L}{\partial W} = \frac{2}{n} X^\top (\hat{Y} - Y), \quad \frac{\partial L}{\partial b} = \frac{2}{n} \sum_{i=1}^{n} (\hat{y}_i - y_i)$$

Gradient descent update

$$W = W - \eta \frac{\partial L}{\partial W} \quad b = b - \eta \frac{\partial L}{\partial b}$$

Early stopping criterion

$$\Delta \text{RMSE} < tol \;\; \text{for } p \text{ consecutive epochs } \;\; \Rightarrow \;\; stop$$

---

How it works

• Model: Linear regression with standardized features
• Loss: MSE and RMSE (root mean squared error)
• Optimizer: Gradient descent with early stopping
• Visualization: Interactive Plotly plots + 3D GIF of the regression plane on Matplotlib

---

Quickstart

1. Install dependencies:

   pip install -r requirements.txt

2. Run the notebook:

   jupyter lab notebooks/main.ipynb

3. Outputs (GIF, figures) will appear in the assets/ folder.

---

Results (sample)

• Gradient descent

• Final RMSE: ~42,000($)
• 3D visualization of the regression plane:

---

License