ML Hands-On Problem Set
20 comprehensive machine learning topics across 4 disciplines. Work through 5 progressive difficulty tiers per topic with real-world Kaggle, Hugging Face, and benchmark datasets.
Linear Regression
California Housing
Kaggle“Clean, continuous target, mild nonlinearity — good for seeing the normal equation and GD agree.”
Derive the closed-form (normal equation) solution for simple linear regression by minimizing MSE with calculus.
By hand, fit a line to 5 hand-picked (x, y) points using the normal equation.
Explain why MSE, not MAE, gives you the normal equation's nice closed form.
List and explain all 5 classical assumptions of linear regression (linearity, independence, homoscedasticity, normality of residuals, no multicollinearity). For each, describe what happens to your model if it's violated.
What's the geometric interpretation of least squares as a projection?
Implement multiple linear regression using the closed-form normal equation (matrix inversion) in NumPy — no sklearn.
Implement the same model using batch gradient descent. Plot cost vs. iteration and tune the learning rate until it converges cleanly.
Implement stochastic gradient descent and mini-batch gradient descent versions of the same problem. Compare convergence speed and noise in the loss curve across all three.
Implement residual analysis from scratch: plot residuals vs. fitted values, and a Q-Q plot to check normality, without using statsmodels.
Fit linear regression on a real dataset (e.g. California housing) using sklearn. Report R², MAE, RMSE on a held-out test set.
Standardize/normalize features and show, empirically, why this matters for gradient descent convergence speed but not for the normal equation's answer.
Detect multicollinearity using VIF (variance inflation factor) and show what happens to your coefficients when you drop a collinear feature.
Deliberately engineer a dataset where linear regression underfits (nonlinear relationship) and one where it overfits (too many polynomial features, too little data). Show the learning curves for both.
Inject an outlier into your dataset and show how much it distorts the fitted line. Then implement a simple robust alternative (e.g. Huber loss) from scratch and compare.
Take a dataset that visibly violates homoscedasticity (variance grows with x). Show the effect on confidence intervals and propose a fix (transform y, weighted least squares).
Wrap your trained model in a minimal FastAPI service with a /predict endpoint, input validation, and a health check.
Implement online (incremental) linear regression that updates its weights as new data streams in, without retraining from scratch (recursive least squares or SGD-based).
Research task: read about Bayesian linear regression and implement a version that returns a predictive distribution, not just a point estimate. Compare its uncertainty estimates on out-of-distribution inputs.