Machine Learning and Statistical Models

🔒 Full text at this link requires payment or institutional access. Unmarked links may be unchecked.

Linear and generalized linear models

• Least squares and ridge - Least squares minimizes squared prediction errors; ridge adds a squared-coefficient penalty to stabilize fits with correlated predictors. Includes implementation and solver guidance. docs.
• Lasso - Constrains the sum of absolute regression coefficients, shrinking some exactly to zero. Combines estimation and variable selection. Tibshirani 1996.
• Elastic net - Combines L1 and L2 penalties, allowing sparse coefficients while retaining ridge-style shrinkage. A useful alternative to lasso for correlated predictors. Zou and Hastie 2005; docs.
• Generalized linear models - Connects an exponential-family response mean to a linear predictor through a link function. Unifies models fitted by iteratively weighted linear regression. Nelder and Wedderburn 1972.
• Logistic regression - Models binary class probability through a logistic link applied to a linear score. The documented implementation also supports multiclass classification and regularization. docs.
• Generalized linear mixed models - Adds random effects to GLMs for correlated observations and overdispersion. Breslow-Clayton develops approximate inference using penalized quasi-likelihood. Breslow and Clayton 1993.
• Tests as linear models - Worked connections between regression, t-tests, correlation and ANOVA. Rank-based analogues are also discussed; some equivalences are approximate. Lindeløv 2019.

Additive models and adaptive splines

• Generalized additive models - Replaces a GLM's linear predictor terms with smooth functions of individual inputs. Local scoring extends additive regression to different response distributions. Hastie and Tibshirani 1986.
• MARS - Builds regression models from products of spline basis functions, selecting their number, variables and knot locations from data. Separates additive contributions from interactions. Friedman 1991.
• Generalized additive neural networks - Potts's KDD reference on neural additive models. Potts 1999.
• Intelligible classification and regression - Studies additive models whose individual feature contributions can be plotted, comparing fitting methods and the accuracy cost of restricting interactions. Lou et al. 2012.
• GA2M - Extends additive models with selected pairwise interactions. FAST ranks candidate feature pairs, keeping explanations as one- and two-dimensional functions. Lou et al. 2013.
• Intelligible healthcare models - Pneumonia and readmission case studies use additive models with pairwise interactions to expose unexpected learned associations and revise models. Caruana et al. 2015.
• GAMs with autoregressive terms - Combines nonlinear covariate terms with autoregression for serially correlated outcomes. Demonstrated on temperature and daily mortality. Yang et al. 2012.
• Neural Additive Models - Jointly trains a separate neural network for each input feature, then combines their outputs linearly. Learns flexible univariate shapes while preserving additive structure. Agarwal et al. 2021.
• How trustworthy are GAMs? - Shows that similarly accurate GAMs can learn conflicting feature shapes. Compares inductive biases, sparsity and behavior on rare subpopulations; accuracy alone does not validate an explanation. Chang et al. 2021.

Trees and tabular prediction

• CART - Recursively partitions feature space into regions with simple predictions. Depth and pruning control complexity; individual trees can be unstable under changes to training data. docs.
• Random forests - Combines randomized trees, including random feature selection at splits. Analyzes how individual tree strength and dependence affect ensemble error. Breiman 2001.
• Bayesian additive regression trees - Represents a response as a sum of trees with a regularization prior that keeps individual trees weak. Bayesian backfitting samples the model and its predictive uncertainty. Chipman et al. 2010.
• XGBoost - Gradient-boosted trees with sparsity-aware split finding and a weighted quantile sketch for approximate tree learning. The paper also addresses cache access and distributed computation. Chen and Guestrin 2016.
• CatBoost - Uses ordered boosting and categorical-feature processing to address prediction shift caused by target leakage during training. Prokhorenkova et al. 2018.
• TabPFN: original classifier - Pretrains a transformer on synthetic classification tasks. At inference, labeled training rows provide context for predictions without updating the model weights; the original scope is small tabular datasets. Hollmann et al. 2022.
• TabPFN: classification and regression - Extends prior-data-fitted tabular prediction to regression, categorical inputs and missing values. A pretrained transformer performs in-context prediction from labeled examples. Hollmann et al. 2025.
• TabM - Builds parameter-sharing ensembles of multilayer perceptrons for tabular prediction. Joint training produces multiple predictions without storing a separate full network for every ensemble member. Gorishniy et al. 2025.
• TabICL - Combines representations of table rows with a transformer for in-context classification. Synthetic-task pretraining supports prediction on new tables without task-specific weight updates. Qu et al. 2025.

Observation protocols

• Repeated measurements - More readings with independent noise at fixed inputs can harm new-input predictions. Priors coupling input density and responses can count repeated locations as fresh input evidence; TabPFN-v3 also depends on row count. Liu et al. 2026.
• Repetition evidence - TabPFN-v3 (8-member ensemble), six 1D functions, 20 seeds, 12 sites, 64 readings/site, noise SD 0.05: median new-input MSE 0.015 (one reading), 0.049 (separate rows), 0.0094 (means). Count reset weakens gains at SD 0.30/0.50. Liu et al. 2026.

Compression of tabular models

• Shallow TFMs - Uses affine block replacement, following ReplaceMe, with per-feature maps for dual-axis models. Across 171 classification datasets and eight models, halving depth loses at most 0.002 median AUROC per model. Cannistraci and Vogt 2026.
• Compression limits - On 41 regression datasets, retaining one block costs TabPFN-3 0.132 median R2. Regression windows use labelled calibration R2; classification uses output KL. Maps are fitted to the target table's activations. Cannistraci and Vogt 2026.
• Compression cost - MIMIC-III: one-block TabPFN-3 on RTX 5090 gives 1.2x speedup for 512-query re-encoding, 2.4x for a cached single query. On i9-14900K (fp32, batch 512), re-encoding gains are 1.4x for TabPFN-3 and 12.4x for TabPFNv2. Cannistraci and Vogt 2026.

Local and conditional-independence classifiers

• Nearest-neighbor classification - Predicts from nearby labeled examples. Cover and Hart bound asymptotic one-neighbor error by twice Bayes error under their assumptions; this is neither a finite-sample guarantee nor a guarantee for an arbitrary learned distance. Cover and Hart 1967.
• Naive Bayes - Multiplies class-conditional feature probabilities under an independence model. Domingos and Pazzani show that correct class decisions can survive some dependence violations even when estimated probabilities are inaccurate. Domingos and Pazzani 1997.

Support-vector methods

• Support-vector networks - Constructs a linear decision surface in a nonlinear feature space and extends the approach to nonseparable training data. A foundation for kernel-based classification. Cortes and Vapnik 1995.
• SVM implementations - Linear and kernel classification, regression and outlier detection. Decision scores are not probabilities; probability estimation adds a separate fitting step. docs.

Probabilistic prediction, survival and treatment effects

• Quantile regression forests - Extends random forests to estimate the conditional response distribution and its quantiles, rather than only a conditional mean. Meinshausen 2006.
• Extreme NO2 forecasting - Compares probabilistic models for high-pollution episodes, including threshold-exceedance probabilities. A case study in evaluating forecast distributions rather than only mean predictions. Vasseur and Aznarte 2020.
• Kaplan-Meier - Estimates a survival curve from incomplete lifetime observations using a product-limit estimator. Independent censoring is a substantive assumption, explicitly discussed in the paper. Kaplan and Meier 1958.
• Cox regression - Models covariate effects as multipliers of an unspecified baseline hazard. Handles censored event times without choosing a parametric baseline distribution. Cox 1972.
• DeepSurv - Uses a neural network for the risk function in a Cox proportional-hazards model, allowing nonlinear covariate interactions in survival prediction. Katzman et al. 2018.
• Treatment-effect meta-learners - Builds estimators of conditional average treatment effects from existing regression learners. Introduces the X-learner and studies settings with unequal treatment-group sizes. Künzel et al. 2019.
• Simpson's paradox - Constructs examples where conditioning on additional variables reverses an association. Illustrates the dependence of aggregated comparisons on the variables included in an analysis. Fenton et al. 2019.
• NGBoost - Boosts parameters of a conditional probability distribution using natural gradients, extending boosting from point estimates to probabilistic regression. Duan et al. 2020.

Classical forecasting and latent states

• Forecasting: Principles and Practice, second edition - Historical R reference covering forecasting methods, diagnostics and evaluation, with examples based on the forecast package. Hyndman and Athanasopoulos 2018.
• Forecasting: Principles and Practice, third edition - Updated forecasting reference using tsibble and fable. Covers data exploration, statistical models, uncertainty and forecast evaluation. Hyndman and Athanasopoulos 2021.
• ARIMA and seasonal ARIMA - Combines differencing, autoregressive terms and moving-average errors. Seasonal extensions add corresponding terms at seasonal lags. Reference: FPP3, Chapter 9. Hyndman and Athanasopoulos 2021.
• Holt-Winters and ETS - Smooths level, trend and seasonality recursively. ETS supplies statistical state-space formulations and forecast distributions. Reference: FPP3, Chapter 8. Hyndman and Athanasopoulos 2021.
• STL decomposition - Uses Loess smoothing to separate trend, seasonal and remainder components. Seasonal variation may change over time; robust fitting limits outlier influence on the smooth components. Hyndman and Athanasopoulos 2021.
• Dynamic regression - Combines regression predictors with ARIMA errors. Forecasts require future predictor values or scenarios; ordinary intervals condition on these values rather than propagating their uncertainty. Hyndman and Athanasopoulos 2021.
• Vector autoregression - Models several series jointly, regressing each on lags of all series. Stationarity and differencing determine the appropriate specification. Hyndman and Athanasopoulos 2021.
• GARCH - Generalizes ARCH by allowing past conditional variances into the current variance equation. Studies stationarity, estimation and testing for time-varying volatility. Bollerslev 1986.
• Kalman filtering - Formulates linear filtering through state transitions and recursive error-covariance updates. Treats stationary and nonstationary prediction problems in a common framework. Kalman 1960.
• Hidden Markov models - Rabiner reviews Markov models with hidden states and probabilistic observations, with applications to speech recognition. A reference for the model's statistical structure and inference problems. Rabiner 1989.
• Infinite HMM - Uses Dirichlet processes to extend HMMs to countably many possible hidden states, integrating out transition parameters rather than fixing a finite state count in advance. Beal et al. 2001.
• Dynamic time warping - Aligns sequences by optimizing a nonlinear time-warping path with dynamic programming. Sakoe-Chiba studies path constraints for spoken-word matching; this is an alignment method, not a forecasting model. Sakoe and Chiba 1978.

Neural forecasting

• DeepAR - Trains an autoregressive recurrent network across related time series to produce probabilistic forecasts, sharing statistical information across series. Salinas et al. 2017.
• N-BEATS - Uses fully connected blocks with forward and backward residual connections for forecasting. Provides generic and explicitly interpretable configurations. Oreshkin et al. 2019.
• Temporal Fusion Transformer - Combines recurrent processing and attention for multi-horizon forecasts. Separately accommodates static covariates, known future inputs and historically observed variables. Lim et al. 2019.
• Chronos - Scales and quantizes time-series values into tokens, then trains transformer language models with cross-entropy. Sampling token sequences produces probabilistic forecasts. Ansari et al. 2024.
• PatchTST - Treats time-series patches as tokens and processes each channel separately with shared embedding and transformer weights; designed for forecasting and self-supervised pretraining. Nie et al. 2023.

Dynamical and mechanistic models

• Simple discrete dynamics - Shows how simple first-order nonlinear difference equations can produce stable equilibria, cycles or irregular fluctuations. A foundation for studying deterministic complexity. May 1976 🔒.
• Predator-prey inference - Fits Lotka-Volterra differential equations with a statistical observation model to lynx-hare pelt counts. Demonstrates Bayesian parameter inference and posterior predictive checks. docs.
• Lorenz dynamics - Studies a deterministic nonlinear convection model whose nearby initial states can diverge. Connects nonperiodic dynamics to limits of long-range prediction. Lorenz 1963.
• Bass diffusion - Models initial product adoption using innovative and imitative behavior: purchase timing depends on previous buyers. The original study tests the model on consumer durables. Bass 1969.
• Bass model retrospective - Reviews the origins, applications and extensions of the product-diffusion model, identifying subsequent research directions. Bass 2004.
• SIR epidemic dynamics - Models infection and removal in a susceptible population under simplifying assumptions about contact and immunity. Epidemics can end while susceptible individuals remain. Kermack and McKendrick 1927.
• DICE - Describes an integrated climate-economy model for dynamic optimization of climate policy. Documents the structure and assumptions behind its cost-benefit analysis. Nordhaus 1992.

Evaluation

• Conformal prediction overview - Reviews calibration-score methods for prediction sets with finite-sample marginal coverage under exchangeability. This guarantee does not imply coverage at each individual input. Angelopoulos and Bates 2021.
• Forecast evaluation - Compare against naïve and seasonal-naïve baselines. Evaluate held-out point forecasts and distributions separately; scaled errors support comparisons across series with different units. Hyndman and Athanasopoulos 2021.
• Rolling-origin validation - Repeatedly trains on past observations and evaluates later ones. Use forecast horizons matching deployment; fitting residuals do not measure future forecast accuracy. Tashman 2000, Hyndman and Athanasopoulos 2021.

Software

• rpart - R implementation of recursive partitioning for classification, regression and survival trees, based on CART. code.
• party - R toolbox for conditional-inference trees, forests and model-based recursive partitioning. Supports response types including censored and multivariate outcomes. code.
• mgcv - R implementation of generalized additive and additive mixed models, with smoothing-parameter estimation and several response-distribution families. code.

Related topics

• Neural Networks
• Ensemble Learning
• Bayesian Networks
• Interpretability
• Optimization
• Clustering