Adaptive Ridge Parameter Tuning for Large-Scale Regression with Expensive Validation

• Date: 12. 11. 2026
• Time: 14:00 - 15:30
• Address:
  Sokolovská 83, Praha
• Room: K3
• Speaker: Andreas Stathopoulos

Ridge regression requires tuning a regularization parameter that balances data fidelity against sensitivity to noise. At large scale the problem is typically addressed through iterative methods that access the matrix through matrix vector products. Still, this tuning can be considerably more expensive than solving a single least-squares problem, since many candidate regularization parameters may need to be considered and evaluated. Moreover, in applications such as cross-validation, the evaluation itself can be expensive since it involves a large validation set that is set aside from training or a user provided evaluation function. We present an adaptive method for tuning the ridge parameter while minimizing both solver and evaluation costs. The method exploits the shift invariance of Krylov subspaces to approximate solutions for multiple ridge parameters from a common subspace. Rather than prescribing a fixed set of shifts or solving all candidate systems to a common accuracy, it progressively determines which shifts are sufficiently converged to provide reliable evaluations. Inexpensive residual- and training-based quantities are used to estimate evaluation convergence, with optional validation subsampling when training and validation behavior differ. Exact evaluations are performed selectively as the search progresses, and local optimization and stopping criteria are used to avoid unnecessary Krylov iterations and validation evaluations. We illustrate the method on kernel ridge regression and a range of standard least-squares test problems, examining the spectral and noise regimes that determine its effectiveness.