Most regression models give you a single number back. A Gaussian Process gives you a number and a confidence interval around it — and for a lot of real engineering problems, that second part is the one that actually matters.
The problem
I worked on predicting the heating load (HL) and cooling load (CL) of residential buildings from eight structural inputs — relative compactness, surface area, wall area, roof area, overall height, orientation, glazing area, and glazing area distribution. The dataset (768 buildings, from the UCI repository) is small and the relationships between inputs are non-trivial, which makes it a good candidate for a non-parametric model that doesn’t assume a fixed functional form up front.
Why a GP instead of a point estimate
Random forests and gradient boosting will happily give you a heating-load prediction for a building shape they’ve never seen — with no signal that they’re extrapolating. A GP posterior widens naturally in regions with less training support, which means the uncertainty itself is informative: it tells you when to trust the number and when to go get more data.
Using GPy, I trained two independent GPs — one per target — with an RBF kernel using Automatic Relevance Determination (ARD), so each input dimension gets its own lengthscale. Shorter lengthscales mean the model is more sensitive to that variable.
ker1 = GPy.kern.RBF(8, ARD=True) + GPy.kern.White(8)
m1 = GPy.models.GPRegression(X_train, y_train[['Y1']], ker1)
m1.optimize(messages=True, max_f_eval=1000)What the lengthscales revealed
Orientation and glazing-area distribution came out with the largest lengthscales for both targets — meaning the model was least sensitive to them, the opposite of what I expected going in. Compactness, surface area, wall area, and height carried roughly equal weight. That’s a useful sanity check before trusting a model in production: if the “important” variables it settles on don’t match domain intuition, that’s worth digging into before shipping.
Kernel search beat the RBF baseline
The RBF+White baseline was solid but not the best available. I swept combinations of linear, periodic, and RBF kernels (added and multiplied) and re-fit for both targets. A linear × periodic kernel cut heating-load MAE and MSE by roughly 55% and 74% relative to baseline; for cooling load, linear + RBF gave the best result. There was no single winning kernel across both targets — which is itself the finding: don’t assume one covariance structure fits every output of a multi-target problem, even when the outputs are physically related.
Full notebook: GPs.md