Is there any way to choose a priori parameters to implement some kind of transfer learning in MLR(Hyperopt) Bayesian optimization by Gaussian process? My intention is to use the last month hyperparameter tuning experiments of my ML models as priors to the gaussian process fit today (like in Google Vizier bayes optim)
About Google Vizier (link to the paper) Transfer Learning:
When doing black-box optimization, users often run studies that are similar to studies they have run before, and we can use this fact to minimize repeated work. Vizier supports a form of Transfer Learning which leverages data from prior studies to guide and accelerate the current study. For instance, one might tune the learning rate and regularization of a machine learning system, then use that Study as a prior to tune the same ML system on a different data set. [...] our strategy is to build a stack of Gaussian Process regressors, where each regressor is associated with a study, and where each level is trained on the residuals relative to the regressor below it. Our model is that the studies were performed in a linear sequence, each study using the studies before it as priors.
The bottom of the stack contains a regressor built using data from the oldest study in the stack. The regressor above it is associated with the 2nd oldest study, and regresses on the residual of its objective relative to the predictions of the regressor below it. Similarly, the regressor associated with the i ^{th} study is built using the data from that study, and regresses on the residual of the objective with respect to the predictions of the regressor below it.
Comments
Post a Comment