AttributeError: 'Sum' object has no attribute 'gradient_x' when using sci-kit learn RBF for sci-kit optimize gp_minimize

[mahdiabdollahpour]

I still get error when using sci-kit learn RBF for sci-kit optimize gp_minimize, any idea how to solve it?

here is the code to reproduce it

from numpy import mean
from pandas import read_csv
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.svm import SVC
from skopt.space import Integer
from skopt.space import Real
from skopt.space import Categorical
from skopt.utils import use_named_args
from skopt import gp_minimize
from skopt import gp_minimize, dummy_minimize
from sklearn.gaussian_process.kernels import RBF


import time
import numpy as np
from skopt.learning import GaussianProcessRegressor
from skopt.learning.gaussian_process.kernels import Matern

# define the space of hyperparameters to search
search_space = list()
# search_space.append(Real(1e-6, 100.0, 'log-uniform', name='C'))
# search_space.append(Categorical(['linear', 'poly', 'rbf', 'sigmoid'], name='kernel'))
# search_space.append(Integer(1, 5, name='degree'))
search_space.append(Real(1e-6, 100.0, 'log-uniform', name='gamma'))

other_kernel = RBF(length_scale=1.0)
base_estimator = GaussianProcessRegressor(
    kernel= other_kernel,
    normalize_y=True, noise="gaussian",
    n_restarts_optimizer=0)

# define the function used to evaluate a given configuration
@use_named_args(search_space)
def evaluate_model(**params):

    return 0.7


# load dataset
url = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/ionosphere.csv'
dataframe = read_csv(url, header=None)
# split into input and output elements
data = dataframe.values
X, y = data[:, :-1], data[:, -1]
print(X.shape, y.shape)
# perform optimization
result = gp_minimize(evaluate_model, search_space,base_estimator=base_estimator)
# summarizing finding:
print('Best Accuracy: %.3f' % (1.0 - result.fun))
print('Best Parameters: %s' % (result.x))

The error log

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-5-795400b0eace>](https://localhost:8080/#) in <cell line: 48>()
     46 print(X.shape, y.shape)
     47 # perform optimization
---> 48 result = gp_minimize(evaluate_model, search_space,base_estimator=base_estimator)
     49 # summarizing finding:
     50 print('Best Accuracy: %.3f' % (1.0 - result.fun))

18 frames
[/usr/local/lib/python3.10/dist-packages/skopt/learning/gaussian_process/gpr.py](https://localhost:8080/#) in predict(self, X, return_std, return_cov, return_mean_grad, return_std_grad)
    344 
    345             if return_mean_grad:
--> 346                 grad = self.kernel_.gradient_x(X[0], self.X_train_)
    347                 grad_mean = np.dot(grad.T, self.alpha_)
    348                 # undo normalisation

AttributeError: 'Sum' object has no attribute 'gradient_x'

package versions

scikit-learn==1.2.2
scikit-optimize==0.9.0

Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] on linux

[alipsgh]

@mahdiabdollahpour try

from skopt.learning.gaussian_process.kernels import RBF

instead of

from sklearn.gaussian_process.kernels import RBF

It seems that Scikit-Optimize has a wrapper that implements gradient_x: https://github.com/scikit-optimize/scikit-optimize/blob/master/skopt/learning/gaussian_process/kernels.py#L68

[mahdiabdollahpour]

@mahdiabdollahpour try

from skopt.learning.gaussian_process.kernels import RBF

instead of

from sklearn.gaussian_process.kernels import RBF

It seems that Scikit-Optimize has a wrapper that implements gradient_x: https://github.com/scikit-optimize/scikit-optimize/blob/master/skopt/learning/gaussian_process/kernels.py#L68

It solves the problem.

[alipsgh]

You may want to change the default Kernel from Matern to RBF (or any other) by editing the utils file: https://github.com/scikit-optimize/scikit-optimize/blob/master/skopt/utils.py#L380

if is_cat:
    other_kernel = HammingKernel(length_scale=np.ones(n_dims))
else:
    # other_kernel = Matern(
    #     length_scale=np.ones(n_dims),
    #     length_scale_bounds=[(0.01, 100)] * n_dims, nu=2.5)
    other_kernel = RBF(length_scale=np.ones(n_dims), length_scale_bounds=[(0.01, 100)] * n_dims)