Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

运行hypergbm/examples/01.quick-start.ipynb时,trial会报错cannot pickle '_hashlib.HASH' object #104

Open
calculatechen opened this issue Dec 1, 2023 · 3 comments

Comments

@calculatechen
Copy link

System information

  • OS Platform and Distribution : centos 7.6
  • Python version: 3.8.18
  • HyperGBM version: 0.3.0

Describe the current behavior
运行到下面这行代码时,
experiment = make_experiment(train_data.copy(), target='target', reward_metric='precision', pos_label=1,random_state=1234, max_trials=20)
会报错
Traceback (most recent call last):
File "/home/user1/anaconda3/envs/automl1/lib/python3.8/site-packages/hypernets/dispatchers/in_process_dispatcher.py", line 95, in dispatch
trial = hyper_model._run_trial(space_sample, trial_no, X, y, X_eval, y_eval, X_test, cv, num_folds, model_file,
File "/home/user1/anaconda3/envs/automl1/lib/python3.8/site-packages/hypernets/model/hyper_model.py", line 111, in _run_trial
estimator.save(model_file)
File "/home/user1/anaconda3/envs/automl1/lib/python3.8/site-packages/hypergbm/hyper_gbm.py", line 645, in save
pickle.dump(self, output, protocol=pickle.HIGHEST_PROTOCOL)
TypeError: cannot pickle '_hashlib.HASH' object
只有第一次trial能正常运行,之后的都会出现这个报错。但如果不用jupyter,而是用python运行时,上面内容就不会输出。我不清楚这个报错是否会对精度产生影响呢?

@lixfz
Copy link
Collaborator

lixfz commented Dec 3, 2023

感谢您的反馈!

我们已经在最新代码中修复这个问题,稍后会发布。

目前你可以暂时将HyperGBM和Hypernets升级到最新代码以避免在运行时出现‘cannot pickle '_hashlib.HASH' object’ 这样的错误。

例如:

 pip install  --force-reinstall --no-deps  git+https://github.com/DataCanvasIO/Hypernets
 pip install  --force-reinstall --no-deps  git+https://github.com/DataCanvasIO/HyperGBM

@gyhbest
Copy link

gyhbest commented Dec 10, 2023

安装了上面的新版本后。在jupyter里运行还是有问题。
运行test 代码:
experiment = make_experiment(train_data, target='target', reward_metric='precision',max_trials=20)
estimator = experiment.run()
报错:
2-10 21:54:28 E hypernets.m.hyper_model.py 103 - run_trail failed! trail_no=12
12-10 21:54:28 E hypernets.m.hyper_model.py 105 - Traceback (most recent call last):
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/hypernets/model/hyper_model.py", line 94, in _run_trial
ret_data = estimator.fit_cross_validation(X, y, stratified=True, num_folds=num_folds, shuffle=False,
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/hypergbm/hyper_gbm.py", line 263, in fit_cross_validation
X = self.fit_transform_data(X, y, verbose=verbose)
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/hypernets/tabular/cache.py", line 203, in _cache_call
result = fn(*args, **kwargs)
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/hypergbm/hyper_gbm.py", line 223, in fit_transform_data
X = self.data_pipeline.fit_transform(X, y)
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/hypernets/tabular/dataframe_mapper.py", line 313, in fit_transform
Xt = _call_fit(transformers.fit_transform, Xt, y)
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/hypernets/tabular/dataframe_mapper.py", line 39, in _call_fit
return fit_method(X, y, **kwargs)
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/sklearn/base.py", line 1152, in wrapper
return fit_method(estimator, *args, **kwargs)
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/sklearn/pipeline.py", line 471, in fit_transform
Xt = self._fit(X, y, **fit_params_steps)
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/sklearn/pipeline.py", line 377, in _fit
X, fitted_transformer = fit_transform_one_cached(
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/joblib/memory.py", line 353, in call
return self.func(*args, **kwargs)
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/sklearn/pipeline.py", line 957, in _fit_transform_one
res = transformer.fit_transform(X, y, **fit_params)
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
data_to_wrap = f(self, X, *args, **kwargs)
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/sklearn/base.py", line 919, in fit_transform
return self.fit(X, y, **fit_params).transform(X)
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/sklearn/base.py", line 1145, in wrapper
estimator._validate_params()
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/sklearn/base.py", line 640, in _validate_params
self.get_params(deep=False),
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/sklearn/base.py", line 195, in get_params
value = getattr(self, key)
AttributeError: ['mean radius', 'mean texture', 'mean perimeter', 'mean area', 'mean smoothness', 'mean compactness', 'mean concavity', 'mean concave points', 'mean symmetry', 'mean fractal dimension', 'radius error', 'texture error', 'perimeter error', 'area error', 'smoothness error', 'compactness error', 'concavity error', 'concave points error', 'symmetry error', 'fractal dimension error', 'worst radius', 'worst texture', 'worst perimeter', 'worst area', 'worst smoothness', 'worst compactness', 'worst concavity', 'worst concave points', 'worst symmetry', 'worst fractal dimension']: 'FloatOutputImputer' object has no attribute 'keep_empty_features'

12-10 21:54:34 W hypernets.t.cache.py 200 - AttributeError: ['mean radius', 'mean texture', 'mean perimeter', 'mean area', 'mean smoothness', 'mean compactness', 'mean concavity', 'mean concave points', 'mean symmetry', 'mean fractal dimension', 'radius error', 'texture error', 'perimeter error', 'area error', 'smoothness error', 'compactness error', 'concavity error', 'concave points error', 'symmetry error', 'fractal dimension error', 'worst radius', 'worst texture', 'worst perimeter', 'worst area', 'worst smoothness', 'worst compactness', 'worst concavity', 'worst concave points', 'worst symmetry', 'worst fractal dimension']: 'FloatOutputImputer' object has no attribute '_fit_dtype'
Traceback (most recent call last):
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/hypernets/tabular/cache.py", line 192, in _cache_call
result = tfn(*args[1:], **kwargs) # exclude args[0]==self
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/hypergbm/hyper_gbm.py", line 240, in transform_data
X = self.data_pipeline.transform(X)
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/hypernets/tabular/dataframe_mapper.py", line 267, in transform
Xt = transformers.transform(Xt)
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/sklearn/pipeline.py", line 696, in transform
Xt = transform.transform(Xt)
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
data_to_wrap = f(self, X, *args, **kwargs)
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/hypernets/tabular/sklearn_ex.py", line 619, in transform
return super().transform(X).astype(np.float64)
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
data_to_wrap = f(self, X, *args, **kwargs)
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/sklearn/impute/_base.py", line 530, in transform
X = self._validate_input(X, in_fit=False)
File "/user/anaconda3/envs/py39_test/lib/python3.9/site-packages/sklearn/impute/_base.py", line 305, in _validate_input
if not in_fit and self._fit_dtype.kind == "O":

@lixfz
Copy link
Collaborator

lixfz commented Dec 11, 2023

烦请提供一下您的运行环境及HyperGBM安装过程,谢谢
@gyhbest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants