"generate_test" function with Bedrock client (for Claud03)

[MelissaMokhtari]

Issue Type

Bug

Source

source

Giskard Library Version

2.11.0

Giskard Hub Version

1.3.0

OS Platform and Distribution

Linux 5.10.215-203.850.amzn2.x86_64 x86_64

Python version

3.10.14

Installed python packages

accelerate==0.30.1
aiohttp==3.9.5
aiosignal==1.3.1
albumentations==1.4.7
amazon-textract-caller==0.2.3
amazon-textract-response-parser==1.0.2
amazon-textract-textractor==1.7.11
aniso8601==9.0.1
annotated-types==0.6.0
ansi2html==1.9.1
anthropic==0.23.1
anyio==4.3.0
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
asttokens==2.4.1
async-lru==2.0.4
async-timeout==4.0.3
attrs==23.2.0
autovizwidget==0.21.0
awscli==1.32.101
Babel==2.14.0
beautifulsoup4==4.12.3
bert-score==0.3.13
bitsandbytes==0.43.1
bleach==6.1.0
blinker==1.7.0
blis==0.7.11
bokeh==3.4.0
boto3==1.34.106
botocore==1.34.106
Brotli==1.1.0
cached-property==1.5.2
captum==0.6.0
catalogue==2.0.10
certifi==2024.2.2
cffi==1.16.0
chardet==5.2.0
charset-normalizer==3.3.2
chromedriver-autoinstaller==0.6.4
click==8.1.7
cloudpathlib==0.16.0
cloudpickle==2.2.1
colorama==0.4.4
comm==0.2.2
confection==0.1.4
contextlib2==21.6.0
contourpy==1.2.0
cryptography==42.0.5
cssselect==1.2.0
cycler==0.12.1
cymem==2.0.8
dataclasses-json==0.6.6
datasets==2.19.1
debugpy==1.8.1
decorator==5.1.1
defusedxml==0.7.1
Deprecated==1.2.14
dill==0.3.8
dirtyjson==1.0.8
distro==1.9.0
docker==6.1.3
docopt==0.6.2
docutils==0.16
dparse==0.6.3
editdistance==0.8.1
entrypoints==0.4
environment-kernels==1.2.0
evaluate==0.4.2
exceptiongroup==1.2.0
executing==2.0.1
faiss-cpu==1.8.0
fastjsonschema==2.19.1
feedfinder2==0.0.4
feedparser==6.0.11
filelock==3.13.3
Flask==3.0.2
Flask-RESTful==0.3.10
fonttools==4.50.0
fqdn==1.5.1
frozenlist==1.4.1
fsspec==2024.3.1
giskard==2.11.0
gitdb==4.0.11
GitPython==3.1.43
gmpy2==2.1.2
google-pasta==0.2.0
greenlet==3.0.3
griffe==0.45.0
gssapi==1.8.3
gym==0.26.2
gym-notices==0.0.8
h11==0.14.0
h2==4.1.0
hdijupyterutils==0.21.0
hpack==4.0.0
html2text==2020.1.16
httpcore==1.0.4
httpx==0.27.0
huggingface-hub==0.23.0
hyperframe==6.0.1
idna==3.6
imageio==2.34.0
importlib-metadata==6.11.0
importlib_resources==6.4.0
ipykernel==6.29.3
ipython==8.22.2
ipywidgets==8.1.2
isoduration==20.11.0
itsdangerous==2.1.2
jedi==0.19.1
jieba3k==0.35.1
Jinja2==3.1.3
jmespath==1.0.1
joblib==1.3.2
json5==0.9.24
jsonpatch==1.33
jsonpath-ng==1.6.1
jsonpointer==2.4
jsonschema==4.21.1
jsonschema-specifications==2023.12.1
jupyter==1.0.0
jupyter_client==8.6.1
jupyter-console==6.6.3
jupyter_core==5.7.2
jupyter-events==0.10.0
jupyter-lsp==2.2.4
jupyter_server==2.13.0
jupyter_server_terminals==0.5.3
jupyterlab==4.1.5
jupyterlab_pygments==0.3.0
jupyterlab_server==2.25.4
jupyterlab_widgets==3.0.10
kiwisolver==1.4.5
krb5==0.5.1
langchain-community==0.0.38
langchain-core==0.1.52
langchain-openai==0.1.7
langcodes==3.4.0
langdetect==1.0.9
langsmith==0.1.59
language_data==1.2.0
lazy_loader==0.4
Levenshtein==0.25.1
lightning==2.2.4
lightning-utilities==0.11.2
llama-index==0.10.37
llama-index-agent-openai==0.2.5
llama-index-callbacks-openinference==0.1.4
llama-index-cli==0.1.12
llama-index-core==0.10.37
llama-index-embeddings-bedrock==0.1.5
llama-index-embeddings-huggingface==0.2.0
llama-index-embeddings-openai==0.1.9
llama-index-indices-managed-llama-cloud==0.1.6
llama-index-legacy==0.9.48
llama-index-llms-anthropic==0.1.11
llama-index-llms-bedrock==0.1.8
llama-index-llms-huggingface==0.2.0
llama-index-llms-openai==0.1.19
llama-index-multi-modal-llms-openai==0.1.6
llama-index-program-openai==0.1.6
llama-index-question-gen-openai==0.1.3
llama-index-readers-file==0.1.22
llama-index-readers-llama-parse==0.1.4
llama-index-readers-nougat-ocr==0.1.3
llama-index-readers-web==0.1.15
llama-parse==0.4.3
llamaindex-py-client==0.1.19
llvmlite==0.42.0
lxml==5.2.2
marisa-trie==1.1.1
Markdown==3.6
MarkupSafe==2.1.5
marshmallow==3.21.2
matplotlib==3.8.3
matplotlib-inline==0.1.6
minijinja==2.0.1
mistune==3.0.2
mixpanel==4.10.1
mlflow-skinny==2.12.2
mpi4py==3.1.5
mpmath==1.3.0
multidict==6.0.5
multiprocess==0.70.16
munch==4.0.0
munkres==1.1.4
murmurhash==1.0.10
mypy-extensions==1.0.0
nbclient==0.10.0
nbconvert==7.16.3
nbformat==5.10.3
nest_asyncio==1.6.0
networkx==3.2.1
newspaper3k==0.2.8
nltk==3.8.1
notebook==7.1.2
notebook_shim==0.2.4
nougat-ocr==0.1.17
num2words==0.5.13
numba==0.59.1
numpy==1.26.4
nvgpu==0.10.0
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==8.9.2.26
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-ml-py==12.535.133
nvidia-nccl-cu12==2.20.5
nvidia-nvjitlink-cu12==12.4.127
nvidia-nvtx-cu12==12.1.105
onnx==1.16.0
openai==1.30.1
opencv-python==4.9.0.80
opencv-python-headless==4.9.0.80
orjson==3.10.3
outcome==1.3.0.post0
overrides==7.7.0
packaging==23.2
pandas==1.5.3
pandocfilters==1.5.0
parso==0.8.3
pathos==0.3.2
patsy==0.5.6
pexpect==4.9.0
pickleshare==0.7.5
pillow==10.2.0
pip==24.0
pkgutil_resolve_name==1.3.10
platformdirs==4.2.0
playwright==1.43.0
plotly==5.20.0
ply==3.11
pox==0.3.4
ppft==1.7.6.8
preshed==3.0.9
prometheus_client==0.20.0
prompt-toolkit==3.0.42
protobuf==4.25.3
psutil==5.9.8
ptyprocess==0.7.0
pure-eval==0.2.2
py4j==0.10.9.5
pyarrow==15.0.2
pyarrow-hotfix==0.6
pyasn1==0.5.1
pybind11==2.11.1
pybind11-global==2.11.1
pycparser==2.21
pydantic==2.7.1
pydantic_core==2.18.2
pyee==11.1.0
pyfunctional==1.5.0
pygame==2.5.2
Pygments==2.17.2
pynndescent==0.5.12
pynvml==11.5.0
pyparsing==3.1.2
pypdf==4.2.0
pypdfium2==4.30.0
PySocks==1.7.1
pyspark==3.3.0
pyspnego==0.10.2
python-dateutil==2.9.0
python-dotenv==1.0.1
python-json-logger==2.0.7
python-Levenshtein==0.25.1
pytorch-lightning==2.2.4
pytz==2024.1
PyYAML==6.0.1
pyzmq==25.1.2
qtconsole==5.5.1
QtPy==2.4.1
rapidfuzz==3.9.0
referencing==0.34.0
regex==2024.5.15
requests==2.31.0
requests-file==2.0.0
requests-kerberos==0.14.0
requests-toolbelt==1.0.0
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rpds-py==0.18.0
rsa==4.7.2
ruamel.yaml==0.18.6
ruamel.yaml.clib==0.2.8
s3fs==0.4.2
s3transfer==0.10.1
safetensors==0.4.3
sagemaker==2.219.0
sagemaker_pyspark==1.4.5
schema==0.7.5
scikit-image==0.23.2
scikit-learn==1.4.1.post1
scipy==1.11.4
sconf==0.2.5
seaborn==0.13.2
selenium==4.21.0
Send2Trash==1.8.2
sentence-transformers==2.7.0
sentencepiece==0.2.0
sentry-sdk==2.2.0
seqeval==1.2.2
setuptools==69.2.0
sgmllib3k==1.0.0
shap==0.40.0
six==1.16.0
slicer==0.0.7
smart-open==6.4.0
smclarify==0.5
smdebug-rulesconfig==1.0.1
smmap==5.0.1
sniffio==1.3.1
sortedcontainers==2.4.0
soupsieve==2.5
spacy==3.7.4
spacy-legacy==3.0.12
spacy-loggers==1.0.5
span-marker==1.5.0
sparkmagic==0.21.0
spider-client==0.0.11
SQLAlchemy==2.0.30
sqlparse==0.5.0
srsly==2.4.8
stack-data==0.6.2
statsmodels==0.14.1
striprtf==0.0.26
sympy==1.12
tabulate==0.9.0
tblib==3.0.0
tenacity==8.2.3
termcolor==2.4.0
terminado==0.18.1
text-generation==0.7.0
thinc==8.2.3
threadpoolctl==3.4.0
tifffile==2024.5.10
tiktoken==0.7.0
timm==0.5.4
tinycss2==1.2.1
tinysegmenter==0.3
tldextract==5.1.2
tokenizers==0.19.1
tomli==2.0.1
torch==2.3.0
torch-model-archiver==0.7.1b20230208
torch-workflow-archiver==0.2.12b20240314
torchaudio==2.1.0
torchdata==0.7.0
torchmetrics==1.4.0.post0
torchserve==0.8.2b20230828
torchtext==0.16.0
torchvision==0.16.0
tornado==6.4
tqdm==4.66.2
traitlets==5.14.2
transformers==4.40.2
trio==0.25.1
trio-websocket==0.11.1
triton==2.3.0
typer==0.9.4
types-python-dateutil==2.9.0.20240316
typing_extensions==4.10.0
typing-inspect==0.9.0
typing-utils==0.1.0
tzdata==2024.1
ujson==5.9.0
umap-learn==0.5.6
unicodedata2==15.1.0
uri-template==1.3.0
urllib3==2.2.1
wasabi==1.1.2
wcwidth==0.2.13
weasel==0.3.4
webcolors==1.13
webencodings==0.5.1
websocket-client==1.7.0
Werkzeug==3.0.1
wheel==0.43.0
widgetsnbextension==4.0.10
wrapt==1.16.0
wsproto==1.2.0
XlsxWriter==3.2.0
xxhash==3.4.1
xyzservices==2023.10.1
yarl==1.9.4
zipp==3.17.0
zstandard==0.22.0

Current Behaviour?

A bug happened!
When launching the "generate_testset" after defining the bedrock Client for Claud03, I get an error saying:  "MainThread giskard.rag  ERROR    Encountered error in question generation: An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row. Skipping.
2024-05-14 08:10:06,421 pid:12762 MainThread giskard.rag  ERROR    An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row"
And just under it a "KeyError: "None of ['id'] are in the columns"" 
PS: I don't use giskard hub

Standalone code OR list down the steps to reproduce the issue

import giskard
from giskard.llm.client.bedrock import ClaudeBedrockClient
import boto3
bedrock_runtime= boto3.client("bedrock-runtime",
                              aws_access_key_id=aws_access_key_id,
                              aws_secret_access_key=aws_secret_access_key,
                               region_name=region_name)

bedrock_runtime_embed= boto3.client("bedrock-runtime",
                              aws_access_key_id=aws_access_key_id,
                              aws_secret_access_key=aws_secret_access_key,
                               region_name=region_name)

claud_client=ClaudeBedrockClient(bedrock_runtime, model="anthropic.claude-3-haiku-20240307-v1:0")#"anthropic.claude-3-sonnet-20240229-v1:0"
giskard.llm.set_default_client(claud_client)

from giskard.llm.embeddings.bedrock import BedrockEmbedding
embed_client=BedrockEmbedding(bedrock_runtime_embed, model="amazon.titan-embed-text-v1")
giskard.llm.embeddings.set_default_embedding(embed_client)
#Question generation

from giskard.rag import KnowledgeBase, generate_testset, QATestset
import pandas as pd

text_nodes = splitter(documents)
knowledge_base_df = pd.DataFrame([node.text for node in text_nodes], columns=["text"])
#knowledge_base_df["id"]=knowledge_base_df.index
knowledge_base = KnowledgeBase(knowledge_base_df)

testset = generate_testset(knowledge_base,
                           num_questions=3,
                           agent_description="TEST")#here the error

Relevant log output

2024-05-07 08:27:45,145 pid:12138 MainThread giskard.rag  INFO     Finding topics in the knowledge base.
2024-05-07 08:27:53,602 pid:12138 MainThread giskard.rag  INFO     Found 1 topics in the knowledge base.
Generating questions:   0%|          | 0/3 [00:00<?, ?it/s]
2024-05-07 08:27:54,321 pid:12138 MainThread giskard.rag  ERROR    Encountered error in question generation: An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row. Skipping.
2024-05-07 08:27:54,330 pid:12138 MainThread giskard.rag  ERROR    An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row
Traceback (most recent call last):
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/base.py", line 57, in generate_questions
    yield self.generate_single_question(knowledge_base, *args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/simple_questions.py", line 96, in generate_single_question
    generated_qa = self._llm_complete(messages=messages)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/base.py", line 42, in _llm_complete
    out = self._llm_client.complete(
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/llm/client/bedrock.py", line 72, in complete
    response = self._client.invoke_model(body=body, modelId=self.model, accept=accept, contentType=contentType)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py", line 565, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py", line 1021, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row
2024-05-07 08:27:54,530 pid:12138 MainThread giskard.rag  ERROR    Encountered error in question generation: An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row. Skipping.
2024-05-07 08:27:54,538 pid:12138 MainThread giskard.rag  ERROR    An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row
Traceback (most recent call last):
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/base.py", line 57, in generate_questions
    yield self.generate_single_question(knowledge_base, *args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/simple_questions.py", line 96, in generate_single_question
    generated_qa = self._llm_complete(messages=messages)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/base.py", line 42, in _llm_complete
    out = self._llm_client.complete(
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/llm/client/bedrock.py", line 72, in complete
    response = self._client.invoke_model(body=body, modelId=self.model, accept=accept, contentType=contentType)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py", line 565, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py", line 1021, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row
2024-05-07 08:27:54,722 pid:12138 MainThread giskard.rag  ERROR    Encountered error in question generation: An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row. Skipping.
2024-05-07 08:27:54,730 pid:12138 MainThread giskard.rag  ERROR    An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row
Traceback (most recent call last):
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/base.py", line 57, in generate_questions
    yield self.generate_single_question(knowledge_base, *args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/simple_questions.py", line 96, in generate_single_question
    generated_qa = self._llm_complete(messages=messages)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/base.py", line 42, in _llm_complete
    out = self._llm_client.complete(
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/llm/client/bedrock.py", line 72, in complete
    response = self._client.invoke_model(body=body, modelId=self.model, accept=accept, contentType=contentType)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py", line 565, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py", line 1021, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/tmp/ipykernel_12138/3725230573.py in ?()
----> 1 testset = generate_testset(knowledge_base,
      2                            num_questions=tuning_parameters["number_questions"],
      3                            agent_description="TEST")

~/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/testset_generation.py in ?(knowledge_base, num_questions, question_generators, language, agent_description)
     97             "question_generators": [qg.__class__.__name__ for qg in question_generators],
     98             "knowledge_base_size": len(knowledge_base._documents),
     99         },
    100     )
--> 101     return QATestset(questions)

~/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/testset.py in ?(self, question)
     43     def __init__(self, question: Sequence[QuestionSample]):
     44         self._questions = question
---> 45         self._dataframe = pd.DataFrame.from_records([question.to_dict() for question in self._questions]).set_index(
     46             "id"
     47         )

~/anaconda3/envs/python3/lib/python3.10/site-packages/pandas/core/frame.py in ?(self, keys, drop, append, inplace, verify_integrity)
   6105                     if not found:
   6106                         missing.append(col)
   6107 
   6108         if missing:
-> 6109             raise KeyError(f"None of {missing} are in the columns")
   6110 
   6111         if inplace:
   6112             frame = self

KeyError: "None of ['id'] are in the columns"

[kevinmessiaen]

Thanks for the report, this issue has been released in Giskard 2.13.0: #1937