Describe the bug
When executing the Evaluator.start_trial
method in the autorag library, a RecursionError: maximum recursion depth exceeded
occurs. This issue arises specifically when using HuggingfaceEmbedding
with the BAAI/bge-m3
model for embedding, following the instructions in the README's "Step 3: Use Custom LLM & Embedding Model | Use Custom Model."
To Reproduce
Steps to reproduce the issue:
Evaluator
object, specifying qa_data_path
, corpus_data_path
, and project_dir
.evaluator.start_trial
method with a configuration file (config.yaml
).HuggingfaceEmbedding
with BAAI/bge-m3
, as described in the README's "Step 3: Use Custom LLM & Embedding Model | Use Custom Model."Expected behavior
The Evaluator
should successfully start the QA evaluation using the configuration provided, utilizing HuggingfaceEmbedding
with the BAAI/bge-m3
model, and return the evaluation results.
Full Error Log
Ingesting VectorDB... ββββββββββββββββββββββββββββββββββββββββ 100% 1/1 0:03:47
Evaluating... ββββββββββββββββββββββββββββββββββββββββ 0% 0/3 0:00:03
modules.json:β100%
β349/349β[00:00<00:00,β11.9kB/s]
config_sentence_transformers.json:β100%
β123/123β[00:00<00:00,β6.83kB/s]
README.md:β100%
β15.8k/15.8kβ[00:00<00:00,β1.01MB/s]
sentence_bert_config.json:β100%
β54.0/54.0β[00:00<00:00,β5.09kB/s]
config.json:β100%
β687/687β[00:00<00:00,β25.7kB/s]
tokenizer_config.json:β100%
β444/444β[00:00<00:00,β19.7kB/s]
sentencepiece.bpe.model:β100%
β5.07M/5.07Mβ[00:00<00:00,β48.7MB/s]
tokenizer.json:β100%
β17.1M/17.1Mβ[00:00<00:00,β44.4MB/s]
special_tokens_map.json:β100%
β964/964β[00:00<00:00,β99.2kB/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]Exception in thread Thread-12:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/IPython/core/formatters.py", line 224, in catch_format_error
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
0%| | 0/77 [00:00<?, ?it/s]
r = method(self, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/IPython/core/formatters.py", line 909, in __call__
printer = self.lookup(obj)
File "/usr/local/lib/python3.10/dist-packages/IPython/core/formatters.py", line 392, in lookup
if obj_id in self.singleton_printers:
File "/usr/local/lib/python3.10/dist-packages/traitlets/traitlets.py", line 700, in __get__
return self.get(obj, cls)
RecursionError: maximum recursion depth exceeded
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.10/dist-packages/rich/live.py", line 32, in run
self.live.refresh()
File "/usr/local/lib/python3.10/dist-packages/rich/live.py", line 237, in refresh
with self.ipy_widget:
File "/usr/local/lib/python3.10/dist-packages/ipywidgets/widgets/widget_output.py", line 109, in __enter__
self._flush()
File "/usr/local/lib/python3.10/dist-packages/ipywidgets/widgets/widget_output.py", line 132, in _flush
sys.stderr.flush()
File "/usr/local/lib/python3.10/dist-packages/rich/file_proxy.py", line 53, in flush
self.__console.print(output)
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 1678, in print
with self:
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 864, in __exit__
self._exit_buffer()
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 822, in _exit_buffer
self._check_buffer()
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 2019, in _check_buffer
self._write_buffer()
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 2035, in _write_buffer
display(self._buffer, self._render_buffer(self._buffer[:]))
File "/usr/local/lib/python3.10/dist-packages/rich/jupyter.py", line 91, in display
ipython_display(jupyter_renderable)
File "/usr/local/lib/python3.10/dist-packages/IPython/core/display.py", line 327, in display
publish_display_data(data=format_dict, metadata=md_dict, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/IPython/core/display.py", line 119, in publish_display_data
display_pub.publish(
File "/usr/local/lib/python3.10/dist-packages/ipykernel/zmqshell.py", line 115, in publish
self._flush_streams()
File "/usr/local/lib/python3.10/dist-packages/ipykernel/zmqshell.py", line 83, in _flush_streams
sys.stderr.flush()
File "/usr/local/lib/python3.10/dist-packages/rich/file_proxy.py", line 53, in flush
self.__console.print(output)
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 1678, in print
with self:
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 864, in __exit__
self._exit_buffer()
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 822, in _exit_buffer
self._check_buffer()
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 2019, in _check_buffer
self._write_buffer()
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 2035, in _write_buffer
display(self._buffer, self._render_buffer(self._buffer[:]))
File "/usr/local/lib/python3.10/dist-packages/rich/jupyter.py", line 91, in display
ipython_display(jupyter_renderable)
File "/usr/local/lib/python3.10/dist-packages/IPython/core/display.py", line 327, in display
publish_display_data(data=format_dict, metadata=md_dict, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/IPython/core/display.py", line 119, in publish_display_data
display_pub.publish(
File "/usr/local/lib/python3.10/dist-packages/ipykernel/zmqshell.py", line 115, in publish
self._flush_streams()
File "/usr/local/lib/python3.10/dist-packages/ipykernel/zmqshell.py", line 83, in _flush_streams
sys.stderr.flush()
File "/usr/local/lib/python3.10/dist-packages/rich/file_proxy.py", line 53, in flush
self.__console.print(output)
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 1678, in print
with self:
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 864, in __exit__
self._exit_buffer()
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 822, in _exit_buffer
self._check_buffer()
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 2019, in _check_buffer
self._write_buffer()
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 2035, in _write_buffer
display(self._buffer, self._render_buffer(self._buffer[:]))
File "/usr/local/lib/python3.10/dist-packages/rich/jupyter.py", line 91, in display
ipython_display(jupyter_renderable)
File "/usr/local/lib/python3.10/dist-packages/IPython/core/display.py", line 327, in display
publish_display_data(data=format_dict, metadata=md_dict, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/IPython/core/display.py", line 119, in publish_display_data
display_pub.publish(
File "/usr/local/lib/python3.10/dist-packages/ipykernel/zmqshell.py", line 115, in publish
self._flush_streams()
File "/usr/local/lib/python3.10/dist-packages/ipykernel/zmqshell.py", line 83, in _flush_streams
sys.stderr.flush()
File "/usr/local/lib/python3.10/dist-packages/rich/file_proxy.py", line 53, in flush
self.__console.print(output)
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 1678, in print
with self:
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 864, in __exit__
self._exit_buffer()
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 822, in _exit_buffer
self._check_buffer()
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 2019, in _check_buffer
self._write_buffer()
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 2035, in _write_buffer
display(self._buffer, self._render_buffer(self._buffer[:]))
File "/usr/local/lib/python3.10/dist-packages/rich/jupyter.py", line 91, in display
ipython_display(jupyter_renderable)
File "/usr/local/lib/python3.10/dist-packages/IPython/core/display.py", line 327, in display
publish_display_data(data=format_dict, metadata=md_dict, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/IPython/core/display.py", line 119, in publish_display_data
display_pub.publish(
File "/usr/local/lib/python3.10/dist-packages/ipykernel/zmqshell.py", line 115, in publish
self._flush_streams()
File "/usr/local/lib/python3.10/dist-packages/ipykernel/zmqshell.py", line 83, in _flush_streams
sys.stderr.flush()
File "/usr/local/lib/python3.10/dist-packages/rich/file_proxy.py", line 53, in flush
self.__console.print(output)
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 1678, in print
with self:
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 864, in __exit__
self._exit_buffer()
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 822, in _exit_buffer
self._check_buffer()
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 2019, in _check_buffer
self._write_buffer()
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 2035, in _write_buffer
display(self._buffer, self._render_buffer(self._buffer[:]))
File "/usr/local/lib/python3.10/dist-packages/rich/jupyter.py", line 91, in display
ipython_display(jupyter_renderable)
File "/usr/local/lib/python3.10/dist-packages/IPython/core/display.py", line 327, in display
publish_display_data(data=format_dict, metadata=md_dict, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/IPython/core/display.py", line 119, in publish_display_data
display_pub.publish(
File "/usr/local/lib/python3.10/dist-packages/ipykernel/zmqshell.py", line 115, in publish
self._flush_streams()
File "/usr/local/lib/python3.10/dist-packages/ipykernel/zmqshell.py", line 83, in _flush_streams
sys.stderr.flush()
File "/usr/local/lib/python3.10/dist-packages/rich/file_proxy.py", line 53, in flush
self.__console.print(output)
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 1678, in print
with self:
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 864, in __exit__
self._exit_buffer()
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 822, in _exit_buffer
---------------------------------------------------------------------------
RecursionError Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/IPython/core/formatters.py](https://localhost:8080/#) in catch_format_error(method, self, *args, **kwargs)
223 try:
--> 224 r = method(self, *args, **kwargs)
225 except NotImplementedError:
28 frames
RecursionError: maximum recursion depth exceeded
During handling of the above exception, another exception occurred:
RecursionError Traceback (most recent call last)
... last 11 frames repeated, from the frame below ...
[/usr/local/lib/python3.10/dist-packages/rich/file_proxy.py](https://localhost:8080/#) in flush(self)
51 output = "".join(self.__buffer)
52 if output:
---> 53 self.__console.print(output)
54 del self.__buffer[:]
55
RecursionError: maximum recursion depth exceeded in __instancecheck__
self._check_buffer()
File "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 2019, in _check_buffer
self._write_buffer()
Code where the bug occurred
import autorag
from autorag import LazyInit
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
autorag.embedding_models['upstage'] = LazyInit(HuggingFaceEmbedding, model_name="BAAI/bge-m3")
from autorag.evaluator import Evaluator
evaluator = Evaluator(
qa_data_path='/content/eli5_data/qa_sample.parquet',
corpus_data_path='/content/eli5_data/corpus_sample.parquet',
project_dir='/content/project_dir'
)
evaluator.start_trial('/content/config.yaml')
AutoRAG YAML file (config.yaml):
%%writefile config.yaml
vectordb:
- name: chroma_upstage
db_type: chroma
client_type: persistent
collection_name: upstage
embedding_model: upstage
path: ${PROJECT_DIR}/resources/chroma
node_lines:
- node_line_name: retrieve_node_line
nodes:
- node_type: retrieval
strategy:
metrics: [retrieval_f1, retrieval_recall, retrieval_ndcg, retrieval_mrr]
top_k: 3
modules:
- module_type: vectordb
vectordb: chroma_upstage
- module_type: bm25
- module_type: hybrid_rrf
weight_range: (4,80)
- node_line_name: post_retrieve_node_line
nodes:
- node_type: prompt_maker
strategy:
metrics:
- metric_name: meteor
- metric_name: rouge
- metric_name: sem_score
embedding_model: upstage # Use upstage embedding model
modules:
- module_type: fstring
prompt: "Read the passages and answer the given question. \n Question: {query} \n Passage: {retrieved_contents} \n Answer : "
- node_type: generator
strategy:
metrics:
- metric_name: meteor
- metric_name: rouge
- metric_name: sem_score
embedding_model: upstage # Use upstage embedding model
modules:
- module_type: llama_index_llm
llm: upstage # Use upstage LLM
batch: 4
Desktop (please complete the following information):
Additional context
I used Step 3: Use Custom LLM & Embedding Model | Use Custom Model from the Colab Tutorial provided in your README.md, and in this step, I only modified the following code:
import autorag
from autorag import LazyInit
from llama_index.embeddings.upstage import UpstageEmbedding
autorag.embedding_models['upstage'] = LazyInit(UpstageEmbedding, model_name="solar-embedding-1-large")
to
import autorag
from autorag import LazyInit
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
autorag.embedding_models['upstage'] = LazyInit(HuggingFaceEmbedding, model_name="BAAI/bge-m3")
Pay now to fund the work behind this issue.
Get updates on progress being made.
Maintainer is rewarded once the issue is completed.
You're funding impactful open source efforts
You want to contribute to this effort
You want to get funding like this too