At vectordb.py, llamaindex UpstageEmbedding class is instance of OpenAIEmbedding.
So it tires to truncate token to 8000, but tiktoken cannot tokenize it because solar embedding model is not capable of tokenize in tiktoken.
Add model_name condition at truncate openai token also.
Just change colab demo to BAAI
Marker-Inc-Korea / AutoRAG
AutoML tool for RAG
1.7k
How does funding with Polar work?
1
Pay now to fund the work behind this issue.
2
Get updates on progress being made.
3
Maintainer is rewarded once the issue is completed.
FAQ
Backer
You're funding impactful open source efforts
Contributor
You want to contribute to this effort
Maintainer
You want to get funding like this too