From now, we tokenize each answers using 'gpt2' tokenizer, and it is not selectable.
For more accuracy, we have to tokenize each generator results using corresponding tokenizer to LLM models.
It can be useful to use tokenlog library.
But have to implement LlamaIndex support at tokenlog library.
Last question is...
Is it worth it?
Marker-Inc-Korea / AutoRAG
AutoML tool for RAG
1.7k
How does funding with Polar work?
1
Pay now to fund the work behind this issue.
2
Get updates on progress being made.
3
Maintainer is rewarded once the issue is completed.
FAQ
Backer
You're funding impactful open source efforts
Contributor
You want to contribute to this effort
Maintainer
You want to get funding like this too