Hey AutoRAG devwlopers,

Thanks for the great work on AutoRAG. It's been a valuable tool for optimizing RAG pipelines. I thought there are a couple of feature requests that could make it even more powerful and efficient:

1. Batch API Support:
   With OpenAI's recent release of the Batch API, there's an opportunity to reduce the cost of generating QA data from corpus data by up to 50%. Integrating Batch API support into AutoRAG would be good for users working with large datasets and limited budgets.

2. Incremental Data Addition/Subtraction:
   The current AutoRAG pipeline requires creating corpus data and generating QA data from it sequentially, as I understood. However, there are scenarios where users may have additional data to incorporate into an existing RAG pipeline, or need to remove or replace it(like renewed docs) It would be great to know if there are any plans or ezisting methods to support incremental data addition in the future or how this could be implemented.

These feature requests may require significant development efforts, but they would greatly enhance the usability and efficiency of AutoRAG. If there's anything that can be done to help or provide further clarification, please let me know.

Thanks for considering these feature requests. Looking forward to hearing your thoughts.