Implementation

• Look into LiteLLM (OpenAI, Anthropic, OpenAI Azure support) + Ollama
• Add LLM connection setup to user settings + global default
• Replace OpenAI setup popup by LLM provider setup
• Set default LLM provider on convo create
• Add LLM provider selection drop down in conversations based on already setup providers + add more button (so people realize they can add more)
• Build an abstraction that takes as input an LLM provider (that's already setup)
• Add the current active LLM for current chat to QueryOptions and populate it in the conversation service
• Use the LLM provider in QueryOptions instead of the current OpenAI-specific implementation in the query graph

How do we manage which model is used in a new conversation?

• Global default -> set in user settings
• On creating new conversation, set the LLM to the global default
• User can change LLM per chat (dropdown like ChatGPT)
• Save LLM used per chat

What will the model abstraction look like?

 model = state.options.get_model()
 model.bind_tools(tools)
 model.invoke(messages)