LLM Model Config
Configuration examples
OpenAI
For OpenAI configuration, create .env files containing the API key
# This file contains a configuration section relevant to LLM, not the entire config
llm:
type: openai
params:
prompt_template: |
Contex information is provided below. Given only the context and not prior knowledge, provide detailed answer to the question and references to the provided context. If answer isn't in the context, say you don't know.
When answering questions, take into consideration the history of the chat converastion, which is listed below under Chat History. The chat history is in reverse chronological order, so the most recent exhange is at the top.
### Context:
---------------------
{context}
---------------------
### Question: {question}
model_kwargs:
temperature: 0.0
model_name: gpt-4o-mini
An example of OpenAI gpt4o-mini is shown below:
# This file contains a configuration section relevant to LLM, not the entire config
llm:
type: openai
params:
prompt_template: |
Contex information is provided below. Given only the context and not prior knowledge, provide detailed answer to the question and references to the provided context. If answer isn't in the context, say you don't know.
When answering questions, take into consideration the history of the chat converastion, which is listed below under Chat History. The chat history is in reverse chronological order, so the most recent exhange is at the top.
### Context:
---------------------
{context}
---------------------
### Question: {question}
model_kwargs:
temperature: 0.0
model_name: gpt-4o-mini
Ollama + Litellm
# This file contains a configuration section relevant to LLM, not the entire config
llm:
type: openai
params:
prompt_template: |
Contex information is provided below. Given only the context and not prior knowledge, provide detailed answer to the question and references to the provided context. If answer isn't in the context, say you don't know.
### Context:
---------------------
{context}
---------------------
### Question: {question}
model_kwargs:
temperature: 0.2
model: "any"
api_key: "any"
base_url: "http://0.0.0.0:8000"
Huggingface
# This file contains a configuration section relevant to LLM, not the entire config
llm:
type: huggingface
params:
model_name: microsoft/phi-2 # tiiuae/falcon-7b-instruct
cache_folder: /storage/llm/cache
prompt_template: |
### Instruction:
Use the following pieces of context to answer the question at the end. If answer isn't in the context, say that you don't know, don't try to make up an answer.
### Context:
---------------
{context}
---------------
### Question: {question}
load_8bit: False
trust_remote_code: False
device: auto
pipeline_kwargs:
do_sample: True
max_new_tokens: 512
num_return_sequences: 1
model_kwargs:
do_sample: True
temperature: 0.01
Reference
- class llmsearch.models.config.AzureOpenAIModelConfig(*, prompt_template: str, deployment_name: str, model_name: str, model_kwargs: dict = {}, openai_api_type: str = 'azure', openai_api_version: str = '2023-05-15', openai_api_base: str)
- model_computed_fields = {}
A dictionary of computed field names and their corresponding ComputedFieldInfo objects.
- model_config = {'protected_namespaces': ()}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- model_fields = {'deployment_name': FieldInfo(annotation=str, required=True), 'model_kwargs': FieldInfo(annotation=dict, required=False, default={}), 'model_name': FieldInfo(annotation=str, required=True), 'openai_api_base': FieldInfo(annotation=str, required=True), 'openai_api_type': FieldInfo(annotation=str, required=False, default='azure'), 'openai_api_version': FieldInfo(annotation=str, required=False, default='2023-05-15'), 'prompt_template': FieldInfo(annotation=str, required=True)}
Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo].
This replaces Model.__fields__ from Pydantic V1.
- class llmsearch.models.config.HuggingFaceModelConfig(*, cache_folder: Path | str | None = None, tokenizer_name: str | None = None, model_name: str, prompt_template: str, load_8bit: bool = False, trust_remote_code: bool = False, tokenzier_kwargs: dict = {}, model_kwargs: dict = {}, pipeline_kwargs: dict = {})
- model_computed_fields = {}
A dictionary of computed field names and their corresponding ComputedFieldInfo objects.
- model_config = {'protected_namespaces': ()}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- model_fields = {'cache_folder': FieldInfo(annotation=Union[Path, str, NoneType], required=False), 'load_8bit': FieldInfo(annotation=bool, required=False, default=False), 'model_kwargs': FieldInfo(annotation=dict, required=False, default={}), 'model_name': FieldInfo(annotation=str, required=True), 'pipeline_kwargs': FieldInfo(annotation=dict, required=False, default={}), 'prompt_template': FieldInfo(annotation=str, required=True), 'tokenizer_name': FieldInfo(annotation=Union[str, NoneType], required=False), 'tokenzier_kwargs': FieldInfo(annotation=dict, required=False, default={}), 'trust_remote_code': FieldInfo(annotation=bool, required=False, default=False)}
Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo].
This replaces Model.__fields__ from Pydantic V1.
- class llmsearch.models.config.LlamaModelConfig(*, model_path: Path | str, prompt_template: str, model_init_params: dict = {}, model_kwargs: dict = {})
- model_computed_fields = {}
A dictionary of computed field names and their corresponding ComputedFieldInfo objects.
- model_config = {'protected_namespaces': ()}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- model_fields = {'model_init_params': FieldInfo(annotation=dict, required=False, default={}), 'model_kwargs': FieldInfo(annotation=dict, required=False, default={}), 'model_path': FieldInfo(annotation=Union[Path, str], required=True), 'prompt_template': FieldInfo(annotation=str, required=True)}
Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo].
This replaces Model.__fields__ from Pydantic V1.
- class llmsearch.models.config.OpenAIModelConfig(*, prompt_template: str, model_kwargs: dict = {})
- model_computed_fields = {}
A dictionary of computed field names and their corresponding ComputedFieldInfo objects.
- model_config = {'protected_namespaces': ()}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- model_fields = {'model_kwargs': FieldInfo(annotation=dict, required=False, default={}), 'prompt_template': FieldInfo(annotation=str, required=True)}
Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo].
This replaces Model.__fields__ from Pydantic V1.