Config file¶

Index mapping¶

You can define each index in the schema block of the configuration:

schema:
  <your-index-name>:
    config:
      <index configuration>
    store:
      <store configuration>
    fields:
      <field definitions>

Index configuration¶

An example of index configuration:

schema:
  index-name:
    config:
      flush:
        duration: 5s # how frequently new segments are created
      hnsw:
        m: 16 # max number of node-node links in HNSW graph
        efc: 100 # beam width used while building the index
        workers: 8 # how many concurrent workers used for HNSW merge ops

Fields:

flush.duration: optional, duration, default 5s. Index writer will periodically produce flush index segments (if there are new documents) with this interval.
hnsw.m: optional, int, default 16. How many links should HNSW index have? Larger value means better recall, but higher memory usage and bigger index. Common values are within 16-128 range.
hnsw.efc: optional, int, default 100. How many neighbors in the HNSW graph are explored during indexing. Bigger the value, better the recall, but slower the indexing speed.
hnsw.workers: optional, int, default = number of CPU cores. How many concurrent workers to use for index merges.

Store configuration¶

TODO

Fields definitions¶

TODO

ML Inference¶

See ML Inference overview and RAG Search for an overview of use cases for inference models.

Embedding models¶

Example of a full configuration:

inference:
  embedding:
    your-model-name:
      provider: onnx
      model: nixiesearch/e5-small-v2-onnx
      file: model.onnx
      max_tokens: 512
      batch_size: 32
      prompt:
        query: "query: "
        doc: "passage: "

Fields:

provider: required, string. As for v0.3.0, only the onnx provider is supported.
model: required, string. A Huggingface handle, or an HTTP/Local/S3 URL for the model. See model URL reference for more details on how to load your model.
prompt: optional. A document and query prefixes for asymmetrical models.
file: optional, string, default is to pick a lexicographically first file. A file name of the model - useful when HF repo contains multiple versions of the same model.
max_tokens: optional, int, default 512. How many tokens from the input document to process. All tokens beyond the threshold are truncated.
batch_size: optional, int, default 32. Computing embeddings is a highly parallel task, and doing it in big chunks is much more effective than one by one. For CPUs there are usually no gains of batch sizes beyong 32, but on GPUs you can go up to 1024.

LLM competion models¶

Example of a full configuration:

inference:
  completion:
    your-model-name:
      provider: llamacpp
      model: Qwen/Qwen2-0.5B-Instruct-GGUF
      file: qwen2-0_5b-instruct-q4_0.gguf
      prompt: qwen2
      system: "You are a helpful assistant, answer only in haiku."
      options:
        n_threads: 8
        n_gpu_layers: 100
        n_parallel: 8
        cont_batching: true
        flash_attn: true
        use_mmap: true
        use_mlock: true
        no_kv_offload: false
        seed: 42

Fields:

provider: required, string. As for v0.3.0, only llamacpp is supported. Other SaaS providers like OpenAI, Cohere, mxb and Google are on the roadmap.
model: required, string. A Huggingface handle, or an HTTP/Local/S3 URL for the model. See model URL reference for more details on how to load your model.
file: optional, string. A file name for the model, if the target model has multiple. A typical case for quantized models.
prompt: required, string. A prompt format used for the LLM. See Supported LLM prompts for more details
system: optional, string, default empty. An optional system prompt to be prepended to all the user prompts.
options: optional, obj. A set of llama-cpp specific options. See llamacpp reference on options for more details.