LLM Provider

LLM provider is a top-level primitive in Arch, helping developers centrally define, secure, observe, and manage the usage of of their LLMs. Arch builds on Envoy’s reliable cluster subsystem to manage egress traffic to LLMs, which includes intelligent routing, retry and fail-over mechanisms, ensuring high availability and fault tolerance. This abstraction also enables developers to seamlessly switching between LLM providers or upgrade LLM versions, simplifying the integration and scaling of LLMs across applications.

Below is an example of how you can configure llm_providers with an instance of an Arch gateway.

Example Configuration
 1version: v0.1
 2
 3listener:
 4  address: 0.0.0.0 # or 127.0.0.1
 5  port: 10000
 6  # Defines how Arch should parse the content from application/json or text/pain Content-type in the http request
 7  message_format: huggingface
 8
 9# Centralized way to manage LLMs, manage keys, retry logic, failover and limits in a central way
10llm_providers:
11  - name: OpenAI
12    provider: openai
13    access_key: OPENAI_API_KEY
14    model: gpt-4o
15    default: true
16    stream: true
17
18# default system prompt used by all prompt targets
19system_prompt: You are a network assistant that just offers facts; not advice on manufacturers or purchasing decisions.

Note

When you start Arch, it creates a listener port for egress traffic based on the presence of llm_providers configuration section in the arch_config.yml file. Arch binds itself to a local address such as 127.0.0.1:51001/v1.

Arch also offers vendor-agnostic SDKs and libraries to make LLM calls to API-based LLM providers (like OpenAI, Anthropic, Mistral, Cohere, etc.) and supports calls to OSS LLMs that are hosted on your infrastructure. Arch abstracts the complexities of integrating with different LLM providers, providing a unified interface for making calls, handling retries, managing rate limits, and ensuring seamless integration with cloud-based and on-premise LLMs. Simply configure the details of the LLMs your application will use, and Arch offers a unified interface to make outbound LLM calls.

Example: Using the OpenAI Python SDK

from openai import OpenAI

# Initialize the Arch client
client = OpenAI(base_url="http://127.0.0.1:51001/v1")

# Define your LLM provider and prompt
llm_provider = "openai"
prompt = "What is the capital of France?"

# Send the prompt to the LLM through Arch
response = client.completions.create(llm_provider=llm_provider, prompt=prompt)

# Print the response
print("LLM Response:", response)