Listener

Listener is a top level primitive in Arch, which simplifies the configuration required to bind incoming connections from downstream clients, and for egress connections to LLMs (hosted or API)

Arch builds on Envoy’s Listener subsystem to streamline connection managemet for developers. Arch minimizes the complexity of Envoy’s listener setup by using best-practices and exposing only essential settings, making it easier for developers to bind connections without deep knowledge of Envoy’s configuration model. This simplification ensures that connections are secure, reliable, and optimized for performance.

Downstream (Ingress)

Developers can configure Arch to accept connections from downstream clients. A downstream listener acts as the primary entry point for incoming traffic, handling initial connection setup, including network filtering, gurdrails, and additional network security checks. For more details on prompt security and safety, see here.

Upstream (Egress)

Arch automatically configures a listener to route requests from your application to upstream LLM API providers (or hosts). When you start Arch, it creates a listener for egress traffic based on the presence of the listener configuration section in the configuration file. Arch binds itself to a local address such as 127.0.0.1:12000/v1 or a DNS-based address like arch.local:12000/v1 for outgoing traffic. For more details on LLM providers, read here.

Configure Listener

To configure a Downstream (Ingress) Listner, simply add the listener directive to your configuration file:

Example Configuration
 1version: v0.1
 2
 3listener:
 4  address: 0.0.0.0 # or 127.0.0.1
 5  port: 10000
 6  # Defines how Arch should parse the content from application/json or text/pain Content-type in the http request
 7  message_format: huggingface
 8
 9# Centralized way to manage LLMs, manage keys, retry logic, failover and limits in a central way
10llm_providers:
11  - name: OpenAI
12    provider: openai
13    access_key: $OPENAI_API_KEY
14    model: gpt-4o
15    default: true
16    stream: true
17
18# default system prompt used by all prompt targets