Prompts

Arch’s primary design point is to securely accept, process and handle prompts. To do that effectively, Arch relies on Envoy’s HTTP connection management, subsystem and its prompt handler subsystem engineered with purpose-built LLMs to implement critical functionality on behalf of developers so that you can stay focused on business logic.

Arch’s prompt handler subsystem interacts with the model subsytem through Envoy’s cluster manager system to ensure robust, resilient and fault-tolerant experience in managing incoming prompts.

Messages

Arch accepts messages directly from the body of the HTTP request in a format that follows the Hugging Face Messages API. This design allows developers to pass a list of messages, where each message is represented as a dictionary containing two key-value pairs:

Role: Defines the role of the message sender, such as “user” or “assistant”.

Content: Contains the actual text of the message.

Prompt Guard

Arch is engineered with Arch-Guard, an industry leading safety layer, powered by a compact and high-performimg LLM that monitors incoming prompts to detect and reject jailbreak attempts - ensuring that unauthorized or harmful behaviors are intercepted early in the process.

To add jailbreak guardrails, see example below:

Example Configuration

version: v0.1

listener:
  address: 0.0.0.0 # or 127.0.0.1
  port: 10000
  # Defines how Arch should parse the content from application/json or text/pain Content-type in the http request
  message_format: huggingface

# Centralized way to manage LLMs, manage keys, retry logic, failover and limits in a central way
llm_providers:
  - name: OpenAI
    provider: openai
    access_key: $OPENAI_API_KEY
    model: gpt-4o
    default: true
    stream: true

# default system prompt used by all prompt targets
system_prompt: You are a network assistant that just offers facts; not advice on manufacturers or purchasing decisions.

prompt_guards:
  input_guards:
    jailbreak:
      on_exception:
        message: Looks like you're curious about my abilities, but I can only provide assistance within my programmed parameters.

Note

As a roadmap item, Arch will expose the ability for developers to define custom guardrails via Arch-Guard, and add support for additional safety checks defined by developers and hazardous categories like, violent crimes, privacy, hate, etc. To offer feedback on our roadmap, please visit our github page

Prompt Targets

Once a prompt passes any configured guardrail checks, Arch processes the contents of the incoming conversation and identifies where to forwad the conversation to via its prompt target primitve. Prompt targets are endpoints that receive prompts that are processed by Arch. For example, Arch enriches incoming prompts with metadata like knowing when a user’s intent has changed so that you can build faster, more accurate RAG apps.

Configuring prompt_targets is simple. See example below:

Example Configuration

version: v0.1

listener:
  address: 0.0.0.0 # or 127.0.0.1
  port: 10000
  # Defines how Arch should parse the content from application/json or text/pain Content-type in the http request
  message_format: huggingface

# Centralized way to manage LLMs, manage keys, retry logic, failover and limits in a central way
llm_providers:
  - name: OpenAI
    provider: openai
    access_key: $OPENAI_API_KEY
    model: gpt-4o
    default: true
    stream: true

# default system prompt used by all prompt targets
system_prompt: You are a network assistant that just offers facts; not advice on manufacturers or purchasing decisions.

prompt_guards:
  input_guards:
    jailbreak:
      on_exception:
        message: Looks like you're curious about my abilities, but I can only provide assistance within my programmed parameters.

prompt_targets:
  - name: information_extraction
    default: true
    description: handel all scenarios that are question and answer in nature. Like summarization, information extraction, etc.
    endpoint:
      name: app_server
      path: /agent/summary
    # Arch uses the default LLM and treats the response from the endpoint as the prompt to send to the LLM
    auto_llm_dispatch_on_response: true
    # override system prompt for this prompt target
    system_prompt: You are a helpful information extraction assistant. Use the information that is provided to you.

  - name: reboot_network_device
    description: Reboot a specific network device
    endpoint:
      name: app_server
      path: /agent/action
    parameters:
      - name: device_id
        type: str
        description: Identifier of the network device to reboot.
        required: true
      - name: confirmation
        type: bool
        description: Confirmation flag to proceed with reboot.
        default: false
        enum: [true, false]

error_target:
  endpoint:
    name: error_target_1
    path: /error

# Arch creates a round-robin load balancing between different endpoints, managed via the cluster subsystem.
endpoints:
  app_server:
    # value could be ip address or a hostname with port
    # this could also be a list of endpoints for load balancing
    # for example endpoint: [ ip1:port, ip2:port ]
    endpoint: 127.0.0.1:80
    # max time to wait for a connection to be established
    connect_timeout: 0.005s

Intent Matching

Arch uses fast text embedding and intent recognition approaches to first detect the intent of each incoming prompt. This intent matching phase analyzes the prompt’s content and matches it against predefined prompt targets, ensuring that each prompt is forwarded to the most appropriate endpoint. Arch’s intent matching framework considers both the name and description of each prompt target, and uses a composite matching score between embedding similarity and intent classification scores to enchance accuracy in forwarding decisions.

Intent Recognition: NLI techniques further refine the matching process by evaluating the semantic alignment between the prompt and potential targets.
Text Embedding: By embedding the prompt and comparing it to known target vectors, Arch effectively identifies the closest match, ensuring that the prompt is handled by the correct downstream service.

Agentic Apps via Prompt Targets

To support agentic apps, like scheduling travel plans or sharing comments on a document - via prompts, Arch uses its function calling abilities to extract critical information from the incoming prompt (or a set of prompts) needed by a downstream backend API or function call before calling it directly. For more details on how you can build agentic applications using Arch, see our full guide here:

Note

Arch-Function is a collection of dedicated agentic models engineered in Arch to extract information from a (set of) prompts and executes necessary backend API calls. This allows for efficient handling of agentic tasks, such as scheduling data retrieval, by dynamically interacting with backend services. Arch-Function achieves state-of-the-art performance, comparable with frontier models like Claude Sonnet 3.5 ang GPT-4, while being 100x cheaper ($0.05M/token hosted) and 10x faster (p50 latencies of 200ms).

Prompting LLMs

Arch is a single piece of software that is designed to manage both ingress and egress prompt traffic, drawing its distributed proxy nature from the robust Envoy. This makes it extremely efficient and capable of handling upstream connections to LLMs. If your application is originating code to an API-based LLM, simply use the OpenAI client and configure it with Arch. By sending traffic through Arch, you can propagate traces, manage and monitor traffic, apply rate limits, and utilize a large set of traffic management capabilities in a centralized way.

Attention

When you start Arch, it automatically creates a listener port for egress calls to upstream LLMs. This is based on the llm_providers configuration section in the arch_config.yml file. Arch binds itself to a local address such as 127.0.0.1:12000.

Example: Using OpenAI Client with Arch as an Egress Gateway

import openai

# Set the OpenAI API base URL to the Arch gateway endpoint
openai.api_base = "http://127.0.0.1:12000"

# No need to set openai.api_key since it's configured in Arch's gateway

# Use the OpenAI client as usual
response = openai.Completion.create(
   model="text-davinci-003",
   prompt="What is the capital of France?"
)

print("OpenAI Response:", response.choices[0].text.strip())

In these examples, the OpenAI client is used to send traffic directly through the Arch egress proxy to the LLM of your choice, such as OpenAI. The OpenAI client is configured to route traffic via Arch by setting the proxy to 127.0.0.1:12000, assuming Arch is running locally and bound to that address and port. This setup allows you to take advantage of Arch’s advanced traffic management features while interacting with LLM APIs like OpenAI.