Agentic Workflow

Arch helps you easily personalize your applications by calling application-specific (API) functions via user prompts. This involves any predefined functions or APIs you want to expose to users to perform tasks, gather information, or manipulate data. This capability is generally referred to as function calling, where you have the flexibility to support “agentic” apps tailored to specific use cases - from updating insurance claims to creating ad campaigns - via prompts.

Arch analyzes prompts, extracts critical information from prompts, engages in lightweight conversation with the user to gather any missing parameters and makes API calls so that you can focus on writing business logic. Arch does this via its purpose-built Arch-Function - the fastest (200ms p90 - 10x faser than GPT-4o) and cheapest (100x than GPT-4o) function calling LLM that matches performance with frontier models.

Single Function Call

In the most common scenario, users will request a single action via prompts, and Arch efficiently processes the request by extracting relevant parameters, validating the input, and calling the designated function or API. Here is how you would go about enabling this scenario with Arch:

Step 1: Define Prompt Targets

Prompt Target Example Configuration

version: v0.1
listener:
  address: 127.0.0.1
  port: 8080 #If you configure port 443, you'll need to update the listener with tls_certificates
  message_format: huggingface

# Centralized way to manage LLMs, manage keys, retry logic, failover and limits in a central way
llm_providers:
  - name: OpenAI
    provider: openai
    access_key: $OPENAI_API_KEY
    model: gpt-3.5-turbo
    default: true

# default system prompt used by all prompt targets
system_prompt: |
  You are a network assistant that just offers facts; not advice on manufacturers or purchasing decisions.

prompt_targets:
    - name: network_qa
      endpoint:
        name: app_server
        path: /agent/network_summary
      description: Handle general Q/A related to networking.
      default: true
    - name: reboot_devices
      description: Reboot specific devices or device groups
      endpoint:
        name: app_server
        path: /agent/device_reboot
      parameters:
        - name: device_ids
          type: list
          description: A list of device identifiers (IDs) to reboot.
          required: true
    - name: device_summary
      description: Retrieve statistics for specific devices within a time range
      endpoint:
        name: app_server
        path: /agent/device_summary
      parameters:
        - name: device_ids
          type: list
          description: A list of device identifiers (IDs) to retrieve statistics for.
          required: true  # device_ids are required to get device statistics
        - name: time_range
          type: int
          description: Time range in days for which to gather device statistics. Defaults to 7.
          default: 7

# Arch creates a round-robin load balancing between different endpoints, managed via the cluster subsystem.
endpoints:
  app_server:
    # value could be ip address or a hostname with port
    # this could also be a list of endpoints for load balancing
    # for example endpoint: [ ip1:port, ip2:port ]
    endpoint: host.docker.internal:18083
    # max time to wait for a connection to be established
    connect_timeout: 0.005s

Step 2: Process Request Parameters

Once the prompt targets are configured as above, handling those parameters is

Parameter handling with Flask

from flask import Flask, request, jsonify

app = Flask(__name__)


@app.route("/agent/device_summary", methods=["POST"])
def get_device_summary():
    """
    Endpoint to retrieve device statistics based on device IDs and an optional time range.
    """
    data = request.get_json()

    # Validate 'device_ids' parameter
    device_ids = data.get("device_ids")
    if not device_ids or not isinstance(device_ids, list):
        return (
            jsonify({"error": "'device_ids' parameter is required and must be a list"}),
            400,
        )

    # Validate 'time_range' parameter (optional, defaults to 7)
    time_range = data.get("time_range", 7)
    if not isinstance(time_range, int):
        return jsonify({"error": "'time_range' must be an integer"}), 400

    # Simulate retrieving statistics for the given device IDs and time range
    # In a real application, you would query your database or external service here
    statistics = []
    for device_id in device_ids:
        # Placeholder for actual data retrieval
        stats = {
            "device_id": device_id,
            "time_range": f"Last {time_range} days",
            "data": f"Statistics data for device {device_id} over the last {time_range} days.",
        }
        statistics.append(stats)

    response = {"statistics": statistics}

    return jsonify(response), 200


if __name__ == "__main__":
    app.run(debug=True)

Parallel & Multiple Function Calling

In more complex use cases, users may request multiple actions or need multiple APIs/functions to be called simultaneously or sequentially. With Arch, you can handle these scenarios efficiently using parallel or multiple function calling. This allows your application to engage in a broader range of interactions, such as updating different datasets, triggering events across systems, or collecting results from multiple services in one prompt.

Arch-FC1B is built to manage these parallel tasks efficiently, ensuring low latency and high throughput, even when multiple functions are invoked. It provides two mechanisms to handle these cases:

Step 1: Define Prompt Targets

When enabling multiple function calling, define the prompt targets in a way that supports multiple functions or API calls based on the user’s prompt. These targets can be triggered in parallel or sequentially, depending on the user’s intent.

Example of Multiple Prompt Targets in YAML:

Prompt Target Example Configuration

version: v0.1
listener:
  address: 127.0.0.1
  port: 8080 #If you configure port 443, you'll need to update the listener with tls_certificates
  message_format: huggingface

# Centralized way to manage LLMs, manage keys, retry logic, failover and limits in a central way
llm_providers:
  - name: OpenAI
    provider: openai
    access_key: $OPENAI_API_KEY
    model: gpt-3.5-turbo
    default: true

# default system prompt used by all prompt targets
system_prompt: |
  You are a network assistant that just offers facts; not advice on manufacturers or purchasing decisions.

prompt_targets:
    - name: network_qa
      endpoint:
        name: app_server
        path: /agent/network_summary
      description: Handle general Q/A related to networking.
      default: true
    - name: reboot_devices
      description: Reboot specific devices or device groups
      endpoint:
        name: app_server
        path: /agent/device_reboot
      parameters:
        - name: device_ids
          type: list
          description: A list of device identifiers (IDs) to reboot.
          required: true
    - name: device_summary
      description: Retrieve statistics for specific devices within a time range
      endpoint:
        name: app_server
        path: /agent/device_summary
      parameters:
        - name: device_ids
          type: list
          description: A list of device identifiers (IDs) to retrieve statistics for.
          required: true  # device_ids are required to get device statistics
        - name: time_range
          type: int
          description: Time range in days for which to gather device statistics. Defaults to 7.
          default: 7

# Arch creates a round-robin load balancing between different endpoints, managed via the cluster subsystem.
endpoints:
  app_server:
    # value could be ip address or a hostname with port
    # this could also be a list of endpoints for load balancing
    # for example endpoint: [ ip1:port, ip2:port ]
    endpoint: host.docker.internal:18083
    # max time to wait for a connection to be established
    connect_timeout: 0.005s