Tech Overview

• Terminology
• Threading Model
• Listener
  • Downstream (Ingress)
  • Upstream (Egress)
  • Configure Listener
• Prompts
  • Messages
  • Prompt Guard
  • Prompt Targets
  • Prompting LLMs
• Model Serving
  • Local Serving (CPU - Moderate)
  • Cloud Serving (GPU - Blazing Fast)
• Request Lifecycle
  • Terminology
  • Network topology
  • High level architecture
  • Configuration
  • Request Flow (Ingress)
  • Request Flow (Egress)
• Error Target
  • Key Concepts
  • Error Header Example
  • Best Practices and Tips