This document summarizes common issues and solutions for the AI gateway during usage.
Product Inquiries
What Is an AI Gateway?
AI Gateway provides capabilities for large model gateways (unified access and intelligent scheduling for multiple models) and MCP gateways (AI transformation and protocol conversion for traditional applications), targeting large model and intelligent scenarios.
What Is the Difference Between a Cloud Native Gateway and an AI Gateway?
Cloud Native Gateway addresses scenarios involving traditional microservices and cloud-native traffic management, solving problems related to the access, routing, rate limiting, circuit breaking, and security protection of north-south and east-west traffic.
AI Gateway provides capabilities such as unified model access, intelligent routing, protocol conversion (MCP/OpenAI), and token-level rate limiting. It targets large model and intelligent scenarios, focusing on solving core challenges enterprises face when accessing, scheduling, and managing multiple AI models, including protocol complexity, governance difficulties, uncontrollable costs, and high barriers to legacy system transformation.
Both belong to the cloud-native intelligent gateway product family and can be independently selected based on business needs.
AI Gateway Usage
What Problems Does an AI Gateway Solve?
AI Gateway focuses on solving core challenges enterprises face when accessing, scheduling, and managing multiple AI models, including protocol complexity, governance difficulties, uncontrollable costs, and high barriers to legacy system transformation. As the traffic entry point and governance hub for enterprise intelligent architecture, it helps enterprises efficiently, securely, and cost-effectively integrate and utilize AI capabilities through unified protocol adaptation, intelligent routing and scheduling, and comprehensive observability.
What Protocols Does an AI Gateway Support?
AI Gateway is 100% compatible with the open-source gateway ecosystem and fully adapts to standard AI protocols, supporting MCP, OpenAI, SSE, and others. It provides conversion for traditional protocols such as RESTful and gRPC, enabling a single gateway to handle all traffic.
How to Invoke Large Models Such as Hunyuan via an AI Gateway
The core process consists of six steps:
1. Create a Model Key: Securely configure the API key required for accessing large models in the gateway.
2. Create a Model Service: Add a model provider (such as Hunyuan), associate a model key, and select a model protocol (such as OpenAI-compatible).
3. Create a Model API: Create an API that provides capabilities externally, bind it to a model service, and configure the request protocol, routing, and Base Path.
4. Create a Consumer: Create a caller identity and add API Key credentials for it.
5. Create a Consumer Group and Grant Permissions: Group consumers and grant them permissions to access the model API.
6. Obtain the Access Address and Initiate a Call: Obtain the gateway entry address and request path from the console, and initiate the call using consumer credentials (API Key).
Call Example:
curl -i -X POST <Access Address> \\
-H "Content-Type: application/json" \\
-H "Authorization: Bearer <API_KEY>" \\
-d '{
"model": "<MODEL_NAME>",
"messages": [
{"role": "user", "content": "Hello, please introduce yourself."}
]
}'
How to Use a Custom Model Provider to Integrate with Third-Party Model Platforms
1. Create an API Key on a third-party model platform.
2. Create a model key in the AI gateway and paste the third-party API Key.
3. Create a model service, select "Custom Provider" → enter a third-party OpenAPI-compatible endpoint → select the default model.
4. Create a model API, specify the Base Path and request protocol, and bind it to a model service.
5. Grant permissions for this model API within the consumer group.
6. Initiate the call using the consumer API Key via the gateway's public network entry.
How Does an AI Gateway Control Model Invocation Costs and Ensure Stability?
AI Gateway provides capabilities for rate limiting, circuit breaking, and degradation across multiple dimensions, from consumers and APIs to models:
It supports token-level rate limiting (TPM/TPH/TPD), QPM-level rate limiting, and concurrent request rate limiting.
It supports intelligent scheduling policies such as multi-model weight routing and model name routing.
It supports Fallback configuration between model services, automatically switching to a backup service when the primary model service is abnormal.
It provides multi-dimensional monitoring of API call latency, Token consumption, and model costs, facilitating Ops and cost optimization.
How Does an AI Gateway Implement Multi-Tenant Isolation and Permission Management?
AI Gateway assigns independent API Keys to different tenants through a multi-level permission model based on consumers and consumer groups. This enables secure isolation and convenient sharing of AI capabilities across different teams and projects, supporting platform-based operations.
What Is an MCP Gateway?
MCP Gateway addresses the scenario of AI-enabling legacy applications. Using a protocol conversion engine, it automatically wraps standard APIs provided by existing business systems into standardized tools (MCP Tools) that AI applications can invoke. This achieves "zero-code transformation" AI enablement for business capabilities, allowing legacy systems to quickly gain the ability to be invoked by AI Agents via the MCP protocol.