Scenarios
The degradation policy is used to automatically downgrade service quality when the backend model service becomes abnormal, ensuring system availability. By configuring the degradation policy, you can:
Degradation is automatically triggered when the service becomes abnormal.
View degradation trigger records.
The degradation policy is a key measure for ensuring system stability. When used in conjunction with cross-service Fallback, it enables a complete disaster recovery solution.
This document guides you on how to configure and manage degradation policies in AI Gateway.
Prerequisites
The AI Gateway instance has been created.
The model API has been created.
The model service has been created.
Operation Steps
Degradation policies are configured when you create or edit a model API. For details, see Model API. Step 1: Go to the API Details Page
2. On the instance list page, click the ID of the gateway instance you want to configure to go to its basic information page.
3. In the left sidebar, click Model Management, and then click the Model API tab.
4. Click the target model API name (the API that has been created) to go to its details page.
5. On the API details page, locate the Degradation Policy section. Click the Configure Degradation Policy button in the upper-right corner to go to the degradation policy configuration dialog.
Step 2: Configure Downgrade Trigger Conditions
In the configuration dialog, enable the degradation switch and configure the following parameters:
|
Trigger condition | Yes | Select the degradation trigger condition. Multiple selections are supported: • Service Unavailable: Triggers when the primary model service returns errors such as 500 / 502 / 503. • Connection Timeout: Triggers when a request exceeds the timeout period without a response. • Rate Limit (HTTP 429): Triggers when the primary model service triggers rate limiting. | Service unavailable
Rate Limiting
|
Step 3: Configure the Fallback Model Service
1. Add backup model services sequentially (in order).
2. After degradation is triggered, the gateway attempts to call the backup model services sequentially in the specified order.
3. After the configuration is complete, click OK to save.
Viewing the Downgrade Status
On the API details page, you can view historical degradation records in the Degradation Records section:
Degradation Trigger Time
Degradation Reason (Specific Trigger Conditions and Thresholds)
Primary Service
Final Service Used
Degradation Result