Scenarios
The parameter rewrite policy supports adding, editing, or deleting parameters in the request Body and Query before the request is forwarded to the backend model service. It applies to the following scenarios:
Parameter Injection: Automatically adds necessary parameters (such as fixed temperature and max_tokens) to all requests.
Parameter Override: Overrides the parameter values passed from the client to uniformly control model behavior.
Parameter Cleanup: Deletes sensitive or unnecessary parameters to prevent parameter leakage.
Protocol Adaptation: Adapts to parameter differences across vendors (for example, rewriting top_p to topP).
Multi-Version Compatibility: Supports parameter format conversion across different API versions.
Note:
Parameter rewriting occurs after request routing and before the request is forwarded to the backend model service. It only affects the forwarded request and does not affect the response received by the client.
Prerequisites
1. The AI gateway instance has been created and is in a running state.
2. The model API has been created.
3. Understand the parameter formats and requirements supported by the backend model service.
Operation Steps
Parameter Rewriting Configuration
Step 1: Go to the Model API Details Page
2. On the instance list page, click the ID of the gateway instance you want to configure to go to its basic information page.
3. In the left sidebar, click Model Management, and then click the Model API tab.
4. Click the target model API name to go to the API details page.
5. Switch to the Parameter Rewriting Tab.
Step 2: Parameter Rewriting Configuration
1. In the Parameter Rewriting Tab, view the current configuration status.
2. Click Configure Parameter Rewriting Rules to go to the configuration dialog.
3. In the configuration dialog, enable the Whether to Enable switch.
4. Select the parameters that need to be rewritten, and configure the parameter rewriting rules.
Operation Type Description:
|
Addition | Add a new parameter key-value pair. | Inject default parameters, such as temperature: 0.7 |
Editing | Modify the value of an existing parameter. | Override parameters passed from the client. |
Deletion | Delete specified parameters. | Clean up sensitive or unnecessary parameters. |
5. Click the Add button to configure multiple parameter rewriting rules. The rules are executed sequentially in the order they are configured.
Step 3: Save the Configuration
Click OK to save the parameter rewriting rules. The configuration takes effect immediately.
Typical Configuration Examples
Example 1: Unifying Temperature Parameters
Scenario: Centrally controls the temperature parameter for all requests to prevent clients from setting it arbitrarily.
Configuration:
|
Body parameter | Editing an alarm policy | temperature
| 0.7
|
Effect: Regardless of the temperature value passed by the client, 0.7 is used in the end.
Example 2: Cleaning Up Sensitive Information
Scenario: Deletes user identification information from requests to protect user privacy.
Configuration:
|
Body parameter | Deleting an alarm policy | user
| - |
Body parameter | Deleting an alarm policy | metadata.user_id
| - |
Effect: The user identification field in the request is deleted and is not forwarded to the backend model service.
Example 3: Adapting to Azure OpenAI
Scenario: Adapts to the api-version Query parameter requirements of Azure OpenAI.
Configuration:
|
Query parameter | Addition | api-version
| 2024-02-01
|
Effect: ?api-version=2024-02-01 is automatically added to all requests.
Must-Knows
1. Execution Timing: Parameter rewriting occurs after request routing and before the request is forwarded to the backend model service.
2. Execution Order: Multiple rules are executed sequentially in the order they are configured. It is recommended to arrange dependent rules in the correct order.
3. Body Parameter Restriction: Parameter rewriting is only supported for JSON-formatted request bodies.
4. Parameter Key Path: Supports simple parameter key names (for example, temperature). Nested paths (for example, messages[0].content) are not currently supported.
5. Interaction with Caching: Parameter rewriting occurs after the cache query. The cache Key is calculated based on the original request.
6. Relationship with Logs: The LLM Log records the rewritten request body (if body collection is enabled).
7. Parameter Value Type: In the current version, parameter values are uniformly processed as strings. Number, Boolean, and other type markers are not currently supported.