Home

Traffic Splitting with Audiences and Variations

Suggest Edits

Qwak provides multiple ways to split traffic and run A/B tests, via Audiences and Variations.

1. Segmenting traffic with Audiences

Audiences are a powerful tool for categorizing traffic based on predefined conditions set as metadata on the client side. They are configured at the Qwak Account level and are applicable to all real-time models deployed within an Account. Requests that do not match any specific Audience conditions are directed to the fallback Audience by default.

How to create an Audience

Audiences are created using YAML files and managed via the Qwak CLI. Below is an example of creating an Audience to categorize users from New York aged between 10 and 30.

Please note that these details are not automatically fetched, they will have to be filled in from the client side at request time, in the metadata/header fields.

The first condition is a match of the location field to be new-york. This match has to be exact, it's not a regex.
The second condition is a binary one, where the age key will have to be in the range specified with first_operand and second_operand. 

api_version: v1
spec:
  audiences:
    - name: New-York
      description: Users from New York aged 10-30
      conditions:
        unary:
          - key: location
            operator: UNARY_OPERATOR_TYPE_EXACT_MATCH
            operand: new-york
        binary:
          - key: age
            operator: BINARY_OPERATOR_TYPE_RANGE_MATCH
            first_operand: 10
            second_operand: 30

Name: a name for the audience as it will be shown on the platform.
Description: a description for the audience.
Conditions: a list of conditions with AND operand between them.

Defining Conditions

The following condition types are available when creating an Audience:

Unary operators:

  1. UNARY_OPERATOR_TYPE_EXACT_MATCH: Exact value match.
  2. UNARY_OPERATOR_TYPE_SAFE_REGEX_MATCH: Regular expression match.
  3. UNARY_OPERATOR_TYPE_PRESENT_MATCH: Presence of a key.
  4. UNARY_OPERATOR_TYPE_PREFIX_MATCH: Prefix match.
  5. UNARY_OPERATOR_TYPE_SUFFIX_MATCH: Suffix match.
  6. UNARY_OPERATOR_TYPE_CONTAINS_MATCH: Substring match.

Binary operator:

  1. BINARY_OPERATOR_TYPE_RANGE_MATCH: The value is between the saved values.

Registering and managing Audiences

Register an Audience from a config file

Apply an Audience configuration to the Qwak platform using the CLI command:

qwak audiences create -f <file_path>

Retrieve a list of all Audiences 

qwak audiences list

Returns a list of audience ids and names.

Retrieve the details of a specific Audience:

qwak audiences get --audience-id <audience_id>

To get the audience-id use the list command first, to retrieve all audiences with their IDs.

📘

Tip

For a full list of available options and parameters, run qwak audiences --help.


Directing traffic to Audiences

In the following examples, we demonstrate how to route requests to specific Audiences.

Using the Python Client

from qwak_inference import RealTimeClient

model_id = <your_model_id>
feature_vector = <...>

client = RealTimeClient(model_id=model_id)

client.predict(feature_vector, metadata: {"location":"new-york", "age": 25})

Using REST calls

curl --location --request POST 'https://models.<your_env>.qwak.ai/v1/1_hour_model/predict' \
--header 'Content-Type: application/json' \
--header 'location: new-york' \
--header 'age: 25' \
--header 'Authorization: Bearer <Auth Token>' \
--data '{
  "columns": ["feature_1", "feature_2"],
  "index": [0],
  "data": [[0.0, 0.0]]
}'

🚧

Please note that Audience information is not stored in Analytics, meaning that the source of a request from a specific audience cannot be tracked.


2. Splitting traffic with Variations

Variations serve as a traffic splitting mechanism, operating at a higher level than Audiences and featuring the following characteristics:

  • Each Variation corresponds to a specific model Build.
  • Variations always receive traffic from an Audience, so creating an Audience is a prerequisite for using Variations.
  • Variations can distribute traffic by percentages from different audiences or between different builds, providing an effective A/B testing feature. Variations can also be designated as Shadow Variation, which duplicates a percentage of live traffic to other builds for testing purposes.
  • Each Variation can direct traffic to at most one deployed Build.
  • The Variation to which a request was directed is stored in Analytics under a separate column named variation_name.

Assigning Audience Traffic to Variations

You can control traffic distribution either during realtime deployment or by editing your current realtime deployment settings. When deploying a model with multiple variations, audiences are assigned to specific variations, including the fallback audience and variation.

When distributing traffic from an audience to multiple variations, the percentages must always total 100% of that audience's traffic. The only exception is the Shadow Variation, which can receive less than 100% of live traffic.

Here's an example from the UI: 90% of traffic from the new-york audience is directed to a new-york variation with a Build ID starting with 6f3..., while the remaining 10% goes to the beta variation. Requests without audience metadata are processed by the fallback audience, shown in the UI as All other invocations.

The current variations appear under your model's overview (Traffic Control section):


Fallback Variation: Receives traffic that doesn't match any audience.

Connecting Audiences to Variations: Audiences can be linked to one or more variations, with traffic between variations distributed randomly based on defined percentages.

To modify traffic configuration:

  1. Edit the deployment of the desired build.
  2. Adjust the percentage of traffic for each variation.


Enabling Variations with the default Audience

In certain scenarios, you may not require the traditional traffic categorization provided by Audience conditions. For such cases, Qwak offers support for a default Audience, which lacks conditions but enables the utilization of Variations for all requests.

Below is an example of configuring the default Audience to enable Variations:

api_version: v1
spec:
  audiences:
    - name: default
      description: All traffic

To register this Audience configuration, please refer to the Audiences section above.


Undeploying a multi-Variation Realtime Model

Once you have more than one build deployed, when you undeploy an existing build, you must specify how to split the traffic after the undeployment.

Undeploying using the UI

To undeploy a variation:

  1. In the build view, click the options icon next to a deployed build and select Undeploy.

  1. Redistribute the traffic between the remaining variations and then click Undeploy.

Undeploying using the CLI

To undeploy a model with variation from the CLI, run the following command:

qwak models undeploy \ 
    --model-id <model-id> \
    --variation-name <variation-name> \
    --from-file <config-file-path>

With a configuration file as follows:

realtime:
  variation_name: <The variation name being undeployed>
   audiences:
    - id: <remaining audience id>
      routes:
        - variation_name: <other existing variation>
          weight: 80
          shadow: false
        - variation_name: <other existing variation 2>
          weight: 20
          shadow: false
  fallback_variation: <other existing variation>

If you use --variation-name in the CLI command, you don't have to pass the variation_name in the configuration file.

When undeploying from 2 variations to one, you don't have to pass any variation-related data -all the traffic will pass to the remaining variation.

📘

Note

By default the undeploy command is executed asynchronously, which means that the command does not wait for the undeployment to complete.

To execute the command in sync, use the --sync flag.

Updated 3 months ago