Custom audiences
Use qualification examples to filter annotators before they work on your tasks. This lets you build annotator pools specialized for the requirements of your annotation workflows.
- Is the hand anatomy correct?✓ correct
- Does this hand have 6 fingers?✓ yes
- Are the thumb proportions plausible?✗ no
- Is the wrist angle anatomical?✓ yes
Choose the level of annotator specialization for your workflows.
Global audiences maximize speed for general tasks. Curated and custom audiences add increasing levels of task-specific filtering, qualifications and domain specialization.
Creating a custom audience.
Create the audience
An empty audience. Labelers join only after passing the qualification examples you add next.
audience = client.audience.create_audience(
name="Prompt Alignment",
)Add qualification examples
Questions with known correct answers. Labelers must match the truth to enter the pool.
audience.add_compare_example(
instruction="Which follows the prompt?",
datapoint=[img_a, img_b],
truth=img_a,
context=prompt,
)Assign a job
The job runs only against labelers who qualified. Reuse the audience across as many jobs as you like.
job = audience.assign_job(
job_definition,
)
results = job.get_results()Review every qualification example before you publish.
If an example has a wrong or ambiguous truth value, qualification inverts: good labelers who answer correctly get filtered out, while bad labelers who match the incorrect answer pass through. Every truth must be unambiguous before the audience goes live.
End-to-end workflow.
Create the audience, attach qualification examples, define a job, and ship it. The same audience object is reusable across every job you assign to it later.
from rapidata import RapidataClient
client = RapidataClient()
# 1. Create the audience
audience = client.audience.create_audience(
name="Custom Prompt Alignment Audience",
)
# 2. Add qualification examples
DATAPOINTS = [
["assets/flux_sign.jpg", "assets/mj_sign.jpg"],
["assets/flux_duck.jpg", "assets/mj_duck.jpg"],
["assets/flux_book.jpg", "assets/mj_book.jpg"],
]
PROMPTS = [
"A sign that says 'Diffusion'.",
"A psychedelic duck with glasses.",
"A small blue book sitting on a large red book.",
]
for prompt, datapoint in zip(PROMPTS, DATAPOINTS):
audience.add_compare_example(
instruction="Which image follows the prompt more accurately?",
datapoint=datapoint,
truth=datapoint[0],
context=prompt,
)
# 3. Assign a job to the qualified audience
job_definition = client.job.create_compare_job_definition(...)
job = audience.assign_job(job_definition)
results = job.get_results()Qualification examples should match the real tasks.
If your production task uses specific instructions, answer formats, or interaction settings, use the same setup for qualification examples. Annotators should qualify under the same conditions they will see in production.
from rapidata import NoShuffle
audience.add_classification_example(
instruction="How well does the image match the description?",
answer_options=[
"1: Not at all",
"2: A little",
"3: Moderately",
"4: Very well",
"5: Perfectly",
],
datapoint="assets/email-4o.png",
truth=["5: Perfectly", "4: Very well"],
context="A laptop screen with readable text...",
settings=[NoShuffle()],
)Build once. Use forever.
Audiences are first-class objects in the SDK. Find them by name, fetch them by id, and assign new jobs against the same qualified pool — without re-running the qualification pass.
# Find by name
audiences = client.audience.find_audiences(
"Custom Prompt Alignment Audience"
)
# Or fetch by id
audience = client.audience.get_audience_by_id(
"aud_8f3a..."
)
# Run another job against the same pool
job = audience.assign_job(new_job_definition)