We fund research that needs human feedback.
Rapidata grants up to $50,000 in annotation credits per research project. More can be discussed for specific topics such as Online RLHF. If your study needs human judgment on images, video, audio or text, the grant will cover the collection. Apply by email with your research question and information you deem relevant about your project.
Credits per project, more for selected topics
Human responses at list pricing
To a decision, every application is read
Responses can be obtained in real-time, with up to 6K+/minute
What we fund
The program is meant for research that would benefit from substantially more human feedback than is normally practical to collect.
This can be a benchmark, an evaluation study, a preference dataset, an online training loop, an ablation that needs repeated human evaluation, or something we have not thought of yet.
The funding is provided as Rapidata platform credits, it is not a cash grant. Grant size depends on the study, task complexity and amount of human feedback required.
Research should normally be intended for public release.
If Rapidata contributes annotation credits to the work, we ask that Rapidata is acknowledged as the platform used to collect the human feedback. On specific topics, we are also interested in co-autorship opportunities, in this case we would also assign some of our researchers to the project.
A $50,0001 grant buys 12,500,000 responses at list pricing of $4 per 1,000 responses.2 Some examples below showcase how some or all of the amount could theoretically be spent.
1,000 prompts/round
8 candidates/prompt -> 25 match-ups
Grant size is set per study. Topics that overlap with Rapidata's own research interests can receive well over $50,000.
Research we are particularly interested in
A few areas where we are currently especially interested in supporting experiments are listed below. We accept applications outside these topics
Human evaluation of image, video and audio models
Particularly where existing automated metrics or model judges are not sufficient.
Online human feedback
Online RLHF, preference optimization and validation against fresh model generations. More on our Online RLHF page.
Human feedback vs. reward models or model judges
Understanding where the proxies stop matching actual human judgment.
Fine-grained and population-dependent preferences
Including cases where different groups systematically disagree.
New ways of collecting richer feedback
Beyond a single “A or B?” preference.
Close to our own research
Projects that overlap closely with research we are already working on can receive substantially more support and become a closer collaboration rather than only a grant.
What you can collect
Grant credits work across the dashboard, the Python SDK and the API. Latency to obtain feedback is a few seconds, and judgments can be collected continuously.
Every response includes the annotator's origin and reliability score, so you can report exactly which population you measured.
Tasks can include images, video, audio and text, including combinations of modalities.
Some of our annotation features are listed below.
Pairwise comparison
Show two outputs side by side and ask which one people prefer, which aligns best with a prompt or which falls better under a specific criterion.
Any modalityScoring
Score outputs independently on criteria such as prompt faithfulness, physical coherence, naturalness, artifacts or aesthetics.
Any modalityRanking
Collect rankings across several candidate outputs or model generations.
Any modalityClassification
Collect labels, taxonomies, or validation decisions at scale.
Any modalityLocate & draw
Annotators click or box the exact region of interest.
ImagesFree text
Open-ended answers for error analysis, or other tasks.
Any modalityAnnotation Quality
The annotations collected through the research program use the same Rapidata quality infrastructure as our commercial projects and our own public benchmarks. Research grants do not use a separate or lower-quality annotator pool.
Annotator reliability is continuously evaluated through seeded validation tasks, and reliability information is available at response level. This allows researchers to filter, weight or further analyse responses when needed.
For papers using Rapidata, the annotation setup can for example be described as:
“Human annotations were collected using Rapidata. Annotator quality is continuously assessed through validation tasks, with reliability information tracked at the response level.”
The same methodology is used for the human evaluations published on Benchmark.ai, where more information on annotation quality, validation and aggregation is available.
Publishing the dataset?
If the annotations collected through the grant are released openly as part of a dataset or benchmark, tell us.
We are happy to fund follow-up experiments on open research, particularly when the first collection creates something other researchers can build on.
The collected annotation data remains yours. We ask for an acknowledgement of Rapidata in work that uses the grant.
How to Apply
Send one paragraph
Email info@rapidata.ai.
Include your name and affiliation, what you are trying to measure, the type of human feedback you need, an approximate scale, and a link to previous work if you have any.
Get an answer in two working days
We normally answer within two working days.
Sometimes we'll ask for more information or suggest a different annotation setup if we think it would make the experiment stronger.
Start collecting
If approved, the credits are added directly to your Rapidata account.
You can run the study through the dashboard, Python SDK or API. For larger or more unusual setups, we can help with the experimental setup before launch.
$ pip install -U rapidata
from rapidata import RapidataClient
client = RapidataClient()
audience = client.audience.get_audience_by_id("global")
job_definition = client.job.create_compare_job_definition(
name="Aesthetic preference v3",
instruction="Which image follows the prompt more accurately?",
datapoints=IMAGE_PAIRS,
responses_per_datapoint=2000,
contexts=PROMPTS,
)
job = audience.assign_job(job_definition)
results = job.get_results()Questions
Who owns the annotations?
You do. Rapidata claims no IP, licence or co-authorship (unless discussed otherwise) on data collected under a grant. We ask to be acknowledged in publications that use it.
Do I know who answered?
Yes. Every response carries the annotator's origin and reliability score, so you can break results down by population after collection.
How is the grant size decided?
Case by case, from the study you describe: the reach it needs, the task type, the responses per datapoint that make the result significant, and how long the collection runs. Topics that overlap with our own research interests can receive well over $50,000.
How do you keep quality high at that volume?
Annotators carry a reliability score, updated continuously through seeded validation tasks. Each response ships with that score, so you can weight or filter in your own analysis.
I already spent my annotation budget. Can I still apply?
Yes. Grants cover new collection, including a rerun of an earlier study at a scale that supports publication.
Apply via email, and we'll be in touch within 2 days
1. $50,000+ per researcher applies to eligible projects. A project is eligible when the results are intended for public release as a peer-reviewed paper, preprint, open benchmark, openly licensed dataset or public thesis; the applicant is affiliated with an academic or non-profit research institution, or is an independent researcher with a verifiable publication record; the study requires human judgment; and the annotations will not be resold, sublicensed or used to train closed commercial models. Rapidata decides eligibility at its own discretion and sizes each award to the study described in the application. Awards are issued as platform credits, are non-transferable, hold no cash value and expire twelve months after issue. One active grant per researcher.
2. Response counts are illustrative. 12,500,000 is what a $50,000 allocation buys at list pricing of $4 per 1,000 responses. Effective volume varies with task complexity, media length and the number of responses per datapoint.