Hamed Khosravi
← All publications

Beyond “AI Helps Humans”: Decision-Targeted Evaluation Design for Human–Agent Teams in the Agentic Era

Hamed Khosravi, Xiaoming Huo

Human–AI Coevolution Workshop (HAIC), NeurIPS 2026

Paper Code

How a fixed replay budget divides between the two baselines a team has to beat. The design shifts effort toward whichever comparison is harder to sett
How a fixed replay budget divides between the two baselines a team has to beat. The design shifts effort toward whichever comparison is harder to settle, instead of splitting the budget evenly between them.

The problem

Showing that a human–AI team does better than nothing is not the claim anyone actually wants. The claim worth making is that the team beats the human working alone and beats the AI working alone — two comparisons, usually of very unequal difficulty, paid for out of one evaluation budget.

What we do

Treat the study design itself as the decision. Allocate the replay budget across the two comparisons according to how much each one still needs, rather than splitting it evenly.

What it buys you

The same number of evaluated cases produces a sharper answer on the comparison that was actually in doubt, so a complementarity claim either holds up or fails for a reason, rather than for want of budget in the wrong place.

Cite

@inproceedings{khosravi2026beyond,
  title     = {Beyond ``{AI} Helps Humans'': Decision-Targeted Evaluation Design for Human--Agent Teams in the Agentic Era},
  author    = {Khosravi, Hamed and Huo, Xiaoming},
  booktitle = {NeurIPS Workshop on Human-AI Coevolution (HAIC)},
  year      = {2026}
}