Can AI Agents Produce Booking-Ready Flight Recommendations?

A preliminary evaluation of five consumer-facing AI travel agents across seven real-world flight-shopping scenarios

Preliminary report · 1 November 2026 · TravelLabs.ai · Independent research note (not peer reviewed)

Data release: headline results and method overview only. Scenarios, transcripts, reference answers, scoring instructions and per-task results are held back and available to research partners.

Abstract

A flight answer that looks plausible is not necessarily one a traveler can book. We tested five consumer-facing AI travel agents on seven real flight-shopping requests (35 recorded runs) and scored each run three ways: a weighted rubric, a Booking Viability verdict, and a separate Trust Gate for fabricated or unverified claims. Two language-model judges graded every run and a human expert audited the consequential cases. Average rubric scores ranged from 54.0% to 90.9%. The more telling spread was in outcomes: two agents produced a bookable answer in 6 of 7 tasks, one in none. This is a small descriptive pilot, not evidence of stable rankings.

Key takeaways

• A high score is not a bookable answer. Agents ranked third and fourth were within a point on rubric score, yet produced bookable answers in 4 and 3 of 7 tasks.
• Honest gaps and invented facts are different failures. Most failures were incomplete answers. Fabricated or stale claims were concentrated in one agent.
• How a ticket is built matters as much as which flights are on it. In one expert-verified case, two one-way tickets cost $7,325 against $3,180 for a round trip.
• Even strong agents shared blind spots. The weaknesses we saw most often were not about finding flights at all (Section 5.4).
• LLM judges need human audit. A targeted human review of 30 checkpoint cases corrected 15 of them. The remaining ~2,800 judge grades were not individually audited.

1. Research question and motivation

The question is not whether an AI system can mention plausible flights. It is whether the final response, given the traveler’s constraints and what the agent could verify, supports the next booking decision. Travel punishes small errors: the right city can hide the wrong airport, a marketing cabin name can be mistaken for a seat, a fare can cover one passenger instead of two, and two one-way fares can cost far more than the round trip.

A single composite score cannot capture this. A fluent answer can earn credit for clarity and coverage while omitting a return flight, and an honest agent can admit it cannot verify live inventory yet still fail to deliver a usable recommendation. TravelLabs therefore reports task success and trust separately from the rubric score.

2. Study design and scope

The first TravelBench track covers flight research and shopping, not payment or post-booking servicing. We recorded one run per agent and scenario. The seven expert-built tasks range from a simple query to multi-traveler, multi-city premium-cabin itineraries. They probe different research skills and are not a sample of typical travel demand.

Six agents were tested (42 recordings). Five are reported here (35 recordings): ChatGPT Dot, Muse, Grok Bot, Claude Cowork and Gemini Spark. A sixth was excluded from all public reporting because its terms of service prohibit use in benchmarking. The exclusion rests on those terms, not on its results, and no results for that agent are published or implied.

We evaluate complete agent experiences (model, instructions, tools, browsing and interface), not standalone model ability. Each run started in a fresh chat with an instruction to forget prior history and ignore personal context, and we gave no information about the traveler. This was intentional: it puts every agent on a level field and tests the “naked” agent. In real use, agents will usually know more about a traveler’s preferences and situation, and may do better or worse with that context. Testers chose no model; each product ran at its default (Table 1).

Table 1. Test conditions. Models are as disclosed by the vendor or confirmed by TravelLabs, not verified at run level.

Run timing, tools and interfaces were not perfectly uniform, and fares change by the hour. This is an observational product assessment, not a controlled simultaneous bake-off. One condition deserves a flag: the Gemini runs used a temporary (incognito) chat. To test whether that mattered, we re-ran one task in a standard chat on 2 November 2026 and saw the same behavior: no live fare lookup, the manager’s outdated price and schedule repeated back as if current, and price ranges in place of quotes. The Gemini app offered no separate browsing or agent mode, so the Gemini row reflects the product a regular user gets, which, unlike the other four agents, was not equipped to check live fares. The original runs are the ones scored.

3. Evaluation framework

A frozen rubric of 34 task-dependent checkpoints, each weighted by its consequence, produces a percentage per run. An agent’s score is the mean of its seven runs, averaged across the two judges. That figure is a normalized score under this design, not the probability that a booking will succeed.

Three further signals are kept separate, consistent with multi-metric evaluation practice (Bommasani et al., 2023). Booking Viability is an expert verdict on whether the final response meets the mandatory requirements for the requested stage. The Trust Gate flags fabricated or misleading claims about inventory, policies or verification; honestly saying live prices could not be checked is not a hallucination. Price Competitiveness is experimental and is reported only when a like-for-like comparison can be verified.

Two LLM judges, ChatGPT and Claude, graded every run independently, an approach with documented strengths and biases (Zheng et al., 2023). Two of the products tested are built by the judges’ developers; we discuss that conflict in Section 7. A human review then examined 30 checkpoint cases, deliberately chosen and not random, to concentrate on likely errors: mostly cases where the judges disagreed, plus some where they agreed, to test whether both could be wrong. It corrected 15 of the 30. Because the sample was enriched with suspicious cases, the 50% correction rate is not an error rate for the benchmark; the rest of the grades were not individually audited. An expert reviewer also ruled on booking and trust verdicts the judges could not settle, and remaining judge differences were kept, not forced into agreement.

4. Preliminary leaderboard

Table 2. Five publicly eligible agents; seven runs each. Differences between Grok Bot and Claude Cowork are too small to read as a ranking.

Figure 1. Rubric means and Booking PASS counts. A display of the descriptive sample, not a significance test.

5. Findings

5.1. Actionability changes how a high score should be read

Rubric quality and practical completion are related but not interchangeable. Two agents reached six bookable answers; the middle of the table earned respectable scores with far fewer actionable outcomes. A missing return segment, a missing traveler-specific price or an unconfirmed cabin can make a helpful answer unusable. Composite averages hide such hard constraints, and the reverse also holds: an honest partial answer that discloses uncertainty is not a fabrication.

5.2. Trust errors need their own signal

Trust failures were unsupported claims about what exists, what is available, or whether something had been verified. They differ in kind from ranking real options poorly. Most were concentrated in one agent. Expert review also overturned an initial trust accusation where the evidence supported a real flight but not full booking readiness. Over-detecting hallucinations distorts a benchmark as much as under-detecting them.

5.3. A valid booking can still be an expensive one

In one expert-verified case, an agent priced a trip as two separate one-way fares totalling about $7,325. A verified round-trip construction came to about $3,180: a difference of $4,145, or 130% above the reference. This is not an identical-flight comparison, because the agent’s outbound combined two carriers while the reference used one. We report it because the gap is too large for that to explain; a single one-way leg alone cost more than the entire round trip. Viability asks whether an itinerary is legitimate. Price Competitiveness asks whether the traveler paid an avoidable premium. We report one verified case and do not extrapolate an average.

Figure 2. One historical fare-construction comparison (not identical flights). Itinerary details withheld.

5.4. What agents still get wrong

Beyond the scores, review of the recordings surfaced recurring weaknesses that matter more to a paying traveler than any single rubric item. We describe them here as themes only:

• Quoting before understanding. Requests that sound simple often hide details a quote depends on, and most agents did not ask for them.
• Ticket construction. Choosing between round trips, one-ways and multi-city structures was a frequent source of avoidable cost.
• Carrier judgment. Premium travelers care which airline they fly; few agents reasoned about carrier quality.
• Itinerary comfort. Some options were technically valid but poorly built, such as needlessly long connections when a better one existed.
• Presentation and breadth. Flights were often shown as prose, with few options, rather than the scannable comparison travelers expect.

The first two are partly captured by our rubric today. The last three are not captured well by a percentage score, and we are building dedicated measures for them (Section 9).

5.5. Judge disagreement is an audit signal

The two judges disagreed on many checkpoints, especially where quotes were incomplete, availability was unclear or assumptions were disclosed. Their agent means differed by under five points and produced the same ordering. We corrected clear factual mistakes with human-reviewed evidence and documented defensible differences. Some cases where both judges agreed still looked wrong on review, so agreement is not correctness.

6. What the benchmark measures, and what it does not

The leaderboard maps observed strengths and weaknesses on seven tasks. It shows that important classes of error occurred and that some agents produced bookable answers more often in this sample. It cannot establish how any agent will perform on a future trip, which model is better, or the order of agents with close scores. Booking PASS means completion at the research stage, not a ticket issued; we did not validate checkout, payment or after-sale servicing. Scoring is limited by what the preserved transcripts show.

7. Limitations and threats to validity

• Small sample. Seven scenarios, one run per agent and task, no confidence intervals, and no claim of stable rankings.
• No user context. Every run started from scratch, with no traveler profile or history. That makes comparisons fair but is not how agents will often be used; personalization could change results in either direction.
• Task selection. Hand-built, premium and complex cases; not representative of ordinary travel demand.
• Environment and timing. Agents had different tools and interfaces, and fares changed. The Gemini app has no agent or browsing mode, so it was tested as a chat product without live fare access; a one-task re-run outside temporary chat behaved the same.
• Judge bias and conflict of interest. Two LLM judges can share blind spots. Two tested products are built by the judges’ developers, and research shows LLM evaluators can favor their own outputs (Panickssery et al., 2024). Agent order was identical across judges and the ChatGPT judge did not rate ChatGPT Dot higher than the Claude judge did, which is reassuring but not proof of no bias.
• Judge access to reviewer notes. The compiled transcript packets included annotator remarks in places. Except for two late-defined checkpoints re-graded blind, the judges were not blind to them.
• Targeted, not exhaustive, human audit. The reviewed cases were chosen for likely errors, so they say little about the error rate of the remaining grades.
• Evidence gaps. A historical screenshot does not prove ticketability at checkout, and some captured interface elements were incomplete.
• Financial metric sparsity. Only one comparison supports a quantified premium, and it is not identical-flight.
• Interests. TravelLabs plans to license evaluation data to AI developers. No tested vendor reviewed or paid for this report.

8. Reproducibility and responsible disclosure

TravelLabs keeps versioned scoring workbooks, the rubric, adjudication logs and agent records. Every weighted score in this report was recomputed from the underlying checkpoint grades before publication. Prompts, reference answers, transcript evidence and scoring wording are not part of this release. This is not open data and we do not claim external replication. Future audits will be issued as versioned corrections. Vendors named here are welcome to send corrections or context.

9. What comes next

TravelBench v2.1 extends the flight track with measures we think a score alone cannot carry: Presentation Clarity (can a traveler see and compare options at a glance), Itinerary Comfort (are the options sensibly built compared with the best available alternative) and Request Understanding (does the agent ask before it quotes). We are also moving to repeated runs for reliability, and to new tracks for transaction execution, disruption recovery, award flights, hotels and luxury travel.

Available to research partners: per-task and per-checkpoint results, a taxonomy of failure types with frequencies, reference answers, the v2.1 measures, and repeated-run reliability data. Contact TravelLabs.ai to discuss access.

10. Conclusion

Can AI agents produce booking-ready flight recommendations? Sometimes. The best two did in six of seven tasks, the weakest in none, and rubric scores alone would not have shown that. Rubric quality, Booking Viability and Trust gave different readings, and a verified fare-construction case shows why cost needs its own measure. We offer these findings as a framework for debate, not a final ordering of AI travel systems.

References

Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023. https://arxiv.org/abs/2306.05685

Panickssery, A., Bowman, S. R., and Feng, S. (2024). LLM Evaluators Recognize and Favor Their Own Generations. NeurIPS 2024. https://arxiv.org/abs/2404.13076

Bommasani, R., Liang, P., and Lee, T. (2023). Holistic Evaluation of Language Models. Annals of the New York Academy of Sciences. https://doi.org/10.1111/nyas.15007

Product names and trademarks belong to their respective owners. TravelLabs evaluates product behavior in recorded scenarios and does not imply sponsorship, affiliation or endorsement. Rankings and fare examples are historically bounded.

@techreport{travellabs2026bookingready,
  author = {{TravelLabs Research}},
  title = {Can AI Agents Produce Booking-Ready Flight Recommendations?},
  institution = {TravelLabs},
  year = {2026},
  number = {Working Paper 01},
  version = {1.0},
  url = {https://travellabs.ai/research/can-ai-agents-produce-booking-ready-flight-recommendations}
}