Six lessons from 150 machine learning models at Booking.com
In 2019 three Booking.com data scientists looked back at about 150 customer-facing models that had passed live tests and wrote down what made them work. Most of the lessons are about process, not algorithms.
Booking.com runs on experiments. In 2019 three of its data scientists, Lucas Bernardi, Themis Mavridis and Pablo Estevez, studied "about 150 successful customer facing applications of Machine Learning, developed by dozens of teams", each "validated through rigorous Randomized Controlled Trials", the formal name for A/B tests. Their paper at KDD, a leading data mining conference, gives six lessons. The headline is not an algorithm: "an iterative, hypothesis driven process, integrated with other disciplines was fundamental to build 150 successful products enabled by Machine Learning."
The problem: a good model is not yet a good product
Booking.com has more data than most: "over 171M reviews in more than 1.5M properties". It also has high stakes. A bad song can be skipped, but "once you arrive to an accommodation that does not meet your expectations", you cannot undo the trip.
Teams could build models. The hard part was knowing which ones made the business better: a model can score well on held-out data and still do nothing, or harm, once real travellers use it. The paper studies only the successes, not the "many associated failures" behind them.
How they built it: six lessons
1. Machine learning is a Swiss knife. The models fall into families, from the traveller's preferences and context (flexible dates? a family trip?) to interface tuning and content such as review summaries. Some "act as a meaningful semantic layer": a general score many teams reuse. On average each such model "generated twice as many use cases as the specialized ones" (the paper gives no counts). The payoff was uneven.
2. Offline performance is just a health check. Booking.com compared 23 pairs of models: the gain in an offline score (ROC AUC or mean reciprocal rank) against the change in a business metric in a live test. The correlation was about zero, a Pearson coefficient of -0.1 with a 90% confidence interval from -0.45 to 0.27. In their words, "increasing the performance of a model, does not necessarily translates to a gain in value". Reasons include value that saturates, an "uncanny valley" where predictions that know users too well become "unsettling", and models that over-optimize a proxy, "just driving clicks". Hence: "Offline model performance metrics are only a health check, to make sure the algorithm does what we want to." The sample is small, and the authors "do not want to overstate the generality of this result". Offline gains do not prove value; they still catch broken models.
3. Design the problem before solving it. "It is not obvious what flexibility means". A dates-flexibility model could predict how many date options a traveller weighs, or the chance they change dates. The authors found that "often the best problem is not the one that comes to mind immediately and that changing the set up is a very effective way to unlock value". One destination recommender went through eleven successful versions, with "6 successful algorithm iterations and 4 different setups", from predicting last-minute trips to predicting overlap between destinations.
4. Time is money. Booking.com slowed pages on purpose for test users and found that "an increase of about 30% in latency costs more than 0.5% in conversion rate (a relevant cost for our business)". Four separate tests that made pages faster all raised conversion. Models add delay: "Even mathematically simple models have the potential of introducing relevant latency." So the teams spread copies of models across a cluster, built their own fast engine for linear models, kept models sparse, precomputed and cached what they could, and bundled requests.
5. Watch the predictions when labels come late. A model that predicts a "special request" only learns the truth for users who book. A model that predicts a review waits until after the stay, "which can be months later". So Booking.com reads the shape of the scores it serves: for a yes-or-no model, "the RDC of an ideal model should have one peak at 0 and one peak at 1", RDC being a histogram of outputs. A single bump in the middle, or a shape that differs from training, may signal trouble. "It is a heuristic method, it cannot prove or disprove a model has high quality", and "It does not work for estimators or rankers".
6. Experiment design pays off. "The large majority of the successful use cases of machine learning studied in this work have been enabled by sophisticated experiment designs". First, triggered analysis: compare only the users a change could reach, because everyone else adds noise. Second, a design that keeps a slow model from hiding a good idea.
To compare two models, they count only users where the models disagree. In the paper's toy example, a better model that fixes half the mistakes of one that is 80% accurate, while adding 5% new ones, disagrees with it on at most 15% of cases.
What has changed since
Scale. An InfoQ report on a QCon London 2026 talk says that, as of 2024, Booking.com's inference platform had "more than 480 machine learning models, 400 billion predictions per day, and a latency of less than 20 milliseconds". The talk put the success rate of its A/B tests at "less than 25%", and described a newer method: "preselect with interleaving and validate with A/B testing".
Lesson 2, applied to old systems. A 2025 paper by Booking.com's Castelli, Chouliaras and Goldenberg describes a quality framework for "hundreds of ML systems running in production, generating billions of predictions every hour". Its "block-out" protocol tests a system against a simple benchmark in an A/B test, and "led to a cleanup of dozens of ML systems previously believed to bring value".
Language models arrived on top. The 2023 AI Trip Planner was "Built upon the foundation of Booking.com's existing machine learning models that recommend destination and accommodation", with OpenAI's ChatGPT API on top. Experiment design stayed on the data science blog, from sequential testing in 2023 to "Scaling Experimentation Quality at Booking.com" in March 2026.
Why it matters for your business
Indonesian travel apps share some of this in public. Traveloka told iTnews in 2022: "After one month of A-B tests, the results show that Personalize delivered 13 percent and 66 percent better clickthrough rates" than a third-party tool and its in-house model. That is a real test, but of clicks, the proxy lesson 2 warns about. In July 2026, tiket.com and Microsoft said its in-house AI assistant CRATER handles "more than 75,000 customer interactions every month" and "automates up to 87% of all customer inquiries". No control group is mentioned, so those are volumes, not measured effects.
For a hotel group, a dealer or a retailer, the lesson holds: prove a model in a live test against a control, on a business number, with its delay counted. Less traffic than Booking.com means fewer, bigger tests and triggered analysis to cut the noise.
One duty of care: scores such as family trip or price sensitivity are profiles. Indonesia's personal data protection law (UU No. 27 of 2022, Article 10) gives people the right to object to decisions based only on automated processing, including profiling, that have legal effects or a significant impact on them. A hotel suggestion is unlikely to reach that bar; a price or a refusal might. Record which profiles you compute and why. We are engineers, not lawyers.
How we would build it today
- Start with a few shared scores. Three to five from your own booking or sales data, such as flexible dates, family trip and price sensitivity, served once to every team.
- Let the live test decide. Offline metrics are a health check; the verdict is the A/B result on bookings, cancellations or margin.
- Write the problem three ways first. Will they book within 7 days? Will they change dates? Each leads to a different product. Revisit both around Lebaran and school holidays, when behaviour shifts.
- Budget the delay. Precompute and cache, and run one test with added delay to learn what slowness costs you on Indonesian mobile networks.
- Chart your scores daily. Stays, reviews and cancellations arrive late; a shift in the score histogram is your early warning.
- Design the test. Use an off-the-shelf experiment tool with triggered analysis, a control that never calls the model, and guardrails on latency and cancellations. For chatbots and other language model features, add a golden set of examples and an automated judge offline, then still test live, and keep a human fallback.
The lasting lesson from Booking.com: the model is the cheap part. What scales is the process that tells you whether it worked.