What building Copilot taught GitHub about shipping an LLM product
Before chat windows and coding agents, GitHub spent years getting one thing right, a suggestion that appears while you type. Its 2023 lessons on scoping, testing and cost have aged better than the architecture, and they apply to any company putting a language model in front of its users.
In September 2023 the team behind GitHub Copilot wrote down how they took an AI coding assistant from idea to product. At the time Copilot did one main thing: it suggested code while a developer typed. Since then it has gained chat, a choice of models, agents that open pull requests on their own, and a new way of billing. The architecture in the post is history now. The lessons are not, and in 2026, when every company seems to be building something on a language model, they are worth reading as a checklist.
The problem: help without making people change how they work
Developers are, in GitHub's words, "consistently crunched for time". Most coding assistants of the day completed a single line, and GitHub wanted to suggest whole functions inside the editor. It also tried generating entire commits and dropped the idea because the quality was not good enough. The team's mantra sets the bar for everything that followed: "It's a bug if you have to change the way you code when using GitHub Copilot."
How they built it: find it, nail it, scale it
The post organizes the work in three phases, loosely based on the "Nail It, Then Scale It" startup framework.
Find it. Pick one kind of user and one problem, and balance what is ambitious against what the model can do well today. Whole functions were ambitious enough to matter and narrow enough to get right.
Nail it. The team dogfooded Copilot and ran A/B experiments on almost every change. Several beliefs did not survive contact with users. A web page for trying out the models failed because developers had to leave their editor to use it, so the experience moved into the editor as gray "ghost text" that the developer can accept or ignore. Using the developer's other open files as context raised the share of accepted suggestions by 5 percent, as GitHub reports. "The GitHub and OpenAI teams initially believed every coding language would require its own fine-tuned AI model", and one model turned out to cover many. Chat was shelved because "users had much higher expectations for the capabilities and quality of coding suggestions than GitHub Copilot could deliver at the time". It came back later as Copilot Chat. Because the model's output is probabilistic, every comparison needed statistics rather than a demo.
Scale it. A waitlisted preview brought in developers of all experience levels. Production moved from OpenAI's API to Microsoft Azure. When users reported that quality had dropped, the team added a guardrail metric, the share of multi-line suggestions against single-line ones, next to its main measures, the acceptance rate and later how much accepted code was still there after a while. Cost forced a design change too. An early version generated ten suggestions for every request, although most people took the first one. GitHub's research lead Idan Gazit put it this way: "It was like paying to calculate the results that appear on the second page of a search engine." Ghost text shows one suggestion, and lower randomness plus caching cut both variation and compute. Filters were added for suggestions that could contain insecure patterns, for offensive content, and for matches with public code longer than about 150 characters. GitHub sold to individuals first, and Copilot for Business followed eight months later, as the timeline at the top of this article shows.
What has changed since
The metric grew up. In October 2025 GitHub described a new custom completions model and reported "20% more accepted and retained characters", a "12% higher acceptance rate", three times the throughput and 35 percent lower latency. It also explained why acceptance alone stopped being enough: optimizing for it "could lead to incorrectly favoring a high volume of simple and short suggestions". The rule for shipping stayed the same: only "when statistically significant improvements hold up under real developer workloads".
Testing became infrastructure. The A/B habit now comes with offline benchmarks that run before anything reaches users. On 5 October 2026 GitHub released ReviewBench, an open set of 219 pull requests in 19 languages for testing AI code review. GitHub reports that the benchmark predicted the direction of a later A/B test, in which an ensemble of models raised the share of review comments developers acted on by 8 percent and cut the cost per review by 8 percent.
One model became many, and suggestions became agents. The chat idea that was shelved shipped at the end of 2023. Developers can now choose between models from several providers, and agents can take an issue, work on it and open a pull request.
Cost became part of the product. Agents changed the economics. GitHub moved Copilot to usage-based billing from 1 June 2026, because "a quick chat question and a multi-hour autonomous coding session can cost the user the same amount". Code completions stay included. In September it shared how it cuts agent costs, and the headline lesson echoes 2023: "Optimize the completed task, not the tool call. Shorter output is not cheaper if the agent spends more turns recovering what was removed." One prompt change was stopped mid-experiment when agents started working one after another instead of in parallel.
Productivity claims deserve the same care. The post repeats GitHub's 2022 figure of developers finishing a task up to 55 percent faster, which came from one experiment in which 95 developers wrote the same small program. In a 2025 randomized study by METR, 16 experienced open-source developers working on their own projects, mostly with Cursor rather than Copilot, took 19 percent longer with AI tools, while believing they had been faster. Both results can be true in their settings, which is why measuring your own work beats quoting anyone's average.
Why it matters for your business
In Indonesia the language-model projects we see most often are customer service on WhatsApp, assistants that answer from internal documents, and tools that draft sales messages or reports. They fail in the same places Copilot almost did.
They start too broad, as "a chatbot that answers everything", instead of one job done well. They ask people to open a new app, when the equivalent of the developer's editor is WhatsApp or the CRM the team already lives in. They measure the wrong thing: a bot's "resolution rate" looks like Copilot's raw acceptance rate, and the guardrail is whether the customer had to ask again. And they get surprised by cost, because a long conversation or an agent with tools costs far more than a quick answer, and per-seat pricing hides it until the invoice arrives. If the prompts carry customers' personal data, Indonesia's PDP law applies to where that data goes, too.
How we would build it today
- One user, one job, inside the tool they already use. A sales team's follow-up messages in the CRM, not an "AI portal".
- A test set before the first prompt. One or two hundred real cases with the answer you expect, run on every change of prompt or model. That is ReviewBench at the scale of one company.
- Roll out like an experiment. One main metric plus a guardrail, compared against a group that does not have the feature yet.
- Put the model behind your own gateway. Switching models is then a configuration change, and every call is logged with its cost.
- Measure cost per completed task. Cache what repeats, show one good answer instead of ten, and keep a person on anything that sends, pays or deletes.
The most useful thing in GitHub's post is not any single technique. It is the habit of treating every belief about the model, the users and the cost as something to test, and of changing the metric when the product outgrows it.