How Zillow taught its spam filter to skip the voicemail greeting
Zillow already had a spam filter for messages. Pointed at voicemail transcripts, it missed about half of the spam, because it kept reading the property manager's greeting instead of the caller. In 2023 Zillow described SpectroBrain, a small audio model that finds where the greeting ends. Here is how it works, what the numbers show, and what it means for businesses that live on calls and voice notes.
On Zillow Rentals, renters call and message property managers through the platform. Spammers use the same channels. According to Zillow, they "have called our users to pitch tax debt relief services or cryptocurrency investments". In a May 2023 post on Zillow's tech blog, Winston Quock described SpectroBrain, a system built to find them in voicemail. Its best idea is not a smarter spam model. It is cleaning up what the existing model reads.
The problem: the filter was reading the greeting
Zillow already had a spam classifier for email and text messages, based on RoBERTa and, per the post, "in production for several years". It learned from users' spam reports. Phone calls had no such feedback, because of "the lack of a direct mechanism for telephone users to report spam to the platform". Labels for voice were scarce.
The obvious first attempt was to transcribe each voicemail and run the text model on it. It produced "about 50% false negative rates": half the spam slipped through. The culprit was the start of each recording. Before a caller says anything, there is usually a greeting or an automated phone menu. Zillow calls it the preamble and says "as many as 80% of phone calls through our platform begin with preambles". Many recordings are nothing else: "pure preamble messages account for 54% of all messages". A text model fed a leasing office's greeting and a list of menu options has little left to judge.
How they built it
Find the switch point. The task became finding the moment the preamble ends and the caller starts talking, then cutting there. A message that is only preamble is approved without further work. For the rest, only the caller's part goes to transcription, with OpenAI's Whisper used as is, and then to the existing text classifier. The diagram at the top of this article shows the flow. Only the cutting step is new.
Learn from audio nobody labeled. Zillow labeled just 530 messages by hand, and 48 of them were spam. That is far too few to train an audio model from scratch. So the team first trained an encoder on about 50,000 unlabeled recordings, about 1,000 hours of audio once each was trimmed to 240 seconds. It hid parts of each recording's audio features and asked the model to fill them back in, the same trick BERT uses for text and wav2vec for speech. The encoder learns what greetings, menus and live speech sound like without a single label. This is the "semi-supervised" in the title: self-supervised learning on unlabeled audio, then a small supervised model on top.
Three small experts instead of one big one. On top of the encoder, a tiny classifier scores each slice of about a third of a second as preamble or voice. A change-point search proposes a handful of candidate switch points from those scores. A gradient-boosted tree model (CatBoost) then judges whether whole segments sound like voice, and the candidate where the right side sounds most like a caller and the left side least wins. Each piece is small and data-efficient, which matters when the labeled set is a few hundred messages.
A warning about off-the-shelf parts. In one example, Whisper turned several seconds of silence at the end of a message into spam-sounding text, "despite the fact that the model does attempt to detect silence". A borrowed model can fail in ways your own tests have to catch.
What it delivered
Zillow reports the effect on the spam classifier. With preambles left in, it missed 31 percent of spam and wrongly flagged 20 percent of legitimate messages. With preambles cut by SpectroBrain, the misses fell to 12 percent and the false alarms to 15 percent. With preambles cut by a person, the figures were 7 and 14 percent, which shows how much of the problem was the preamble itself.
Read this as a promising experiment. The labeled set is small, the post does not say how many messages are behind the table, and it does not say whether SpectroBrain runs in production or what happens to a flagged caller. The post's own conclusion is modest and right: semi-supervised learning "does not diminish the needs or benefits of collecting more labeled data". Its value is starting a "model-data flywheel", in which a working first model helps collect better labels for the next one.
What has changed since
Fewer phone numbers to abuse. In May 2026, Zillow removed phone numbers from free rental listings, as property-management writers reported. Peter Lohmann quotes a Zillow spokesperson: "Zillow removed personal phone numbers from rental listings as a proactive safety measure", citing AI tools that make contact details easy to harvest. Paid listings keep a number, and Lohmann sees the change as a squeeze on free listings. Either way, more first contact now happens in channels Zillow can see and filter. We found no follow-up from Zillow on SpectroBrain itself.
Phones guard calls themselves. Google's Scam Detection on Android now checks live calls for scam patterns, and Google says that processing "is all done on-device", with no conversation audio or transcription stored or sent to its servers.
Indonesia counts its scams. The Indonesia Anti-Scam Centre, led by the financial regulator OJK, received 637,055 reports between 22 November 2024 and 31 July 2026, blocked about Rp724.1 billion and returned Rp204.3 billion to victims, OJK reports. In January 2026 the communications ministry issued Regulation No. 7 of 2026 on mobile subscriber registration, which, per the ministry, uses face recognition and caps prepaid numbers at three per identity per operator.
Why it matters for your business
In Indonesia, leads arrive as calls, WhatsApp chats and, very often, voice notes. Sales teams lose hours to spam, fake inquiries and scammers posing as buyers or as the brand itself. Zillow's lesson carries over to all of these channels.
You may not need a new model. If you already sort text messages, by rules, a classifier or a language model, transcription lets the same logic handle voice. But clean the input first. For call recordings that means cutting greetings and phone menus. For voice notes it means trimming silence and background noise, which, as Zillow found, can turn into invented words. And start labeling now: a few hundred examples, tagged by the people who handle the leads, is enough to begin.
Recordings are personal data under Indonesia's PDP law (Law No. 27 of 2022), and the law treats biometric data as a specific, more sensitive category. It defines biometric data as physical, physiological or behavioural characteristics that allow unique identification, such as face images or fingerprints. Transcribing what a caller says is a different act from recognising who they are by voice. Keep your system on the first side unless you have a strong reason and a clear legal basis. We are engineers, not lawyers, so treat this as a prompt to ask yours.
How we would build it today
- Label a few hundred real messages. Spam, scam, genuine lead and other, from calls, chats and voice notes alike.
- Transcribe with a model that knows your languages. Indonesian, English and the mix in between. Test it on your own audio, and drop silent or near-silent stretches before transcription.
- Strip the boilerplate. Detect where greetings and phone menus end in call recordings, and trim silence and music in voice notes.
- Classify the text. Start with a language model and a clear prompt, or a small fine-tuned classifier. Return a label with a confidence, and send uncertain cases to a person.
- Let your team label as they work. One-tap spam and not-spam buttons in the CRM turn every decision into training data, Zillow's flywheel.
- Flag before you block. Deprioritise suspected spam first, keep a way to restore wrongly flagged leads, and log every decision.
- Protect the recordings. Mask names and phone numbers in stored transcripts, keep recordings only as long as needed, and do not identify people by voice.
SpectroBrain's lesson is a humble one. Before you build a smarter model, look closely at what you are feeding the one you have. Zillow's filter was failing because it was reading greetings, and a small, careful cut fixed most of the problem.