Domain Fronting and Domain Fuzzing
- 14 min read - Text OnlyCharles Miller delivered his masters thesis at the Crypto and Privacy village at DEF CON. He introduced domain fronting, where a client contacts a Content Distribution Network (CDN) endpoint under one name, and the endpoint forwards traffic to another destination. This exploits a gap between TLS and HTTP as implemented by CDNs. His research found ways to identify endpoints at scale, even when Azure, Cloudflare, and Google clamp down on domain fronting on their infrastructure. Domain fronting is an important tool used by Tor to circumvent censorship in places that block encrypted SNI.
This talk summary is part of my DEF CON 31 series. The talks this year have sufficient depth to be shared independently and are separated for easier consumption.
I first learned that domain fronting existed on a thread about Azure blocking domain fronting. It was memorable because of this comment (archived) from Filippo:
Filippo Valsorda
It's been many years, and I am still angry and disappointed by Cloudflare's decision to block domain fronting and drop Lantern as a customer. Lantern was one of the most effective Great Firewall bypass proxies at the time, and Cloudflare was expanding in China. (I was at Cloudflare at the time, but I don't have private information on the deliberation. I strongly considered quitting over it, maybe I should have, but I was junior back then.)
The CEO even came on HN to try to frame it as an abuse mitigation, accusing Lantern of exploiting Cloudflare and arguing that they were not a customer. That was obviously false because you need to have a Cloudflare zone configured for domain fronting to work. They were a customer as much as the targeted hate websites they strenuously defend.
Charles Miller completed a masters program in information systems with a focus on cyber security at the University of Maryland Baltimore County, where his thesis focused on domain fronting and finding supported endpoints with python. For the written thesis, see Domain Fronting Through Microsoft Azure and CloudFlare: How to Identify Viable Domain Fronting Proxies (archived). He now works for CrowdStrike as an analyst.
What is domain fronting?
Domain fronting is when a request is sent to a server with one origin, but the request is forwarded to a different origin for processing. This is a quirk where different origins can be requested through the same connection with one origin in the TLS SNI extension and the other in an HTTP Host header. Domain fronting obscures the intended origin to a network operator or censurer by signaling that it is for a safe destination in DNS requests and the unencrypted SNI extension. Like Tor, the intended destination is encrypted and private, as it is sent within the encrypted TLS connection. Domain fronting is a way to obscure traffic, escape surveillance, and bypass censorship.
It works by sending a different HTTP Host header than the Subject Name Identifier (SNI) indicated in the TLS session. (See What is SNI? How TLS server name indication works for more detail on SNI.) The Host header is required by HTTP and is often processed by a different layer than the TLS session. Network appliances may examine traffic and filter traffic by inspecting unencrypted content. SNI is not encrypted and can be used to filter traffic.
Within the TLS session, the Host header is encrypted. A domain fronting request could go to the DNS host for allowed.example with allowed.example in the TLS SNI, while the Host header specifies forbidden.example. The host for allowed.example then forwards the request to forbidden.example. The forwarded address may or may not be on the same infrastructure as allowed.example.
Content Distribution Networks (CDNs) exist to route traffic and to cache traffic in local points of presence. Many HTTPS frontends, including CDNs, make assumptions like "If the SNI is for a host, and I have the certificate for the hostname allowed.example, then I can serve this request" and a later component say "this request is for forbidden.example, so I will request forbidden.example's origin." CDNs serve many tenants and the barrier to join the CDN network is quite low.
Domain fronting is possible because of the multi-tenant nature of CDNs and the inherent design of proxying traffic to tenant origins.
Discovering domain fronting endpoints
Which endpoints would forward requests to a different origin like forbidden.example when connected over TLS as allowed.example?
For a while, many domains would use CNAME records in DNS, which easily identified shared infrastructure by CDN providers. However, CNAMEs increase latency by requiring another DNS request. Now, Azure, AWS, and Cloudflare prefer DNS A records which respond with IPs to their nearby points of presence.
Charles first scanned for DNS CNAMEs to identify common infrastructure that supported domain fronting. With the industry's move away from CNAMEs in CDN resources, he needed another way to identify supported infrastructure.
The next attempt was to see which name servers and SOA Records signaled a likelihood of domain fronting support. As Azure and Cloudflare changed their policies, this turned out to be unreliable.
Charles then found that while nginx.com did not support domain fronting, for some reason www.nginx.com did. By trying many different subdomains like email.allowed.example with a script, he could find suitable domain fronting targets. In other words — fuzzing the internet!
Azure announcement
Many existing Azure CDN endpoints support domain fronting. Microsoft announced that they now block this capability on newly-created endpoints by default. Pre-existing endpoints are not migrated and there is no configuration or toggle for customers to change domain fronting themselves. Azure customers must contact support to reduce the risk that their hostnames are used by malicious traffic.
Charles then searched for subdomains of microsoft.com and found several that were vulnerable. He reported 28 instances, including security.microsoft.com. The security concern was closed as not important and a week later security.microsoft.com was fixed. The other 27 were not.
A recording
This talk was recorded from a seat in the audience and is available to view right now. If you'd like to see how Charles finishes domain fronting and how more endpoints can be found, see the video below.
Final thoughts
Domain fronting is a neat quirk. I also feel that it is too surprising to be considered a feature. On technical grounds: domain fronting is a defect. If we are to get better privacy with HTTPS, we should use technology intended to provide that security goal as intentional behavior.
Encrypted SNI is being standardized and is undergoing experimentation in Google Chrome, it can be accessed under the flag #encrypted-client-hello. A security goal of encrypted SNI is privacy from network operators.
When encrypted SNI spreads more widely, I expect we will need other techniques to clamp down on malicious behavior by malware. Unfortunately, the same techniques will be used for censorship too.
One technique used by the People's Republic of China is to block all traffic using encrypted SNI (archived). Encrypted SNI provides enough privacy that censoring state actors find it dangerous. That said, censorship resistance is not a security goal of encrypted SNI.
Tor has mechanisms to circumvent censorship in the People's Republic of China. One of the methods uses domain fronting on Azure (archived) with meek-azure to circumvent censorship. There are other censorship circumvention methods too. A security goal of Tor is censorship resistance.
The research Charles did will be beneficial to Tor if their list of meek-azure endpoints shrink.
Conflicted
I feel conflicted.
This surprise functionality is used to circumvent censorship and is used to obfuscate attacks against Ukraine (archived). Bypassing censorship is not malicious, to me, while hiding traffic that harms people is malicious. Some countries consider both to be malicious.
I have not thought about censorship resistance before in this way. I am not sure where to start or where to go when it is clear that honest attempts at privacy like encrypted SNI get cut off. Censorship resistance requires dishonest behavior to work, and that sucks.