[Cryptography] Holy Heartbleed Batman - an Internet-scale attack on TLS crypto downloads
iang at iang.org
Sun Apr 26 10:25:03 EDT 2015
26 April 2015
Published here to resist censorship.
Surveillance system used for censorship in Europe
Censorship attack combines packet injection and Heartbleed
We all know there is censorship online. It happens in China. It happens
to "terrorists". But we don't believe it will happen to us.
As Eben Moglen and Kaspersky have pointed out, companies
developing crypto are prime targets no matter where they are. So you
don't have to be a bad guy for the NSA to attack you. You just have to
protect people from the NSA. Even protecting yourself is often enough.
NSA prefers their victims to be defenseless.
Detection in the wild
In early 2015 people were still downloading our ISO file for GoodCrypto.
But suddenly installations stopped.
After a lot of checking we noticed that the downloads got HTTP 200
result codes, but the lengths were all too short. This isn't supposed to
happen. A 200 result means success. These weren't successful downloads,
but the web logs said they were. Ordinary log checks didn't show the bug.
Finding the vuln
Downloads from goodcrypto.com to goodcrypto.com worked. Downloads from
another site at a different datacenter in the same country worked. A
little further away in the network, downloads failed but the server
logged a "Success" status code.
The obvious answer was a server misconfiguration. We couldn't find one.
A server side packet dump showed the client just dropped the connection
in the middle of the download.
We couldn't get a browser to download the whole ISO file. The browser
thought it came in fine, but the file was incomplete.
So was it a browser bug? We tried other browsers. They couldn't download
The wget program often helps debug downloads. It doesn't have the same
malware issues as browsers, because wget doesn't support malware vectors
often tells you why it failed.
When we tried wget, it detected errors, retried, and finally succeeded.
It said the error was a bad length field in a TLS packet. That didn't
make sense at first because we thought TLS packets were error corrected
We searched for other bug reports like this. They were all during
session initiation, not in the middle of a long download.
But our searching led to Heartbleed. Modifying SSL/TLS length fields is
exactly how it works.
Wasn't Heartbleed fixed in 2014?
In 2014 when we all heard about Heartbleed many servers were vulnerable.
But OS providers fixed it fast. Our own servers get regular security
How were we seeing it now?
Servers around the world were fixed fast, but clients were vulnerable
too. The Heartbleed news coverage was all about servers. Servers got
fixed. Many clients didn't. A client side Heartbleed attack is sometimes
called Reverse Heartbleed.
Packet evidence shows MITM
Was our server cracked? We're pretty careful, so that doesn't happen
often. But we checked, and checked again. Even though we don't usually
have any packet logs, we ran download tests with simultaneous packet
logs on both the server and client.
The server packet logs showed an ordinary number of bad packets, all
error corrected. During the download the client dropped the connection.
The client packet logs were very different. There was a surprising
number of bad incoming packets. Almost none of these bad packets showed
in the server logs for the same session. These packets appeared to be
injected into the packet stream. This is an MITM/MOTS attack,
specifically a packet injection attack.
Finally wget reported that a TLS packet with a bad length field got
through and caused the TLS connection to break. Now we knew this was a
MITM variant of Reverse Heartbleed.
Working around the censorship
We added simple instructions to our Download page:
Somebody (Hi NSA!) is trying to censor GoodCrypto downloads.
But don't worry. The workaround for their super duper advanced network
attack is: Just use wget.
If you're on windows you need wget.exe.
At a command prompt:
wget --no-check-certificate (URL appeared here)
Because you will verify the file hash, "--no-check-certificate" is ok in
this rare case, and Windows needs it.
We also strengthened the encouragement to verify the file, and asked
visitors to let us know when the attackers change tactics.
The attacker injected packets by forging our site's IP address. If a
site visitor notices the packets, they will think that the attack is
from us. The attacker didn't just shift blame away from themselves. They
framed someone they don't like. Faking the evidence to blame an attack
on someone else, especially someone the attacker doesn't like, is a
classic False Flag operation.
An MITM attack like this requires impersonation, so the benefits to the
attacker of a False Flag are built right in.
One way to censor anyone is to attack their reputation.
Censorship by a nation state
Even though the download broke, the browser didn't complain. It looked
like a successful download. The server showed an HTTP 200 result code.
Neither the client nor server detected the attack.
Server packet logs didn't show anything unusual except the abrupt client
disconnection. The attack didn't show at all during downloads from a
client that was close to the server in network space, such as another
nearby datacenter. It wasn't an attack on the server or the hosting
We set up another server in a different country and the attack
continued. It wasn't an attack by the hosting country. We were nowhere
near our transfer limits for either server, so it was not traffic
shaping by hosting providers.
The clients were all over the world. It wasn't separate attacks launched
from close to individual clients.
The attack appears to be from someone sitting on the net pipes and
injecting packets. This requires huge resources. The U.S., U.K.,
Canadian, and Chinese spooks do this. Ordinary criminals don't have the
Because the goodcrypto.com servers are in Europe, China is an unlikely
suspect. GCHQ and CSE are dependent on the U.S. for their QUANTUM
capability. That means it was likely an NSA attack, either directly or
But which nation really doesn't matter. There is no known way to protect
against specific nations. You have to protect against all or none.
Who is vulnerable
Anyone who publishes on the web is vulnerable to this form of
censorship. Even if you just use HTTP, the Chinese censorship method of
a simple RST works.
Update: The attack is now intermittent. Exposing them often helps. When
it's paused they may be avoiding forensics, or just changing tactics.
Let us know.
More information about the cryptography