Ukrainian server seizure — a commentary and state of the industry

Yegor Sak -

This article is a follow up to the OpenVPN CA sunset article, which outlined a double server seizure incident that occurred in Ukraine. Though no user data was or is at risk on those servers, this is a follow-up explaining the fixes and measures we have put in place to further protect our users. We then dive into analysis of other VPN providers.

As of July 20th, 2021, the situation has been fully remedied. The implemented changes are as follows:

  • All required keys for server functionality are now stored and exist solely in memory after they are put into operation.
  • All data centers have unique short lived certificates and keys (with a 90 day expiry) generated from our new CA which are rotated every 75 days
  • Each server certificate has uniquely identifying Common Name + SANs
  • New OpenVPN client configurations validate X509 extended key attribute metadata in the server certificates using the common name which is unique to the server location

Since no customer data and, as of July 20, no private keys are ever stored on the server itself, this issue cannot occur again, regardless of the encryption state of the server. In the event of a server seizure, there is nothing useful for anyone to take.

But we are still taking things a step further. Our ultimate solution is a new in-memory system which ensures that all data is completely ephemeral. Once this system is released, a technical article will be released that will explain precisely how in-memory data protection is implemented.

In the meantime, the measures put into place adequately mitigate this and other similar scenarios. While we know that disk encryption is a worthwhile precaution to take in general, we feel that this type of infrastructure is meant to be treated as ephemeral, which is why the in-memory stack approach is the best path forward as it doesn’t require a hard disk to be present.

The rest of the article will be fairly technical and will take a deep dive into the OpenVPN infrastructure of common VPN providers to provide some insights into the state of the consumer VPN industry.

In short, it’s not good.

To disclose or not to disclose… a no brainer question

When a company suffers a security breach (by virtue of a malicious 3rd party or government seizure), it has one of two choices to make:

1. Fully disclose the issue, accept responsibility, and provide a solution within a reasonable amount of time. This has financial and reputational implications that may be hard to recover from.

OR

2. Say nothing, quietly resolve the problem (or not, as you will see later), and hope it never surfaces. If it does surface, throw as much PR and marketing staff as you can and deny the true impact of the issue. The benefit of this is that if the issue never surfaces online (and there’s a very high chance that it won’t), you don’t lose any customers and nobody is the wiser.

If you’re a dishonest company, the 2nd option is a no brainer. Especially if it’s a government seizure, since seized material rarely ends up leaked online for profit/lulz. It dies in some court case that never sees the light of day.

We’ve chosen to go with option 1: disclose everything. If you’re a regular reader of this blog, you know we have strong opinions about the VPN industry, and doing anything other than being 100% transparent goes against everything Windscribe stands for.

Basic ethics may be commonplace in other industries, but in the consumer VPN industry the “norms” are different. Most VPN companies are operated by anonymous individuals, through offshore shell companies that they claim are for “your protection”, because being based in Panama, BVI or Isle of Man somehow puts you above the law (it absolutely doesn’t). In reality, this is all marketing, and the only people those offshore companies protect are the owners… from paying taxes. All for your safety…

With all of the above being commonplace, there are virtually no incentives for them to be honest because there is an extremely high chance they will not get caught for any wrongdoing. Let’s dive into some recent examples.

NordVPN 2019 Hack

This section of the blog article is brought to you by NordVPN (Nord’s lawyers: This comment is satire, the rest is fact).

Back in 2019, NordVPN’s server certificate + key was publicly leaked as a result of a breach of a single server in Finland back in 2018. NordVPN vehemently denied the seriousness of this breach, claiming that it only affected that single server. We don’t have access to the OpenVPN configs they used back in 2019, however what we DO have access to is what’s currently available on their website. One might imagine that 2 years later, everything should be in a better shape from a security perspective. So let’s have a look.

Using this handy trick, you can download the remote certificate presented by the OpenVPN server. Once you do, you will discover something like this:

Just the relevant parts

From this, you can determine the following:

  • The server presents a unique certificate with the Common Name (CN) being the hostname of a random server in the US. This is a good start.
  • The certificate is good for 1 year. This is not ideal, but not terrible. In theory, if this certificate + matching private key were to leak someone could impersonate a single server for 1 year.

However, this is just one side of the story. The client that connects to this server using a downloaded OpenVPN config file looks like this:

Just the relevant parts

What you will notice is the lack of x509 verification. If that was in place, you would see a verify-x509-name parameter in the config.

Why is this important?

This works exactly like your browser when you access a HTTPS website. Say you visit reddit.com, when you enter the URL into your address bar, your browser connects to the Reddit servers which sends a TLS certificate over the wire for reddit.com. Your browser then checks the certificate to see if reddit.com is indeed present in the common name or SANs (subject alternative names), that it is not expired, and that it was signed by a publicly trusted certificate authority (CA). If these conditions are true the website will load. If they are not true then you will be presented with an insecure connection error.

The OpenVPN client, by default, does NOT verify that the server you are connecting to is the server that you expect it to be (ie. the hostname you connect to is in the certificate’s common name). The only thing it does is verify is:

  • The certificate has been issued/signed by the Certificate Authority that is trusted inside the <ca> block in the config
  • The certificate is not expired

Unless x509 verification is in place, the client will trust ANY server that presents a certificate that was generated by the Certificate Authority as long as it’s not expired. With that in mind, a breach of a single server, regardless of the unique certificate being deployed there, gives the attacker the ability to impersonate ANY other server for that VPN provider. This makes unique certificates and keys entirely useless, and offers no security benefits whatsoever.

So, assuming back in 2018 NordVPN didn’t perform x509 verification, which is a reasonable assumption since they don’t do this now, the breach of a single server would have given an attacker the ability to impersonate all servers on their network. Mind you, the technical requirements to execute such an attack are quite high, but it was possible nevertheless. This is equally as true now, in 2021.

Now you may be thinking, none of this matters because all of their servers run entirely in-memory, and if they were to be seized or breached, there is nothing of value to take. These claims may very well be true, however there is no way for you to verify them, or the fact that what they did was implemented correctly. What you CAN verify are the points mentioned above. If they were not properly implemented, especially after a serious breach that was brushed under the rug by their legion of PR and marketing people, can you really trust them?

None of this was raised in the multiple audits that NordVPN has undergone, which just goes to show that being audited is not a sign that everything is being done correctly. It’s a very good idea to do one, but it does not guarantee anything.

Additionally, in-memory-only VPN servers are great, but it’s not a bulletproof solution. If a government agency wants something bad enough, there are ways to dump the server’s memory while it’s running. If your only defense is “Everything is in RAM bro, trust me”, this is not a worthwhile solution.

TorGuard 2019 Hack

Another VPN provider suffered a virtually identical data breach around the same time as NordVPN. Now that 2 years have passed, let’s see if anything was actually learned from that unfortunate experience.

Ohh boy. This is the worst case scenario.

  • No identifying information in the Common Name, which is just “server”.
  • The certificate is good for 10 years.

A breach of a single server would allow the attacker to impersonate ANY other server until 2029. X509 verification is also impossible with this certificate, since there is nothing unique in it.

TorGuard also claims to operate entirely from in-memory/ramdisk servers. Can you verify that this is the case or that it was correctly implemented? No you cannot. What you can verify is that the current OpenVPN infrastructure is virtually as insecure as it was back in 2018–2019.

Update: TorGuard attempted to downplay the issue claiming their CA private key was not on the server. No VPN keeps CA private key on the server (as there is no point to do that). However a cert + matching private key minted by the CA was on the server. Since the cert has no identifying information in it, it’s effectively as good as a CA private key being leaked. You can’t generate new certs with it, but the key + cert you have is basically “god mode”, and can be used to impersonate ANY server. So either they don’t know what they’re talking about (bad), or lied about it (also bad).

PIA’s “Preventative Security Update” 2016

Private Internet Access suffered a Russian server seizure back in 2016, much like Windscribe in Ukraine just last month. The situation is identical, but see if you can spot the difference in how it was presented.

The move has nevertheless prompted a “preventative” security update, which will see Private Internet Access rotate its certificates. “Furthermore,” it said, “we’re updating our client applications with improved security measures to mitigate circumstances like this in the future, on top of what is already in place.

There is absolutely no reason to do this, unless your certificates and keys were compromised, which could be used to impersonate ANY other server. They failed to mention this crucial tidbit.

So 5 years later, has anything changed? Let’s see.

At first glance, that looks pretty reasonable.

  • Certificates are valid for 6 months.
  • There is unique data in the certificate that identifies a server or location.

Now let’s look at the OpenVPN client config.

No X509 verification, which makes that fancy certificate useless to prevent the exact same problem. 5 years later, not much was learned. But hey, everything is in RAM, so don’t worry about it. </sarcasm>

Surfshark

Although Surfshark was never publicly compromised, and never announced a server seizure taking place, they’re a fairly popular provider according to Youtubers who are paid to promote it. Let’s see what they’re doing.

This suggests:

  • The certificates are valid for up to 1 year, not great but not terrible either.
  • There is a unique Common Name in the certificate, which can be used to perform X509 verification. Is it being performed?

Nope it is not. A seizure of a single server will result in the attacker’s ability to impersonate any other server Surfshark operates. They too claim that the VPN servers are now all RAM based. Can this be proven to be implemented correctly, or at all? Nope. Can it be proven that their OpenVPN implementation is insecure? Yup.

ExpressVPN

This particular provider has been around for a long time, and was never publicly compromised. They did have a high profile server seizure in Turkey, back in 2017. The investigation revealed that there were no useful logs on the server regarding the alleged activity. This is not as impressive as you think. What was omitted from the discussion is the fact that the server was seized, along with everything else on it. This included private keys and certificates as well, since they announced their “TrustedServer” infrastructure only 2 years after that. The effect of this seizure is identical, and would have resulted in Turkish authorities being able to impersonate any ExpressVPN server. So, 4 years have passed since then. What does it look like now?

The above OpenVPN server certificate suggests the following:

  • The certificate is valid until 2071. I guess they wanted to make sure their servers outlive all the staff that currently work at ExpressVPN. This is not good.
  • The server common name appears to be unique, so in theory you could do X509 verification. Is it being done?

At first glance, yes! However once you dig deeper, you will notice a fatal flaw. To explain why, let’s look into the OpenVPN manual.

verify-x509-name Server name-prefix in the ExpressVPN config looks strangely familiar, as if it was copy/pasted directly from the OpenVPN manual. Now considering all ExpressVPN servers (that we tested) have a common name that starts with the word “Server”, this configuration parameter is completely useless and does not offer any security whatsoever. Kudos for trying, booooo for not trying hard enough.

Update: ExpressVPN denies the severity of the issue, however they miss the point. All OpenVPN configs that are used by users outside of their apps are currently vulnerable to the issue mentioned above. This is due to incorrect X509 verification, and the fact that the client cert + key + TLS key are identical between all locations. A compromise of a single server’s private key + cert would allow the attacker to impersonate any ExpressVPN server. You can confirm this yourself by downloading an OpenVPN config from one location, and then change the “remote” line to any other location. The config will still connect. Since the configs use DNS to get an IP address of a server, if an attacker posses a private key from any machine, and spoofs *.expressnetw.com to an IP they control, the victim will connect to that server and not be aware that it’s malicious. They cannot deny this, because it’s verifiably true.

The fact that Lightway protocol is secure from this attack (not verified by us), or the fact that the official apps may not be vulnerable to this type of attack is outside the scope as it was never in question. Our own apps were not vulnerable to this either, even with a breached key. However custom config users were at risk, hypothetically. Same applied at the time of their 2017 server seizure.

Perfect-Privacy

This provider suffered a public server seizure back in 2016. The company stated no customer data was obtained, which again, is not that impressive. What was conveniently omitted from the statement is that the drives that were seized and copied had to have contained private keys, which were in the hands of Dutch authorities. Since Perfect-Privacy does not perform X509 verification in 2021 (although they do have unique client cert + key for each location which accomplishes a similar goal), and their RAM-only infrastructure didn’t exist then, it’s safe to assume that those keys could have been used to impersonate any other server, for the lifetime of the certificate.

Update: PP disputes this claim as it appears the RAM infrastructure was in place at the time. We figured that this would be made front and center in the disclosure announcement that no keys were leaked, and the seized drives had nothing on them. Customer data is never stored on VPN servers themselves, by default.

The above statement is very ambiguous. Multiple 1TB drives were copied, the drives were returned and wiped, yet not a single mention is made of what was on those multiple 1TB drives other than “no customer data”.

This could very well be “fine”, the drives could have been empty and private keys were not leaked, but nothing in the communication makes this evident. That’s all we’re pointing out.

So all hope is lost?

I can probably keep going and test every single provider out there (hey VPN review sites, how about getting on top of this?), but unfortunately I do not have all week. Is anyone actually operating OpenVPN correctly? The answer is yes.

IVPN

If you fetch IVPN configs, you will notice they do use X509 verification. Here is an example line from their New York location: verify-x509-name us-ny name-prefix

When you look at the certificate, you will see the following:

  • The expiry of the certificate is 10 years, not super great but less important in this case.
  • There is unique information in the certificate, which is being correctly verified via x509 verification parameter. This is really good.

Windscribe (as of July 20 2021)

Since this IS a Windscribe blog, we of course have to shill ourselves, now that we got our house in order. Before July 20th 2021, we suffered from the exact same issues which are currently plaguing almost all of the VPN providers out there. We’ve spent an ungodly amount of time researching the best possible solution. Here is what Windscribe’s certificates look like now, example from Amsterdam:

You will notice the following:

  • Our certificates are valid for 3 months
  • There is unique data in the certificate that uniquely identifies a specific datacenter. This data is used to perform X509 verification as follows:

The above configuration assures the following:

  • Your OpenVPN client cryptographically verifies that it’s connecting to the expected Windscribe server location.
  • Seizure of a certificate + key cannot allow an attacker to impersonate any other server location. Since our policy is to terminate all servers in a location where a seizure occurs, this effectively means it cannot be used for anything worthwhile after the seizure occurs.
  • Even if the certificate + key were somehow compromised, it would only be valid for a maximum of 3 months.

Other providers that we tested that perform X509 verification also include:

  • VyprVPN
  • IPVanish

Conclusion

The main take-away from this article is that humans make mistakes, as they’re typically the weakest point of any computer system. Just like all the VPN providers above, we’ve made a mistake. Ours happened early on in Windscribe’s 5 year existence, and we reaped the “fruits” of that mistake years later. What I find inexcusable is hiding the fact that a mistake was made, and delivering a “solution” that is equally as faulty, many years later, as virtually all VPN providers that we’ve tested have done. We’ve learned about the problem, and delivered a network-wide solution in less than 4 weeks that is aligned with best industry practices. Many of the other companies have 10x the staff we do, and were around for 2–3x longer. The fact that the above issues are present in their infrastructure years after these events occurred is simply unacceptable.