UPDATE — July 20th, 2021
We’ve completed phase 2 of the OpenVPN CA sunset process mentioned below. If you have trouble connecting and use the OpenVPN protocol, simply restart the app which will resolve the issue. There should be no impact/anything to do for most people.
If you are using custom configs, you may need to re-download them or remove the “compress” flag if you downloaded them after July 8th 2021.
This completes the CA sunset process. The final step will happen on August 3rd 2021 when compression will be completely deprecated on the servers. If you download the new configs now, they will no longer contain the compression flag and will work now and after August 3rd.
We are making changes to our OpenVPN certificate authority which involves client side changes some of which are breaking depending on if you use our applications or not. The majority of you using our client applications will not need to do anything beyond waiting and/or logging out and back into the app.
The following is a lengthy article with a lot of technical detail, however here is a quick summary of actions that are required on your part.
If you are using:
- IKEv2 (default protocol in all our apps) or Wireguard: No impact and no action required.
- Browser Extension: No impact and no action required.
- OpenVPN with our Windows and Mac apps: No impact, potential connection issues may occur which can be resolved by restarting or updating the app.
- OpenVPN with Mobile (Android/iOS) apps: No impact, your client will update within 24 hours or on full app restart.
- Linux CLI app: No impact. If you’re unable to connect, simply log out and log back into the app.
- OpenVPN 2.4 or newer outside of our apps: You will need to download a new configuration from our website and configure it on any devices that use the current configurations before July 20th 2021 17:00 UTC. You can do this right now.
- OpenVPN 2.3.18 or older outside of our apps: There is a problem with the way that validation works with our transitionary cross-signed certificate chain (during the sunsetting period) in older versions of OpenVPN and as such you will have to wait until July 20th 2021 17:00 UTC when we sunset the current CA and then download a new configuration from our website. Your current configuration will abruptly stop working at that time. You can download the new configs right now, and simply start using them once the current ones stop working.
What is changing?
We’re sunsetting (discontinuing) our current OpenVPN certificate authority in favor of a brand new one that follows industry best practices, including the use of an intermediate certificate authority (CA). With this new intermediate CA we have already generated and deployed short-lived server certificates used by the servers you connect to with OpenVPN. This follows best practices, similar to LetsEncrypt (which we use for IKEv2 and our browser extension), where a certificate is never valid for longer than 90 days. We have already deployed the new certificate chain on our servers which is cross signed by the old certificate authority allowing users to gracefully transition over prior to the sunset date.
The impact for those of you not using our desktop/mobile client applications with old or custom configurations is that you will be unable to connect to our servers come the date of sunset which is slated for July 20th 2021 17:00 UTC. Custom configuration users will need to go to our website and generate/download the new configuration files and CA trust (bundled in one, or separately if you wish). You can do this right now, ahead of the deadline. The new client CA trust and configuration will work with the existing certificate bundle sent by the server upon connection (pre-sunset) as well as the new bundle (post-sunset). The exception to this if you are using OpenVPN <= 2.3.6 whereby the trust will not work properly (please see the note below).
If you’re using our desktop client applications for Windows, MacOS, iOS or Android, you don’t have to do anything provided you’re running the latest versions of the mobile apps, and any version of the Windows and Mac apps. If you have trouble, simply restart or update the app which will fetch the new configuration files.
If you’re running the deprecated Linux CLI client, you will have to log out and log back in to get the new configs (the 2.0 Linux GUI app is coming shortly).
Why is it changing?
There are four reasons for the change. All are aimed to improve security, compatibility and long term maintainability of our OpenVPN infrastructure. The first one is of the highest priority.
Server seizure incident
On June 24th 2021 our monitoring systems alerted us that two servers in Ukraine had gone offline. When engaging with our provider for those servers, we were informed that the two servers had been seized as part of an investigation of activity that occurred 12 months prior. The hosting provider failed to inform us of a preliminary hearing that took place earlier this year, during which a judgement was rendered to seize the two servers in question.
We have no reason to believe that the servers were compromised or that there was any unauthorized access before seizure. As we do not log VPN traffic, no customer data from those servers while in operation are at any risk.
On the disk of those two servers was an OpenVPN server certificate and its private key. Although we have encrypted servers in high sensitivity regions, the servers in question were running a legacy stack and were not encrypted. We are currently enacting our plan to address this.
Impact
In extremely limited cases, an entity in possession of the private key, with a very high level of resources and access to your network could impersonate a Windscribe VPN server and capture VPN tunnel traffic running through it. The Ukrainian authorities have the hypothetical ability to impersonate a Windscribe OpenVPN server only if all 4 of the following conditions are met:
- The attacker has control over your network and can intercept all communications (privileged position for MITM attack)
- You are using a legacy DNS resolver (legacy DNS traffic is unencrypted and subject to MITM)
- The attacker has the ability to manipulate your unencrypted DNS queries (the DNS entries used to pick an IP address of one of our servers)
- You are NOT using our Windscribe applications (our apps connect via IP and not DNS entries)
The potential impact for the user if all of the above conditions are true:
- An attacker would be able to see unencrypted traffic inside of your VPN tunnel
- Encrypted conversations like HTTPS web traffic or encrypted messaging services would not be affected
- An attacker would be able to see the source and destinations of traffic
It’s important to remember that:
- Most internet traffic is encrypted (HTTPS) inside of your VPN tunnel
- No historical traffic is at risk thanks to PFS (perfect forward secrecy) which prevents decryption of historical traffic, even if one possesses the private key for a server
- No other protocols supported by our servers are affected, only OpenVPN
Timeline of Disclosure
The moment we were informed of the seizure, we immediately began working to address it. We needed to balance informing our users and ensuring that a plan for addressing the issue was well understood, developed, and ready to be implemented.
There were two major considerations: First, the proficiency and resources an attacker would require to build a campaign with the exposed server certificate and key. And second, the impact that sunsetting the previous Certificate Authority (which was not compromised) that issued it would have on all of our existing users. Between these two points we wanted to have a tested and ready to deploy path forward, with questions answered and available for all before we flipped the switch in the most expedient and responsible amount of time possible.
We needed to specifically test out how our certificate sunsetting strategy would work with as many possible end user configurations as we could out in the wild, while at the same time having no knowledge about their setup due to the technical realities and privacy that our service provides. We had to understand how different versions of OpenVPN behaved with different certificate signing extensions and methods, so that we could support as broad of a client base as possible in the most secure and forward thinking fashion while minimizing the amount of inconveniences that result from this migration (no, we didn’t just use easy-rsa to redo our CA in 5 minutes and call it a day).
Corrective Actions
We have been working on in-memory based servers for some time, and building out our automation for a new PKI and provisioning infrastructure.
The simple truth is that these safeguards were not in place when the server seizure occurred. This should not have happened and we understand that it hurts the trust you all have placed in us. The plans to upgrade our server stack were deferred in order for us to grow our team and build the foundations that would allow us to execute the planned improvements. We have grown our team over 50% in the last year and are now ready to make this a reality.
In-memory servers
We will transition all Windscribe servers to operate as in-memory servers with no hard disk backing. This means that our servers and any data they contain or generate, live and die in RAM and cannot be accessed after a machine has been shut off or rebooted.
The real challenge is not the in-memory OS itself, as we’ve perfected that a long time ago and are currently using it in the ControlD infrastructure (a DNS service we made). The issue is rather building the new provisioning system to actually deploy and maintain the Windscribe server stack, which is quite complicated relative to ControlD.
This new system will allow us to rapidly iterate on everything that’s running on a typical Windscribe server (over 40 different components and services), and deploy and manage services orders of magnitude faster.
We expect to start testing this new system in the coming weeks, and transition the entire server fleet by early Fall 2021.
UPDATE: As a stopgap measure, we’ve eliminated the need to store the private keys on disk ahead of the entirely in-memory server stack which is currently being worked on. This eliminates the chance of a similar issue reoccurring until we get this new infrastructure into production.
New server stack
Once the new provisioning system is in place and our entire server fleet is running the in-memory stack, we will have a clear path to deploy a lot of new features both internally and customer facing. Some of this includes:
- Wireguard as the primary protocol — We’ve been working on a fork of Wireguard for the last 9 months which will allow us to make it the default protocol in all our applications.
- Resilient authentication backend — This will allow VPN servers to function even if there is a complete outage of our core API and primary infrastructure.
- New application features — Ability to rotate your IP without disconnecting, request a specific IP (effectively static IPs for everyone), multi-hop, client side R.O.B.E.R.T. rules that are not stored in any database, and a whole lot more.
Audit
We will have a third party assessment completed to validate that our next-gen server stack and provisioning system is sound. We want to guarantee that our service delivers on all of the promises that we make by virtue of being a VPN provider. We are confident in hitting our goals, and we want you to be able to trust us when we say we have delivered on our promises in the form of a third-party audit. We aim to have this completed by the end of 2021.
Other Reasons for the Change
Although the main reason we’re doing the CA sunset is a paranoid level precaution because of the above mentioned incident, we’d like to take this “opportunity” to further improve the Windscribe OpenVPN infrastructure to tackle some long standing issues. These include:
CA expiry date compatibility issues
When we started Windscribe, back in 2016 and created the current certificate authority, we set the expiry date of the CA to the year 2040. This created compatibility issues on some devices that use LibreSSL library, which suffers from the 2038 problem. This prevents affected devices from connecting to Windscribe. A new CA will solve this problem.
x509 certificate validation options
In order to provide validation that you are connecting to the location you intend to connect to, we plan to start using the — verify-x509-name in combination with the existing — remote-cert-tls server OpenVPN options. This is similar to how your browser will check the domain name you browse is a part of the TLS certificate served to you upon connection to the website. This was previously impossible because the server certificates were generic and did not contain metadata unique to each server/location. This will further ensure that you’re communicating with an OpenVPN Windscribe server you think you’re using.
Compression phaseout
Our current OpenVPN server and client configurations use the compress parameter which is now deprecated. In very narrow circumstances this can allow a hypothetical attacker to infer the contents of the encrypted connection while using insecure HTTP websites (not HTTPS). This is unrelated to the CA sunset but since we’re going to have to restart all OpenVPN servers and distribute new configurations anyway, it makes sense to remove this as well. On the date of sunset (July 20th 2021 17:00UTC) when we restart our OpenVPN servers a second time, we will begin the phased removal of OpenVPN compression. Users of our desktop applications will not notice anything as we have a mechanism for transitioning away from compression gracefully. Custom config users should remove all compress options from their configurations (or simply download the config from our website). Again, our application users will not be required to do anything.
Conclusion
For most of you, these changes will not be noticeable (or have any impact at all), however for others there may be cause for inconvenience. We hope that you will bear with us as we go through the steps to ensure that we have a solid foundation to deliver on our promises, regardless of the VPN protocol you use.
Sincerely,
The Windscribe Team
