r/Android icon

Go to Android

Let's Encrypt and DNS over TLS Hell on Android

Archived post. New comments cannot be posted and votes cannot be cast.

Recently, after the DST Root CA X3 Root certificate expiry, I started facing issues with Android's Private DNS (DNS over TLS) connecting to my Adguard Home server that uses Let's Encrypt certificates.

This issue appears to be prevalent in all Android devices, connecting to DNS over TLS servers, specifically ones that use Let's Encrypt TLS certificates (which is pretty much everyone who self-hosts).

After many days of lost sleep debugging this issue, I decided to write something up to document my findings and to help other DoT self-hosters out there resolve this strange issue.

Edit: Thank you for the silver and the take my energy award!

More replies

More replies

Let's Encrypt did ask ACME client developers to transition to the correct chain when they came up with the work around: Extending Android Device Compatibility for Let's Encrypt Certificates

I’m an ACME client developer, what do I need to do?

If your client handled the X3 to R3 transition smoothly, then you shouldn’t need to take action. Ensure that your client correctly uses the intermediate certificate provided by the ACME API at the end of issuance, and doesn’t retrieve intermediates by other means (e.g. hardcoding them, reusing what is on disk already, or fetching from AIA URLs).

I think Google will continue to not enforce notAfter field of trust anchors because android certificate store is still not a part of mainline modules and fragmentation will revive this issue again in the next decade. This was the first and last occurence when android fragmentation had almost broken the old android devices.

It might be that android still uses an older version of openssl. NodeJS 6 and older have a similar problem caused by an old version of openssl which incorrectly checks only the very first certificate chain (which is invalid).

More replies

Holy shit! I too spent countless hours wondering why the Private DNS on my Android devices stopped working with my PiHole instance! And that's around the same time when I migrated the instance to another server so I thought there was something I messed up! Thanks OP!

Oh so that's why my self-written, self-hosted DNSoTLS server stopped working. I assumed I somehow misconfigured certbot, and being too lazy to figure it all out just switched to AdGuard DNSoTLS.

This fucked up my Android TV, since I made some manual entry of my DVB-T2 channels over IP using https with let's encrypt. Kodi would not play it at all, and I have to use "Ignore HTTPS" in other apps.

Thanks for the workaround. I was actually thinking of using other free HTTPS cert providers after this issue

Has anyone else been having issues loading some websites on Android 7.0 and below ever since DST Root CA X3 expired?

I know that ISRG Root X1 is supposed to be cross-signed for old Android compatibility, and it does mostly seem to be working. MOST Let's Encrypt websites (for example, Stack Exchange or Let's Encrypt's own website) do load perfectly fine.

However, there are a couple that just don't load. For example, OBS Studio's homepage is broken (https://obsproject.com).

This is what I get on Chrome 95, Android 7.0 - https://i.imgur.com/Rvjrlua.png

As mentioned before, the cross sign works perfectly on stack exchange, and you can see the cross sign here in action - https://i.imgur.com/GlcoGOn.png

OBS Project also loads fine in Firefox, which uses its own root certificate stores, and thus trusts ISRG Root X1 already, without having to rely on the DST Root CA X3 cross sign. It seems like on these websites, the cross sign is broken for some reason, and I still don't understand why.

So based on what I see. It's precisely the same issue. The OBS Studio site that you showed uses the Alternate chain, which means the last certificate in the chain is a self-signed ISRG Root X1 cert. This is definitely not compatible with phones that received their last updates before 2016.

Old Android phones will only remain functional on the default trust chain whereby the last certificate in the chain is DST Root CA X3.

In other words, some parts of the internet is probably not accessible by the Android 7 phone. Specifically the ones with the alternate chain Let's Encrypt certificates.

More replies More replies

Now I know why browsing on my phone becomes really slow whenever I use dns.adguard.com.

Actually tbh this probably doesn't have anything to do with the speed but more likely the latency of Adguard's DNS. Their physical servers are likely far away from your current location. But that's just my guess.

More replies

and I thought it was just me. Ffs.

This was a very wonderful read, regardless of your experience with LetsEncrypt or personally run DNS stuff