How to Ignore Invalid and Self-Signed SSL Certificates Using cURL

If you‘ve ever encountered an error message when trying to access an HTTPS URL with cURL, it may be due to an invalid or self-signed SSL certificate. In this in-depth guide, we‘ll explore what invalid certificates are, how to bypass them with cURL‘s -k option, and discuss the security implications of doing so.

Content Navigation show

Understanding Invalid and Self-Signed Certificates

When a client (like a web browser) makes an HTTPS request to a server, the server responds with an SSL/TLS certificate that verifies its identity. This certificate is signed by a trusted certificate authority (CA) and contains information like the server‘s public key and domain name. The client checks that the certificate is valid and matches the requested hostname before establishing a secure connection.

An invalid certificate means there is a problem with the cert presented by the server. Some common causes include:

  • The certificate has expired and is no longer valid
  • The hostname or domain in the certificate doesn‘t match the requested URL
  • The certificate is self-signed and not from a trusted CA
  • There are issues with the certificate chain or intermediate certs

A self-signed certificate is one that is signed by the owner of the website rather than a trusted third-party CA. While self-signed certs are easy and free to generate, they do not provide the same level of trust and security as a properly signed certificate from a reputable CA. Self-signed certificates are often used for testing and internal sites.

According to a 2022 study by security firm Qualys, over 1.2% of the top 1 million websites had expired certificates, while 15% were using certificates not signed by a public CA. Another report by Venafi found that 76% of organizations have experienced an outage or security incident due to certificate issues. This shows that certificate problems are surprisingly common on the web.

Bypassing Certificate Errors with cURL‘s -k Flag

By default, cURL will refuse to connect to a server with an invalid or untrusted certificate:

$ curl https://expired.example.com
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

To ignore cert errors and proceed anyway, use the -k or –insecure option:

$ curl -k https://expired.example.com
<!doctype html>
<html>
  <head>
    <title>Welcome to expired.example.com!</title>

The -k flag tells cURL to skip the certificate validation checks and continue with the request. It‘s important to note that this disables all certificate verification, not just expiration checks. It also bypasses other important validations like:

  • Hostname mismatches, where the cert doesn‘t match the URL‘s domain
  • Certificate revocation status
  • Usage of weak hashing algorithms like MD5 or SHA-1
  • Missing or invalid intermediate certificates in the chain

Here‘s an example of the step-by-step process of accessing a site with an invalid cert using cURL:

  1. Make a request to the URL with an untrusted certificate:

    $ curl https://self-signed.example.com

  2. cURL detects the invalid cert and prints an error message:

    curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

  3. Make the request again with the -k option to ignore the cert error:

    $ curl -k https://self-signed.example.com

  4. The request succeeds and cURL outputs the response body:

    <!doctype html>
<html>
  <head>
    <title>Welcome to self-signed.example.com!</title>

So while the -k flag allows you to bypass certificate errors, it‘s crucial to understand the security implications.

The Risks of Ignoring Certificate Warnings

SSL/TLS certificates are a fundamental component of HTTPS and help provide the security and trust users expect from the web. Improperly configured or invalid certificates can expose your data to interception and spoofing.

Some key reasons certificates are important:

  • Authentication: Certs allow clients to verify the server‘s identity and avoid connecting to impostor sites.
  • Encryption: SSL/TLS certificates enable encryption of data in transit, preventing snooping of sensitive info.
  • Integrity: Certificates include a cryptographic signature that allows detection of tampering or corruption.

When you bypass certificate validation with cURL‘s -k option, you‘re disabling all of these critical security checks. An attacker could potentially:

  • Intercept the connection with a man-in-the-middle attack and impersonate the server
  • Spoof a legitimate website to steal login credentials or distribute malware
  • Eavesdrop on the data sent over the connection, which may contain sensitive info
  • Tamper with the response body, injecting false info or malicious code

A 2017 academic study found that 5% of HTTPS sites were accessible with common HTTPS interception tools due to improper validation. In 2019, security researchers discovered a widespread MITM vulnerability in TLS proxies affecting companies like Kaspersky and Symantec. Ignoring certificate errors makes you more susceptible to these types of risk.

While the -k flag may be helpful for debugging or testing with self-signed certs, it should never be used in production or with untrusted sites. Instead, consider adding self-signed test certs to your local trust store so they can be properly verified without disabling overall validation.

How SSL/TLS Validation Works

To better understand the role of certificates, let‘s take a closer look at how clients validate SSL/TLS certs:

  1. Upon connecting, the server sends its certificate chain, including its leaf cert and any intermediates up to a root.

  2. The client confirms the certificate is signed by a trusted root certificate authority in its trust store.

  3. It checks that the certificate is valid for the requested hostname and has not expired.

  4. The client verifies the integrity of the cert by checking its digital signature.

  5. For extended validation (EV) certs, the client confirms the certificate meets additional identity verification criteria.

  6. The client checks if the certificate has been revoked using OCSP or CRL.

If any of these checks fail, the client will halt the connection and show an error. The exact behavior varies by client – browsers show warnings and require explicit user action to proceed, while tools like cURL and wget will refuse to connect without a bypass option.

Behind the scenes, cURL uses a set of root CA certs provided by the Mozilla NSS library. You can see the paths with:

$ curl-config --ca
/etc/ssl/certs/ca-certificates.crt

When you use the -k option, cURL essentially skips steps 2-6 in the validation process and trusts whatever certificate the server provides.

Best Practices for Secure cURL Usage

Here are some tips for using cURL safely and avoiding the need for -k:

  1. Only use -k for local testing or with self-signed certs you‘ve manually reviewed and trust. Never use it in production or with third-party sites.

  2. For internal servers, add your self-signed certs to cURL‘s CA bundle so they can be properly verified:

    $ curl --cacert /path/to/your/root-ca.crt https://internal.example.com

  3. Keep your cURL version up-to-date to get the latest security patches and protocol support. Vulns in cURL are rare but have happened, such as CVE-2022-35252 and CVE-2021-22945.

    $ curl --version
curl 7.81.0

  4. If you need to inspect a suspicious site, use -v to view the cert details without bypassing validation:

    $ curl -v https://untrusted.example.com
...
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.

  5. Harden your cURL configuration by creating a ~/.curlrc file with strict defaults like:

    cacert = "/path/to/trusted-certs.pem" 
capath = "/dev/null"

  6. Consider using a separate network security tool for scanning and auditing HTTPS configs, such as testssl.sh, sslscan, or sslyze. These tools provide detailed analysis without the risks of -k.

With the proper precautions, cURL remains an invaluable tool for interacting with the web and APIs. However, it‘s critical to understand its security model and implications of the options we use.

Comparing Other Tools

Different HTTP clients and libraries take varying approaches to handle certificate errors:

  • Web browsers will show a warning page and require explicit user action to proceed past an invalid cert. For example, in Chrome you must click "Advanced" and then "Proceed to example.com (unsafe)"

  • The wget tool has an –no-check-certificate option that behaves similarly to cURL‘s -k.

  • The Python requests library allows disabling of cert verification with verify=False, though it displays a warning.

  • In Node.js, the https.request() method has a rejectUnauthorized option that defaults to true. Setting it to false disables cert validation.

  • The OpenSSL command line tool allows bypassing validation with the -no_check_certificate flag for its s_client command.

Here‘s an example of the wget command ignoring cert errors:

$ wget --no-check-certificate https://self-signed.example.com

--2022-03-04 09:26:07--  https://self-signed.example.com/
Resolving self-signed.example.com... 203.0.113.100
Connecting to self-signed.example.com|203.0.113.100|:443... connected.
WARNING: The certificate of ‘self-signed.example.com‘ is not trusted.
WARNING: The certificate of ‘self-signed.example.com‘ hasn‘t got a known issuer.
The certificate‘s owner does not match hostname ‘self-signed.example.com‘
HTTP request sent, awaiting response... 200 OK
Length: 235 [text/html]
Saving to: ‘index.html‘

index.html            100%[=======================>]     235  --.-KB/s    in 0s

2022-03-04 09:26:07 (18.1 MB/s) - ‘index.html‘ saved [235/235]

Like cURL, wget shows a warning about the untrusted issuer and hostname mismatch but proceeds with the request when passed the –no-check-certificate option.

SSL/TLS Trends and Vulnerabilities

Over the past decade, HTTPS adoption has skyrocketed thanks to initiatives like Let‘s Encrypt and Google‘s prioritization of HTTPS in search rankings. As of March 2022, W3Techs reports that 81.8% of websites use HTTPS by default, up from just 31.6% in 2017.

However, this rapid growth has also led to increased scrutiny of HTTPS misconfigurations and vulnerabilities. Major attacks and issues disclosed in recent years include:

  • Heartbleed (2014): A buffer over-read in OpenSSL that allowed theft of private keys and memory contents
  • POODLE (2014): A weakness in the obsolete SSLv3 protocol enabling man-in-the-middle attacks
  • DROWN (2016): An attack on servers supporting SSLv2 that allows decryption of TLS connections
  • ROBOT (2018): A vulnerability allowing RSA decryption and signing in TLS versions before 1.3

The industry has responded with faster migration to newer protocols, stricter defaults, and automation of certificate issuance and renewal. TLS 1.3, standardized in 2018, removes legacy features and provides significant security and performance improvements over TLS 1.2. As of February 2022, Cloudflare reports that 80% of HTTPS traffic uses TLS 1.3.

However, many older and vulnerable servers remain online. SSL Pulse reports that as of March 2022, 25.1% of surveyed sites still support SSLv3, while 9.1% are vulnerable to Heartbleed. Continued effort is needed to identify and remediate these risks.

Conclusion

Invalid SSL certificates are a common stumbling block when accessing HTTPS URLs with cURL. While the -k or –insecure flag offers a quick workaround, it‘s crucial to understand the major security implications of disabling certificate validation. Bypassing these checks leaves you vulnerable to MITM attacks, spoofing, and eavesdropping.

As we‘ve seen, invalid and self-signed certificates are surprisingly widespread on the web. Expired certs, non-public CAs, and hostname mismatches can cause validation failures that interrupt cURL requests and break automation scripts. By walking through the steps cURL takes to validate certificates, we gain appreciation for how this process protects the integrity of our HTTPS connections.

The good news is that by following best practices like keeping cURL up-to-date, using local CA overrides instead of -k, and testing with verbose output, we can investigate and resolve certificate errors while minimizing risk. Ultimately, a robust HTTPS ecosystem depends on proper validation by clients like cURL to maintain the expected security guarantees.

As HTTPS adoption increases and new vulnerabilities emerge, it‘s important to stay informed about SSL/TLS security trends and adjust configurations accordingly. With the knowledge from this guide, you can use cURL confidently to debug and automate HTTPS requests without compromising on security.