How to Send HTTP Headers Using cURL: The Ultimate Guide

If you work with APIs or do any kind of web scraping, mastering cURL is an essential skill. cURL is a powerful command-line tool for sending HTTP requests and transferring data. One of the most useful things you can do with cURL is set custom HTTP headers on your requests. Headers provide important metadata about the request and allow you to do things like authentication, set your user agent, specify content types, and much more.

Content Navigation show

In this guide, we‘ll dive deep into how to send HTTP headers with cURL. I‘ll explain what headers are, show you the syntax for setting them, and share lots of examples. By the end, you‘ll be a cURL header expert!

What is cURL?

First, a quick overview of cURL if you‘re not already familiar. cURL is an open source command line tool and library for transferring data with URLs. It supports a wide range of protocols including HTTP, HTTPS, FTP, SFTP, and many others.

Some common uses for cURL include:

  • Downloading files
  • Sending data to APIs
  • Testing API endpoints
  • Debugging web applications
  • Web scraping

cURL has a ton of features, but one of the most useful is the ability to easily set any HTTP headers you want on your requests.

What is an HTTP Header?

An HTTP header is a field of an HTTP request or response that passes additional context and metadata about the request or response. Headers are case-insensitive name-value pairs in clear-text string format, separated by a colon. The end of the header section is indicated by an empty field, resulting in the transmission of a blank line.

Here‘s an example of what HTTP headers look like:

Host: api.example.com
User-Agent: curl/7.77.0
Accept: */*
Content-Type: application/json
Authorization: Bearer abcxyz12345

There are numerous standard HTTP headers that have defined meanings for requests and responses. Some common headers include:

  • Host – the domain name of the server, used by servers with multiple domains to determine which site to show
  • User-Agent – identifies the client application making the request, like the browser or a bot
  • Accept – informs the server what media types the client can understand
  • Content-Type – indicates the media type of the request body being sent to the server
  • Authorization – sends credentials to authenticate the client with the server
  • Cookie – contains stored HTTP cookies to send with the request
  • Referer – the URL of the previous web page from which a link to the currently requested page was followed

There are many other standard headers as well as custom headers used by specific APIs. Being able to inspect and set these request headers yourself is very useful for development, testing, and scraping.

Setting HTTP Headers with cURL

Now let‘s get to the good part – how to actually set HTTP headers using cURL! It‘s quite easy using the -H or --header option.

The basic syntax looks like this:

curl -H "Header-Name: header-value" https://example.com

You simply use -H followed by the header name and value separated by a colon. Let‘s look at a concrete example. If I wanted to set a custom header called X-API-KEY with a value of 12345abcde, I would do this:

curl -H "X-API-KEY: 12345abcde" https://api.example.com/data

Using -H sets the header for this specific request only. The custom header will be sent to the server along with the other default headers cURL normally sends.

To set multiple headers, you can simply specify the -H option multiple times, like this:

curl -H "X-API-KEY: 12345abcde" -H "Content-Type: application/json" https://api.example.com/data

This will send both the X-API-KEY and Content-Type headers with the specified values. You can set as many headers as you need to.

Sometimes you may want to override one of the default headers curl sets rather than add a new custom one. For example, curl automatically sends a User-Agent header with its version number. But some servers may block requests from cURL. To work around this, you can manually set your own User-Agent string:

curl -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0" https://example.com

This will override the default cURL user agent with a Firefox browser user agent, making the request seem like it‘s coming from a regular web browser.

Important Headers to Know

While you can set any arbitrary HTTP header using cURL, there are some headers that are especially useful to know about. Let‘s go through a few of the most important ones.

User-Agent

As I mentioned above, the User-Agent header identifies the client making the HTTP request. It typically includes the application name, version, operating system, and more. Setting the User-Agent to mimic a browser is a very common practice in web scraping. Some websites will block requests from unknown user agents.

Here‘s an example of spoofing a Chrome user agent:

curl -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.0 Safari/537.36" https://example.com

Referer

The Referer header contains the URL of the previous web page from which a link to the currently requested page was followed. In other words, it tells the server what URL "referred" the client to that page. Some websites use the Referer header for analytics, logging, or caching purposes. Occasionally a website might block requests that are missing a Referer or have an unknown Referer.

To set the Referer in cURL, it‘s just another -H option:

curl -H "Referer: https://google.com" https://example.com

This will make it look like the user clicked a link from https://google.com to get to https://example.com, even though the request is coming directly from cURL.

Authentication Headers

Many APIs and websites require authentication using HTTP headers. The most common authentication types are Basic Auth and Bearer tokens.

To send a username and password using Basic Auth, you can build the Authorization header like this:

curl -H "Authorization: Basic $(echo -n "username:password" | base64)" https://api.example.com

The username and password are joined with a colon, base64 encoded, and then passed in the Authorization header with the "Basic" prefix.

If authenticating with a Bearer token, it looks very similar:

curl -H "Authorization: Bearer eyJhGciOiXVCJ9..." https://api.example.com

Just prefix the token with "Bearer" as the Authorization header value.

Cookies

HTTP Cookies are commonly used to store session data and provide authentication. You can view cookies sent by a server using cURL‘s -v verbose option for viewing headers. To send cookies in a request, you can either use the Cookie header or the --cookie option:

# Using -H
curl -H "Cookie: session_id=abc123; user_token=xyz789" https://example.com

# Using --cookie
curl --cookie "session_id=abc123; user_token=xyz789" https://example.com

Both accomplish the same result of sending the specified cookies to the server.

Content-Type

When sending data in a POST or PUT request, you should normally set the Content-Type header to specify the format of the request body. This tells the server how to interpret the data.

For example, if sending JSON data, you would set the Content-Type to "application/json":

curl -X POST -H "Content-Type: application/json" -d ‘{"username":"john","password":"secret"}‘ https://api.example.com/login

Form encoded data would use "application/x-www-form-urlencoded" instead:

curl -X POST -H "Content-Type: application/x-www-form-urlencoded" -d ‘username=john&password=secret‘ https://api.example.com/login

The -d option is used to pass the POST data and the -X specifies the HTTP method as POST.

cURL Header Tips

Here are a few final tips for working with headers in cURL:

  • Use -v or --verbose to see the full headers (request and response) for debugging purposes. This is super useful for inspecting exactly what is being sent and received.

  • Remember you can combine setting headers with other cURL options like -d for request bodies, -b for cookies, -X to set the HTTP method, etc.

  • If a site uses redirects, you may need to tell cURL to follow them to the final destination using -L or --location. Though keep in mind, headers like Authorization and Cookies may not get passed through by default on the redirects.

  • For readability, consider using the --header long-form option instead of -H when setting many headers. You can also use a line continuation character to spread a single cURL command over multiple lines:

curl --header "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0" \ 
     --header "Referer: https://google.com" \
     --header "Authorization: Bearer eyJhGciOiXVCJ9..." \
     https://api.example.com/endpoint

Conclusion

In this guide we‘ve taken a deep dive into using cURL to set HTTP headers. We covered the basics of what headers are and how to set them with the -H option. Then we looked at several of the most important headers to know, including User-Agent, Referer, Authorization, Cookies, and Content-Type.

Being able to view and customize headers is incredibly valuable for working with web APIs, testing applications, and scraping websites. Mastering this skill with cURL will serve you well.

To learn more, I recommend practicing with different header combinations, inspecting the headers on websites you use with your browser‘s developer tools, and studying API documentation to learn what headers different services expect.

Let me know if you have any other questions! I hope this guide has been helpful for understanding and working with HTTP headers in cURL.