:httpc cheatsheet

:httpc HTTP client is part of Erlang standard library, and as such can be easily used in Elixir code too. One particular advantage of using :httpc is that you can do a lot with it even with a minimal number of external dependencies. I prefer to use :httpc in one-off scripts and in production.

The :httpc reference is available at the Erlang standard library documentation website, here:

Erlang -- httpc

For now, I just wanted to publish an initial version of compilation of my own notes. Later I intent to update this post by adding notes to existing examples, adding references and even more examples. Feel free to modify the contents of the post too!

These examples are self-contained, can be copied to a script.exs file and executed via elixir script.exs command. Enjoy! :slight_smile:

Index:

  • sending a GET request returning a JSON payload
  • sending a POST request with JSON request payload
  • sending a POST request with no payload
  • downloading a file
  • uploading a file
  • low-level tracing of HTTP interaction

General notes

  • for :httpc to work, :inets app needs to be started beforehand; in case you intend to interact with HTTPS endpoints (which is what you want to do in most cases when making requests on public internet), :ssl app needs to be started as well; all examples bellow start both apps,

  • all examples below pass a big-looking “parameter” called http_request_opts - this is how you make sure :httpc performs validation of the server’s TLS certificate; more on why this is necessary is described in “Erlang standard library: inets” page of “Secure Coding and Deployment Hardening Guidelines” by ERLef; without params under ssl key, you are likely to see the following warning:

    [warning] Description: 'Authenticity is not established by certificate path validation'
         Reason: 'Option {verify, verify_peer} and cacertfile/cacerts is missing'
    
  • :httpc appears to like char lists a lot; a common mistake I made in the beginning when using :httpc is feeding it with "https://example.com" (a string) URL, while it should be a 'https://example.com' URL

Sending a GET request returning a JSON payload

Example

:inets.start()
:ssl.start()

url = 'https://httpbin.org/get'
headers = [{'accept', 'application/json'}]

http_request_opts = [
  ssl: [
    verify: :verify_peer,
    cacerts: :public_key.cacerts_get(),
    customize_hostname_check: [
      match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
    ]
  ]
]

:httpc.request(:get, {url, headers}, http_request_opts, [])

Sending a POST request with JSON request payload

Example

Mix.install([:jason])

:inets.start()
:ssl.start()

url = 'https://httpbin.org/post'
headers = []
content_type = 'application/json'
body = Jason.encode!(%{hello: "world"})

http_request_opts = [
  ssl: [
    verify: :verify_peer,
    cacerts: :public_key.cacerts_get(),
    customize_hostname_check: [
      match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
    ]
  ]
]

:httpc.request(:post, {url, headers, content_type, body}, http_request_opts, [])

Sending a POST request with no payload

Example

:inets.start()
:ssl.start()

url = 'https://httpbin.org/post'
headers = []
content_type = ''
body = ''

http_request_opts = [
  ssl: [
    verify: :verify_peer,
    cacerts: :public_key.cacerts_get(),
    customize_hostname_check: [
      match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
    ]
  ]
]

:httpc.request(:post, {url, headers, content_type, body}, http_request_opts, [])

Downloading a file

Example

:inets.start()
:ssl.start()

url = 'https://www.rfc-editor.org/rfc/pdfrfc/rfc1149.txt.pdf'
headers = []

path_to_file =
  System.tmp_dir!()
  |> Path.join("rfc1149.txt.pdf")
  |> String.to_charlist()

http_request_opts = [
  ssl: [
    verify: :verify_peer,
    cacerts: :public_key.cacerts_get(),
    customize_hostname_check: [
      match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
    ]
  ]
]

{:ok, :saved_to_file} =
  :httpc.request(:get, {url, headers}, http_request_opts, [stream: path_to_file])

Uploading a file

Example

Mix.install([:multipart])

:inets.start()
:ssl.start()

part = Multipart.Part.file_field("/tmp/rfc1149.txt.pdf", "my_file")

multipart =
  Multipart.new()
  |> Multipart.add_part(part)

content_length =
  multipart
  |> Multipart.content_length()
  |> Integer.to_string()
  |> String.to_charlist()

content_type =
  multipart
  |> Multipart.content_type("multipart/form-data")
  |> String.to_charlist()

url = 'https://httpbin.org/anything'
headers = [{'Content-Length', content_length}]
payload = Multipart.body_binary(multipart)

http_request_opts = [
  ssl: [
    verify: :verify_peer,
    cacerts: :public_key.cacerts_get(),
    customize_hostname_check: [
      match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
    ]
  ]
]

:httpc.request(:post, {url, headers, content_type, payload}, http_request_opts, [])

Notes

  • some HTTP clients offer convenient API around sending file payloads; UX varies on the library API design, and sometime can serve as a source of confusion (see this discussion, for instance),

  • because design of :httpc appears to make little-to-no assumptions regarding the semantics of request payload, one can fairly say that :httpc simply send bytes. As such, we should be able to “manually” construct any payload, including one to upload a file, and :httpc will just handle it,

  • multipart package, originally mentioned by its author @engineeringdept here focuses on just putting together a bunch of bytes that look like a correct “HTTP payload containing a file upload”,

Low-level tracing of HTTP interaction

Example

:inets.start()
:ssl.start()

url = 'https://httpbin.org/get'
headers = []

http_request_opts = [
  ssl: [
    verify: :verify_peer,
    cacerts: :public_key.cacerts_get(),
    customize_hostname_check: [
      match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
    ]
  ]
]

:httpc.set_options([verbose: :debug])
:httpc.request(:get, {url, headers}, http_request_opts, [])

Notes

  • :debug option value will easily generate several megabytes of information, while :trace value can potentially generate several tens of megabytes, so use it with care.
19 Likes

Nit:

There is no need to set the “Content-Type” header in a request that does not include a body. If you want to indicate to the server that you are ready to accept a JSON response body, us the “Accept” header instead:

headers = [{'accept', 'application/json'}]
3 Likes

Just appreciation response. I just started exploring escript yesterday and was trying out elixir clients and they kept failing on castore and then moved to httpc. Though it worked but it kept showing following warning

[warning] Description: 'Authenticity is not established by certificate path validation'
     Reason: 'Option {verify, verify_peer} and cacertfile/cacerts is missing'

There is actually no working solution when you search online but your post has the the solution I needed and now its works like a charm without any warning.

Thanks and gotta love the elixir/erlang/beam community

3 Likes

Thanks, this is really great! Instant browser bookmark from me.

There is actually no working solution when you search online

There is a solution online, though it might not be immediately found if you’re searching for Elixir, and not Erlang: Erlang standard library: ssl | EEF Security WG. In the page body you’ll find

%% Erlang (OTP 25 or later)
ssl:connect("example.net", 443, [
    {verify, verify_peer},
    {cacerts, public_key:cacerts_get()},
    {depth, 3},
    {customize_hostname_check, [
        {match_fun, public_key:pkix_verify_hostname_match_fun(https)}
    ]}
]).

The Secure Coding and Deployment Hardening Guidelines, by the Erlang Ecosystem Foundation’s Security Working Group, is a great read, btw.

2 Likes