Req application/octet-stream tar file mock test

Questions / Help
, ,
1

Hello friends, Imagine you want to download a tar file from hex.pm website.

I have this function and do not want to install req hex plugin. as you see I find the "contents.tar.gz" and it is the binary of file.

  @spec download(download_type, pkg) :: okey_return | error_return
  def download(:hex, %{app: app, tag: tag_name}) when not is_nil(tag_name) do
    case build_url("https://repo.hex.pm/tarballs/#{app}-#{tag_name}.tar") do
      %Req.Response{status: 200, body: body} ->
        converted = Map.new(body, fn {key, value} -> {to_string(key), value} end)
        {:ok, converted["contents.tar.gz"]}

      _ ->
        mix_global_err()
    end
  end

And downloader function:

  defp build_url(location) do
    [base_url: location]
    |> Keyword.merge(Application.get_env(:mishka_installer, :downloader_req_options, []))
    |> Keyword.merge(Application.get_env(:mishka_installer, :proxy, []))
    |> Req.request!()
  end

Now I want to mock it, so this is the place I can not be able to fix.

One of my function test is what does not work:

body = [
        {~c"VERSION", "3"},
        {~c"CHECKSUM", "1D5EC32825E5AF1B5CF58933F4C918909BC79E4391EBE2B9315B02705145B18F"},
        {~c"metadata.config", "{<<\"app\">>,<<\"mishka_installer\">>}.\n{<<\"build_tools\">>"},
        {~c"contents.tar.gz", <<31, 139, 8, 0, 0, 0>>}
      ]

      body_iolist_to_binary =
        Enum.reduce(body, "", fn {key, value}, acc -> acc <> "#{key}:#{value}\n" end)
        |> :erlang.iolist_to_binary()

      Req.Test.expect(Downloader, fn conn ->
        conn
        |> Plug.Conn.put_resp_content_type("application/octet-stream")
        |> Plug.Conn.send_resp(200, body_iolist_to_binary)
      end)

If I put body directly I have error!! I have no idea how can simulate the hex output.

It should be noted my function works as well, my problem is located in testing this

Thank you in advance

2

A tar file is not some kind of map, like you’re building here in your test. The body you get from Req has the tar already extracted, which gives you a map (see Req.Steps — req v0.5.0). In your mock you need to build a proper .tar file, e.g. by using :erl_tar.

1 Like
3

I thought about it before, And I send a tar file binary like this

 body = File.read!("mishka_developer_tools-0.1.5.tar")

      Req.Test.expect(Downloader, fn conn ->
        conn
        |> Plug.Conn.put_resp_content_type("application/octet-stream")
        |> Plug.Conn.send_resp(200, body)
      end)

# error
     ** (Protocol.UndefinedError) protocol Enumerable not implemented for <<100, 101, 112, 108, 111, 121, 109, 101, 110, 116, 47, 100, 101, 118, 47, 101, 120, 116, 101, 110, 115, 105, 111, 110, 115, 47, 109, 105, 115, 104, 107, 97, 95, 100, 101, 118, 101, 108, 111, 112, 101, 114, 95, 116, 111, 111, 108, 115, 45, 48, ...>> of type BitString. This protocol is implemented for the following type(s): Date.Range, File.Stream, Function, GenEvent.Stream, HashDict, HashSet, IO.Stream, Jason.OrderedObject, List, Map, MapSet, Range, Req.Response.Async, Stream
     code: Downloader.download(:hex, %{app: "mishka_installer", tag: "0.0.4"})

but it did not work and the tar output of me is a binary, not same like hex

another code i tested with erl_tar.create

      body = File.read!("mishka_developer_tools-0.1.5.tar")
      {:ok, files} = :erl_tar.extract({:binary, body}, [:memory])

      Req.Test.expect(Downloader, fn conn ->
        conn
        |> Plug.Conn.put_resp_content_type("application/octet-stream")
        |> Plug.Conn.send_resp(200, files)
      end)

  1) test Download Mock Test ===> Downloade hex with version (MishkaInstallerTest.Installer.DownloaderTest)
     test/installer/downloader_test.exs:11
     ** (ArgumentError) errors were found at the given arguments:

       * 1st argument: not an iodata term

     code: Downloader.download(:hex, %{app: "mishka_installer", tag: "0.0.4"})
     stacktrace:
       :erlang.iolist_to_binary(

These are my test i could not be able to send the send_resp

4

This looks to be a bug. Tracing in Req it determines tar as format on the real request, but bin for the mocked request – hence no automatic decoding happening. You could work around this by disabling the built in decoding and doing it manually.

1 Like
5
1 Like
6

Ohh, thank you.

I just put my other test that can be done

      tar_file_path = "deployment/dev/extensions/mishka_developer_tools-0.1.5.tar"
      tar_binary = File.read!("deployment/dev/extensions/mishka_developer_tools-0.1.5.tar")

      Req.Test.expect(Downloader, fn conn ->
        conn
        |> Plug.Conn.put_resp_content_type("application/octet-stream")
        |> Plug.Conn.put_resp_header(
          "content-disposition",
          "attachment; filename=\"#{Path.basename(tar_file_path)}\""
        )
        |> Plug.Conn.send_resp(200, tar_binary)
      end)
7 
conn
|> Plug.Conn.put_resp_content_type("application/octet-stream", nil)
|> Plug.Conn.send_resp(200, body)

This would be another workaround, where the content type won’t define a charset, which I guess in your case makes it match the actual response.

1 Like
8

Thank you.

Your code returns ** (Req.ArchiveError) tar unpacking failed: Checksum failed, for now I decode_body: false

It works for now

code
code1200×650 40.8 KB

And my test

code
code1200×824 55.6 KB

9

Even if you end up not using req_hex, I’d still consider using hex_core to get more realistic tests. Here is an example:

Mix.install([
  :req,
  :plug,
  :hex_core
])

Req.Test.expect(Downloader, fn conn ->
  {:ok, %{tarball: tarball}} =
    :hex_tarball.create(
      %{
        "name" => "foo",
        "version" => "1.0.0"
      },
      [
        {~c"mix.exs",
         """
         defmodule Foo.MixProject do
           use Mix.Project
           def project do
             [app: :foo, version: "1.0.0"]
           end
         end
         """}
      ]
    )

  conn
  |> Plug.Conn.put_resp_content_type("application/octet-stream", nil)
  |> Plug.Conn.send_resp(200, tarball)
end)

Req.get!(url: "/foo.tar", plug: {Req.Test, Downloader})
|> IO.inspect()

Thanks @LostKobrakai for reporting the format detection bug btw. application/octet-stream; charset=utf-8 is not a proper content-type for tar, these bytes are not utf8. I’ll see if I can make any improvements anyway though.

2 Likes
10

Good point. I didn’t really think about that tbh. So Plug.Conn.put_resp_content_type(conn, "application/octet-stream", nil) is actually the way to go then, not just a workaround.

2 Likes