Plug test messing up body request

[henriquesati]

I’m trying to test my API endpoints that communicate with a docker container SQL. When running insomnia calls, the api behavior is fine, but now trying to automate the tests i’m experiencing the following issues
the endpoint is ignoring some fields, although i’m sending it corretly, (same request are being mane on imnsonia and it is working)
i’m not being able to set my logs on the console even using Logger
I tried to to parse my body request and log to see what is going on, but was also unable to log the request_body parsed, also is like the console just ignores it
test.file

defmodule TestesPay.MyRouterTest do
  use ExUnit.Case, async: false
  use Plug.Test

  alias TestesPay.MyRouter

  setup_all do
    :ok = Ecto.Adapters.SQL.Sandbox.checkout(MyApp.Repo)
    Ecto.Adapters.SQL.Sandbox.mode(MyApp.Repo, {:shared, self()})
    :ok
  end

  test "POST /api/create_contract com dados válidos" do
   body = %{
      "chain" => "1",
      "coin" => "ETH",
      "transaction_value" => 2,
      "transaction_charged" => 6.78,
      "ref" => "REF123",
      "ref_fee" => "2.0",
      "payload" => "data_payload"
    }

    conn = conn(:post, "/api/create_contract", body)
    |> put_req_header("content-type", "application/json")
    |> MyRouter.call(%{})

    IO.inspect(conn, label: "Connection after MyRouter.call")
    IO.inspect(conn.resp_body, label: "body response")

    assert conn.status == 200
  end

  test "POST /api/create_contract com dados inválidos" do
    body = %{
      "chain" => "example_chain"
    }
    |> Jason.encode!()

    conn = conn(:post, "/api/create_contract", body)
    |> put_req_header("content-type", "application/json")
    |> MyRouter.call(%{})

    IO.inspect(conn, label: "Connection after MyRouter.call")
    IO.inspect(conn.resp_body, label: "response body")

    assert conn.status == 400
    assert Jason.decode!(conn.resp_body)["msg"] == "expected_error_message"
  end
end

my plug:

defmodule MyApp.PlugServer.Plugs.CheckRequestFields do
  def init(options), do: options
  alias Plug.Conn

  def call(%Plug.Conn{request_path: path} = conn, opts) do
    {:ok, body_as_json, conn} = Plug.Conn.read_body(conn, opts)
    if path in opts[:paths] do
      verify_tuple = verify_request_fields!(body_as_json, opts[:fields], MyApp.Validate.CustomRules.get_rules(conn.request_path)) ##cuidado com essa linha ao começar muitos roteamentos
  
      conn = case verify_tuple do
        {:ok, _} -> 
          conn
          |> Conn.assign(:resp, verify_tuple)
          |> Conn.assign(:status, 200)
        {:error, _} -> 
          conn
          |> Conn.assign(:resp, verify_tuple)
          |> Conn.assign(:status, 400)
      end
      conn
    else
      conn = Conn.assign(conn, :resp, {:error, %{"msg" => "algo deu errado em call check request fields"}})
      conn
    end
  end


    defp verify_request_fields!(body, fields, rules) do
      {:ok, data} = Jason.decode(body)
        with {:ok, response} <- Validate.validate(data, rules) do
          IO.inspect(response)
            {:ok, response}
        else
          {:error, array_of_errors} ->
            error_map = %{
              "field" => List.first((List.first(array_of_errors).path)),
              "msg" => List.first(array_of_errors).message
            }
                {:error, error_map}
        end
  end
end

and my router:

defmodule TestesPay.MyRouter do
  use Plug.Router
  alias MyApp.PlugServer.Plugs.CheckRequestFields
  
  plug CheckRequestFields, fields: ["chain", "coin", "coin", "transaction_value", "transaction_charged", "ref", "ref_fee", "payload"], paths: ["/api/create_contract"]
  plug :match
  plug :dispatch


  post "/api/create_contract" do
    with {:ok, response} <- conn.assigns.resp,
    {:ok, json_response} = Jason.encode(response),
    {:ok, changeset} = MyApp.ControllerCreateContract.create(response)
     do
      conn
      |> put_resp_content_type("application/json")
      |> resp(conn.assigns.status, json_response)
      |>send_resp()
    else
      {:error, response} ->    
        {:ok, json_response} = Jason.encode(response)
        conn
      |> put_resp_content_type("application/json")
      |> resp(conn.assigns.status, json_response)
      |>send_resp()
    end
  end

  match _ do
    send_resp(conn, 404, "oops")
  end
end

config.file

use Mix.Config


config :testes_pay, MyApp.Repo,
  pool: Ecto.Adapters.SQL.Sandbox,
  database: "teste2",
  username: "pedri",
  password: "1234",
  hostname: "localhost",
  port: 8080

config :testes_pay, ecto_repos: [MyApp.Repo]

config :testes_pay, TestesPay.MyRouter,
  http: [port: 4002],
  server: true

# config :logger,
# backends: [:console],
# compile_time_purge_matching: :debug

output: ` └─ lib/plug_server/router.ex:13:11: TestesPay.MyRouter.do_match/4

Connection after MyRouter.call: %Plug.Conn{
adapter: {Plug.Adapters.Test.Conn, :…},
assigns: %{
status: 400,
resp: {:error, %{“field” => “coin”, “msg” => “is required”}}
},
body_params: %Plug.Conn.Unfetched{aspect: :body_params},
cookies: %Plug.Conn.Unfetched{aspect: :cookies},
halted: false,
host: “www.example.com”,
method: “POST”,
owner: #PID<0.478.0>,
params: %{},
path_info: [“api”, “create_contract”],
path_params: %{},
port: 80,
private: %{
plug_route: {“/api/create_contract”,
#Function<1.74211796/2 in TestesPay.MyRouter.do_match/4>}
},
query_params: %Plug.Conn.Unfetched{aspect: :query_params},
query_string: “”,
remote_ip: {127, 0, 0, 1},
req_cookies: %Plug.Conn.Unfetched{aspect: :cookies},
req_headers: [{“content-type”, “application/json”}],
request_path: “/api/create_contract”,
resp_body: “{"field":"coin","msg":"is required"}”,
resp_cookies: %{},
resp_headers: [
{“cache-control”, “max-age=0, private, must-revalidate”},
{“content-type”, “application/json; charset=utf-8”}
],
scheme: :http,
script_name: ,
secret_key_base: nil,
state: :sent,
status: 400
}
response body: “{"field":"coin","msg":"is required"}”

1. test POST /api/create_contract com dados inválidos (TestesPay.MyRouterTest)
   test/integration_test.exs:34
   Assertion with == failed
   code: assert Jason.decode!(conn.resp_body)[“msg”] == “expected_error_message”
   left: “is required”
   right: “expected_error_message”
   test/integration_test.exs:48: (test)

2. test POST /api/create_contract com dados válidos (TestesPay.MyRouterTest)
   test/integration_test.exs:13
   ** (MatchError) no match of right hand side value: {:error, %Jason.DecodeError{position: 1, token: nil, data: “–plug_conn_test–”}}
   code: |> MyRouter.call(%{})
   stacktrace:
   (testes_pay 0.1.0) lib/plug_server/plugs/check_request_fields.ex:29: MyApp.PlugServer.Plugs.CheckRequestFields.verify_request_fields!/3
   (testes_pay 0.1.0) lib/plug_server/plugs/check_request_fields.ex:8: MyApp.PlugServer.Plugs.CheckRequestFields.call/2
   (testes_pay 0.1.0) lib/plug_server/router.ex:1: TestesPay.MyRouter.plug_builder_call/2
   test/integration_test.exs:26: (test)

Finished in 0.2 seconds (0.00s async, 0.2s sync)
2 tests, 2 failures`

just to be clear about what I mentioned, I’m not being able to log. When using logger, the console just ignore

    conn = conn(:post, "/api/create_contract", body) |> put_req_header("content-type", "application/json")
           |> MyRouter.call(%{})

    {:ok, body_as_json, conn} = Plug.Conn.read_body(conn)
    {:ok, data} = Jason.decode(body_as_json)

    Logger.info("params@! ---------------------------------------------->>>>>")  # Log params
    Logger.info("Conexão após chamada para MyRouter.call:", conn)  # Log connection details
    Logger.info("Corpo da resposta:", conn.resp_body) 

[al2o3cr]

@henriquesati

  test "POST /api/create_contract com dados válidos" do
   body = %{
      "chain" => "1",
      "coin" => "ETH",
      "transaction_value" => 2,
      "transaction_charged" => 6.78,
      "ref" => "REF123",
      "ref_fee" => "2.0",
      "payload" => "data_payload"
    }

    conn = conn(:post, "/api/create_contract", body)
    |> put_req_header("content-type", "application/json")
    |> MyRouter.call(%{})

This test is missing the |> Jason.encode! on the end of the pipe for body (compared to the other one), is that intentional?

[henriquesati]

@henriquesati

|> Jason.encode!()

I think its probably a mistake after too much attempts. Accordingly with Plug.test docs: "The params_or_body field must be one of:

• nil - meaning there is no body;
• a binary - containing a request body. For such cases, :headers must be given as option with a content-type;
• a map or list - containing the parameters which will automatically set the content-type to multipart. The map or list may contain other lists or maps and all entries will be normalized to string keys;"
  So I dont think I need to encode with Jason

[henriquesati]

I tried out encoding the body as json as @al2o3cr pointed out, I dont know why it works because Plug test docs says the maps are accepted (as I already said too)
this worked for me

  test "POST /api/create_contract com dados válidos" do
    body = Jason.encode!(%{
      "chain" => "1",
      "coin" => "ETH",
      "transaction_value" => 2,
      "transaction_charged" => 6.78,
      "ref" => "REF123",
      "ref_fee" => 2.0,
      "payload" => "data_payload"
    })

    conn = conn(:post, "/api/create_contract", body)
    |> put_req_header("content-type", "application/json")
    |> TestesPay.MyRouter.call(body)
    
    IO.inspect(conn, label: "Connection after MyRouter.call")
    IO.inspect(conn.resp_body, label: "body response")
    assert conn.status == 200
  end

[henriquesati]

some extra modifyings I did
check_request_fields:

def call(%Plug.Conn{request_path: path} = conn, opts) do
    {:ok, body_as_json, conn} = Plug.Conn.read_body(conn, opts)
    if path in opts[:paths] do
      verify_tuple = verify_request_fields!(body_as_json, opts[:fields], MyApp.Validate.CustomRules.get_rules(conn.request_path)) ##cuidado com essa linha ao começar muitos roteamentos
      
      conn = case verify_tuple do
        {:ok, body} -> 
          conn
          |> Conn.assign(:status, 200)
          |> Conn.assign(:resp, verify_tuple)
          |> Conn.assign(:resp_body_as_json, body_as_json)
        {:error, error_body_as_json} -> 
          conn
          |> Conn.assign(:status, 400)
          |> Conn.assign(:resp, verify_tuple)
          |> Conn.assign(:resp_body_as_json, error_body_as_json)
      end
      conn
    else
      conn = Conn.assign(conn, :resp, {:error, %{"msg" => "algo deu errado em call check request fields"}})
      conn
    end
  end

router file:

defmodule TestesPay.MyRouter do
  use Plug.Router
  alias MyApp.PlugServer.Plugs.CheckRequestFields
  
  plug CheckRequestFields, fields: ["chain", "coin", "coin", "transaction_value", "transaction_charged", "ref", "ref_fee", "payload"], paths: ["/api/create_contract"]
  plug :match
  plug :dispatch


  post "/api/create_contract" do
    with {:ok, response} <- conn.assigns.resp,
    {:ok, changeset} = MyApp.ControllerCreateContract.create(response)
     do
      IO.inspect(conn)
      conn
      |> put_resp_content_type("application/json")
      |> send_resp(conn.assigns.status, conn.assigns.resp_body_as_json)
    else
      {:error, json_response} ->    
        conn
        |> put_resp_content_type("application/json")
        |> send_resp(conn.assigns.status, json_response)
    end
  end

  match _ do
    send_resp(conn, 404, "oops")
  end


end

[LostKobrakai]

@henriquesati

I dont know why it works because Plug test docs says the maps are accepted

You can only pass a map if all you care for is setting conn.body_params directly. The test adapter cannot automatically encode that map for the body content, hence it being set to "--plug_conn_test--".

So if you run such a conn through a piece of code, which tries to parse the body it’s going to fail. In that case you need to provide a binary to set the body.

[henriquesati]

Yes I saw on a log this --plug_conn-test-- but I still dont get the usecase, can you provide more info/source?

[LostKobrakai]

Not every test for a plug runs a parser for the body. I’d argue a large majority don’t – the ones, which test individual plugs, controllers, …. Therefore people often just care about the resulting body_params, which you can directly set as a map. conn.body is never read on such tests.

E.g. the bulk of phoenix tests never touch the endpoint.ex, which would run Plug.Parsers / does the read_body – the expectation here being that parsers work (and are tested elsewhere) and don’t need to be tested in your codebase. Also encoding and then parsing params would for the most part be wasted cpu cycles if you don’t need to test the parsing, but just need to set params.

[henriquesati]

Basically if my plug parses the request from json ->map, on the tests I must provide the body as json so the plug can internally parse it?

[LostKobrakai]

Yes. This is a bit tricky to understand from the documentation as the 3rd parameter for conn/3 is params_or_body. A binary sets conn.body and a map sets conn.body_params.

[dimitarvp]

All you say is likely true – not doubting you or the docs – but in my case I have a pipeline in my router like so:

  pipeline :api_v1 do
    plug :accepts, ["json"]
  end

And then in the router again:

  scope "/api/v1/commands", MyAppWeb.API.V1.Commands do
    pipe_through :api_v1

    post "/schedule", CommandController, :schedule
  end

And the test is like so:

  test "succeeds with valid XXX command arguments", %{conn: conn} do
    command = Jason.decode!(@command_fixture_xxx_1)
    conn = post(conn, ~p"/api/v1/commands/schedule", command)
    assert json_response(conn, :ok) == %{}
  end

…and I can post a normal map without having to Jason.encode! it first.

@henriquesati I haven’t followed the entire thread but have you tried with a pipeline that has plug :accepts, ["json"] in it?

[LostKobrakai]

Accepts is not doing any parsing. It does content negotiation for the format the response will be formatted in. That’s completely indenpendent from the content on the request.

Parsing happens in Plug.Parsers in the endpoint. Plug.Parsers will however only run if body_params are still unfetched, so the error here is specific to manually reading the body here. And endpoint and router pipelines also need to be opted into being executed in phoenix tests with Phoenix.ConnTest.bypass_through/1.

So lots of things, which make having a proper conn.body value unnecessary in places.

[henriquesati]

No and I dont think I’ll be able to test it, its my first time using elixir and elixir libraries so it’s been hard to do more complicated things :(, i 'm sticking to what works and making little modifications to a better shaped code/use more plugs core functionalities.
Reading now LostKobrakai last answer to you, I think its the appropriate answer