Using Tesla to perform a lookup on a public website

Is there a way to synthetically create the form-data and encode it to be able to send the POST method from within my Phoenix application and then get a response back from the site for parsing?

Yes, at least with httpoison, here’s a test case that submits a form.

I haven’t used tesla, but Tesla.Middleware.FormUrlencoded and Tesla.Middleware.FormUrlencodedTest (for examples) might be relevant.

I can do a regular post with Tesla. The problem is that the POST is not url-encoded. It appears the JQuey function appends hidden input fields into the form for posting, and then applies gzip to the form, along with the session token from the original page. I’m looking for any assistance in the best way to mimic the functionality of the JQuey function that is appending the data to the form on submission. I think I need to:

1. Load the index.html page to get session tokens
2. Append hidden fields with selection values to the form
3. Post the form to index.html, along with the session token from #1
4. Receive the response (which is actually the page search.html)
5. Parse the response for the items I need.

It is step #2 that I’m having problems with. If anyone has done something similar, I’d appreciate some pointers.

Thanks

Not sure if anyone can help unless they used the exact some code though. If you can paste some code that does this hidden fields addition to the form plus how are they [not] encoded then we could have something to work with.

You might be better off with a browser driver like GitHub - elixir-wallaby/wallaby: Concurrent browser tests for your Elixir web apps. · GitHub or GitHub - HashNuke/hound: Elixir library for writing integration tests and browser automation · GitHub.

Thanks, I’ll check them out…still digging out from 10-days with no connectivity on vacation.

So I’ve made progress using https://github.com/elixir-wallaby/wallaby but am now getting a Wallaby.StaleReferenceError.

The code sample:

def search_for_mp(mp_name) do

    Application.put_env(:wallaby, :base_url, "https://secure.utah.gov/llv/search")
    {:ok, _} = Application.ensure_all_started(:wallaby)

    opts = %{ javascriptEnabled: true,
      loadImages: false,
      version: "",
      rotatable: false,
      takesScreenshot: true,
      cssSelectorsEnabled: true,
      nativeEvents: false,
      platform: "ANY",
      unhandledPromptBehavior: "accept",
      loggingPrefs: %{
        browser: "DEBUG"
      }
    }

    {:ok, session} = Wallaby.start_session([%{capabilities: opts}])

    IO.inspect(session)
    # IO.inspect(@submit_button)
    searchName = Wallaby.Query.xpath("//div[@id='searchName']")

    session
    |> Wallaby.Browser.visit("/index.html")
    |> Wallaby.Browser.find(searchName)
    |> Wallaby.Browser.fill_in(@name_field, with: mp_name)
    |> Wallaby.Browser.click(@submit_button)

  end

When I run the code I get the following error message:

%Inspect.Error.{ message: “got Wallaby.StaleReferenceError with message "The element you are trying to reference is stale or no longer attached to the\nDOM. The most likely reason is that it has been removed with Javascript.\n\nYou can typically solve this problem by using find to block until the DOM is in a\nstable state. …}”

It looks like my use of javascriptEnabled: true in the capabilities map passed to the session is not being picked up, so I suspect that is causing part of the problem, but I can’t find any good references in how to set the session to allow javascript (which the submit button invokes).

Any suggestions appreciated.