ChipCoons

Using Tesla to perform a lookup on a public website

I’m trying to perform a lookup of a public-facing government web site to find licensing information on certain professionals. (See https://secure.utah.gov/liv/search/index.html )

I can load the page using tesla with its mint adaptor but need to now perform a POST, which appears to read part of the form data using JQuery.

The submit button is:

<input type="submit" value="Search" onclick="pageTracker._trackEvent('Search', 'By Name', 'Free'); setProfessions();">

The setProfessions function is:

function setProfessions() {
  $('input:checkbox:checked.licenseType').each(function () {
    $('<input>').attr('type','hidden').attr('value', $(this).val()).attr('name', 'professions').appendTo('form');
  });
}

(It appears the pageTracker._trackEvent script is part of the Google Analytics on the page.)

The request sent over the wire is:

Request
POST /llv/search/search.html HTTP/1.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Origin: https://secure.utah.gov
Cookie: JSESSIONID=fa808df4bb17cee802c57bb3c7d8; TS01bdb7d2=0143bf517010277adab28f63a21b4b94ccd906dc9560f6e061893b1cde8cf4bb926ae1270d0d55201c82143ef6534344f655d9588c6f4af6af8ce1ded94e9257c77dfbee6c; __utma=128287630.1742368486.1591802595.1591810615.1591816490.4; __utmb=128287630.2.9.1591816594569; __utmc=128287630; __utmz=128287630.1591802595.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); _ga=GA1.2.1742368486.1591802595; _gat_UA-103830962-11=1; _gid=GA1.2.731830168.1591802595; TS01959f26=0143bf5170cddb06b2628645b42b516df797f7f4f171eee6ee7ff0066afb789be3cef48b9fd7318eeff57bb54412d734fe511bd2f2; __utmt=1; TS01959f26_77=08f156120cab2800bcd2801cf30c2b76e6ce0d21176587d514fd5c1bdd2dae547dca300aea61a5577e55dc465a5d1fa90860c69dd38240003ea72082daf55dd9e6126a7c4ea30f412fef20a05a890c3a4819469b6c38866480328690ce775fcfb40e66968958ccc12126549da8133a87eb122ba58bf2eb55; fontsize=90%25
Accept-Encoding: gzip, deflate, br
Host: secure.utah.gov
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.2 Safari/605.1.15
Accept-Language: en-us
Referer: https://secure.utah.gov/llv/search/index.html
Connection: keep-alive

So it appears the button sends a post to a different URL with the selection from the form.

My question is:
Is there a way to synthetically create the form-data and encode it to be able to send the POST method from within my Phoenix application and then get a response back from the site for parsing?

They do not have a public API, so I’m stuck trying this round-about approach.

If there is a better set of tools to meet this need (other than Tesla), I’m happy to hear recommendations.

Thanks,

6 comments

/phoenix #http-requests #tesla

0 2388 6

2020-07-12 20:38:09 UTC

First Post!

idi527

Is there a way to synthetically create the form-data and encode it to be able to send the POST method from within my Phoenix application and then get a response back from the site for parsing?

Yes, at least with httpoison, here’s a test case that submits a form.

I haven’t used tesla, but Tesla.Middleware.FormUrlencoded and Tesla.Middleware.FormUrlencodedTest (for examples) might be relevant.

Post #2

Last Post!

ChipCoons

So I’ve made progress using https://github.com/elixir-wallaby/wallaby but am now getting a Wallaby.StaleReferenceError.

The code sample:

def search_for_mp(mp_name) do

    Application.put_env(:wallaby, :base_url, "https://secure.utah.gov/llv/search")
    {:ok, _} = Application.ensure_all_started(:wallaby)

    opts = %{ javascriptEnabled: true,
      loadImages: false,
      version: "",
      rotatable: false,
      takesScreenshot: true,
      cssSelectorsEnabled: true,
      nativeEvents: false,
      platform: "ANY",
      unhandledPromptBehavior: "accept",
      loggingPrefs: %{
        browser: "DEBUG"
      }
    }

    {:ok, session} = Wallaby.start_session([%{capabilities: opts}])

    IO.inspect(session)
    # IO.inspect(@submit_button)
    searchName = Wallaby.Query.xpath("//div[@id='searchName']")

    session
    |> Wallaby.Browser.visit("/index.html")
    |> Wallaby.Browser.find(searchName)
    |> Wallaby.Browser.fill_in(@name_field, with: mp_name)
    |> Wallaby.Browser.click(@submit_button)

  end

When I run the code I get the following error message:

%Inspect.Error.{ message: “got Wallaby.StaleReferenceError with message "The element you are trying to reference is stale or no longer attached to the\nDOM. The most likely reason is that it has been removed with Javascript.\n\nYou can typically solve this problem by using find to block until the DOM is in a\nstable state. …}”

It looks like my use of javascriptEnabled: true in the capabilities map passed to the session is not being picked up, so I suspect that is causing part of the problem, but I can’t find any good references in how to set the session to allow javascript (which the submit button invokes).

Any suggestions appreciated.

Post #7