JSON parses in Javascript but not Elixir

I’m trying to parse a JSON that works in JS but I can’t get it to work in Elixir. I’ve tried with both Jason and Json libraries.

Mix.install([
  {:jason, "~> 1.4"},
  {:json, "~> 1.4"}
])

My workaround at the moment is to run System.cmd("node", []) to do an eval which is not sustainable solution.

The json: JSON parses in Javascript but not Elixir · GitHub

Any ideas/tips for this rookie here?

The JSON is definitely invalid. Are you sure JS doesn’t raise you an error? I just tried it in browser and I see an error.

The only solution is to ask generating a valid JSON file.

That’s not a valid JSON file though.

Thanks for taking a look. I think I butchered the copy-pasta – I’ve updated it with the entire JS line: JSON parses in Javascript but not Elixir · GitHub

FYI: the code to get this is:

defmodule Bubbledb do
  defp get_page(url) do
    case HTTPoison.get(url, [], timeout: 10_000, recv_timeout: 10_000) do
      {:ok, response} ->
        case response.status_code do
          200 ->
            {:ok, response.body}

          _ ->
            {:error, "Failed to fetch #{url}: #{response.status_code}"}
        end

      {:error, %HTTPoison.Error{reason: reason}} ->
        {:error, "Failed to fetch #{url}: #{reason}"}
    end
  end

  def get_raw_data_schema(bubble_app_url) do
    {:ok, body} = get_page(bubble_app_url)

    dynamic_js = 
      Regex.run(~r/\/package\/dynamic_js.*\/dynamic\.js/, body)
      |> hd()

    url = :uri_string.parse(bubble_app_url)
    host = url.host

    data_types_url = "https://#{host}#{dynamic_js}"
    get_page(data_types_url)
  end

  def get_app_data(bubble_app_html) do
    {:ok, re} = Regex.compile("(var app = .*)}'\\);")

    Regex.run(re, bubble_app_html)
    |> List.first()
  end
end

Then run

{:ok,bubble_app_html} = Bubbledb.get_raw_data_schema("https://bubble.io")
Bubbledb.get_app_data(bubble_app_html)

You’re fetching a line of JS, not JSON data:

var app = JSON.parse('{“last_change”:“17561905295”…

The JSON in that line is also inside a single quoted string. All single quotes will be escaped and all escaped double quotes are double escaped, so you can’t just strip away the surrounding JS.

3 Likes

So the data is ok. The problem is in your code. You cannot parse out json stored in an escaped format and try to decode it without previously unescaping it. Putting the code in the browser console, doing JSON.stringify on the result and trying to decode that works just fine.

3 Likes

So this needs to be run inside Javascript? There’s no way to parse this inside of Elixir?

(apologies for my ignorance, still learning)

You should maybe take smaller steps.

Anyway, you are parsing a piece javascript code:

var app = JSON.parse('{"last_change":"17561905295","last_change_as_of":1704382209808} ...')

The actual JSON is the string inside

var app = JSON.parse('<JSON IS HERE>')

So when you take only this part (shortened version):

data.json

{
    "last_change": "17561905295",
    "last_change_as_of": 1704382209808
}
"data.json" #=> "data.json"
|> File.read!() #=> "{\n    \"last_change\": \"17561905295\",\n    \"last_change_as_of\": 1704382209808\n}"
|> Jason.decode!() #=> %{"last_change" => "17561905295", "last_change_as_of" => 1704382209808}
2 Likes

Try changing the capture group in the regex so you only get the JSON string inside the quotes.

Then you will need to do something about the escaped quotes by replacing all escaped single quotes (\') with single quotes ('), and all double escaped double quotes (\\") with single escapes (\"). I’m not sure how complete/correct this step is, but something along those lines.

Then finally you can run it through Jason.decode.

1 Like

I had the livebook still alive: This seems to work:

String.replace(str_between_single_quotes, [~S[\\"], ~S[\']], fn
  ~S[\\"] -> ~S[\"]
  ~S[\'] -> ~S[']
end)
3 Likes

Thanks for all the help!! That worked. I’m gonna rewrite this module and redeploy.

I learned a ton.

A few of them:

  • pattern matching is magic :magic_wand: . I keep telling my wife and children about it, but they don’t get it. I’ll keep trying.
  • |> and |> and |> some more
  • … after all that piping, dump it all on |> dbg()
  • … with kino installed in a Livebook
  • I’ve also learned how to deploy on Fly.io
  • how to install nodeJS in Fly – it’s a simple apt-get install nodejs, but I didn’t know that
  • how to install the latest nodeJS on Fly – it involves something like this:
FROM ${RUNNER_IMAGE}

RUN apt-get update -y && \
  apt-get install -y libstdc++6 openssl libncurses5 locales ca-certificates curl \
  && curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash \
  && export NVM_DIR="$HOME/.nvm" \
  && [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" \
  && nvm install node \
  && apt-get clean && rm -f /var/lib/apt/lists/*_*
  • there are limits to how long a string you can pass to System.cmd