zatae

zatae

How to parse result from Google BigQuery API

I am using google_api to query a BigQuery dataset but I find it very difficult to parse data (TableRow/TableFieldSchema) contained in response.

It looks like there is no built-in tool to convert this to simply convert it to a Map or anything else. I found this solution that works pretty well but it’s really weird to me that there is no simpler way to do it ?

Marked As Solved

adammokan

adammokan

@zatae After looking more at my earlier response, I realized that the LogFlare stuff I linked to was more related to writing to BQ.

My solution to handling the output from BQ is generally based off the same StackOverflow link you posted.

FWIW, here is a gist of a single module I extracted from some production systems in my world. I did just toss in the handling for “JSON” data types, but didn’t test it. Only requirement outside of the BQ library is jason.

Not suggesting this is any better than the StackOverflow solution, but has a few opinionated things going on to meet my needs.

Also Liked

adammokan

adammokan

Yeah, what you are bumping into can be a chore with the semi-official library - but once you’ve used it for a while, its not terrible.

LogFlare has a solution that is similar to one I use in my production systems for turning that output into a list of maps with the correct field names and stuff. That is here. Basically they are pulling the schema data from the response and coercing the rows into the correct shapes, types, snake casing attribute names, etc.

Also worth looking at the BQ related stuff in the LiveBook side of the community. The req/req_bigquery library. That just hits the REST API the same as the google client you point to (which uses Tesla under the hood). Not saying to use that for your project, but they take an another approach to handle that output using the Table.Reader protocol as is becoming common for Livebook-related efforts. There are YouTube videos showing how to use this for grabbing data in a LiveBook.

As to some of the reasons why the main library doesn’t do this for you is that he official google cloud hex packages (that I am aware) of are built by some sort of code generator based on API specs. I think it is just a person or two at Google that sponsor those efforts, so I understand why it is as automated as possible. The negative of this is they lack any sort of normal polish or feel like Elixir libraries. As someone who has written a ton of google cloud-related elixir code over the years, I’ve learned to have the Google REST API docs open while trying to decipher these libraries. Much easier than following the long namespaces around in hexdocs. Last point is that all of the google libraries are just hitting these same REST API endpoints (no matter the language).

Also worth just hitting up hex.pm and tossing BigQuery in the search. I know there are a few efforts in various states over the years. Most people I know just end up building a module to clean up the output and reuse it in many projects.

If I have some free cycles next week, I try to revisit this thread out and toss a gist of my variation on this if needed.

samwiseZ

samwiseZ

Thank you for your service. This gist has saved me a significant amount of time :smile:

Where Next?

Popular in Questions Top

Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
shijith.k
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New

Other popular topics Top

danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 29377 241
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
boundedvariable
I am going through the kafka architecture. All the features what the kafka is providing are already in Erlang. I would like hear your opi...
New
PeterCarter
There are pre-rolled solutions for other frameworks that do work. However, Phoenix does not seem to have these. Have people had good expe...
New
svb
Hi! Currently I want to submit a form by pressing the Enter key. However, since my input field is of type “textarea” this is just adds a...
New

We're in Beta

About us Mission Statement