Table.Reader for Elasticsearch isn't working :(

I’m trying to write a Table.Reader implementation for Snap.SearchResponse from the snap library (GitHub - breakroom/snap: An Elasticsearch client for Elixir) which talks to elasticsearch

This is what I think it should look like

if Code.ensure_loaded?(Table.Reader) do
  defimpl Table.Reader, for: Snap.SearchResponse do
    def init(result) do
      columns = get_columns(result)
      rows = get_rows(result)
      {:rows, %{columns: columns}, rows}
    end

    defp get_columns(response) do
      if response.hits.hits |> Enum.empty?() do
        []
      else
        hits = response.hits.hits
        hits |> List.first() |> Map.fetch!(:source) |> Map.keys()
      end
    end

    defp get_rows(response) do
      hits = response.hits.hits

      hits
      |> Enum.map(fn hit ->
        hit.source |> Map.values()
      end)
    end
  end
end

and here is an Example Snap.SearchResponse

%Snap.SearchResponse{
  took: 1,
  timed_out: false,
  shards: %{"failed" => 0, "skipped" => 0, "successful" => 1, "total" => 1},
  hits: %Snap.Hits{
    total: %{"relation" => "gte", "value" => 10000},
    max_score: 1.0,
    hits: [
      %Snap.Hit{
        index: "products",
        type: nil,
        id: "BH2lkY0BtfUXFCo5NxOU",
        score: 1.0,
        source: %{
          "Country" => "Myanmar",
          "Item Type" => "Beverages",
          "Order Date" => "5/29/2016",
          "Order ID" => "238846909",
          "Order Priority" => "L",
          "Region" => "Asia",
          "Sales Channel" => "Offline",
          "Ship Date" => "6/15/2016",
          "Total Cost" => "284933.77",
          "Total Profit" => "140360.58",
          "Total Revenue" => "425294.35",
          "Unit Cost" => "31.79",
          "Unit Price" => "47.45",
          "Units Sold" => "8963"
        },
        fields: nil,
        explanation: nil,
        matched_queries: nil,
        highlight: nil,
        inner_hits: nil
      },
      %Snap.Hit{
        index: "products",
        type: nil,
        id: "BX2lkY0BtfUXFCo5NxOU",
        score: 1.0,
        source: %{
          "Country" => "Costa Rica",
          "Item Type" => "Cereal",
          "Order Date" => "2/23/2017",
          "Order ID" => "673583209",
          "Order Priority" => "L",
          "Region" => "Central America and the Caribbean",
          "Sales Channel" => "Online",
          "Ship Date" => "2/26/2017",
          "Total Cost" => "830895.45",
          "Total Profit" => "628546.05",
          "Total Revenue" => "1459441.50",
          "Unit Cost" => "117.11",
          "Unit Price" => "205.70",
          "Units Sold" => "7095"
        },
        fields: nil,
        explanation: nil,
        matched_queries: nil,
        highlight: nil,
        inner_hits: nil
      },
      %Snap.Hit{
        index: "products",
        type: nil,
        id: "Bn2lkY0BtfUXFCo5NxOU",
        score: 1.0,
        source: %{
          "Country" => "Cameroon",
          "Item Type" => "Meat",
          "Order Date" => "7/30/2014",
          "Order ID" => "277272450",
          "Order Priority" => "H",
          "Region" => "Sub-Saharan Africa",
          "Sales Channel" => "Offline",
          "Ship Date" => "8/12/2014",
          "Total Cost" => "3623195.15",
          "Total Profit" => "568282.00",
          "Total Revenue" => "4191477.15",
          "Unit Cost" => "364.69",
          "Unit Price" => "421.89",
          "Units Sold" => "9935"
        },
        fields: nil,
        explanation: nil,
        matched_queries: nil,
        highlight: nil,
        inner_hits: nil
      },
      %Snap.Hit{
        index: "products",
        type: nil,
        id: "B32lkY0BtfUXFCo5NxOU",
        score: 1.0,
        source: %{
          "Country" => "Republic of the Congo",
          "Item Type" => "Clothes",
          "Order Date" => "9/23/2015",
          "Order ID" => "593713462",
          "Order Priority" => "H",
          "Region" => "Sub-Saharan Africa",
          "Sales Channel" => "Online",
          "Ship Date" => "10/7/2015",
          "Total Cost" => "66734.08",
          "Total Profit" => "136745.28",
          "Total Revenue" => "203479.36",
          "Unit Cost" => "35.84",
          "Unit Price" => "109.28",
          "Units Sold" => "1862"
        },
        fields: nil,
        explanation: nil,
        matched_queries: nil,
        highlight: nil,
        inner_hits: nil
      }
    ]
  },
  suggest: nil,
  aggregations: nil,
  scroll_id: nil,
  pit_id: nil
}

so i’ve forked snap, added this code and loaded it in a livebook

{:ok, results} = Snap.Search.search(Elastic, "products", %{query: %{match_all: %{}}})

results

I would expect livebook to show me a data table, but instead it just shows me the full Snap.SearchResponse pasted above. I have tried added IO.puts in the Table.Reader init function but to no avail. Any ideas how i can debug this?

update:

Table.to_rows(results)

works, so my Table.Reader works… i don’t understand yet why it doesn’t show as tabular data in livebook

Kino.Shorts.data_table(results)

also works

In order for Livebook to render a struct in a special way, it needs to implement the Kino.Render protocol. See kino_db/lib/kino_db.ex at 0de094fcefb2f28a5c413e95ce551adf00dcb5e7 · livebook-dev/kino_db · GitHub for example.

2 Likes

Wow thanks :slight_smile: I had no idea this is what was wrong.

Do you have any thoughts on the best path forward for livebook + elasticsearch? It’s not exactly tabular data. I mean, it contains tabular data but it also has a great deal of metadata eg. aggregations shards etc… as per the above example.

I didn’t really interact with ES in practice, so I don’t have a strong opinion what would be most useful. For queries I can imagine the score being quite relevant, so perhaps it should also be a part of the table.