Seeking info about Explorer and Table Formats

Hi @ndrluis! I have been learning about Iceberg recently, so it is really cool that you are tackling this.

Regarding the focus of Explorer, I think it is a performant (and potentially distributed) DataFrame engine first with support for multiple backends. The multiple backends is a feature but not the priority. That’s my take though, other members of the Explorer team may have a different perspective.

What is next on my roadmap is to have something like Explorer.BatchFrame which is a collection of dataframes, potentially on several nodes, and then we execute the operations on those frames concurrently. The process to do so is:

1. Get the location of all .parquet, .arrow, etc files
2. Spawn several nodes, each pointing to one the locations above
3. Create a BatchFrame
4. ??? (do all work)
5. Collect or write the results back

With this in mind, where to do you think Iceberg would enter? I believe it can help with step 1 but, what about step 5? Would you want to write the results back in Iceberg format too?

Thank you @josevalim for the response.

You are correct about the first step, and regarding the writing, I believe it would be better to delegate the writing to the Iceberg implementation because it has some details about the metadata and commit process that I don’t think make sense to have on the Explorer side.

Are you thinking of using Arrow in the Explorer.BatchFrame, or would it be an Elixir File IO?

If the idea is to be compatible with Arrow Table/RecordBatch, I can focus on using the Iceberg Rust implementation. However, if the idea is to have something only in Elixir, I believe it would be easier to write the entire Iceberg implementation in Elixir.

The BatchFrame will rather be a collection of existing DataFrames. For example, you could create it manually like this:

Explorer.BatchFrame.new([
  Explorer.DataFrame.from_parquet("foo/1.parquet"),
  Explorer.DataFrame.from_parquet("foo/2.parquet"),
  Explorer.DataFrame.from_parquet("foo/3.parquet"),
  ...
])

Or you could use from_ipc, from_csv, whatever. So it will use whatever Backend you have chosen (most likely Polars) and coordinate work across them. Does this make sense?

Sorry, the question about I/O does not make sense because, since the idea is to be like Daft, the scan responsibility is from Explorer.

From my perspective, it would be easier to delegate these scans to the Iceberg Rust Implementation and pass the RecordBatch/Table Arrow to an Explorer Dataframe. However, if the idea is to distribute the reading process, I will need to learn more about the scan process in the context of Iceberg.

I understand that we could use from_parquet (in the future, it would be good to support ORC files), but I believe that it is not so simple.

There are some details that I don’t have the knowledge of, but I know they exist, such as Logical Plan / Physical Plan and how the partitions and deleted files/rows in the Iceberg context would work in this scan process.

Another thing is that we need to use some features that are not being used today, like the pushdown predicate and statistics in the from_parquet.

But with this idea of Explorer being responsible for the reading process, I can start to dig into these more lower-level things.

I missed an important point. Iceberg uses Avro files for some of its metadata, which contain low-level details specific to Iceberg’s implementation.

Additionally, all implementations (Go, Rust, Python, Java) use Arrow for data delivery to ensure compatibility with other tools. I understand that Explorer currently doesn’t support Arrow RecordBatch or Table, but I believe it’s best to follow the approach used by other implementations.

Once we have the basic implementation to return an Arrow RecordBatch/Table, we can revisit and enable data loading in Explorer in a way that makes sense for Explorer.

Agreed. Explorer may be able to do distribution in the future but it doesn’t mean everyone has to use it. Being able to receive directly from Arrow would be welcome too. We already do something similar in DataFrame.from_query, where we pass a pointer around.

Btw, here is an issue we can keep this discussion moving forward: Add delta lake file support · Issue #752 · elixir-explorer/explorer · GitHub

I also wonder if we should leverage Iceberg via DuckDB: https://duckdb.org/docs/stable/extensions/core_extensions

You can already use DuckDB via GitHub - livebook-dev/adbc: Apache Arrow ADBC bindings for Elixir · GitHub.

I’m really interested on this topic. I was wondering if you find any good alternative. Thank you!