Rich_Morin

Preferred data representation(s) for property graphs?

I have a (very WIP) project that will be making some graph-structured data generally available. I’d really like it to be accessible and convenient for AI systems (e.g., LLMs) and their developers. So, I’d like to know whether there are any preferred data representation(s) for property graphs, including pitfalls, etc.

Background

Internally, the data is maintained in an Elixirish version of GraphQL. This (WIP) approach retains GraphQL’s ability to let clients dynamically request subsets of a published data set, but in a BEAM-friendly manner (Actors, messages, terms).

So, the native data structures are dynamic (i.e., run-time friendly) Elixir terms: nested maps, lists, and scalars, but few atoms or structs. A GraphQL front end is an obvious addition, because the semantics are extremely similar. However, this is sort of a “raw” interface.

The next-level structure is a property graph (i.e., entities and relationships, both with attached attributes). Used by means of a property graph database (think Gremlex or Neo4j), this provides access, navigation, organization, processing, and storage.

Although a property graph can be expressed in terms of maps and such, the underlying data structures can also be finessed by graph query languages such as Cypher.

In summary, there will be a way to generate almost any desired concrete data representation. Which brings us back to the original question: what data format(s) would AI code and coders find the most palatable? (ducks)

-r

4 comments

/nx #data-structures #neo4j #graphql #gremlex #dynamic #deep-learning #graph-database #property-graph

1 712 4

2023-08-14 22:18:55 UTC

Most Liked

polvalente

Nx Core Team

I think it would be nice if we could have an easy way to convert to/from Explorer DataFrames and Nx.Tensors. Those are the two main ML data structures we have in the Nx ecosystem and having a way to leverage them would help bridge the gap from your library to the rest of the ecosystem.

This doesn’t mean they should necessarily be the main output format, but conversion functions would be nice.

Post #1

Last Post!

Rich_Morin

That sounds quite promising, for the GraphQLish API. However, it may not be relevant to the task of getting from a graph database to the outside world. Specifically, Neo4j (or whatever) will have ways to export data. So, the problem may come down to selecting from the available offerings. However, it strikes me that we may be missing something.

Naive speculation…

LLMs are already doing a lot of handling of semi-structured data. I suspect that most of this is in the form of HTML, with the rest being in PDF et al. So, they have all sorts of markup (generally semantic) to digest.

Also, most web sites contain internal links that form a directed graph. Some of these (e.g., Hex.pm, HexDocs.pm, Wikidata/Wikipedia, large commercial sites) have substantial and well-manicured graphs.

These graphs contain a lot of relationship information, which I presume LLMs can exploit to at least some degree (clues welcome…). For example, when they digest Wikipedia (and presumably Wikidata), how much advantage do they take of the links?

Anyway, I wonder whether automagically generated HTML, via LiveView, might not be easily digestible by both humans and LLMs. Nestled in these pages, or perhaps available for “download”, could be all sorts of tensors, etc. Does this make sense?

-r

Post #4