GeoSQL - spatial databases and GIS SQL functions

GeoSQL provides access to spatial data stored in databases such as PostGIS and Spatialite (SQLite3 extension for spatial data).

Spatial data is everywhere and mapping is a key feature in many applications. GeoSQL provides a portable and powerful toolbox for interacting with the spatial databases we store that data in.

Portable GIS functionality

GeoSQL provides access to hundreds of standard and non-standard spatial SQL functions to Ecto. This makes representing and interacting with spatial data easy in Ecto schemas and queries. Two-dimensional data as well as Z, M, and ZM variants are supported along with coordinate system support via SRIDs.

Functions are split out by standards and availability including GeoSQL.MM2 and GeoSQL.MM3. Beyond standards, GeoSQL.Common contains non-standard but widely implemented functions, and database-specific calls are also provided in e.g. the GeoSQL.PostGIS modules. Extensive support for multi-dimensional and topological queries is included. This makes it easy to see in your code which standards and backends a given piece of code requires to function.

Database-specific type extensions are provided along with utilities to e.g. decode geometries as needed.

The underlying geometry library supports WKT, WKB, and GeoJSON standards for easy data interchange. Other formats such as GML and KML are also supported via SQL functions.

defmodule MayApp.SpatialData do
  use Ecto.Schema
  use GeoSQL.MM2
  use GeoSQL.QueryUtils

  schema "spatial_data" do
    field(:name, :string)
    field(:geom, GeoSQL.Geometry)
    field(:point, GeoSQL.Geometry.Point)
    field(:pointz, GeoSQL.Geometry.PointZ)
    field(:linestring, GeoSQL.Geometry.LineString)
    field(:multipointzm, GeoSQL.Geometry.MultiPointZM)
    field(:polygon, GeoSQL.Geometry.Polygon)
    field(:multipolygon, GeoSQL.Geometry.MultiPolygon)
  end

   def features(repo) do
      from(s in __MODULE__, select: s.geom)
      |> repo.all()
   end

   def boundaries(repo) do
      from(s in __MODULE__, select: MM2.boundary(s.geom))
      |> repo.all()
      |> QueryUtils.decode_geometry(repo)
    end
end

High-level workflows

GeoSQL also provides high-level functionality, such as generating Mapbox Vector Tiles directly from a PostGIS database with a single function call. This makes it trivial to, for example, create vector map tile servers with a simple Plug:

defmodule MapTiles.VectorTilePlug do
  @behaviour Plug

  import Plug.Conn
  alias GeoSQL.PostGIS.VectorTiles

  def init(opts), do: opts

  def call(%Plug.Conn{method: "GET", params: %{"x" => x, "y" => y, "z" => z}} = conn, _opts) do
    z = Toolbelt.to_integer(z)
    x = Toolbelt.to_integer(x)
    y = Toolbelt.to_integer(y)

    layers = [
      %VectorTiles.Layer{
        name: "pois",
        source: "nodes",
        columns: %{geometry: :geom, id: :node_id, tags: :tags}
      }
    ]

    result = VectorTiles.generate(MapTiles.Repo, z, x, y, layers, "osm")

    conn
    |> put_resp_content_type("application/vnd.mapbox-vector-tile")
    |> resp(200, result)
    |> send_resp()
    |> halt()
  end
end

More workflows will be provided as the library develops.

Design Goals

  • Ease: fast to get started and hide complexity where possible.
  • Portability: schemas and queries that work across databases, hiding the many (often trivial) differences between them.
  • Completeness: extensive support for GIS SQL functions, not only the most common ones.
  • Clarity: functions organized by their availability and standards compliance.
  • Utility: out-of-the-box support for complete worfklows, to make complex tasks simple.
  • Reliability: unit tested and deployed in real-world applications.
16 Likes

Some shout-outs and gratitude …

geo_postgis

The people at Felt who make awesome mapping products have long maintained the geo and geo_postgis libraries. I’ve even contributed a patch here or there as I’ve used both of those libraries over the years in various projects.

GeoSQL began as a fork of geo_postgis when my needs grew beyond what geo_postgis provides in ways that were largely incompatible with it: spatialite, newer versions of PostGIS, high-level workflows … it’s grown far beyond that initial fork, but that is still where it started and I have immense gratitude for the work put into geo_postgis.

geometry

I moved from the geo library to geometry when I discovered that geo did not support EMPTY geometries. (I have submitted a PR to fix that, though!) While looking to see if there were other alternatives, I found the promising geometry library.

I wasn’t sure if it was still maintained as the last release was around 2 years ago, but thankfully it certainly was! The maintainer, @Marcus, was gracious enough to merge the PRs I showed up with and soon it could easily replace geo in GeoSQL.

What was really impressive was the performance, however: it uses around half the memory of geo and is anywhere from 50% to 1000%+ faster than geo depending on what one is doing with it. Consistently faster with a smaller memory footprint? Yes, please! :slight_smile:

Combined with the nice GeoJSON support and complete access to Z, ZM, and M variants of the geometry types, a big thanks to @Marcus for his work on geometry!

exqlite and ecto_sqlite3

Another key component was @warmwaffles’ SQLite3 libraries. I needed a way to extend the type support for SQLite3 databases with the Spatialite extension, and they brought in the necessary features to make that possible.

Now geometry types and spatial SQL queries work (nearly) as well with Spatialite as they do with PostGIS. I say “nearly” because SQLite is its own “fun” world of peculiar details. It’s why GeoSQL provides decode_geom and wrap_wkb functions … but it works!

The open source GIS community

The people working on things like PostGIS, Spatialite, proj, OSM, etc. get a fraction of the recognition they deserve in my opinion. They are doing amazing work in a very technical and (in terms of data) messy area of applied computer science: GIS. I’m constantly amazed by the work they do and the projects they work on and maintain for the whole world to benefit from. :slight_smile:

6 Likes

So .. where next?

Planned and/or already in progress

  • More SQL functions, for starters While there are over 290 already supported in GeoSQL, there are dozens more I plan to add support for.
  • Curves. Functions for working with curves are already there, but the geometry library does not yet support them. Types along with WTK and WKB encoding/decoding are needed, but we’re already on that and I hope to see that arrive in the coming weeks.
  • Expand the test suite. This is how I have found all sorts of strange edge cases and subtle variations in backends, and with this sort of macro-heavy code it’s pretty much a requirement. MM2 has very good coverage at this point, while MM3 and Common functions both have partial coverage. The PostGIS functions needs more work (though the vector tile functions have decent coverage). I’ll continue working to expand on these.

… and after that?

  • More workflows. The vector tile generation has been a godsend for projects I am currently working on, and I expect to add more “batteries included” modules over time.
  • GeoJSON .. maybe? geometry has very good GeoJSON support, as do the database backends, in particular PostGIS. I suspect there are opportunities here for useful things to be created.

… and things I don’t plan on working much on (but don’t let that stop you)

  • Performance. I have done essentially zero performance related work. The geometry library already gives GeoSQL a fantastic foundation to build upon there, but maybe there are other things that can be done? The vector tiles are reasonable: on the full OSM extract of Switzerland, it spits out tiles in <4ms per tile on my dev machine, which I consider sufficient for the time being. There are janky-looking things going on with the Spatialite types module, but I’ve used that same approach previously in projects where I have done performance measurements and those things are so fast as to be ignorable.
  • More backends. I’d love to see more databases supported, such as MySQL which also has an implementation of the SQL/MM standard. My own needs are covered by PostGIS and Spatialite, however, so this will depend on other motivated individuals.

PRs welcome! :slight_smile:

3 Likes

I released v1.0.0 today. The major version change was due to a breaking change in the API, driven by further experience in production use. The library should be API stable from here; near-term future work will focus on curve geometries and continuing to expand the test suite.

Changelog

  • Breaking changes
    • All SQL/MM functions were collapsed into a single MM module.
      Migration: Replace all instances of MM2 and MM3 with MM.
  • Improvements
    • Expanded unit test suite considerably, improved vector tile generation tests
  • Fixes
    • MM.locate_along was missing a parameter placeholder
3 Likes