Locus: Geolocation and ASN lookup of IP addresses

Hi forum,

I’m pleased to announce the release of locus 1.0.0, an Erlang/OTP library that allows you to pinpoint the country, city or ASN of IPv4 and IPv6 addresses, using MaxMind’s GeoLite2 databases.

The MaxMind databases you choose are downloaded on-demand, cached on the filesystem and updated automatically.

The databases are loaded into memory (mostly) as-is; reference counted binaries are shared with the application callers using ETS tables, and the original binary search tree is used to lookup addresses. The data for each entry is decoded on the fly upon successful lookups.

17 Likes

Hi forum,

Locus 1.1.1 was released today.

Added:

  • OTP 18, 19.0, 19.1 and 19.2 support (version 1.0.x required 19.3 or higher)
  • ability of consulting database metadata, source and version through :get_info
  • ability of subscribing database loader events
  • ability of specifying connect, download start and idle download timeouts
  • ability of turning off caching

Documentation was moved to HexDocs and test coverage was substantially increased.

2 Likes

Great work! That’s a hugely useful dataset to have easy access too.

Without having looked heavily at what you’ve done, what is the size of the database?

Thanks! :slight_smile:

Without having looked heavily at what you’ve done, what is the size of the database?

It depends on the database you pick. MaxMind provides three databases for free:

  • Country database (~3 MiB when uncompressed)
  • City database (~60 MiB when uncompressed; conceptually similar to country database but with more precision)
  • ASN database (~6 MiB when uncompressed)

As the databases are, for the most part, used in their raw form, the expected memory consumption should correspond roughly one-to-one.

They’re not, however, included in the package; they’re downloaded on demand, and caching is leveraged to minimize likelihood of unavailability upon subsequent application boots. My main motivation for this was to avoid some of the shortcomings of egeoip, where database updates were committed to the repository, which grew ever larger.

I do plan to support explicit loading from local filesystem, with or without support for updating from network after having that as a starting point. But because the semantics of this, and all its edge cases, aren’t still quite clear in my head, I’ve been delaying it, as I want to keep things as simple as possible for now.

3 Likes

Whooo awesome there!

Hi forum,

Locus 1.3.0 was released today.

Added:

  • ability of loading databases from local file system
  • type spec of database entries

Fixed:

  • wrong handling of timezones on cached tarballs
  • wrong handling of daylight saving time on conditional HTTP requests

The timezone / DST fixes mentioned above were also backported to earlier versions and tagged under:

  • 1.0.1
  • 1.1.4
  • 1.2.2

Links:

5 Likes

Hello forum,

Locus 1.6.0 was released today.

Added:

  • new API method for validating loaded databases (locus:analyze/1)
  • new command line tool supporting database validation

Changed:

  • safety of database HTTPS downloads was substantially improved by now rejecting expired certificates, mismatched hostnames, self-signed certificates or unknown certificate authorities
  • test coverage using MaxMind’s test data was greatly extended
  • database decoder was thoroughly optimized
  • documentation was mildly improved

Fixed:

  • misguided rejection of UTF-8 strings with non-printable (but valid) codepoints
  • unnecessarily strict refusal to load 2.x database formats succeeding 2.0
  • infinite recursion in maliciously crafted databases due to circular paths

Links:

5 Likes

Hello again,

Locus 1.7.0 was released today.

Added

  • ability of loading databases from uncompressed tarballs (.tar files)
  • ability of loading unpacked databases (.mmdb and .mmdb.gz files)
  • stacktrace of caught exceptions to event reporting (including custom logger)
  • ability of launching database loaders under library consumers’ own supervisors
  • wait_for_loaders/2 API method for concurrently awaiting multiple database loaders

Changed

  • log level of HTTP and filesystem database loading failures from warning to error
  • HTTP and filesystem loaders into a common loader codebase
  • caching of HTTP databases as to store and load compressed .mmdb files rather than tarballs
  • supervision structure as to launch database loaders as transient processes under a new simple_one_for_one supervisor
  • dependency versions:
    • certifi [2.4.2 => 2.5.1]
    • ssl_verify_fun [1.1.4 => 1.1.5]

Removed

  • support for OTP 17.4 and 17.5
  • undocumented support for rebar 2
  • half-baked and unwarranted support for file://-prefixed URLs

Fixed

  • case-sensitive patterning of .mmdb file extensions within tarballs
  • overly verbose logger messages on OTP 21.1+
  • HTTPS certificate validation test cases on OTP 22

Links:

4 Likes

I’m pleased to announce the release of Locus 1.8.0 .

Added

  • support for returning types other than map upon successful lookups

Changed

  • MMDB decoder, which was split into separate tree, data section and analysis modules
  • imported stacktrace_compat version [1.0.2 => 1.1.1]

Removed

  • support for OTP 18

Fixed

  • incidents of locus managerial processes keeping references to old binaries, upon a database update, for a potentially unlimited time (OTP 20+ only)
  • broken logging of playground shell on OTP 21.1+

Links:

5 Likes

Locus 1.9.0, after one month in beta, has been released!

Added

  • support for downloading databases with full awareness of license keys (now mandatory)

Deprecated

  • the use of discontinued https://geolite.maxmind.com/download/geoip/database/GeoLite2-... database URLs

Links:

6 Likes

Locus 1.10.0 was released today (a rather short release cycle this time.)

Added

  • rejection of successful HTTP downloads if body size doesn’t match content-length header
  • checksum verification of databases downloaded directly from MaxMind
  • censorship of license key from database URLs mentioned in logs
  • :await_loader API function which, contrary to :wait_for_loader, will await readiness up to the entire specified interval (rather than return upon the first encountered failure)
  • purging of very large binaries from internally caught exceptions which are known error cases, as to lower the risk of the VM getting OOM-killed when logging formatters get their hands on those very large chunks of data

Changed

  • default behaviour upon failing to load a database, as to retry loading while exponentially backing off (using very short intervals at first)

Deprecated

  • :wait_for_loader and :wait_for_loaders API functions (use :await_loader and :await_loaders instead)

Fixed

  • incomplete spec for locus_loader:event() type
  • wrong spec for locus_maxmind_download:msg() and locus_maxmind_download:event() types

Links:

2 Likes

Hi everyone,

Locus 1.11.0 was released today.

Added

  • support for HTTPS redirections across distinct hostnames
  • support for not censoring license keys from MaxMind URLs mentioned in logs
  • support for censorship of arbitrary query arguments from HTTP URLs mentioned in logs
  • truncation of large HTTP URLs mentioned in logs
  • download_redirected events to HTTP downloads
  • too_many_redirections and invalid_redirection reasons to HTTP download failures
  • mention of compatibility with other providers

Changed

  • consumer-subscribed events as to follow the same URL censorship rules as the built-in logger
  • documented MaxMind database edition format from atoms to tuples while keeping retrocompatibility

Links:

2 Likes

Locus 1.12.0 was released today!

Added

  • support for Erlang/OTP 23

Changed

  • checksum verification algorithm of MaxMind downloads from MD5 to SHA-256

Links:

4 Likes

Hi forum,

Locus 2.0.0, a MaxMind DB reader, was released tonight.

It’s the first breaking release and I used this opportunity to drop most deprecated code as well as polish the existing API.

Added

  • support for retrieving databases using consumer-defined locus_custom_fetchers
  • support for decoding IEEE-754 infinities in MMDB data
  • locus:check/1 to API (which replaces locus:analyze/1 and can be up to 3200% faster :racing_car:)
  • details to MMDB unpacking errors
  • linting checks with rebar3_lint
  • dead code checks with rebar3_hank

Changed

  • :warning: return type of locus:lookup/2 (see MIGRATION.md)
  • database loader to use persistent_term instead of ETS
  • MMDB decoder to perform stricter metadata validations
  • MMDB decoder to not crash upon maps containing duplicate keys
  • databases downloaded through HTTP(S) without a last-modified response header to no longer be cached
  • imported version of tls_certificate_check to ‘~> 1.7’
  • single CT suite covering both filesystem and HTTP sources into one for each
  • test coverage for the better

Removed

  • :warning: locus:wait_for_loader/1 from API (deprecated in 1.10.0 - see MIGRATION.md)
  • :warning: locus:wait_for_loader/2 from API (deprecated in 1.10.0 - see …)
  • :warning: locus:wait_for_loaders/2 from API (deprecated in 1.10.0 - see …)
  • :warning: locus:get_version/1 from API (deprecated in 1.4.0 - see …)
  • :warning: locus:analyze/1 from API (locus:check/1 now fullfils this role - see …)
  • deprecated loader options pre_readiness_update_period and post_readiness_update_period (see …)
  • warnings on the use of discontinued GeoLite2 HTTP URLs

Links:

8 Likes

Locus 2.1.0 is out! (that didn’t take long)

Added

  • locus_mmdb:unpack_tree_data_and_data_section/2 to the API
  • locus_mmdb_data_codec:parse_on_index/3 to the API
  • CI on Windows

Removed

  • no longer warranted run time dependency (public_key)

Fixed

  • wrong return of errors during :check/1 for maps containing maps or arrays which had been checked previously
  • wrong paths in :check/1 errors or warnings concerning maps
  • no longer warranted type definition

(Hopefully stable now!)

Is there any way to use this library without a license from the upstream data provider?

Sure! As long as it’s in MMDB format, locus will know how to parse it[*].

I know of one alternative provider, and it doesn’t require license or registration for downloading their lite MMDB databases (although some restrictions apply):

And you can also build your own databases with arbitrary data (using tools other than locus, although it wouldn’t be too difficult to give it encoding capabilities.)


[*]: With the exception of a single data type which I’ve never seen so far

1 Like