Locus: Geolocation and ASN lookup of IP addresses

Hi forum,

Locus 1.1.1 was released today.

Added:

• OTP 18, 19.0, 19.1 and 19.2 support (version 1.0.x required 19.3 or higher)
• ability of consulting database metadata, source and version through :get_info
• ability of subscribing database loader events
• ability of specifying connect, download start and idle download timeouts
• ability of turning off caching

Documentation was moved to HexDocs and test coverage was substantially increased.

• Overview
• Documentation

Great work! That’s a hugely useful dataset to have easy access too.

Without having looked heavily at what you’ve done, what is the size of the database?

Thanks!

Without having looked heavily at what you’ve done, what is the size of the database?

It depends on the database you pick. MaxMind provides three databases for free:

• Country database (~3 MiB when uncompressed)
• City database (~60 MiB when uncompressed; conceptually similar to country database but with more precision)
• ASN database (~6 MiB when uncompressed)

As the databases are, for the most part, used in their raw form, the expected memory consumption should correspond roughly one-to-one.

They’re not, however, included in the package; they’re downloaded on demand, and caching is leveraged to minimize likelihood of unavailability upon subsequent application boots. My main motivation for this was to avoid some of the shortcomings of egeoip, where database updates were committed to the repository, which grew ever larger.

I do plan to support explicit loading from local filesystem, with or without support for updating from network after having that as a starting point. But because the semantics of this, and all its edge cases, aren’t still quite clear in my head, I’ve been delaying it, as I want to keep things as simple as possible for now.

Whooo awesome there!

Hi forum,

Locus 1.3.0 was released today.

Added:

• ability of loading databases from local file system
• type spec of database entries

Fixed:

• wrong handling of timezones on cached tarballs
• wrong handling of daylight saving time on conditional HTTP requests

The timezone / DST fixes mentioned above were also backported to earlier versions and tagged under:

• 1.0.1
• 1.1.4
• 1.2.2

Links:

• Package
• Documentation

Hello forum,

Locus 1.6.0 was released today.

Added:

• new API method for validating loaded databases (locus:analyze/1)
• new command line tool supporting database validation

Changed:

• safety of database HTTPS downloads was substantially improved by now rejecting expired certificates, mismatched hostnames, self-signed certificates or unknown certificate authorities
• test coverage using MaxMind’s test data was greatly extended
• database decoder was thoroughly optimized
• documentation was mildly improved

Fixed:

• misguided rejection of UTF-8 strings with non-printable (but valid) codepoints
• unnecessarily strict refusal to load 2.x database formats succeeding 2.0
• infinite recursion in maliciously crafted databases due to circular paths

Links:

• Hex package
• Documentation
• GitHub
• GitLab

Hello again,

Locus 1.7.0 was released today.

Added

• ability of loading databases from uncompressed tarballs (.tar files)
• ability of loading unpacked databases (.mmdb and .mmdb.gz files)
• stacktrace of caught exceptions to event reporting (including custom logger)
• ability of launching database loaders under library consumers’ own supervisors
• wait_for_loaders/2 API method for concurrently awaiting multiple database loaders

Changed

• log level of HTTP and filesystem database loading failures from warning to error
• HTTP and filesystem loaders into a common loader codebase
• caching of HTTP databases as to store and load compressed .mmdb files rather than tarballs
• supervision structure as to launch database loaders as transient processes under a new simple_one_for_one supervisor
• dependency versions:
  • certifi [2.4.2 => 2.5.1]
  • ssl_verify_fun [1.1.4 => 1.1.5]

Removed

• support for OTP 17.4 and 17.5
• undocumented support for rebar 2
• half-baked and unwarranted support for file://-prefixed URLs

Fixed

• case-sensitive patterning of .mmdb file extensions within tarballs
• overly verbose logger messages on OTP 21.1+
• HTTPS certificate validation test cases on OTP 22

Links:

• Hex package
• Documentation
• GitHub
• GitLab

I’m pleased to announce the release of Locus 1.8.0 .

Added

• support for returning types other than map upon successful lookups

Changed

• MMDB decoder, which was split into separate tree, data section and analysis modules
• imported stacktrace_compat version [1.0.2 => 1.1.1]

Removed

• support for OTP 18

Fixed

• incidents of locus managerial processes keeping references to old binaries, upon a database update, for a potentially unlimited time (OTP 20+ only)
• broken logging of playground shell on OTP 21.1+

Links:

• Hex package
• Documentation
• GitHub
• GitLab

Locus 1.9.0, after one month in beta, has been released!

Added

• support for downloading databases with full awareness of license keys (now mandatory)

Deprecated

• the use of discontinued https://geolite.maxmind.com/download/geoip/database/GeoLite2-... database URLs

Links:

• Hex package
• Documentation
• GitHub
• GitLab

Locus 1.10.0 was released today (a rather short release cycle this time.)

Added

• rejection of successful HTTP downloads if body size doesn’t match content-length header
• checksum verification of databases downloaded directly from MaxMind
• censorship of license key from database URLs mentioned in logs
• :await_loader API function which, contrary to :wait_for_loader, will await readiness up to the entire specified interval (rather than return upon the first encountered failure)
• purging of very large binaries from internally caught exceptions which are known error cases, as to lower the risk of the VM getting OOM-killed when logging formatters get their hands on those very large chunks of data

Changed

• default behaviour upon failing to load a database, as to retry loading while exponentially backing off (using very short intervals at first)

Deprecated

• :wait_for_loader and :wait_for_loaders API functions (use :await_loader and :await_loaders instead)

Fixed

• incomplete spec for locus_loader:event() type
• wrong spec for locus_maxmind_download:msg() and locus_maxmind_download:event() types

Links:

• Hex package
• Documentation
• Source code (GitHub)
• Source code (GitLab)