Hi forum,
I’m pleased to announce the release of locus 1.0.0, an Erlang/OTP library that allows you to pinpoint the country, city or ASN of IPv4 and IPv6 addresses, using MaxMind’s GeoLite2 databases.
The MaxMind databases you choose are downloaded on-demand, cached on the filesystem and updated automatically.
The databases are loaded into memory (mostly) as-is; reference counted binaries are shared with the application callers using ETS tables, and the original binary search tree is used to lookup addresses. The data for each entry is decoded on the fly upon successful lookups.
17 Likes
Hi forum,
Locus 1.1.1 was released today.
Added:
- OTP 18, 19.0, 19.1 and 19.2 support (version 1.0.x required 19.3 or higher)
- ability of consulting database metadata, source and version through
:get_info
- ability of subscribing database loader events
- ability of specifying connect, download start and idle download timeouts
- ability of turning off caching
Documentation was moved to HexDocs and test coverage was substantially increased.
2 Likes
Great work! That’s a hugely useful dataset to have easy access too.
Without having looked heavily at what you’ve done, what is the size of the database?
Thanks! 
Without having looked heavily at what you’ve done, what is the size of the database?
It depends on the database you pick. MaxMind provides three databases for free:
- Country database (~3 MiB when uncompressed)
- City database (~60 MiB when uncompressed; conceptually similar to country database but with more precision)
- ASN database (~6 MiB when uncompressed)
As the databases are, for the most part, used in their raw form, the expected memory consumption should correspond roughly one-to-one.
They’re not, however, included in the package; they’re downloaded on demand, and caching is leveraged to minimize likelihood of unavailability upon subsequent application boots. My main motivation for this was to avoid some of the shortcomings of egeoip, where database updates were committed to the repository, which grew ever larger.
I do plan to support explicit loading from local filesystem, with or without support for updating from network after having that as a starting point. But because the semantics of this, and all its edge cases, aren’t still quite clear in my head, I’ve been delaying it, as I want to keep things as simple as possible for now.
3 Likes
Hi forum,
Locus 1.3.0 was released today.
Added:
- ability of loading databases from local file system
- type spec of database entries
Fixed:
- wrong handling of timezones on cached tarballs
- wrong handling of daylight saving time on conditional HTTP requests
The timezone / DST fixes mentioned above were also backported to earlier versions and tagged under:
Links:
5 Likes
Hello forum,
Locus 1.6.0 was released today.
Added:
- new API method for validating loaded databases (
locus:analyze/1)
- new command line tool supporting database validation
Changed:
-
safety of database HTTPS downloads was substantially improved by now rejecting expired certificates, mismatched hostnames, self-signed certificates or unknown certificate authorities
- test coverage using MaxMind’s test data was greatly extended
- database decoder was thoroughly optimized
- documentation was mildly improved
Fixed:
- misguided rejection of UTF-8 strings with non-printable (but valid) codepoints
- unnecessarily strict refusal to load 2.x database formats succeeding 2.0
- infinite recursion in maliciously crafted databases due to circular paths
Links:
5 Likes
Hello again,
Locus 1.7.0 was released today.
Added
- ability of loading databases from uncompressed tarballs (
.tar files)
- ability of loading unpacked databases (
.mmdb and .mmdb.gz files)
- stacktrace of caught exceptions to event reporting (including custom logger)
- ability of launching database loaders under library consumers’ own supervisors
-
wait_for_loaders/2 API method for concurrently awaiting multiple database loaders
Changed
- log level of HTTP and filesystem database loading failures from warning to error
- HTTP and filesystem loaders into a common loader codebase
- caching of HTTP databases as to store and load compressed
.mmdb files rather than tarballs
- supervision structure as to launch database loaders as transient processes under a new
simple_one_for_one supervisor
- dependency versions:
-
certifi [2.4.2 => 2.5.1]
-
ssl_verify_fun [1.1.4 => 1.1.5]
Removed
- support for OTP 17.4 and 17.5
- undocumented support for rebar 2
- half-baked and unwarranted support for
file://-prefixed URLs
Fixed
- case-sensitive patterning of
.mmdb file extensions within tarballs
- overly verbose
logger messages on OTP 21.1+
- HTTPS certificate validation test cases on OTP 22
Links:
4 Likes
I’m pleased to announce the release of Locus 1.8.0 .
Added
- support for returning types other than map upon successful lookups
Changed
- MMDB decoder, which was split into separate tree, data section and analysis modules
- imported
stacktrace_compat version [1.0.2 => 1.1.1]
Removed
Fixed
- incidents of
locus managerial processes keeping references to old binaries, upon a database update, for a potentially unlimited time (OTP 20+ only)
- broken logging of playground shell on OTP 21.1+
Links:
5 Likes
Locus 1.9.0, after one month in beta, has been released!
Added
- support for downloading databases with full awareness of license keys (now mandatory)
Deprecated
- the use of discontinued
https://geolite.maxmind.com/download/geoip/database/GeoLite2-... database URLs
Links:
6 Likes
Locus 1.10.0 was released today (a rather short release cycle this time.)
Added
- rejection of successful HTTP downloads if body size doesn’t match
content-length header
- checksum verification of databases downloaded directly from MaxMind
- censorship of license key from database URLs mentioned in logs
-
:await_loader API function which, contrary to :wait_for_loader, will await readiness up to the entire specified interval (rather than return upon the first encountered failure)
- purging of very large binaries from internally caught exceptions which are known error cases, as to lower the risk of the VM getting OOM-killed when logging formatters get their hands on those very large chunks of data
Changed
- default behaviour upon failing to load a database, as to retry loading while exponentially backing off (using very short intervals at first)
Deprecated
-
:wait_for_loader and :wait_for_loaders API functions (use :await_loader and :await_loaders instead)
Fixed
- incomplete spec for
locus_loader:event() type
- wrong spec for
locus_maxmind_download:msg() and locus_maxmind_download:event() types
Links:
2 Likes
Hi everyone,
Locus 1.11.0 was released today.
Added
- support for HTTPS redirections across distinct hostnames
- support for not censoring license keys from MaxMind URLs mentioned in logs
- support for censorship of arbitrary query arguments from HTTP URLs mentioned in logs
- truncation of large HTTP URLs mentioned in logs
-
download_redirected events to HTTP downloads
-
too_many_redirections and invalid_redirection reasons to HTTP download failures
- mention of compatibility with other providers
Changed
- consumer-subscribed events as to follow the same URL censorship rules as the built-in logger
- documented MaxMind database edition format from atoms to tuples while keeping retrocompatibility
Links:
2 Likes
Locus 1.12.0 was released today!
Added
- support for Erlang/OTP 23
Changed
- checksum verification algorithm of MaxMind downloads from MD5 to SHA-256
Links:
4 Likes
Locus 2.1.0 is out! (that didn’t take long)
Added
-
locus_mmdb:unpack_tree_data_and_data_section/2 to the API
-
locus_mmdb_data_codec:parse_on_index/3 to the API
- CI on Windows
Removed
- no longer warranted run time dependency (
public_key)
Fixed
-
wrong return of errors during
:check/1 for maps containing maps or arrays which had been checked previously
-
wrong paths in
:check/1 errors or warnings concerning maps
- no longer warranted type definition
(Hopefully stable now!)
Is there any way to use this library without a license from the upstream data provider?
Sure! As long as it’s in MMDB format, locus will know how to parse it[*].
I know of one alternative provider, and it doesn’t require license or registration for downloading their lite MMDB databases (although some restrictions apply):
And you can also build your own databases with arbitrary data (using tools other than locus, although it wouldn’t be too difficult to give it encoding capabilities.)
[*]: With the exception of a single data type which I’ve never seen so far
2 Likes
)
return type of 
