NatureWhistle v0.4.1 is out!

This release is a pretty significant step forward for the project.

When I first built NatureWhistle, the idea was simple:

Listen to Telemetry. Detect something abnormal. Tell me about it.

As the project grew, I wanted to make it easier to extend without turning the core into a collection of metric-specific logic.

That led to the biggest architectural change in this release: Alert Packs.

The goal is simple: NatureWhistle should provide useful alerts out of the box, while still allowing applications to decide exactly what they want to monitor.

What’s new

Alert Pack architecture

Alerts can now be organised into independent packs rather than being tightly coupled to the core.

v0.4.1 ships with three built-in packs:

• BEAM Pack — VM memory, process memory, ETS memory, binary memory, process count, atom count, port count, and run queue

• Ecto Pack — slow queries, queue time, database execution time, decode and encode time

• Oban Pack — slow jobs/queues, job exceptions, and repeated job failures

The idea is that each pack owns a particular area of observability while NatureWhistle’s core remains responsible for the common alerting machinery.

That also makes the system easier to extend with application-specific Telemetry events.

BEAM Pack

The BEAM pack gives you a set of VM-level alerts without having to build the Telemetry collection yourself.

For example, you can configure a high-memory alert:

config :nature_whistle,
  beam: [
    high_memory: [
      threshold: 1_073_741_824
    ]
  ]

The BEAM pack covers metrics such as:

:memory
:process_memory
:ets_memory
:binary_memory
:process_count
:atom_count
:port_count
:run_queue

The collector periodically gathers the configured VM metrics and emits the corresponding Telemetry events.

So instead of your application having to continuously call :erlang.memory/0, :erlang.system_info/1, etc. and wire those measurements into an alerting system, NatureWhistle handles that part for you.

Disabling an alert

You don’t have to use every alert in a pack.

For example, if you don’t care about process count but want the other BEAM alerts, you can disable that specific alert:

config :nature_whistle,
  beam: [
    high_process_count: false
  ]

This is important because the packs are not all-or-nothing.

You can take the BEAM pack and choose the subset of signals that make sense for your application.

Overriding thresholds

The built-in thresholds are defaults, not rules.

For example:

config :nature_whistle,
  beam: [
    high_process_count: [
      threshold: 100_000
    ]
  ]

This allows the same pack to make sense for applications with very different workloads.

A service legitimately running tens of thousands of processes shouldn’t necessarily be treated the same way as a small Phoenix application.

Ecto Pack

The Ecto pack focuses on database performance.

It provides alerts around things such as:

• slow queries

• queue time

• database execution time

• decode time

• encode time

For example, you can configure a slow-query threshold:

config :nature_whistle,
  ecto: [
    slow_query: [
      threshold: 1_000
    ]
  ]

The important part here is that NatureWhistle isn’t replacing Ecto telemetry.

It is consuming the Telemetry events that Ecto already emits and turning selected conditions into actionable alerts.

So if you’re already using Ecto’s Telemetry events, NatureWhistle can sit on top of them.

Oban Pack

The Oban pack does something similar for background jobs.

It provides alerts around:

• slow jobs

• slow queues

• job exceptions

• repeated job failures

For example:

config :nature_whistle,
  oban: [
    slow_job: [
      threshold: 5_000
    ]
  ]

This gives you a way to monitor the behaviour of your background jobs without having to build a separate alerting layer around Oban’s Telemetry events.

Runtime alert management

One of the things I’m particularly happy with in this release is that alerts don’t have to be entirely defined at application startup.

You can register an alert at runtime:

NatureWhistle.register_alert(%{
  id: :my_custom_alert,
  event: [:my_app, :something, :happened],
  measurement_key: :duration,
  threshold: 1_000
})

Once registered, NatureWhistle takes care of the Telemetry handler required for that alert.

You can also remove it:

NatureWhistle.unregister_alert(:my_custom_alert)

Runtime alerts are intentionally ephemeral.

They exist for the lifetime of the running application and don’t require changing application configuration or restarting the application.

This is useful for application-specific signals that you may want to experiment with or enable dynamically.

For example, your application might emit:

:telemetry.execute(
  [:my_app, :checkout, :stop],
  %{duration: duration},
  %{user_id: user_id}
)

You could then register an alert against that event:

NatureWhistle.register_alert(%{
  id: :slow_checkout,
  event: [:my_app, :checkout, :stop],
  measurement_key: :duration,
  threshold: 2_000
})

Now a checkout taking longer than two seconds can participate in the same alerting pipeline as the built-in BEAM, Ecto and Oban alerts.

That’s the part of the Alert Pack architecture I think is most interesting: the built-in packs are really just a starting point.

Automatic Telemetry handler synchronisation

Runtime registration introduces an important problem.

If an alert is added dynamically, NatureWhistle needs to start listening to its Telemetry event.

If an alert is removed, it shouldn’t keep unnecessary handlers around.

NatureWhistle v0.4 handles that automatically.

When alerts are registered or removed, NatureWhistle synchronises the Telemetry handlers with the alerts currently configured.

So you don’t have to manually attach and detach handlers yourself.

This also means that multiple alerts can share the same Telemetry event without NatureWhistle creating unnecessary duplicate handlers.

Stateful incident tracking

Another major change is that NatureWhistle no longer treats every abnormal Telemetry event as an entirely new incident.

It now tracks the lifecycle of an alert.

Conceptually, an alert can move through states like:

healthy
   │
   │ threshold exceeded
   ▼
breached
   │
   │ condition remains abnormal
   │
   └───────────────┐
                   │
                   │ condition recovers
                   ▼
                healthy

This allows NatureWhistle to distinguish between:

• an incident starting

• an incident remaining active

• an incident recovering

This matters because a production system can generate thousands of abnormal events while one underlying incident is still happening.

You generally don’t want thousands of identical notifications.

Instead, NatureWhistle can notify you when the incident starts and then notify you again when the metric recovers.

Richer alert context

Alerts also carry more context about the Telemetry event that triggered them.

That means notifications can contain useful information about the event rather than simply saying:

CPU threshold exceeded

The intention is to make the notification something you can actually use when investigating a production problem.

BEAM metrics collector

The release also introduces a supervised collector for BEAM metrics.

The collector periodically samples the configured VM metrics and emits them through Telemetry.

This means the built-in BEAM alerts don’t require your application to manually collect those metrics.

The collector is also configurable, so you don’t have to collect every supported BEAM metric if your application doesn’t need them.

The general idea is:

BEAM VM
   │
   ▼
NatureWhistle Collector
   │
   ▼
Telemetry events
   │
   ▼
Alert Packs
   │
   ▼
Incident tracking
   │
   ▼
Notifier

That keeps collection, detection, incident state, and notification as separate concerns.

A few other improvements

There are a number of smaller changes in this release as well:

• BEAM run-queue thresholds are normalised against the number of online schedulers.

• Built-in alerts can be disabled or have their thresholds overridden.

• Runtime failure tracking is supervised.

• Telemetry handler lifecycle is managed dynamically.

• Startup ordering around the BEAM collector was tightened up.

• Alert simulation and testing utilities were improved.

What’s next?

There are still things I want to explore here.

One of the next areas I’m particularly interested in is detecting growing process mailboxes — one of my networks on LinkedIn hinted that.

The challenge is that BEAM applications can have a very large number of processes, so simply monitoring every process individually isn’t necessarily the approach I want to take.

I’m more interested in detecting processes whose mailbox behaviour is trending in the wrong direction and identifying them before they become a larger production problem.

That’s still an area I’m thinking through — I will really appreciate any suggestion on it.

This release is also the result of a lot of feedback and ideas from other Elixir developers.

So thank you to everyone who has taken the time to look at the project, give feedback, suggest improvements, or simply try it out.

For those who haven’t seen NatureWhistle before, the original announcement is here:

GitHub:

Hex:

Feedback, criticism, and contributions are very welcome.