Problem with logging using logstash

I’m using this library

I have the following logstash config

# Sample Logstash configuration for receiving
# UDP syslog messages over port 7777

input {
  udp {
    port => 7777
    queue_size => 10000
    workers => 10
    type => "syslog"

output {
  stdout {}

and I configured logger_logstash_backend in phoenix as:

Configures Elixir’s Logger

config :logger,
  backends: [{LoggerLogstashBackend, :info_log}, :console]

config :logger, :info_log,
  host: "",
  port: 7777,
  level: :info,
  type: "syslog",
  metadata: [
    extra_fields: "go here"

and try to call it like "Here is something: FF"
but I got the error

[error] :gen_event handler {LoggerLogstashBackend, :info_log} installed in Logger terminating
** (MatchError) no match of right hand side value: {:error, :badarg}
    (logger_logstash_backend 3.0.0) lib/logger_logstash_backend.ex:61: LoggerLogstashBackend.log_event/5
    (logger_logstash_backend 3.0.0) lib/logger_logstash_backend.ex:36: LoggerLogstashBackend.handle_event/2
    (stdlib 3.11) gen_event.erl:577: :gen_event.server_update/4
    (stdlib 3.11) gen_event.erl:559: :gen_event.server_notify/4
    (stdlib 3.11) gen_event.erl:561: :gen_event.server_notify/4
    (stdlib 3.11) gen_event.erl:300: :gen_event.handle_msg/6
    (stdlib 3.11) proc_lib.erl:249: :proc_lib.init_p_do_apply/3

Ignoring the fact that you are using version 3.0.0 while there is 4.0.0 already, the problem seems there:

This is caused due to fact that a lot (even default one) backends assumed that logger metadata will include only data that can be safely displayed. This was never a case (as metadata() was defined as keyword() which mean that the value can be any key) and Elixir 1.10 includes metadata that has no JSON representation (namely tuples and sometimes functions). This is bug in the backend.

1 Like

I change to {:logger_logstash_backend, "~> 4.0.0"},

and remove the metadata()

config :logger,
  backends: [{LoggerLogstashBackend, :info_log}, :console]

config :logger, :info_log,
  host: "",
  port: 7777,
  level: :info,
  type: "syslog"

and still got the same error

This wasn’t about your custom metadata, but metadata that is passed to the LoggerLogstashBackend. It seems that there is no way to filter out what data will be passed to JSX as since 1.10 the metadata will contain mfa: {module, function, arity} field which cannot be encoded by JSX.

1 Like

So I need to use another Library to deal with logstash :frowning: , I couldn’t find another good one

Logging to a file and using logstash, filebeat, etc. to watch those files and ship to something else is an option.

It’s a little more infra to setup but a common pattern

To solve my problem I used this library:

with this configuration:

config :logger,
  backends: [
    {LogstashJson.TCP, :logstash}
config :logger, :logstash,
  level: :warn,
  host: {:system, "LOGSTASH_TCP_HOST", "localhost"},
  port: {:system, "LOGSTASH_TCP_PORT", "7777"},
  workers: 2,
  buffer_size: 10_000

and it worked without any problems