din_S
March 27, 2020, 8:51am
1
I’m using this library
I have the following logstash config
# Sample Logstash configuration for receiving
# UDP syslog messages over port 7777
input {
udp {
port => 7777
queue_size => 10000
workers => 10
type => "syslog"
}
}
output {
stdout {}
}
and I configured logger_logstash_backend in phoenix as:
Configures Elixir’s Logger
config :logger,
backends: [{LoggerLogstashBackend, :info_log}, :console]
config :logger, :info_log,
host: "127.0.0.1",
port: 7777,
level: :info,
type: "syslog",
metadata: [
extra_fields: "go here"
]
and try to call it like
Logger.info "Here is something: FF"
[error] :gen_event handler {LoggerLogstashBackend, :info_log} installed in Logger terminating
** (MatchError) no match of right hand side value: {:error, :badarg}
(logger_logstash_backend 3.0.0) lib/logger_logstash_backend.ex:61: LoggerLogstashBackend.log_event/5
(logger_logstash_backend 3.0.0) lib/logger_logstash_backend.ex:36: LoggerLogstashBackend.handle_event/2
(stdlib 3.11) gen_event.erl:577: :gen_event.server_update/4
(stdlib 3.11) gen_event.erl:559: :gen_event.server_notify/4
(stdlib 3.11) gen_event.erl:561: :gen_event.server_notify/4
(stdlib 3.11) gen_event.erl:300: :gen_event.handle_msg/6
(stdlib 3.11) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Ignoring the fact that you are using version 3.0.0 while there is 4.0.0 already, the problem seems there:
|> Keyword.merge(metadata)
|> Enum.into(%{})
|> Map.put(:level, to_string(level))
|> inspect_pids
{{year, month, day}, {hour, minute, second, milliseconds}} = ts
{:ok, ts} = NaiveDateTime.new(
year, month, day, hour, minute, second, (milliseconds * 1000)
)
ts = Timex.to_datetime ts, Timezone.local
{:ok, json} = JSX.encode %{
type: type,
"@timestamp": Timex.format!(ts, "%FT%T%z", :strftime),
message: to_string(msg),
fields: fields
}
:gen_udp.send socket, host, port, to_char_list(json)
end
defp configure(name, opts) do
env = Application.get_env :logger, name, []
This is caused due to fact that a lot (even default one) backends assumed that logger metadata will include only data that can be safely displayed. This was never a case (as metadata() was defined as keyword() which mean that the value can be any key) and Elixir 1.10 includes metadata that has no JSON representation (namely tuples and sometimes functions). This is bug in the backend.
1 Like
din_S
March 27, 2020, 1:13pm
3
I change to {:logger_logstash_backend, "~> 4.0.0"},
and remove the metadata()
config :logger,
backends: [{LoggerLogstashBackend, :info_log}, :console]
config :logger, :info_log,
host: "127.0.0.1",
port: 7777,
level: :info,
type: "syslog"
and still got the same error
This wasn’t about your custom metadata, but metadata that is passed to the LoggerLogstashBackend. It seems that there is no way to filter out what data will be passed to JSX as since 1.10 the metadata will contain mfa: {module, function, arity} field which cannot be encoded by JSX.
1 Like
din_S
March 27, 2020, 1:21pm
5
So I need to use another Library to deal with logstash
Logging to a file and using logstash, filebeat, etc. to watch those files and ship to something else is an option.
It’s a little more infra to setup but a common pattern
din_S
April 6, 2020, 4:38pm
7
To solve my problem I used this library:
with this configuration:
config :logger,
backends: [
{LogstashJson.TCP, :logstash}
]
config :logger, :logstash,
level: :warn,
host: {:system, "LOGSTASH_TCP_HOST", "localhost"},
port: {:system, "LOGSTASH_TCP_PORT", "7777"},
workers: 2,
buffer_size: 10_000
and it worked without any problems
, I couldn’t find another good one
