gastonrey
Broadway Kafka can't connect through SASL
Hi guys, I’m facing an issue trying to connect to kafka using the broadway_kafka connector, I need to pass a user and pass but I’m getting an error that it couldn’t join the given group, looking the output I see the arguments are not being taken as expected, looks like the password position is being held as the callback, even when I pass the :plain key in config:
Output config on start:
mfargs: {:brod_client, :start_link,
[
[{"xxxx.aws.confluent.cloud", 9092}],
HubspotWorkers.HubspotDuplicaetDomainsWorker.Broadway.Producer_0.Client,
[
sasl: {:plain, "user",
#Function<11.44977799/0 in :brod_utils.init_sasl_opt/1>}
]
]},
restart_type: {:permanent, 10},
shutdown: 5000,
child_type: :worker
Output error:
[warning] :brod_client [#PID<0.474.0>] HubspotWorkers.HubspotDuplicaetDomainsWorker.Broadway.Producer_0.Client is terminating
reason: [
{{"xxxxxx.aws.confluent.cloud", 9092},
{{{:kpro_req, #Reference<0.3725464545.3257925635.50906>, :api_versions, 0,
false, []}, :closed},
[
{:kpro_lib, :send_and_recv_raw, 4,
[
file: '/Users/gastonrey/dev/elixir/hubspot_workers/deps/kafka_protocol/src/kpro_lib.erl',
line: 70
]},
{:kpro_lib, :send_and_recv, 5,
[
file: '/Users/gastonrey/dev/elixir/hubspot_workers/deps/kafka_protocol/src/kpro_lib.erl',
line: 81
]},
{:kpro_connection, :query_api_versions, 4,
[
file: '/Users/gastonrey/dev/elixir/hubspot_workers/deps/kafka_protocol/src/kpro_connection.erl',
line: 246
]},
{:kpro_connection, :init_connection, 3,
[
file: '/Users/gastonrey/dev/elixir/hubspot_workers/deps/kafka_protocol/src/kpro_connection.erl',
line: 233
]},
{:kpro_connection, :init, 4,
[
file: '/Users/gastonrey/dev/elixir/hubspot_workers/deps/kafka_protocol/src/kpro_connection.erl',
line: 170
]},
{:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 240]}
]}}
]
[error] GenServer HubspotWorkers.HubspotDuplicaetDomainsWorker.Broadway.Producer_0.Client terminating
** (stop) [{{"pkc-zpjg0.eu-central-1.aws.confluent.cloud", 9092}, {{{:kpro_req, #Reference<0.3725464545.3257925635.50906>, :api_versions, 0, false, []}, :closed}, [{:kpro_lib, :send_and_recv_raw, 4, [file: '/Users/gastonrey/dev/elixir/hubspot_workers/deps/kafka_protocol/src/kpro_lib.erl', line: 70]}, {:kpro_lib, :send_and_recv, 5, [file: '/Users/gastonrey/dev/elixir/hubspot_workers/deps/kafka_protocol/src/kpro_lib.erl', line: 81]}, {:kpro_connection, :query_api_versions, 4, [file: '/Users/gastonrey/dev/elixir/hubspot_workers/deps/kafka_protocol/src/kpro_connection.erl', line: 246]}, {:kpro_connection, :init_connection, 3, [file: '/Users/gastonrey/dev/elixir/hubspot_workers/deps/kafka_protocol/src/kpro_connection.erl', line: 233]}, {:kpro_connection, :init, 4, [file: '/Users/gastonrey/dev/elixir/hubspot_workers/deps/kafka_protocol/src/kpro_connection.erl', line: 170]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 240]}]}}]
(brod 3.16.5) /Users/gastonrey/dev/elixir/hubspot_workers/deps/brod/src/brod_client.erl:622: :brod_client.ensure_metadata_connection/1
(brod 3.16.5) /Users/gastonrey/dev/elixir/hubspot_workers/deps/brod/src/brod_client.erl:343: :brod_client.handle_info/2
(stdlib 4.3.1) gen_server.erl:1123: :gen_server.try_dispatch/4
(stdlib 4.3.1) gen_server.erl:1200: :gen_server.handle_msg/6
(stdlib 4.3.1) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Last message: :init
State: {:state, HubspotWorkers.HubspotDuplicaetDomainsWorker.Broadway.Producer_0.Client, [{"xxxxxx.aws.confluent.cloud", 9092}], :undefined, [], :undefined, :undefined, [sasl: {:plain, "xxxxx", #Function<11.44977799/0 in :brod_utils.init_sasl_opt/1>}], HubspotWorkers.HubspotDuplicaetDomainsWorker.Broadway.Producer_0.Client}
[info] Group member (hubspot_duplicate_domains_1,coor=#PID<0.478.0>,cb=#PID<0.473.0>,generation=0):
failed to join group
reason: {:client_down,
[
{{"xxxxxx.aws.confluent.cloud", 9092},
{{{:kpro_req, #Reference<0.3725464545.3257925635.50906>, :api_versions, 0,
false, []}, :closed},
[
{:kpro_lib, :send_and_recv_raw, 4,
[
file: '/Users/gastonrey/dev/elixir/hubspot_workers/deps/kafka_protocol/src/kpro_lib.erl',
line: 70
]},
{:kpro_lib, :send_and_recv, 5,
[
file: '/Users/gastonrey/dev/elixir/hubspot_workers/deps/kafka_protocol/src/kpro_lib.erl',
line: 81
]},
{:kpro_connection, :query_api_versions, 4,
[
file: '/Users/gastonrey/dev/elixir/hubspot_workers/deps/kafka_protocol/src/kpro_connection.erl',
line: 246
]},
{:kpro_connection, :init_connection, 3,
[
file: '/Users/gastonrey/dev/elixir/hubspot_workers/deps/kafka_protocol/src/kpro_connection.erl',
line: 233
]},
{:kpro_connection, :init, 4,
[
file: '/Users/gastonrey/dev/elixir/hubspot_workers/deps/kafka_protocol/src/kpro_connection.erl',
line: 170
]},
{:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 240]}
]}}
]}
[info] Group member (hubspot_duplicate_domains_1,coor=#PID<0.478.0>,cb=#PID<0.473.0>,generation=0):
Leaving group, reason: :shutdown
My config:
{BroadwayKafka.Producer,
[
client_config: [sasl: {:plain, "user", "password"}],
hosts: "xxxxx.aws.confluent.cloud:9092",
group_id: "xxxx_1",
topics: ["xxxx.events.json"]
]},
What is wrong? Am using broadway_kafka 0.4
Marked As Solved
gastonrey
Solved, forgot the ssl: true, so config is like:
client_config: [
ssl: true,
sasl:
{:plain, "user", "pass"}
]
0
Popular in Questions
** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2733ms. You can configure how long re...
New
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
Hello everyone,
I try to use an Javascript Event Handler in my root.html.leex file.
Therefore I created a function in the app.js file: ...
New
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
Hi all,
I’ve just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage.
I’m trying to use Postgres...
New
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this:
...
New
There are pre-rolled solutions for other frameworks that do work. However, Phoenix does not seem to have these. Have people had good expe...
New
Is there a way to rollback a specific migration and only that one (“skipping” all the other ones)?
Would
mix ecto.rollback -v 200809061...
New
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
New
Other popular topics
Hi,
I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
If I have a post route which an argument:
post /my_post_route/:my_param1, MyController.my_post_handler
How would get the post params ...
New
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: )
Hello all, this is ...
New
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
New
In templates/appointment/index.html.eex:
<%= for appointment <- @appointments do %>
<tr>
<td><%= appoi...
New
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
Hey,
Just curious what are the main benefits of Elixir compared to Clojure?
When is Elixir more useful than Clojure and vice versa?
Th...
New
Update:
How to use the Blogs & Podcasts section
You can post links to your blog posts or podcasts either in one of the Official Blog...
New
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #hex










