darnahsan
Broadway kafka producer not getting all partition assignment from Upstash Kafka
My understanding is that this is an upstash issue. I have a topic with 3 partitions but I only get assignment for 2 partitions. Below is my broadway code. Is there anything in the config that could be causing this missing partition assignment ?
defmodule Maverick.Broadway.Upstash do
@moduledoc """
The Upstash Kafka context.
"""
use Broadway
alias Maverick.Whatsapp.Message, as: WhatsappMessage
def upstash_kafka_port(), do: Application.fetch_env!(:maverick, :upstash_kafka_port)
def upstash_kafka_endpoint(), do: Application.fetch_env!(:maverick, :upstash_kafka_endpoint)
def upstash_kafka_hosts(), do: [{upstash_kafka_endpoint(), upstash_kafka_port()}]
def upstash_kafka_authentication(),
do:
{Application.fetch_env!(
:maverick,
:upstash_kafka_sasl_mechanism
), Application.fetch_env!(:maverick, :upstash_kafka_username),
Application.fetch_env!(:maverick, :upstash_kafka_password)}
def upstash_kafka_group_id(), do: "localhost-00"
def upstash_kafka_topic(), do: "localhost"
def producer_concurrency(), do: 2
def processors_concurrency(), do: 10
def start_link(_opts) do
Broadway.start_link(__MODULE__,
name: __MODULE__,
producer: [
module:
{BroadwayKafka.Producer,
[
hosts: upstash_kafka_hosts(),
group_id: upstash_kafka_group_id(),
topics: [upstash_kafka_topic()],
offset_reset_policy: :earliest,
client_id_prefix: "localhost-maverick",
client_config: [
sasl: upstash_kafka_authentication(),
ssl: [
# from CAStore package
cacertfile: CAStore.file_path(),
verify_type: :verify_peer,
customize_hostname_check: [
match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
]
]
]
]},
concurrency: producer_concurrency()
],
processors: [
default: [
concurrency: processors_concurrency()
]
]
)
end
First Post!
darnahsan
a new topic with 4 partitions got all partition assignments for now seem odd partition count is the problem, could the broadway config in any way be playing a part in this ?
0
Last Post!
mgibowski
Sorry, I was wrong in my previous message:
You should be fine with your original configuration, I don’t know what is the issue.
Regarding your other question - Erlang processes are lightweight processes and are not limited by the number of cores.
0
Popular in Questions
Hello, how can I check the Phoenix version ?
Thanks !
New
Hi everyone!
I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
Credo is smart enough to check for (something like) this:
assert length(the_list) == 0
with this response:
Checking if an enum is empt...
New
Good day to you all.
I have been struggling to get a query involving like and ilike to work.
Can anyone assist me on this, please?
pro...
New
Hello all!
I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New
Other popular topics
Update:
How to use the Blogs & Podcasts section
You can post links to your blog posts or podcasts either in one of the Official Blog...
New
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine)
This is a plugin that adds support for Elixir to JetBrains IntelliJ...
New
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New
Posting this to see if we can make things easier for people to get into Neovim. If you use Neovim and have a favourite distro please let ...
New
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #hex












