Questions & Help>Questions

Consuming Kafka messages from an offset with KafkaEx

#kafka
IanVaughan

IanVaughan

Consuming Kafka messages from an offset with KafkaEx

I have a working Elixir app consuming messages from my Kafka broker, it was really easy thanks to the excellent KafkaEx package.

The question I have revolves around reading from different offsets on a partition.

My GenConsumer is implemented like here KafkaEx — kafka_ex v1.0.1 always reads the latest unread from the last committed offset, which is fine.

  1. While its reading the messages, it would be nice to know the offset position it is at, for performance monitoring, but the state is nil as its up to us to set.
  2. If I wanted to read from a given offset, I guess the KafkaEx.GenConsumer is no good for this? I would have to implement my own GenServer /GenStage with KafkaEx.fetch/KafkaEx.stream in?
View thread on forum
#kafka
4 1250 0

Where Next?

View thread on forum
kafka
Home Questions & Help>Questions
#kafka
4 1250 0
Last post

Popular in Questions Top

hariharasudhan94
Questions & Help>Questions

How to use return statement with if condition in elixir?
lets say i have a sample like a = 20; b = 10; if (a > b) do {:ok, "a"} end if (a < b) do {:ok, b} end if (a == b) do {:ok, "equa...
/phoenix
10 20249 6
New
9mm
Questions & Help>Questions

Elixir way to conditionally update a map
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
#maps
55 20289 29
New
Darmani72
Questions & Help>Questions

Params in the URL and body -- how does Phoenix handle them together?
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
/phoenix#params
17 27034 14
New
Tee
Questions & Help>Questions

Enumerables - how does Enum.reduce work with maps
can someone please explain to me how Enum.reduce works with maps
#erlang
18 14884 13
New
JeremM34
Questions & Help>Questions

How can I check Phoenix version?
Hello, how can I check the Phoenix version ? Thanks !
/phoenix
35 28311 8
New
pmjoe
Questions & Help>Questions

What do you think of Gleam compared to Elixir?
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
#programminguages#gleam
24 17623 10
New
vrod
Questions & Help>Questions

Starship (cross-shell prompt) error - (starship::utils): Executing command "elixir" timed out
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
#starship
8 17418 3
New
sergio_101
Questions & Help>Questions

Pattern matching against a string
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
#pattern-matching
28 27371 14
New
openscript
Questions & Help>Questions

How to set up the Elixir SDK in Intellij IDEA with the intellij-elixir plugin?
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
17 23228 9
New
jononomo
Questions & Help>Questions

Websocket connection works on localhost, but get 403 error when deployed via docker
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
/phoenix#channels
8 26986 12
New

Other popular topics Top

WestKeys
Questions & Help>Questions

Mint vs Finch vs Gun vs Tesla vs HTTPoison etc
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
#http_client
252 22547 30
New
Darmani72
Questions & Help>Questions

Params in the URL and body -- how does Phoenix handle them together?
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
/phoenix#params
17 27034 14
New
baxterw3b
Questions & Help>Questions

Anonymous functions with multiple body
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
19 21782 4
New
pmjoe
Questions & Help>Questions

What do you think of Gleam compared to Elixir?
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
#programminguages#gleam
24 17623 10
New
nobody
Questions & Help>Questions

How to get the server ip address?
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
/phoenix
19 49418 16
New
KronicDeth
News>Announcing

IntelliJ Elixir - Elixir plugin for JetBrain's IntelliJ Platform
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
#library#intellij
289 36352 110
New
Brian
Questions & Help>Questions

Import a module from a file into IEX
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
#iex
35 32683 16
New
AstonJ
Chat & Discussions>Wikis

Phoenix LiveView Info
We’ve put together this wiki for Phoenix LiveView - please feel free to add any info you feel is worth including. What is Phoenix LiveV...
#wiki
220 26048 73
New
JakeBecker
Chat & Discussions>Discussions

ElixirLS - the Elixir Language Server
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
#elixir-ls
1144 54120 245
New
sergio
Chat & Discussions>Discussions

What's a great modern drag and drop javascript library you recommend?
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
/phoenix#javascript
66 19965 25
New
Back to top

Questions & Help>Questions

Questions Questions ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

Honeybadger

Real-time error tracking, performance insights, and observability for devs.

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Honeybadger

Real-time error tracking, performance insights, and observability for devs.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

WyewWorks

Enabling companies to succeed by building software people love.

Software Mansion

The team behind Membrane, Popcorn, LiveDebugger. Available for hire.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Popular Tags

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Honeybadger

Real-time error tracking, performance insights, and observability for devs.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

WyewWorks

Enabling companies to succeed by building software people love.

Software Mansion

The team behind Membrane, Popcorn, LiveDebugger. Available for hire.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement