So far I hadn’t been able to figure out how to get or set topic retention values for kafka topics via brod, using versions up to 3.16.3. I can set it when creating a topic by passing name-value pairs when creating topic.
Is this even possible via Brod, or one must install Java and kafka client libraries and invoke kafka-configs.sh or kafka_topics.sh to do it? It seems like a huge limitation. For example, I can’t drop and re-create a topic with same retention because I don’t have all of the metadata.
So it does indeed return {ok, #{field_name() => field_value()}} but yeah, sorry, I don’t know another way to get that info. You may look into the kafka_protocol application (kpro modules and friends) but you may have to encode and decode the request yourself.
@version 0
@resource_type_topic 2 # I could not find the values in neither code nor documentation (Just INT8 in https://kafka.apache.org/protocol.html#The_Messages_DescribeConfigs), so I got into online Java IDE and did System.out.println(org.apache.kafka.common.resource.ResourceType.TOPIC.ordinal()); "2" came out.
IO.inspect(:kpro_req_lib.describe_configs(@version, [%{resource_type: @resource_type_topic}, %{resource_name: @test_topic}], %{validate_only: true}))
I solved it using kafka_protocol library, although it was quite a brain tease to figure out how to call Erlang from Elixir, by wrapping :kpro_req_lib.describe_configs, :kpro_brokers.with_connection, :brod_utils.request_sync and :kpro_req_lib.alter_configs APIs. It was very helpful to go through the source code for the unit tests included with kafka_protocol library.