sashman

sashman

Inserting data into Elasticsearch using GenStage, with elasticsearch_elixir_bulk_processor

A while ago I was using Java for an event-based Elasticsearch project. Elastic provides a Java-based library called Bulk Processor, this allows for configurable bulk inserts into Elasticsearch. Unfortunately, the configuration can only be set during instance creation. This inspired me to write an Elixir based equivalent processor which allows the configuration to be changed dynamically during run time. And of course, a good chance to utilize GenStages.

I hope this comes in useful to someone dealing with event sourcing and streams of data, I think this would be a good fit as a sink of data for Broadway based consumers.

Example usage:

alias ElasticsearchElixirBulkProcessor.Items.Index
[
  %Index{index: "test_index", source: %{"field" => "value1"}},
  %Index{index: "test_index", source: %{"field" => "value2"}},
  %Index{index: "test_index", source: %{"field" => "value3"}}
]
|> ElasticsearchElixirBulkProcessor.send_requests()

Feedback and comments are most welcome!

https://github.com/sashman/elasticsearch_elixir_bulk_processor

Where Next?

Popular in Announcing Top

OvermindDL1
I created a new library (rather I pulled out a couple files from my big project), it manages an operating system PID file for the BEAM. ...
New
kelvinst
Hey everyone! Well, we made this lib a while ago and now we decided to finally go out and public with it! It’s a tool for creating and m...
New
michalmuskala
Hello everybody. I have just released Jason - a new JSON library. You might be wondering, why do we need a new library? The primary foc...
New
versilov
Could not wait for the missing Elixir ML libraries to appear, so, I wrote one myself, taking https://github.com/sdwolfz/exlearn as a foun...
New
tmbb
I’ve decided to create this topic to discuss optimization possibilities for something like Phoenix LiveView. I’ve created this topic unde...
144 10187 141
New
RobertDober
Earmark is a pure-Elixir Markdown converter. It is intended to be used as a library (just call Earmark.as_html), but can also be used as...
239 12560 134
New
mplatts
With HEEX released we decided to start a components library using Tailwind CSS - check it out here: Petal Components. We also have a boi...
New
type1fool
WebAuthnLiveComponent WebAuthnComponents See this post about renaming the package. Passwordless authentication for Phoenix LiveView app...
New
woutdp
Hi! I wanted to introduce my latest project LiveSvelte. It allows you to render Svelte inside LiveView with end-to-end reactivity. It’s ...
New
Flo0807
Hello everyone! I am excited to share our heart project Backpex with you. After building several Phoenix applications, we realized that...
New

Other popular topics Top

RisingFromAshes
I’ve read in another post that it may be possible with a router helper - but I couldn’t find an appropriate one, and tbh, I’m still just ...
New
hariharasudhan94
Lets say I have map like this fetching from my database %{"_id" => #BSON.ObjectId<58eb1a7a9ad169198c3dXXXX>, "email" => ...
New
hariharasudhan94
lets say i have a sample like a = 20; b = 10; if (a > b) do {:ok, "a"} end if (a < b) do {:ok, b} end if (a == b) do {:ok, "equa...
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1144 53690 245
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New

We're in Beta

About us Mission Statement