sashman

sashman

Inserting data into Elasticsearch using GenStage, with elasticsearch_elixir_bulk_processor

A while ago I was using Java for an event-based Elasticsearch project. Elastic provides a Java-based library called Bulk Processor, this allows for configurable bulk inserts into Elasticsearch. Unfortunately, the configuration can only be set during instance creation. This inspired me to write an Elixir based equivalent processor which allows the configuration to be changed dynamically during run time. And of course, a good chance to utilize GenStages.

I hope this comes in useful to someone dealing with event sourcing and streams of data, I think this would be a good fit as a sink of data for Broadway based consumers.

Example usage:

alias ElasticsearchElixirBulkProcessor.Items.Index
[
  %Index{index: "test_index", source: %{"field" => "value1"}},
  %Index{index: "test_index", source: %{"field" => "value2"}},
  %Index{index: "test_index", source: %{"field" => "value3"}}
]
|> ElasticsearchElixirBulkProcessor.send_requests()

Feedback and comments are most welcome!

https://github.com/sashman/elasticsearch_elixir_bulk_processor

Where Next?

Popular in Announcing Top

aesmail
Hello guys, I have finally made it. I created an admin interface for a framework. It’s been on my todo list for years and with the curre...
New
woutdp
Hi! I wanted to introduce my latest project LiveSvelte. It allows you to render Svelte inside LiveView with end-to-end reactivity. It’s ...
New
mindok
What is ContEx? A pure Elixir server-side data plotting/charting library outputting SVG. It has nice barcharts in particular and works g...
New
Azolo
Hey everyone, I just released WebSockex which is a Elixir WebSocket client. WebSockex strives to work as a OTP special process, be RFC6...
New
josevalim
Hello everyone, We have just released NimbleCSV which is a small and fast CSV parsing library for Elixir. It allows developers to define...
New
achempion
Hi, I would like to tell about my initiative to further maintain and develop Waffle project which is the fork of Arc library. The progre...
New
michalmuskala
Hello everybody. I have just released Jason - a new JSON library. You might be wondering, why do we need a new library? The primary foc...
New
trisolaran
Hi! :waving_hand: I would like to present LiveSelect, a little library that I wrote to easily add a dynamic selection input to your LV f...
198 10858 107
New
New
pkrawat1
Hey guyz We at @aviabird are working on a payment library in elixir/phoenix. We are targeting March 2018 to add 56 Gateways to it. Have...
New

Other popular topics Top

marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 30877 112
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1144 53690 245
New
vegabook
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
klo
Got a question about when to concat vs. prepending items to list then reversing to achieve appending. So i know lists boil down to [1 | ...
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 47930 226
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New

We're in Beta

About us Mission Statement