Missing libraries and tools

swelham · October 30, 2016, 9:28pm

Introduction

After some discussion with a few other members we thought it might be an idea to start a thread where we can post about libraries or tools that we feel are missing. We could also use the thread to organise ourselves into working on any that we might want to get involved with.

We’ll try to keep this first post updated with any projects that get started as a result of the discussions here, with a name of those participating and a link to the github repo

Looking for contributers

OpenPantry
A management/user choice system for small-to-medium size food banks and pantry programs
Authors: bglusman komizutama
Repo: https://github.com/openpantry/open_pantry
Demo/preview: Guest mode on free heroku app

Google API Client - existing libraries are listed in the Google docs here and it would be awesome to see an Elixir library listed on that page!

Let’s Encrypt Client - a client library to support the Let’s Encrypt API (docs and existing clients). This standard is called ACME, so it should probably be named with ACME somewhere in the name to allow for easy finding.

Microsoft Graph API Client - the docs are here and the existing samples and SDK’s are here.

In Progress

OraLixir
An ecto adapter for Oracle
Authors: c-bik
Forum thread: Ecto Oracle Adapter
Repo: https://github.com/c-bik/OraLixir

commerce_billing
A payment processing library for Elixir
Authors: joshnuss
Repo: https://github.com/joshnuss/commerce_billing

cashier
Cashier is an Elixir library that aims to be an easy to use payment gateway, whilst offering the fault tolerance and scalability benefits of being built on top of Erlang/OTP
Authors: swelham
Repo: https://github.com/swelham/cashier

Social Parser
A small library for parsing out common social elements such as hashtags, mentions and urls
Authors: swelham
Repo: https://github.com/swelham/social_parser

blex
A Developer focused Blogging platform
Authors: Harrisonl
Repo: https://github.com/Harrisonl/Blex

grpc-elixir
grpc-elixir is the Elixir implementation of gRPC, which is a high performance, open-source universal RPC framework used and made by Google.
Authors: tony612
Repo: https://github.com/tony612/grpc-elixir

vips
Elixir wrapper for VIPS command line
Authors: schultzer
Repo: https://github.com/Schultzer/vips

Getting involved

Please feel free to update this wiki (or reply to this thread) with your ongoing projects or new ideas.

AstonJ · October 30, 2016, 9:47pm

Nice one Stuart

I second your original suggestion of a paypal library, or perhaps getting involved to add it to GitHub - joshnuss/commerce_billing: A payment processing library for Elixir which is based on ActiveMerchant (they have Paypal on the roadmap so definitely worth pinging Josh to find out the status on that).

swelham · October 30, 2016, 10:10pm

Cool I hadn’t seen that repo before, I will drop him a new issue on there and go from there.

Eiji · October 31, 2016, 9:44am

I’m going to release differ package (I’m working on first version to commit). In my API I will allow compare, patch and revert for all Elixir data types that is supported by Ecto.
After that I’m going to create a library to manage plugins (with allow to disable and enable them).
I’m going to create some libraries for my last planned work: CMS, but I’m not an expert of Elixir and I can’t give you a planned release dates.

swelham · October 31, 2016, 9:52am

That sounds great @Eiji, do you have any links to your work yet? Also we’re not worried about release dates here so no preasure!

This thread is looking to document what libs the community feel are missing and also to be a place where we can post projects and find other people that might like to contribute.

Eiji · October 31, 2016, 10:20am

No, I’m waiting for next release of Elixir to remove some code for example: &List.myers_difference/2. At now I change a Map to sorted List (Keyword) and use my implementation of myers_difference (it’s not algorithm for a Map, but there is no public API for this - there is non-public API in ExUnit). I also need to implement protocol for date and time Elixir data types. Also in next release (?) there is a diff method that returns a number in specified unit.

For plugins I have idea, but I’m working on other projects now.

I think also for PostgreSQL extension for Ecto, because some of features was not implemented, because does not exists in other databases (for example easy query associations in different prefix), but here I will need a help. No problem if someone know how to do it. Feel free to implement this.

Of course complete CMS is at end and I don’t start any job with it (I have already planned it).
I need to have 100+ hours per day

swelham · October 31, 2016, 10:51am

If only we could add a few extra hours into the day!

If you have some time would you be happy to put your plans into a readme that we could link to from this thread? We might be able to get some others involved and start to build up those dev hours

Eiji · October 31, 2016, 11:05am

I don’t need a help for differ.
Don’t know what about plugins library.

If some want help it will be awesome if I got help with PostgreSQL extension for Ecto:

Create and drop schema (easy, but we could provide it as a function instead of execute SQL in every migration)
Helper macro for references to tables inside another schema (Ecto prefix), see: this thread
Helper Repo method to preload association in another schema (Ecto prefix), see this thread.
Add dependency to ecto_enum library.
More PostgreSQL-only features in just simple functions and/or macros.

This helper methods/macros will be really helpful in multi PostgreSQL schema setup.

OvermindDL1 · October 31, 2016, 3:02pm

Something else useful that I think someone was creating but I cannot find anything about yet on hex.pm was a Let’s Encrypt library to auto-refresh the SSL cert.

AstonJ · October 31, 2016, 3:08pm

If anyone’s interested in this and needs inspiration, this is what I use for our cert on the forum: How To Secure HAProxy with Let's Encrypt on CentOS 7 | DigitalOcean (with a Ruby script that checks/renews via a daily cron job)

OvermindDL1 · October 31, 2016, 3:30pm

I’m far more lazy, I just use certbot on my debian server to update my nginx front-end certs. ^.^

swelham · October 31, 2016, 3:42pm

@OvermindDL1 great idea, I have updated the wiki!

AstonJ · October 31, 2016, 3:46pm

That does indeed look easier

AstonJ · October 31, 2016, 4:02pm

If anyone is interested in performant ways of working with images, VIPS bindings for Elixir could be a neat little project (take a look at the Ruby one for inspiration).

swelham · October 31, 2016, 4:17pm

Another great idea, now added to wiki

Eiji · October 31, 2016, 4:27pm

I got an interesting idea!
How about catch messages, save them to database and files?
The table structure:

id Integer
env Atom
type Atom
source Atom
file_info Map
caller_info Map
stacktrace List of Maps
log_fiile_path String
inserted_at - datae and time
updated_at - datae and time

File default structure:

/project_directory
  /logs
    /env
      /type
        timeline.log
        /descriptions
          id.log

Where:

id is a database id.
env is our environment
type is one of: [:error, :warning, :debug] - of course in production configuration developer could set only :error reporting
source is an info about from what it comes, for example: :otp message or :ecto message or :logger call
file_info for example: %{module: "MyProject.MyModule", function: %{name: "my_function", callback: "&my_function/2"}, file_path: "/lib/my_project/my_module.ex", line: "53"}
caller_info for example: %{module: "IO", function: %{name: "puts", callback: "&puts/1", args: "OK"} or: %{module: "Kernel", function: "raise", data: %{message: "Argument first is not an integer", type: "ArgumentError"}}
stacktrace for example: [%{from: "stdlib", file: "lists.erl", line: 1338:, callback: "lists.foreach/2"}]
log_fiile_path is the path generated using config (for the directory structure)
inserted_at is date of first time found message (error, warning, message) - sometimes useful if we know what update applied in specified day and time
updated_at is last date of found message

Example format of timeline.log:

At date_and_time found #id type. Details stored in: “logs/env/type/descriptions/23.log”.

Example format of id.log:

At: “2016-01-01 00:00:00.0000 UTC”
First reported at: “2015-12-12 00:00:00.0000 UTC”
In environment: “production”
Found #23 error
That was called from module: “MyProject.MyModule”, function “&my_function/2”, in file: “/lib/my_project/my_module.ex” at line: 23.
Caller is: module “IO” and function: “&puts/1” with data:
[
“OK”
]
Stacktrace:
(stdlib) lists.erl:1338: :lists.foreach/2
…

Logs descriptions could be optionally to catch them from logs based program like fail2ban.

With following structures we can easily format that message for example for special markup.

This file can be send by mail to service account or automatically add (if inserted) a new bug to for example Github (by public HTTP API) from production server, so it’s possible to get fast help directly after user gets “500 Internal Server Error” message without fill a bug report.

Phone call:
John: “Hey, your site is excellent, but one page is not working. Can you see what’s wrong?”
Support: “Please reload page - it’s already solved :-).”
John: “Wow! Thank you very much!”

Is is possible to catch all Elixir messages (runtime errors, deprecation warnings, IO operations and more) like I described?
What do you think about it?

OvermindDL1 · October 31, 2016, 4:38pm

You can create your own error logger to plug into Elixir Log, just like the file_logger plugin, but log to a DB. Of course you’d want to throttle it, cache recent ones and prevent duplicates and such or you can get overwhelmed with messages if something bad suddenly happens.

A library for that would be cool though…

swelham · October 31, 2016, 4:41pm

Anyone want to come up with a name for that idea?

OvermindDL1 · October 31, 2016, 4:45pm

Well the others follow logger_file_backend and logger_json_file_backend and logger_papertrail_backend and logger_lager_backend and many others that follow logger_*_backend, so probably logger_ecto_backend or so? Or something more generic like logger_cache_callback_backend if it handles the deduplication and caching and such for you, just passed to a callback to then do whatever you want, maybe even to another log channel so something like logger_ecto_backend or logger_file_backend or so could log them as normal? Although maybe that would not be a backend but a filter or so? ^.^

swelham · October 31, 2016, 5:06pm

I agree something more generic like a logger_filter which can handle the caching and deduplication and then have plugable backends such as logger_ecto_backend, logger_file_backend, etc…

This would allow it to be flexible and easily extendable.