Missing libraries and tools

Cool I hadn’t seen that repo before, I will drop him a new issue on there and go from there.

2 Likes

I’m going to release differ package (I’m working on first version to commit). In my API I will allow compare, patch and revert for all Elixir data types that is supported by Ecto.
After that I’m going to create a library to manage plugins (with allow to disable and enable them).
I’m going to create some libraries for my last planned work: CMS, but I’m not an expert of Elixir and I can’t give you a planned release dates.
:smile:

2 Likes

That sounds great @Eiji, do you have any links to your work yet? Also we’re not worried about release dates here so no preasure! :smiley:

This thread is looking to document what libs the community feel are missing and also to be a place where we can post projects and find other people that might like to contribute.

3 Likes

No, I’m waiting for next release of Elixir to remove some code for example: &List.myers_difference/2. At now I change a Map to sorted List (Keyword) and use my implementation of myers_difference (it’s not algorithm for a Map, but there is no public API for this - there is non-public API in ExUnit). I also need to implement protocol for date and time Elixir data types. Also in next release (?) there is a diff method that returns a number in specified unit.

For plugins I have idea, but I’m working on other projects now.

I think also for PostgreSQL extension for Ecto, because some of features was not implemented, because does not exists in other databases (for example easy query associations in different prefix), but here I will need a help. No problem if someone know how to do it. Feel free to implement this.

Of course complete CMS is at end and I don’t start any job with it (I have already planned it).
I need to have 100+ hours per day :smile:

2 Likes

If only we could add a few extra hours into the day!

If you have some time would you be happy to put your plans into a readme that we could link to from this thread? We might be able to get some others involved and start to build up those dev hours :slight_smile:

3 Likes

I don’t need a help for differ.
Don’t know what about plugins library.

If some want help it will be awesome if I got help with PostgreSQL extension for Ecto:

  1. Create and drop schema (easy, but we could provide it as a function instead of execute SQL in every migration)
  2. Helper macro for references to tables inside another schema (Ecto prefix), see: this thread
  3. Helper Repo method to preload association in another schema (Ecto prefix), see this thread.
  4. Add dependency to ecto_enum library.
  5. More PostgreSQL-only features in just simple functions and/or macros.

This helper methods/macros will be really helpful in multi PostgreSQL schema setup.

2 Likes

Something else useful that I think someone was creating but I cannot find anything about yet on hex.pm was a Let’s Encrypt library to auto-refresh the SSL cert.

3 Likes

If anyone’s interested in this and needs inspiration, this is what I use for our cert on the forum: https://www.digitalocean.com/community/tutorials/how-to-secure-haproxy-with-let-s-encrypt-on-centos-7 (with a Ruby script that checks/renews via a daily cron job)

3 Likes

I’m far more lazy, I just use certbot on my debian server to update my nginx front-end certs. ^.^

4 Likes

@OvermindDL1 great idea, I have updated the wiki!

3 Likes

That does indeed look easier :lol:

3 Likes

If anyone is interested in performant ways of working with images, VIPS bindings for Elixir could be a neat little project (take a look at the Ruby one for inspiration).

3 Likes

Another great idea, now added to wiki :thumbsup:

3 Likes

I got an interesting idea!
How about catch messages, save them to database and files?
The table structure:

  1. id Integer
  2. env Atom
  3. type Atom
  4. source Atom
  5. file_info Map
  6. caller_info Map
  7. stacktrace List of Maps
  8. log_fiile_path String
  9. inserted_at - datae and time
  10. updated_at - datae and time

File default structure:

/project_directory
  /logs
    /env
      /type
        timeline.log
        /descriptions
          id.log

Where:

  1. id is a database id.
  2. env is our environment
  3. type is one of: [:error, :warning, :debug] - of course in production configuration developer could set only :error reporting
  4. source is an info about from what it comes, for example: :otp message or :ecto message or :logger call
  5. file_info for example: %{module: "MyProject.MyModule", function: %{name: "my_function", callback: "&my_function/2"}, file_path: "/lib/my_project/my_module.ex", line: "53"}
  6. caller_info for example: %{module: "IO", function: %{name: "puts", callback: "&puts/1", args: "OK"} or: %{module: "Kernel", function: "raise", data: %{message: "Argument first is not an integer", type: "ArgumentError"}}
  7. stacktrace for example: [%{from: "stdlib", file: "lists.erl", line: 1338:, callback: "lists.foreach/2"}]
  8. log_fiile_path is the path generated using config (for the directory structure)
  9. inserted_at is date of first time found message (error, warning, message) - sometimes useful if we know what update applied in specified day and time
  10. updated_at is last date of found message

Example format of timeline.log:

At date_and_time found #id type. Details stored in: “logs/env/type/descriptions/23.log”.

Example format of id.log:

At: “2016-01-01 00:00:00.0000 UTC”
First reported at: “2015-12-12 00:00:00.0000 UTC”
In environment: “production”
Found #23 error
That was called from module: “MyProject.MyModule”, function “&my_function/2”, in file: “/lib/my_project/my_module.ex” at line: 23.
Caller is: module “IO” and function: “&puts/1” with data:
[
“OK”
]
Stacktrace:
(stdlib) lists.erl:1338: :lists.foreach/2

Logs descriptions could be optionally to catch them from logs based program like fail2ban.

With following structures we can easily format that message for example for special markup.

This file can be send by mail to service account or automatically add (if inserted) a new bug to for example Github (by public HTTP API) from production server, so it’s possible to get fast help directly after user gets “500 Internal Server Error” message without fill a bug report.

Phone call:
John: “Hey, your site is excellent, but one page is not working. Can you see what’s wrong?”
Support: “Please reload page - it’s already solved :-).”
John: “Wow! Thank you very much!”
:smile:

Is is possible to catch all Elixir messages (runtime errors, deprecation warnings, IO operations and more) like I described?
What do you think about it?

2 Likes

You can create your own error logger to plug into Elixir Log, just like the file_logger plugin, but log to a DB. Of course you’d want to throttle it, cache recent ones and prevent duplicates and such or you can get overwhelmed with messages if something bad suddenly happens. :wink:

A library for that would be cool though…

2 Likes

Anyone want to come up with a name for that idea? :slight_smile:

2 Likes

Well the others follow logger_file_backend and logger_json_file_backend and logger_papertrail_backend and logger_lager_backend and many others that follow logger_*_backend, so probably logger_ecto_backend or so? Or something more generic like logger_cache_callback_backend if it handles the deduplication and caching and such for you, just passed to a callback to then do whatever you want, maybe even to another log channel so something like logger_ecto_backend or logger_file_backend or so could log them as normal? Although maybe that would not be a backend but a filter or so? ^.^

1 Like

I agree something more generic like a logger_filter which can handle the caching and deduplication and then have plugable backends such as logger_ecto_backend, logger_file_backend, etc…

This would allow it to be flexible and easily extendable.

2 Likes

If I good understand how Logger works it will give me only data from: Logger.* methods like: Logger.info, right?
What about catching:

raise ArgumentError, message: "Bad argument!"

and deprecated warnings?
Of course I mean to only catch and log that situations (not rescue).

1 Like

Those can be caught too. Heck you can plug in :sasl and log everything OTP related if you want (that would be noisy ^.^).

1 Like