Maintainer Wanted: ExAws

Hey folks!

The short version is: I’m looking for a maintainer to take over the operation and maintenance of the ExAws project. CargoSense hasn’t done anything particularly interesting with AWS services in a long time, and I personally lack the time maintain these projects at this point. I have been unable to keep up with developments within AWS’s auth system and this has lead to fragmented support of various features, and PRs I’m unable to meaningfully review. This isn’t the state I want these libraries to be in, but I simply am unable to do anything about it.

To be clear, I am not stepping away from open source. I have a sizable roadmap planned for Absinthe, and there are other, smaller, projects that I’m looking to refresh. However I am trying to focus my efforts, both for my own sanity as well as the health of the projects I’ve created.

The actual maintenance required is reasonably low, but it can also be very critical to people’s operations. My preference would be to find a maintainer who is part of a company utilizing AWS, but if you’re interested please feel free to send me a DM.

58 Likes

Do you have any recommendations for alternative libraries if no one ends up taking over yours?

I wonder how this will effect libraries like waffle, since they depend on ex_aws for file uploads to S3.

ExAws is, as far as I know, the only actively maintained (for some definition of active ;)) AWS library. Someone will pick it up, I’m sure of it :slight_smile:

7 Likes

Thanks for your work on creating and maintaining ExAws! I’ve used it in multiple projects and it’s great :heart:

I wonder if you’d be interested in creating a posting on:

(which happens to be written in Elixir using Phoenix #liveview)

11 Likes

I’m really pleased to see you reaching out to the community to ask for help. This is a huge project that a lot of companies rely on, I’m sure you’ll get somebody stepping up.

I’ll be linking this to a couple people.

19 Likes

Just to point out my disappointment with choice of mascot. Due to name I think that Opossum would be much better choice (in my language it is called just “opos”).

Hiya Ben,

Since I’m already a committer on ExAws.SNS and a heavy user of several other service modules, I would definitely be interested in seeing what my team and I can to support ExAws as well. We’re heavily committed to open source and always looking for ways to contribute back, and this might be a good fit for us.

That includes one or more of us taking on the role of Maintainer if we can swing it, but it would help to have some idea of the number/frequency of pull requests, issues, support needs, etc. so that I can make a case to my supervisor(s) that it would be a worthwhile thing for us to devote some time to.

26 Likes

Hi ben,

I’ve been a long time user of ExAws, and so have a few other people at my company. We’d be potentially interested in helping out with this as well (our company is be more than willing to help with supporting open source projects)

Feel free to contact me if you still need help and want to discuss more

11 Likes

Hi Ben,

I am also happy to assist where possible. Although I work for AWS and love Elixir, I am not a developer anymore (Solutions Architect) - so my productivity would be low. However, hopefully between all of us in the community, we can keep this amazing library healthy.

I have not looked into the code, but most of the other 3rd party libraries generate code from one one of the official SDK, and that allows for easy upgrades as the API moves forward. I am guessing that is not the approach taken with ExAws? For example: https://github.com/inhabitedtype/ocaml-aws

3 Likes

What’s the status of this? Did anyone step up to maintain this project?

Update

Hey everyone. I received a lot of supportive comments from people, and for that I’m very grateful. Unfortunately, all of the comments took some version of the form of “I’d be happy to contribute some, but I don’t have the time to run the project”. To be clear, I think this is a perfectly sensible answer, after all it is basically my answer here. However I’m really not looking for contributors, I’m looking to be done with this project. Also, the more I’ve looked at the project, the more I think it has certain fatal flaws in its approach, and the right course of action isn’t to keep pushing forward in this direction but rather find a new one.

Thus, my current inclination is to simply deprecate ExAws entirely, and advocate that someone write a replacement.

A replacement could utilize the patterns in ExAws that I think are good, while also aiming to auto-generate functionality as people tend to do in other languages with AWS libraries. The bits that are worth keeping from an approach perspective is:

  • Pluggable dependencies (minimize hard coded deps on any specific json codec / http library / etc).
  • Operations as data structures. Instead of Aws.S3.list_objects actually doing the request, have it return a data structure that you pass to a function that actually executes it. This allows much much easier interop with other libraries.

Areas where a new library could improve:

  • Utilize auto-generated functions from the spec to get 100% API coverage immediately
  • Testing strategy. I never came up with a good way to do tests in ExAws and consequently there basically aren’t any. This is bad for all of the obvious reasons, and it’s also a key element of why development stagnated. I had to manually audit or test every change since there wasn’t any regression testing, and as I ran out of time that meant I either had to blindly merge PRs or not merge PRs at all. I chose the latter generally, and this killed the project.
8 Likes

There is Erlang library that does that already. Maybe it could be improved to support pluggable HTTP clients and stuff like that.

EDIT:

It seems that it also has Elixir implementation. Additionally there is erlcloud which also seems pretty active. You could point to these libraries as a potential replacements.

1 Like

Do you have any recommendations on potential replacements that are already developed or partially developed?

Right now one of the most popular file upload libraries for Phoenix (Waffle) depends on ExAws for saving files to S3. I wonder if the author of that library is on the forums. It’s a pretty big hit to lose that library since file uploads (especially to S3) is so popular.

I am still working on article how to manage uploads to S3 without ExAws or Waffle

2 Likes

That would be fantastic, looking forward to it.

Do you plan to cover things like changeset validations on file type (perhaps through mime type or analyzing the first couple of bytes) and file size?

If you want any suggestions on potential features to talk about let me know. I remember opening an issue on Waffle Ecto back in January about supporting md5 tagged file names for cache busting but unfortunately nothing came of it. Waffle Ecto only supports using timestamps in a query string which isn’t as robust.

Also it would be interesting if you included how to swap S3 out for the local file system or a dummy for local development and tests.

Is it possible in your opinion (of course it is, but without too much hoops and headaches) to make calls directly to the AWS API without ExAWS? Is this something you would recommend?

While implementing S3 uploads, I was able to forgo using Waffle and while doing so wondered if I could do without ExAws?

I lot of times with Elixir I find that I can do things without libraries (as opposed to JS which seems to need NPM install for the most minute things). I like this approach.

There are several aspects about interacting with AWS that are difficult to do without a library:

  • Request signing. This is the big one. ExAws has several hundred lines of code dedicated to formatting and signing the parameters according to the AWS signing guidelines. Without this, you can’t make any requests to any AWS service. Copying and pasting these lines into each application or library that wants to talk to AWS is super error prone, since any bug fixes or updates to that code won’t be distributed.

  • Credential management. AWS credentials need to be retrieved via several different mechanisms, and people depend on that working the same way across various services and languages.

Those are really the big ones. There are other benefits to a dedicated library, like handling header boilerplate, or various convenience functions around clunky APIs, but doing AWS request signing and credential management without the help of a library is likely to result in bugs.

That said, perhaps those bits could be extracted (or maybe already exist) as their own library.

3 Likes

I assume you tried some approaches which didn’t work out. Would you like to shed some light on what you tried and why it didn’t work out?