Proposal: ExAws 2.0

Your Libraries & OS Mentoring Libraries
1

Hey folks,

I’m the author and maintainer of ExAws, a library for interacting with AWS APIs in a flexible and convenient fashion. This library plays an important role in people’s applications and, thanks to the many contributions and advice I have received, I think it embodies some good API library design principles.

Unfortunately, I’ve been doing at best a mediocre job as its maintainer. This is for three reasons:

  1. I don’t have enough time these days
  2. AWS is gigantic.
  3. ExAws is gigantic

There isn’t a lot I can do about #1 or #2 at the moment, but I have an idea about #3 that may help:

ExAws Architecture Recap

To review briefly, ExAws is built around 4 core operation structs, that represent the broad categories of API design that AWS uses: JSON, Query, RestQuery, and S3 (yes of course S3 is its own beast, :heart: AWS). Each of these operations implements the ExAws.Operation protocol, which handles the common request logic.

ExAws has resisted using code generation to build its APIs because since we have these structs, you can basically use any service AWS has so long as you supply the config, and build your operation. This could be a bit easier, but code generated functions don’t really bring anything to the table.

Finally, there are individual service modules that exist as Elixir friendly wrappers around each API action and return operation structs.

The Problem

It’s these service modules that are the issue. We’ve got quite a lot of them at this point, and they’re all pretty big, some are huge. Many were contributed (thank you!) but because of that I don’t know much of anything about the AWS service, and after I time I can’t remember anything about the code specific to that service.

If someone makes a PR to update that service, I am ill-equipped to make a good review of it.

v2.0 Proposal

I want to make an ex-aws organization on github. Within that, there would be the following libraries:

  • ex-aws/core: This would house the operation structs, the operation protocol, common request logic, signing logic, and so forth. I would be the primary maintainer of this library, and work to improve the documentation on how to use the operation structs, and how to contribute. On hex this would become :ex_aws_core

  • ex-aws/s3: This would be an example service library that I would also maintain, since it is essential to what we do at Cargosense, and I’m the original author. It would depend on :ex_aws_core, and only contain the specific S3 service module.

  • ex-aws/$SERVICE: There are going to be services that I won’t personally maintain. I simply don’t have the time or knowledge to do so. I will give those who want to maintain $SERVICE full commit access to the repo, so that they can maintain it and improve it. I plan on this being the vast majority of service modules.

  • $YOUR_ACCOUNT/$SERVICE: It’s my goal for the services contained in the ex-aws org to be high quality, largely complete, and maintained by active people. There will be services where that isn’t possible yet, but that’s OK. I want the documentation in core to be easy enough to follow that anyone can create their own little wrappers around core if all they need is a specific helper function or two.

Outside of the library split up, this shouldn’t cause breaking changes for people using ExAws. They’ll need to add some more deps but once they do so, it should be fine.

ExAws is widely used within the community, so I wanted to post this here so that people are aware of the coming changes, and so that I can get some feedback on how people feel about it.

Thoughts?

30 Likes
2

I really like that you are planning to split the different libraries since an application requiring S3 doesn’t always need everything else ExAws currently provides :heart:

1 Like
3

I also think it’s a very good way to go because of what @wmnnd mentioned. I use ExAws in a few projects, only for the S3. Being able to use each service as different modules makes it more lighter and eventually easier to use and create/modify own “service adapters”, just like https://github.com/thoughtbot/bamboo for example.

Looking forward to see it coming! GJ!

4

This is a cool plan. Thanks for keeping us informed.

5

Does this really introduce a re-architecture or does it just split out the architecture into more clearly defined lines? That is to say, to use :ex_aws_s3, will you still build an operation using a function from ExAws.S3 and then execute it via ExAws.Core.request/2? Or will ExAws.S3 provide its own execution function in the new model?

In the first case, it would make more sense to require the user to include :ex_aws_core as a direct dependency since they will be directly dependent on it in code.

In the second case, it makes sense to have the user only depend on :ex_aws_s3. It would concern me, though, as an extensive user of ExAWS. We supplement ExAws with our own request/2 which produces logs with timing and HTTP information. We also rely on substituting the request/2 call for unit testing purposes.

6

Using ExAws should not change at all. That is you’ll still do:

ExAws.S3.list_objects("my_bucket") |> ExAws.request

To support this in your mix.exs file you’d want to do:

[
  {:ex_aws, "~> 2.0"},
  {:ex_aws_s3, "~> 2.0"}
]

:ex_aws_s3 will have :ex_aws as a dependency but, since everyone calls ExAws.request directly, it is proper to include it as a direct dep.

7

Sounds like a good idea.

Would this architecture could make it possible to provide complex wrappers around aws services?

For instance, you could use s3 as a backend to make a simple persistent term storage system with a serialiser/deserialiser layer on top.

1 Like
8

Think of this like an organizational or managerial change, not an architectural one. The architecture of ExAws already enables the kind of thing you’re talking about and, on the whole, I’m very happy with the architecture of ExAws.

What I (and I’m sure many of my users) are less than happy about is the challenge of managing such a large project with such diverse components. In a sense, I’m trying to get the management structure of ExAws to mirror its architecture: a central bit of functionality managed by me, and a bunch of satellite modules managed by not me.

1 Like
9

This sounds like a great plan Ben. Amazon themselves did something similar with v3 of the official Ruby client recently.

10

Thanks for the feedback everyone, the ExAws v2.0 changes are in progress, and I hope to release soon!

At the core of this change is the hope that some of you out there who find a specific AWS service central to what you do can step up and take ownership of that service within ExAws. If you think you are such person, please let me know.

Although general contributions from anyone at any skill level are welcome, and the ExAws code is rather uncomplicated, service maintainers should have enough familiarity with Elixir to promote good coding practices.

Maintainers are NOT required to expand on or otherwise singlehandedly complete the service. The main duty is just being familiar with the AWS reference API docs, and handling issues / PRs. Improved test coverage would be nice if you’re inclined.

I will always be around to answer questions.

Available Services

Some services have clear original contributors, and those people get right of first refusal on the service projects. These projects are available for any who are interested, so please message me privately if that’s you!

  • [ ] ex_aws_dynamo
  • [ ] ex_aws_firehose
  • [ ] ex_aws_kinesis
  • [ ] ex_aws_lambda
  • [ ] ex_aws_sns

More may become available as I hear back from those I’ve pinged for some of the other services. I’ll keep this thread updated.

Thank you everyone for your support!

6 Likes
11

ExAws v2.0.0 has been released. See the list of services here: https://github.com/ex-aws?utf8=✓&q=service&type=&language=

The most important thing to note is that the main ExAws package now has endpoint information for every service on AWS. This makes it much easier to use ExAws for services where there isn’t a helper package yet.

Many folks have stepped forward to help and I really appreciate that. I’m still in the process of getting everyone the right roles. There’s a lot of projects here so it’s taking more time that I would have thought, apologies for the delay!

Thanks for your support everyone!

9 Likes