Contributing to aws-beam/aws-elixir

Given @benwilson512’s comment:

The most productive thing to do for the ecosystem may be to take a look at the packages that depend on ex_aws and see if they’ll accept a pull request to instead use https://github.com/aws-beam/aws-elixir which appears to be actively maintained, and uses code generation to keep up with changes coming from AWS.

7 Likes

That’s a great idea @mbuhot!

I have started looking at aws-beam and what it would be necessary for me personally to migrate to it and I have found some action items:

  1. Migrate from poiston to jason
  2. Make the HTTP client pluggable (similar to what is done in ExAWS)

I was planning to send a PR for those but I decided to post them there as folks are looking for ways to contribute. It is probably worth checking out with the maintainer too if they welcome these changes before moving forward.

8 Likes

I’ve started depot_s3 using ex_aws. If anyone wants to give it a shot at moving to aws-elixir that would be welcome as well:

2 Likes

What about adding documentation?

Based on what’s in the GitHub repo for it, there’s no docs other than 1 example.

Having to install the library and generate your own docs seems like a weird burden to put onto the developer.

Maybe a PR could be made to auto-generate the docs in GitHub Actions?

2 Likes

I would suggest one more thing - instead of generating Elixir and Erlang libraries, generate only one and make the other one wrapper around another.

As most Elixir projects, the docs are on hexdocs: https://hexdocs.pm/aws - The README could make it clearer though, probably a good opportunity for a first PR. Go for it! :+1:

Given it is auto-generated, I don’t see an issue with generating both. Especially because Erlang developers do not want to bring Elixir as a dependency and Elixir developers don’t want to lose the nicely generated ExDoc documentation we have today.

4 Likes

Thanks, I didn’t think to check there because it wasn’t linked.

Although it’s kind of non-intuitive because even after clicking your link, I thought the docs were literally all of what we see on that first hex page you linked. In other words, just a list of services with no other information. There’s no visual hint that those AWS services are links on Hex.

Funny enough the only service I wanted to use is S3, but there’s no docs for S3 too.

I know I fall back to comparing things to Python a lot but this is the documentation for the Python AWS library https://boto3.amazonaws.com/v1/documentation/api/latest/index.html. There’s hundreds of real code examples which I think is fantastic, along with a few practical getting started guides.

This Elixir library’s docs are an API spec. Unfortunately there’s 0 examples on how to use anything. This makes it very hard to figure out how to use it unless you’re already a seasoned veteran.

It’s one of those things where I would contribute if I knew how, but I can’t figure it out because there’s no existing documentation. That and I suppose in this library’s case if the API specs are auto-generated, then maybe there’s no way to modify the docs since the code is generated and is expected to be read-only by humans?

1 Like

There is one issue - both projects register application named aws, which mean that when you will (for some reason) application that uses Erlang dependency that is using Erlang version and you are using Elixir version then there will be name conflict. With making Elixir wrapper over Erlang library it could be mitigated by using different name for the wrapper (as it would cause no problems there).

It seems you have a list of improvements and a reference, so that’s a great opportunity to act on it! I have sent a PR with some changes too. AWS.S3 does exist, the docs may be outdated.

Yeah, this definitely should not be the case. I think it is an orthogonal discussion to wrapping though.

2 Likes

I am not 100% sure as there is some “handwritten” code in the both repos (utilities and credentials). I think that making it unified (either as a additional Erlang repo or by incorporating everything into single repo with wrapper) would help with reducing area of possible bugs.

I wouldn’t even know how to begin. Wouldn’t you need write access to publish the library to update the docs on Hex? I’m also confused how auto-generated code can emit out of date documentation. Aren’t the docs based on the source code or is there a separate step? I’ve never published a package on Hex.

As for adding code examples to improve the docs. It’s a catch-22. I can’t add examples until I learn to use the library but I can’t learn to use the library because there’s no practical examples.

There are 3k commits on boto3 (90+ contributors) vs. ~200 on aws-elixir. Work on boto3 was started in 2014. That was right about the time elixir hit 1.0, but 6 years into python3. What you see for boto takes work and in open source this also means time. We can sure get to a similar state, but one can’t expect it to be there from “day 1”. Therefore I can just support @josevalim. If you see deficits try to open issues or even PRs (and I’ve done so myself a lot of times).

2 Likes

You don’t need write access to the hex package. Opening PRs on github can be done by anyone. If you’re not sure how to interact with the module generation you can ask the maintainers. Also ex_doc does have means of including plain markdown files (the guides you see in popular libraries like ecto or phoenix) without them needing to be part of a module.

Given it’s a wrapper around the official aws apis they’ll likely work quite similar as other languages libraries / the http api aws has docs for. It might not help for this library, but often you can look also at the test-suite how things work.

Edit:
Even if you don’t know things you can ask in issues. E.g. see this one I improved the docs on after having to ask first about what things actually are: https://github.com/elixir-ecto/ecto_sql/issues/257

Right but for the docs to be updated would require someone pushing a new package version to Hex?

Also the top of every source code file says not to touch it because it’s machine generated, example:

# WARNING: DO NOT EDIT, AUTO-GENERATED CODE!
# See https://github.com/aws-beam/aws-codegen for more details.

If the docs are generated by code comments but the files that contain the code are read-only, I can’t think of a way to open a PR on that. The changes would get lost.

This is what I normally do, but yeah aws-elixir has no tests. At least none that show how to use the functions that this library exposes.

In the end this leaves us with no to limited docs, no test suite and no way to update the docs due to things being auto-generated. How would we proceed to improve this given the above constraints? It sounds like an expert would need to crawl through the popular AWS services and write custom guides / docs and then publish them to somewhere outside of the code repo?

But even that isn’t a great experience because it could hard to find and could very easily become outdated vs what’s available in the library.

And as always: try things out. It’s not like the parameters are awefully hard to guess and as mentioned the functions align with common aws api, so “what the functions do” should be not to hard to find out. It’s more a matter of what inputs/outputs are. If you don’t want to rent an actual S3 bucket you can look at the tests for depot_s3 I linked previously. It has integration with a local minio instance.

Edit:
And I want to add: Things need not be correct/complete/perfect on the first shot at it. A good maintainer will help you work through improvements to make them mergeable.

1 Like

Thanks.

Hopefully in time we can pick a semi-blessed solution that ends up having code, tests and documentation in 1 spot so us regular devs can get up and going without having to solve a mystery case to see how to use the library :D.

With that semi-blessed solution maybe other libs that use AWS could start using it too. Right now Waffle is still using exAws as a dependency.

This is starting to feel a bit like a different topic. However it’s worth noting that ExAws’s approach to documentation has always been “use the official AWS docs”. ExAws is a relatively thin wrapper around the underlying API calls. Mandatory parameters are function args, options are options.

1 Like

I think even just filing these improvements as open issues would be a helpful start!

2 Likes

PR Sent :crossed_fingers:

3 Likes

Hello Elixir community.

In my prototyping I am using ExAWS and it works really good for my current needs: S3, STS, SQS but now I see lack for other APIs like autoscaling. I tried ExAWS “custom request” but did not succeed.

So just today I check this library aws-elixir and pool the latest release 0.6.0

When I check AWS.AutoScaling.xyz API I see much less functions present comparing to what is defined on Go side, namely: https://github.com/aws/aws-sdk-go/blob/master/models/apis/autoscaling/2011-01-01/api-2.json

When I check head of git master branch I see it is in par with Go version, namely: https://github.com/aws-beam/aws-elixir/blob/master/lib/aws/auto_scaling.ex

However when I pull version {:aws, "~> 0.6.0"} in mix.exs it just brings limited subset of services and APIs (you can check it here: https://github.com/aws-beam/aws-elixir/releases).

Could you please help to figure out what is the issue? Why the latest version of 0.6.0 aws-elixir released with such limited (and it seems outdated) set of services/APIs?

Thank you.

PS:

I just updated my dependency to:

{:aws, git: "https://github.com/aws-beam/aws-elixir.git"}

and now I have e.g. AWS.AutoScaling.describe_account_limits which is not in v. 0.6 So the question is why this lib is not released with full set of API?