Ecto.Adapters.DynamoDB: An Ecto Adapter for DynamoDB

Looking for some guidance on:

“** (Mix) Expected Ecto.Adapters.DynamoDB to implement Ecto.Adapter.Storage in order to create storage for xxx”

Is there any work being done to implement Adapter.Storage for the DynamoDB adapter? I can see using Postgres or some other storage implementation for Unit Testing, but for integration/acceptance level testing I’d like to be able to hit a (local) DynamoDB instance.

Thanks in advance for pointers to where this is being worked on or why it isn’t

-Eric

The Storage implementation is responsible for creating/dropping database. Correct me if I’m wrong, but I think that doesn’t really make sense in the context of DynamoDB.

In that case, everything should work correctly, if you don’t call the ecto.create and ecto.drop tasks.

Ah, ok, so in the DynamoDB case, even running local, dropping and creating those don’t make much sense.

Running with the ecto adapter for dynamodb throws an error about the ‘storage’ issue and won’t run, which seems odd. Wonder if there’s a way to just override it and say “sure, we handle that” and just make the drop and create no-ops?

I’m also wondering, for what I’m trying to do, if it isn’t just as easy to use the ExAws project directly. I may attack that angle a bit (although the adapter is really nice).

Or I’ll get it in my head to go try to add some fake adapters for storage to the code…pull down master…and get:

Generated ecto_adapters_dynamodb app
** (CompileError) test/test_helper.exs:47: module :eqc_gen is not loaded and could not be found
expanding macro: EQC.using/1
test/test_helper.exs:47: TestGenerators (module)
(elixir) expanding macro: Kernel.use/1
test/test_helper.exs:47: TestGenerators (module)

Thank you for your comments, Eric and Michal. We have yet to visit any
Storage callbacks for the adapter. Eric, if you like, please raise an issue
in our github repository with more details about the error you are
experiencing and how to reproduce it. We’ll be happy to take a look.

Sincerely,
Gilad

@eric - Looks like it’s complaining on missing quick check libraries - Which makes sense, since you won’t have that installed. @alhambra, I thought it compiled by default without having those available? Any compile time flag? Or will we need to tweak to remove?

And I agree with Michael - Creating storage just doesn’t make sense with DynamoDB - The storage layer already ‘exists’ and cannot be deleted - All we allow is the creation of schema/tables via the adapter. In your test code, the easy solution would be to avoid calling the storage functions at all - It will all work fine without them. If that’s not possible, then it seems you’d like a way for the storage functions to work without causing an error.

Michael, do you have any good thoughts on what a ‘correct’ implementation might be? ‘success’ with warning?

Last note - Alhambra1 is correct in that defects for these in the gitlab repo is a good way of ensuring that any requests are tracked and not lost.

@ericheikkila thanks for the heads up on the EQC build error. I wouldn’t have expected that to be a problem for building the adapter code, only for running unit tests, but even in that case it would be best if we could have it automatically skip the QuickCheck stuff and still run any other tests that are available. (And actually, we could probably use something like eqc_suite to also allow for running QuickCheck tests to a limited extent even without a commercial license, but that’d take some doing and will probably be on the back burner for a while.)

Anyway, I’ll try and take a look at adding a quick and dirty fix for the EQC errors sometime in the next couple days. In the mean time you are welcome to just locally rip out any modules that mention EQC, and that should get you up and running for now.

Sorry for the long delay since my last reply! Been a really busy couple of weeks but I finally got around to fixing the adapter test modules so that they skip the EQC code if you don’t have it installed. Let me know if you run into any more issues

It’s been a while since there’s been big news for the adapter, but we’ve just released an alpha version that supports integration with Ecto 3 - ecto_adapters_dynamodb | Hex

Community use and feedback would be much appreciated!

Can anyone further share their experiences, good or bad, with Ecto and DynamoDB (DDB) since the last post? Experiences w/ DDB itself are also welcome.

I would have used Postgres (PG), but was forced hands on a (unfinished) project which stores user records in DDB. The only options I see are:
A) sync DDB to PG via DDB-Streams (likely sending to SQS and then consuming the queue). Or,
B) forego PG for user authentication and use DDB directly, using mix phx.gen.auth-generated code (as is; or with modifications) along with the Ecto DDB adapter.
C) ?

More generally, curious if anyone had to deal with such, or similar, case.

In a related thread, the only hands-on experience was shared by @frumos.