Spur - (Very) Simple Activity streams for Ecto

I’ve been working on a hobby app as a way to learn Elixir and Phoenix (one of approaching Rails horde, I imagine), and the need arose for an activity feed describing actions of various users. I searched around a bit and it didn’t seem there was anything currently existing, so I rolled my own: Spur. It’s a partial extraction of a solution that’s very simple but it’s been good enough for my needs so far. I learned a lot writing it, so hopefully it can at least serve as a learning tool for others.

The functionality is loosely based on this pretty common Rails gem, and although it’s also very simple it relies on two features of Rails that Phoenix/Ecto very consciously omits: callbacks and association polymorphism. The first issue does not pose much of a challenge since Multi provides a very nice and explicit (if a bit bulky to my Rails trained eyes) API for grouping related operations. The core of this lib is extracting the main actions that activities usually need to track: inserts, updates, and deletes:

%Context.Resource{}
|> Context.Resource.changeset(params)
|> Spur.update

Polymorphism is another story. Initially I tried to follow the recommendations in the Ecto docs, and that worked alright in my own app, but they proved to be pretty abstraction proof because they require the Activity model to be aware of context specific modules that would be tracked. I briefly considered trying to solve that through configuration, but that would involve an amount of Elixir meta-programming that’s still a bit daunting for me at the moment, and more importantly, I had the thought that activity tracking actually shouldn’t rely on associations, because by its very nature describes an ephemeral relationship. For exactly this reason I usually use some sort of revision tracking in conjunction with it in order to support things like undo, etc. So instead I kept the schema as simple as possible, basically just specifying some required and recommended fields, along with the usual catch all meta field. Along with wrapping the activity and the object change in a transaction, the main sugar it sprinkles is the ability to pass a function that it executes and save the result automatically on the activity, which is useful for things like storing changes.

%Context.Resource{}
|> Context.Resource.changeset(params)
|> Spur.update(fn resource, params -> add_resource_data_to_meta(params) end)

Anyway, it is a good example of how getting acquainted with the Elixir way has made me more thoughtful about how I approach domain modeling, instead of relying on Rails “cheats” like association polymorphism. Of course, I’d greatly appreciate any tips on how to improve my Elixir idioms and suggestions for directions to take this. One idea I’ve been playing with, inspired by revisionair, which I’m using for revisions, is abstracting the underlying storage logic, but I’m not sure how useful that would be since most of the value here is pretty Ecto specific.

9 Likes

Release 0.2 includes support for associating an activity with an audience.

2 Likes

Release 0.4 includes some minor optimizations and (first feature request) the ability to use a custom name for the activities table.

1 Like

Awesome work. Do you know about ex_audit?
It looks like both of library work by “hijacking” standard Repo and use Ecto.Multi to append an operation for inserting the change.

I generally don’t fee very comfortable using this approach(wrapping the original db operation with this additional trace record db operation into 1 DB transaction). I was wondering what if instead of wrapping them in a Ecto.Multi. Could we only instead send msg to a dedicated GenServer and this genserver role is to do diffing and persist them in db? We can also do batching as well.

The pro of it is we are completing separating the original db operation and the trace record db operation(In fact we can even swap storage engine)

1 Like

I remember checking out ex_audit when I was looking for a library to track changes. At the time I decided to go with revisionair (as I mentioned in my post) but it looks like the former has matured a lot in the meantime. In truth the scope of Spur is much more limited in comparison with either of those libraries. It does not track changes at all, although it is designed to work in conjunction with code that does, so that you can, for example, see data changes that are associated with standard crud activities. But the core purpose of Spur is simply to create a stream that is itself agnostic about the current or past state of any data. It’s base use case is to arbitrarily create ‘activity’ records at any point in the flow of an application, even those that do not modify other data (page views, contacts, etc). More info about the domain is in the w3 spec.

You are certainly right that the API for ex_audit looks similar to the (optional) ‘callback’ support in Spur, though. On my part, there wasn’t a lot of thought put into that decision, it simply resulted from the abstraction of my app logic, which used Multi in the standard way. But your suggestion is very interesting, especially since decoupling from Ecto is one of the stretch goals for this project. As you point out, there’s no inherent reason for tracking operations to use the same transaction or even data store. Coming from Ruby, I was not very familiar with the concept behind GenServer with I started work on Spur, but it does sound like a promising, more Elixir-y direction to go into. As soon as I have some time, I’ll definitely play around with it and see what I come up with. If you feel like taking a look at the code and giving me any more specific suggestions, I’d greatly appreciate it!

1 Like

With “hijacking is bad” I totally disagree, the Repo is the best place to control this from architectural point of view. But, exploring Ecto I found that it doesn’t use the same API to work with nested assocs(like if you delete a record with dependent has_many it wont’t call delete_all/2 from passed repo, it calls directly the adapter) and moreover doesn’t pass opts recursively to nested insert/update/delete calls. After opening an issue I received the suggestion to implement it in prepare_changes(called in the same transaction) which I personally doesn’t accept and it is a lack of Ecto guarantees and design.

Can you elaborate on the “architectural point of view” you’re defending here? I certainly think that in most cases you will want to wrap the “parent” change in the same transaction as the activity tracking (or other changes tracking), but it doesn’t seem necessary to enforce that, at least in the case of the problem Spur is trying to solve. I think what @bruteforcecat is suggesting might still involve a multi operation but would allow the user to decide how to implement it, either via an Ecto operation or PubSub update or whatever else.

I don’t know what your background is, but coming from Rails one thing I’ve noticed is that the ‘Elixir way’ (if there is such a thing) does not invest so heavily in the principle of “convention over configuration”, specifically in those cases where there are significant complexity/clarity costs for some “magical” convenience, favoring instead to be a bit more explicit about what’s going on. Personally I am very influenced by the Rails approach to make things really easy for the “80%” of use cases, and I have no patience for the approach in libraries like React where the user is forced to jump through innumerable hoops to implement the simplest and most common features. But I also appreciate the idea that things can be easy and empowering without sweeping every last detail under the rug (to occasionally become infested with bugs).

The influence of Ruby/Rails is obvious in the current design of Spur (the original gem inspiration is a Rails gem), but a large part of the reason I’ve become excited about Elixir is what it contributes to these high level architectural questions. so I’d certainly be interested in your thoughts on that in terms of how to improve this library.