Advice on analytics for web traffic

Just to confirm what you mean said another way, it’s current license does not prohibit self hosting?

Edit: Ah I see now. Wow this is amazing!

Yep, could however change in the future just like it did with fathom.

Starting to look at Plug.Conn.req_headers and realizing that everything I care about is already in that.
My thought at this point is to make a plug that just async’s a request into postgres. Maybe I use mnesia as a buffer and create some worker that migrates the records to postgres.

But be aware that while everything might be there doesn’t necessarily mean you’re allowed to store it indefinitely like that. E.g. IP addesses are personal data by the gdpr. Given a timestamp and the IP there are means to linking it back to a telephone line and therefore likely a person, even if those means are likely not available to yourself.

All I’m interested are in the basics about the agent, the sec-fetch-user, the referer, and the request_path.
Maybe I use the IP to look up some geo and then I drop the IP on the floor. That’s less than what plausible is doing. I think I will be ok with GDPR. Still thanks for the heads up. I don’t really even care about bounce rate.

The only thing this does is put the onus on me to make it looks nice, welp there goes my free time.

I was gonna see what I can do to incorporate it into my live dashboard

There is also Matomo which allows self-hosting. Also Fathom Lite is still available for self-hosting.

Thanks for mentioning Plausible @LostKobrakai. I’ve been working on it for about a year now.

I’ve thought a lot about offering a self-hosted solution. To be clear, the license allows you to self-host and I’m not stopping anyone. When I say that plausible does not offer a self-hosted solution, it means that I have not taken up the responsibility for making it easy to install and upgrade. If you want to grab the code and run it, by all means go ahead.

I’m wary of offering a self-hosted solution because:
a) The product is still early stage and it would take away from my time focusing on new features
b) It’s likely that the infrastructure will evolve to a point where self-hosting is just not realistic

For instance, I’m currently moving from Postgres to Clickhouse as the database so that I can accomodate sites with >10m pageviews per month. In the future, it might be necessary to add Kafka in the data pipeline to better handle big traffic spikes etc.

There’s a tension between making the product easy to self-host vs making it cost-efficient for large customers and a free tier. For example, Matomo uses a standard MySQL database and it’s very easy to self-host. On the other hand, their hosted service is crazy expensive. With 10m monthly pageviews you’re looking at €849 per month, and that only includes 6 months data retention.

If they built a more streamlined data pipeline, it would make their product much harder to self-host but it would bring the cost of the hosted service down.

Hope that clarifies things a bit. I’m not ruling out a self-hosted version in the future, potentially with different database and ingestion adapters to simplify the infra requirements for self-hosting. However, at the moment I’m focusing on the core business and especially scaling up to larger customers.

I saw a tutorial on Youtube with Plangora:

First of all - great job. I need to check this out for one of my side-projects. Looks awesome.

Out of interest, have you thought about using TimescaleDB? It is just PSQL extension so the migration should be simpler. In addition there are works on adding columnar store to PostgreSQL as well, so this may be interesting for you.