Recommendations to parse Atom and RSS feeds?

Just to note, fast_rss seems to use rustler_precompiled which means in many cases you won’t need the rust compiler

Last time I ran into this I didn’t like any of the options so I just wrote the parsers by hand using Saxy, which worked great.

I am not a fan of the NIF approach for stuff like this because the BEAM’s guarantees are actually very useful for parsing arbitrary internet content in a multitenant system. Throwing them away feels unwise. At some point I will probably write a new RSS library but it’s not a top priority at the moment.

You can properly write a faster parser in Elixir anyway, NIF comes with huge cost especially if you are moving data back and forth. Good rule of thumb, most BIFs are very simple, I wonder why

RSS/Atom feeds are small, simple XML files, Floki with its pure Elixir parser works very well on them. I tracked 10s of thousands feeds and have not seen any problem.

Because they are meant to be timesliced! But yeah, in general if you’re writing code in a beamlang it’s because you want to take the multitenant side of the tradeoffs rather than the batch processing side.

Does the HTML parser support CDATA? I would think not, right?

It’s very common to embed content in RSS feeds using CDATA (e.g. HTML content). Saxy handles it fine.

Oh, I totally overlooked that. Good to know, thank you.

I didn’t know about Saxy and Floki, thank you @garrison and @derek-zhou. I’ll look at both options.

Real men use :xmerl!

More on topic, I agree with building your own bespoke thing, especially with a more or less nailed standard like RSS and Atom. They’re not as big.

My impression was that real men would be blowing up their atom tables, but it seems like maybe the xmerl sax parser does use strings? I’ll keep that in mind.

It is actually somewhat annoying to parse RSS in practice. The spec is extended in a bunch of places and there is a lot of weird stuff out there in the real world. I do think there’s room for a good RSS package, I just didn’t find one that I liked at the time. I would like to write one but I currently have a storage engine-shaped backlog that I’m trying to squash before the new year

I am a recovering perfectionist (long process). To me you either settle for one good ready-made complete software package, use a library and patch over its imperfections, or roll your own and enrich it on demand.

I have found maintaining other people’s stuff a thankless work that very rarely materializes time and energy savings over long-enough usage so I end up doing either option one or option three. (And option one is very rarely useful if you want to integrate stuff in your own program so you’d use f.ex. Port and such, which becomes a hassle after an hour of work.)

And yes if memory serves :xmerl does not use atoms but it has been years and I might not remember well. Same as for JSON, people overdo the parsing conveniences, I often parse JSON as recursive string-keyed maps and just work with that; Elixir’s excellent pattern-matching does multiple things at once there and is unbeaten by any language in those areas except maybe OCaml / Haskell / Rust (if you can stomach the increased number of coding lines).