ericlathrop

I’ve recently gotten back into reading RSS feeds only to find a bunch of sites don’t generate them any more. I use RSS-bridge to fill some of the gaps, but there’s still some sites it didn’t support. I made my own self-hosted RSS feed generator in Elixir called Living in Syndication. I push builds to Docker Hub so you can quickly get it up and running. I’d love PR’s that contribute new RSS feed generators.

Here’s a screenshot:

screenshot

Just copy the RSS urls from the buttons into your reader app, and you’re ready to go.

From the readme:

Living in Syndication

Generate RSS feeds for sites that don’t have them.

Many sites don’t provide RSS feeds these days. This project is a self-hosted app that will scrape the HTML of various sites and create RSS feeds for you. This app is similar to RSS-Bridge, but supports different sites. My favorite RSS Reader is Miniflux.

Supported Sties

Installation

Builds of Living in Syndication are published to Docker Hub.

You can run a build with Docker like this:

$ docker run -it -p 4001:80 ericlathrop/living_in_syndication:v0.1.0

If you want to make your own build:

Clone this repo.
Install Elixir
Install Rust
Run mix deps.get inside the repo.
Run MIX_ENV=prod mix release inside the repo.
The build will be located in the _build/prod folder inside the repo. You can copy it to a server and run rel/living_in_syndication/bin/living_in_syndication start to start the server.

See the Dockerfile for more complete build instructions.

Help Wanted: Adding More Sites

I’d love PRs for adding more supported sites. Please look at the eBird generator as an example.

11 comments

#library #rss #living-in-syndication

10 3362 11

2024-02-10 10:54:33 UTC

Showing Posts 1 to 10

Show Best Posts
Show All (oldest first)
Show All (newest first)

Eiji

Do you have a list of sites you want to support or you just want to support as much as possible?

Post #1

ericlathrop

Two specific sites I’d like to see supported are Twitter and Instagram. There’s already support for RSS generators having parameters (for example asking for twitter usernames). I would like to support as many sites as possible. If it gets unwieldy to maintain, then some generators could be broken out into some plugin system.

Post #2

Eiji

First time used Twitter API. Since you did not declared (like Ecto.Schema or just documentation) I was not always sure if I pass a correct data or to a correct field. Also I wrote everything in format I saw on Twitter (see item.title). Let me know if that works as you expected. If needed I can update it anytime you want.

Code is obviously working, but I think that you may want to adjust some query parameters. For this I made it really customizable by query, params and comment_params.

iex> Feeds.TwitterAPIv2.get_channel(%{bearer_token: "MY_SECRET_BEARER_TOKEN", query: "from:josevalim"})    
%Rss.Channel{
  atom_link: "https://api.twitter.com/2/tweets/search/recent?expansions=attachments.media_keys%2Cauthor_id&media.fields=url&query=from%3Ajosevalim&tweet.fields=created_at%2Clang%2Creferenced_tweets%2Csource&user.fields=description%2Cprofile_image_url%2Curl",
  category: nil,
  cloud: nil,
  copyright: nil,
  description: "Chief Adoption Officer at @dashbit. Creator of @elixirlang. Tweets once in a while. Rarely livestreams at https://t.co/lfDi8qz7jD.",
  docs: "https://www.rssboard.org/rss-specification",
  generator: "Name of My Project",
  image: "https://pbs.twimg.com/profile_images/667973757181431808/IdQgSAH__normal.jpg",
  items: [
    %Rss.Item{
      author: "José Valim",
      category: nil,
      comments: [],
      description: "<p>RT @bostonvaulter: I'm pleased to announce DepViz, a new tool to visualize elixir compilation dependencies that I released today at  #CodeB…</p>",
      enclosure: nil,
      guid: "1325712715285934081",
      link: "https://twitter.com/josevalim/status/1325712715285934081",
      pubDate: #DateTime<2020-11-09 08:11:56.000-05:00 EST America/New_York>,
      source: "Twitter Web App",
      title: "José Valim @josevalim · 9 Nov"
    },
    %Rss.Item{
      author: "José Valim",
      category: nil, 
      comments: [],
      description: "<p>RT @kipcole9: Published an introduction to Language Tags for Elixir programmers interested in I18n at https://t.co/904hbWxWN2 #myelixirstat…</p>",
      enclosure: nil,
      guid: "1325712696151564288",
      link: "https://twitter.com/josevalim/status/1325712696151564288",
      pubDate: #DateTime<2020-11-09 08:11:51.000-05:00 EST America/New_York>,
      source: "Twitter Web App",
      title: "José Valim @josevalim · 9 Nov"
    },
    %Rss.Item{
      author: "José Valim",
      category: nil,
      comments: [],
      description: "<p>RT @kry10_limited: New blog post: Overview of the Kry10 Secure Platform\n(Mostly about the OS)\n\nhttps://t.co/B9A7D6ucuV\n\n#seL4 #Erlang #Elix…</p>",
      enclosure: nil,
      guid: "1325712563661910016",
      link: "https://twitter.com/josevalim/status/1325712563661910016",
      pubDate: #DateTime<2020-11-09 08:11:20.000-05:00 EST America/New_York>,
      source: "Twitter Web App",
      title: "José Valim @josevalim · 9 Nov"
    },
    %Rss.Item{
      author: "José Valim",
      category: nil,
      comments: [],
      description: "<p>RT @remote: Mental health matters most.\n\nWhen you work remotely, remember:\n\n❌ Stop working on breaks. Work is not a break.\n❌ Don't overcomm…</p>",
      enclosure: nil,
      guid: "1324731327640461313",
      link: "https://twitter.com/josevalim/status/1324731327640461313",
      pubDate: #DateTime<2020-11-06 15:12:15.000-05:00 EST America/New_York>,
      source: "Twitter Web App",
      title: "José Valim @josevalim · 6 Nov"
    }
  ],
  language: nil,
  lastBuildDate: #DateTime<2020-11-09 08:11:56.000-05:00 EST America/New_York>,
  link: "https://t.co/OFJNxHxlKf",
  managingEditor: nil,
  pubDate: #DateTime<2020-11-09 08:11:56.000-05:00 EST America/New_York>,
  rating: nil,
  skipDays: nil,
  skipHours: nil,
  textInput: nil,
  title: "José Valim (@josevalim) / Twitter",
  ttl: nil,
  webMaster: nil
}

Note: For this to work you need a Twitter developer account, but since everyone can create it you can simply ask about Bearer authorization token.

Post #3

ericlathrop

I’m glad you posted here, my GitLab notification settings were messed up and I never got an email about the PR. I’ll continue the discussion on the PR.

Post #4

ericlathrop

I pushed v0.1.2 last night with Instagram support!

Post #5

ericlathrop

I pushed v0.1.3 with Twitter support!

Post #6

ericlathrop

I just pushed v0.1.4 with support for showing the releases for a GitHub repository. Makes it easy to know when a new version comes out, and what the changes are!

Post #7

ericlathrop

I pushed v0.2.0 with support for Twitter polls, plus several bugfixes.

[0.2.0] - 2020-12-12

Fixed

Don’t crash if embedded tweet cannot be found
Don’t crash if Instagram returns no posts
Fix Instagram user mix up when searched user isn’t first in results
Fixed crash when Twitter sends suggestions

Added

Support Twitter poll cards

Post #8

ericlathrop

I pushed v0.3.0 with some mitigations for Instagram rate limiting, Twitter user-agent blocking, and a variety of bugfixes.

[0.3.0] - 2020-12-23

Fixed

Fix URLs generated by AbcbirdsOrg feed
Fix crash in ApnewsCom feed
Fix crash in AudubonOrg feed

Changed

Instagram now returns a “429 too many requests” error when we’ve been rate limited
Instagram is now cached for 24 hours, to reduce rate limiting
Instagram now only requires one HTTP request per feed instead of two
Lie to Twitter & Instagram about user agent, to prevent blocking of scraping
GitHubComReleases now just redirects to unadvertised feed

Post #9