makeitrein

makeitrein

Hey all, just started picking up Elixir last week and am writing a scraper as a learning project.

Baby step #1 is extracting the number from a URL on the target web page… here’s what I’ve written:

# url is in "https://xxyyzz.com/xxyyzz.383254/" format... goal is to extract 383254
  def get_id_from_url(url), do: Regex.run(~r"\d+\/", url) |> Enum.at(0) |> Integer.parse |> elem(0)

This seems a bit clunky of a function to me for a simple integer extraction… is there a better way of going about this?

Showing Posts 1 to 10

hassan

hassan

How about

iex(10)> "https://xxyyzz.com/xxyyzz.383254/" |> String.replace(~r/[^\d]/, "")
"383254"
iex(11)>
dimitarvp

dimitarvp

defmodule Test do
  def match_string("https://xxyyzz.com/xxyyzz." <> suffix) do
    case Integer.parse(suffix) do
      {number, "/"} when is_integer(number) ->
        IO.puts "suffix is #{number}"

      _ ->
        IO.puts "cannot parse suffix: #{suffix}"
    end
  end
end

Test it in iex:

iex> Test.match_string "https://xxyyzz.com/xxyyzz.383254/"
suffix is 383254
:ok
iex> Test.match_string "https://xxyyzz.com/xxyyzz.383254/!"
cannot parse suffix: 383254/!
:ok

You can abuse Elixir’s allowed syntax of pattern matching on a string suffix (you cannot pattern-match strings in the middle of the bigger string though, have that in mind). Not sure if I am not taking your example too literally but if I understood you correctly, that’s how I would approach the problem.

NobbZ

NobbZ

You can use hd instead of Enum.at. that saves you some characters.

Also you can use String.to_integer instead of piping through Integer.parse and elem.

But that’s the way to go.

makeitrein

makeitrein OP

Many thx for suggestions! Realized my URL example is a bit lacking… the xxyyzz is representative of any numbers and letters of unknown length, so just stripping non-digits won’t work, nor will the pattern matching (I think, awol from computer).

I guess the challenge is to find a series of numbers at the end of a string that starts with a period and ends with a slash… String.to_integer looks like it might be what I need, good enough for government work.

dimitarvp

dimitarvp

Nope! We have enough borked gov’t systems. Let’s do better.

If you give us a few examples and/or explain the whole URL schemata then I can help you better.

makeitrein

makeitrein OP

Hah, fair game ^^

https://foster.com/death-pancake.1468/ === 1468

https://hkd33.net/mr-rogers101.690153/ === 690153

https://space-force911.gov/sauce-master.13257777/=== 13257777

Here’s a selection of random URLs… all begin with https, have a base domain, followed by the username of the person who submitted the domain followed by a period followed by the id of the post followed by a trailing slash…

Only the id of the post is relevant so we can ignore the base domain, username, period, and ending slash…

dimitarvp

dimitarvp

So all URLs always end in a number plus a forward slash? No exceptions?

makeitrein

makeitrein OP

Correctamundo

dimitarvp

dimitarvp

(EDIT 1: Account for invalid values.)
(EDIT 2: Trim empty strings when splitting.)
(EDIT 3: Included explanations.)

defmodule Test do
  def extract_id(url) when is_binary(url) do
    url
    |> String.split(~w(. /), parts: 1000, trim: true)
    |> List.last
    |> parse_id
    |> fetch_id
  end

  defp parse_id(nil), do: :error
  defp parse_id(x) when is_binary(x), do: Integer.parse(x)

  defp fetch_id({number, ""}) when is_integer(number), do: number
  defp fetch_id(:error), do: :error
end

Test it:

iex> urls = ["https://foster.com/death-pancake.1468/", "https://hkd33.net/mr-rogers101.690153/", "whatever_dude", "https://space-force911.gov/sauce-master.13257777/"]

iex> urls |> Enum.map(&Test.extract_id/1)
[1468, 690153, :error, 13257777]

Breaking it down:

  • ~w(. /) equals [".", "/"] (so String.split is called with multiple separators).

  • parts: 1000 is used to prevent denial-of-service attacks, in case somebody manages to smuggle huge strings to your code. trim: true removes empty strings from the result. Check String.split docs.

  • "https://foster.com/death-pancake.1468/" |> String.split(~w(. /), parts: 1000, trim: true) yields this:

["https:", "foster", "com", "death-pancake", "1468"]

…so we are calling List.last on it to give us the desirable piece of data.

  • Our internal function parse_id has to also handle invalid data:
    • If String.split returns [], List.last would return nil.
    • If String.split returns ["single_invalid_url"], List.last would return "single_invalid_url".

Both cases would make our internal function parse_id to return :error. (Integer.parse will return :error if you supply it a string that does NOT start with an integer.)

  • The fetch_id internal function uses function heads instead of if or case to extract successful integer parsing and return it, or react to an :error return value and just pass it down the line to your consumer code.

  • One caveat: notice that fetch_id matches on {number, ""} when is_integer(number) which means the function will be called only if a full integer string is passed, namely “123” or “456” will succeed but “123xyz” will not. If you expect URLs like “https://whatever.man/1234abcd”, this code won’t work.

makeitrein

makeitrein OP

Dang, that’s some good looking code! Copy and pasting in 3, 2, 1…

Where Next? Top

Trending in Questions Top

water
I’m aware of 2 such projects, are there others? git: GitHub - team-alembic/realworld: A fullstack Phoenix LiveView application with back...
New
katta
I having some trouble figuring out if I have set myself too strict of standards for my production server. Currently I can handle 75% of r...
New
peck
Hi everyone. My team and I have been working on a fairly modest app based around video streaming and chat, but we’ve landed a customer t...
New
bradley
I really like the adapter patterns that ecto, nebulex, waffle, etc. use and would love find something similar for a key management servic...
New
unaware8150
Hello folks! So at work, we are seeing some situations where we have to define some “fixed” strings that are used across the codebase in...
New
Alvinkariuki
How Can I Optimise Compile Time Dependencies I have been building an elixir application for about 2 years now. Many modules and files ha...
New
leoschweizer
Hi, we run different LLM models across a cluster, with each node loading a configurable subset. Model-specific queues let nodes consume ...
New

Other Trending Topics Top

KristerV
Hey. Is there anyone here who creates agents in their apps? Not talking about using agents, but creating them. I’m finding it pretty diff...
New
mudasobwa
I fully migrated to my own harness from Anthropic/Gemini and I think it’s time to share it. Welcome DSH, the DeepSeek Harness, fully writ...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
juhalehtonen
There has been a thread to discuss the Stack Overflow Developer Survey on this forum every year since 2018, so here’s yet another one for...
New
budgie
I love Elixir. It’s one of 2 programming languages I’ve ever fallen in love with. But I don’t use it anymore. Serverless was the promis...
New
ausimian
Emily is an Elixir library that runs Nx computations on Apple’s MLX. Install it as the default Nx backend and Nx, defn, Axon, Nx.Serving,...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews