makeitrein

makeitrein

Hey all, just started picking up Elixir last week and am writing a scraper as a learning project.

Baby step #1 is extracting the number from a URL on the target web page… here’s what I’ve written:

# url is in "https://xxyyzz.com/xxyyzz.383254/" format... goal is to extract 383254
  def get_id_from_url(url), do: Regex.run(~r"\d+\/", url) |> Enum.at(0) |> Integer.parse |> elem(0)

This seems a bit clunky of a function to me for a simple integer extraction… is there a better way of going about this?

Showing Posts 1 to 10

hassan

hassan

How about

iex(10)> "https://xxyyzz.com/xxyyzz.383254/" |> String.replace(~r/[^\d]/, "")
"383254"
iex(11)>
dimitarvp

dimitarvp

defmodule Test do
  def match_string("https://xxyyzz.com/xxyyzz." <> suffix) do
    case Integer.parse(suffix) do
      {number, "/"} when is_integer(number) ->
        IO.puts "suffix is #{number}"

      _ ->
        IO.puts "cannot parse suffix: #{suffix}"
    end
  end
end

Test it in iex:

iex> Test.match_string "https://xxyyzz.com/xxyyzz.383254/"
suffix is 383254
:ok
iex> Test.match_string "https://xxyyzz.com/xxyyzz.383254/!"
cannot parse suffix: 383254/!
:ok

You can abuse Elixir’s allowed syntax of pattern matching on a string suffix (you cannot pattern-match strings in the middle of the bigger string though, have that in mind). Not sure if I am not taking your example too literally but if I understood you correctly, that’s how I would approach the problem.

NobbZ

NobbZ

You can use hd instead of Enum.at. that saves you some characters.

Also you can use String.to_integer instead of piping through Integer.parse and elem.

But that’s the way to go.

makeitrein

makeitrein OP

Many thx for suggestions! Realized my URL example is a bit lacking… the xxyyzz is representative of any numbers and letters of unknown length, so just stripping non-digits won’t work, nor will the pattern matching (I think, awol from computer).

I guess the challenge is to find a series of numbers at the end of a string that starts with a period and ends with a slash… String.to_integer looks like it might be what I need, good enough for government work.

dimitarvp

dimitarvp

Nope! We have enough borked gov’t systems. Let’s do better.

If you give us a few examples and/or explain the whole URL schemata then I can help you better.

makeitrein

makeitrein OP

Hah, fair game ^^

https://foster.com/death-pancake.1468/ === 1468

https://hkd33.net/mr-rogers101.690153/ === 690153

https://space-force911.gov/sauce-master.13257777/=== 13257777

Here’s a selection of random URLs… all begin with https, have a base domain, followed by the username of the person who submitted the domain followed by a period followed by the id of the post followed by a trailing slash…

Only the id of the post is relevant so we can ignore the base domain, username, period, and ending slash…

dimitarvp

dimitarvp

So all URLs always end in a number plus a forward slash? No exceptions?

makeitrein

makeitrein OP

Correctamundo

dimitarvp

dimitarvp

(EDIT 1: Account for invalid values.)
(EDIT 2: Trim empty strings when splitting.)
(EDIT 3: Included explanations.)

defmodule Test do
  def extract_id(url) when is_binary(url) do
    url
    |> String.split(~w(. /), parts: 1000, trim: true)
    |> List.last
    |> parse_id
    |> fetch_id
  end

  defp parse_id(nil), do: :error
  defp parse_id(x) when is_binary(x), do: Integer.parse(x)

  defp fetch_id({number, ""}) when is_integer(number), do: number
  defp fetch_id(:error), do: :error
end

Test it:

iex> urls = ["https://foster.com/death-pancake.1468/", "https://hkd33.net/mr-rogers101.690153/", "whatever_dude", "https://space-force911.gov/sauce-master.13257777/"]

iex> urls |> Enum.map(&Test.extract_id/1)
[1468, 690153, :error, 13257777]

Breaking it down:

  • ~w(. /) equals [".", "/"] (so String.split is called with multiple separators).

  • parts: 1000 is used to prevent denial-of-service attacks, in case somebody manages to smuggle huge strings to your code. trim: true removes empty strings from the result. Check String.split docs.

  • "https://foster.com/death-pancake.1468/" |> String.split(~w(. /), parts: 1000, trim: true) yields this:

["https:", "foster", "com", "death-pancake", "1468"]

…so we are calling List.last on it to give us the desirable piece of data.

  • Our internal function parse_id has to also handle invalid data:
    • If String.split returns [], List.last would return nil.
    • If String.split returns ["single_invalid_url"], List.last would return "single_invalid_url".

Both cases would make our internal function parse_id to return :error. (Integer.parse will return :error if you supply it a string that does NOT start with an integer.)

  • The fetch_id internal function uses function heads instead of if or case to extract successful integer parsing and return it, or react to an :error return value and just pass it down the line to your consumer code.

  • One caveat: notice that fetch_id matches on {number, ""} when is_integer(number) which means the function will be called only if a full integer string is passed, namely “123” or “456” will succeed but “123xyz” will not. If you expect URLs like “https://whatever.man/1234abcd”, this code won’t work.

makeitrein

makeitrein OP

Dang, that’s some good looking code! Copy and pasting in 3, 2, 1…

Where Next? Top

Trending in Questions Top

katta
I having some trouble figuring out if I have set myself too strict of standards for my production server. Currently I can handle 75% of r...
New
brecabral
Documentation While reading the Scoped Routes section, I noticed that the documentation currently refers to a problem without explainin...
New
achenet
Hello, I’m trying to build a basic Phoenix web-app, and I’d like to use Tailwind. However, when I launch mix phx.server, I get an error...
New
kpanic
Hi everyone, I am toying with the idea of building a “match maker” for giving personal help to people that wants to start coding. I sta...
New
asweet-confluent
I recently noticed that Elixir’s Logger defaults its primary log level to :debug when no :logger, :level application configuration is pre...
New
Cxx-mlr
I’m working on a small exercise involving update_in/3, and I came up with this solution: data = %{ name: "Periodic Table", category:...
New
ChrisAmelia
I’ve got trouble wrapping my head around the order in which functions are called in this snippet (from Phoenix’s authentication): toke...
New

Other Trending Topics Top

GenericJam
Edit: 2026 May 15 - This post is archived. Mob is alive!! Main docs: mob v0.7.11 — Documentation A bit of explanation for the slightly c...
New
JesseHerrick
Hey, I’m Jesse and I’m the main contributor behind Dexter, a full-featured, lightning-fast Elixir LSP optimized for large codebases. It s...
New
mudasobwa
I am happy to introduce the very α version of the new programming language compiled to BEAM. Welcome Cure. It has literally three kille...
New
garrison
Hobbes is a low-level distributed database for the Elixir programming language. Hobbes provides a simple, safe, and scalable storage lay...
New
mhanberg
Hi everyone! The first release candidate for the Expert language server project is now available! We’ve published a press release detai...
New
budgie
A little off-topic, but I feel like people here have a good head on their shoulders. I used to be quite good at making software. Was luc...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews