adamwight

adamwight

MediaWiki Client - read and write Wikipedia, follow recent edits, access machine learning analytics

Seeking early adopters and reviewers of the mediawiki_client library (docs), which intends to wrap all of the major access methods for Wikipedia content.

The supported APIs are functionally complete and the library includes tests. My current goal is to get feedback on the exposed interfaces, and stabilize on an idiomatic Elixir style to begin the 1.x releases.

Examples

Watch all Wikimedia sites and print a link to the next 6 articles as they are created:

Wiki.EventStreams.start_link(streams: "page-create")
Wiki.EventStreams.stream()
|> Stream.take(6)
|> Enum.each(fn event -> IO.puts(event["meta"]["uri"]) end)
# https://www.wikidata.org/wiki/Q111963009
# https://en.wikipedia.org/wiki/Rhinella_xerophylla
# https://fr.wiktionary.org/wiki/stachst_aus
# https://commons.wikimedia.org/wiki/File:St_Swithin,_Quenington_-_Wall_monument_-_geograph.org.uk_-_3514076.jpg
# https://commons.wikimedia.org/wiki/File:Nebria_brevicollis_(Carabidae)_-_(imago),_Henshuisterveld,_the_Netherlands.jpg
# https://commons.wikimedia.org/wiki/Category:Protected_areas_of_Russia/5630230

Get current site statistics for German Wikipedia:

|> Wiki.Site.get!("dewiki")
|> Wiki.Action.new()
|> Wiki.Action.get!(
  action: :query,
  meta: :siteinfo,
  siprop: :statistics
)
# %Wiki.Action.Session{
#   ...
#   result: %{
#     "batchcomplete" => true,
#     "query" => %{
#       "statistics" => %{
#         "activeusers" => 19687,
#         "admins" => 188,
#         "articles" => 2583285,
#         "edits" => 211219883,
#         "images" => 130199,
#         "jobs" => 0,
#         "pages" => 7163473,
#         "queued-massmessages" => 0,
#         "users" => 3715461
#       }
#     }
#   },
#   ...
# }

Fetch all information about Douglas Adams contained on Wikidata:
```elixir
Wiki.Site.get!("wikidatawiki")
|> Wiki.Action.new()
|> Wiki.Action.get!(
    action: :wbgetentities,
    ids: "Q42"
)
# result: %{
# 	"entities" => %{
# 		"Q42" => %{
# 			"aliases" => %{
# 				"uk" => [
# 					%{"language" => "uk", "value" => "Дуглас Ноел Адамс"},
# 					%{"language" => "uk", "value" => "Адамс Дуглас"}
# 				],
# 				"pt" => [
# 					%{"language" => "pt", "value" => "Douglas Noël Adams"},
# 					%{"language" => "pt", "value" => "Douglas Noel Adams"}
# 				],

Request ORES scoring for an edit to predict whether it was spam or not:

Wiki.Ores.new("enwiki")
|> Wiki.Ores.request!(
  models: ["damaging"],
  revids: 456789
)
# %{
#   "enwiki" => %{
#     "models" => %{
#       "damaging" => %{"version" => "0.5.1"}
#     },
#     "scores" => %{
#       "456789" => %{
#         "damaging" => %{
#           "score" => %{
#             "prediction" => false,
#             "probability" => %{
#               "false" => 0.9784615344695441,
#               "true" => 0.021538465530455946
#             }
#           }
#         }
#       }
#     }
#   }
# }
"""

Most Liked

adamwight

adamwight

I’ve spun out a streaming bzip2 codec as part of the work to support Elixir processing of Wikipedia dumps.

Example usage:

Mix.install([:bzip2])

:inets.start()
:ssl.start()
tmpfile = '/tmp/articles.xml.bz2'
url = 'https://dumps.wikimedia.org/testwiki/20220501/testwiki-20220501-pages-articles.xml.bz2'

{:ok, :saved_to_file} = :httpc.request(:get, {url, []}, [], [stream: tmpfile])

File.stream!(to_string(tmpfile), [], 900 * 1024)
|> Bzip2.decompress!()
|> Enum.into("")
#  ...
#  <page>
#    <title>Wikipedia:Help</title>
#    <ns>4</ns>
#    <id>80</id>
#    <redirect title="Main Page" />
#    <revision>
#      <id>438551</id>
#      <parentid>438171</parentid>
#      <timestamp>2020-07-01T07:11:36Z</timestamp>
#      <contributor>
#        <username>JohanahoJ</username>
#        <id>37147</id>
#      </contributor>
#      <comment>Changed redirect target from [[Wikipedia:Main Page]] to [[Main Page]]</comment>
#      <model>wikitext</model>
#      <format>text/x-wiki</format>
#  ...

I would have liked to give a fully streamed example, but :httpc requires more than a dab of glue to do so, and I can’t find any other currently working library to stream HTTP.

adamwight

adamwight

Now with Livebook examples.

Also, at day job my team is writing a complementary library to parse wiki article HTML dumps, which works already but will be finalized in a few weeks, then we run it on all available dumps.

We’re looking at article footnotes for our particular use case, but it can be generalized to run whatever other parsing modules.

Where Next?

Popular in Announcing Top

josevalim
EDIT: since Ecto 3.0 final version is out, this post was amended to use the final versions in the instructions below. Hi everyone, We a...
New
restlessronin
The repo is at GitHub - cyberchitta/openai_ex: Community maintained Elixir library for OpenAI API · GitHub. Docs are at OpenaiEx User Gu...
152 10342 134
New
maltoe
Hello! Came here to announce ChromicPDF, a pet project PDF generator I’ve been working on for the past few months. Why another PDF gener...
New
oltarasenko
Dear Elixir community, After a year of development, bug fixes, and improvements, we are proudly ready to share the release of Crawly 0.1...
New
kelvinst
Hey everyone! Well, we made this lib a while ago and now we decided to finally go out and public with it! It’s a tool for creating and m...
New
blatyo
The best overview for how things are tied together is this presentation. Modules and functions are pretty well documented at this point, ...
New
kip
Image is an image processing library for Elixir. It is based upon the fabulous vix library that provides a libvips wrapper for Elixir. I...
622 18934 194
New
Qqwy
Hello everyone, I wrote a small library today called MapDiff. It returns a map listing the (smallest amount of) changes to get from map...
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 36352 110
New
marcuslankenau
I feel kind of stuck with the absence of a proper xml library for Elixir. Currently I use SweetXML which was ok for me more or less to pa...
New

Other popular topics Top

vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
AstonJ
Posting this to see if we can make things easier for people to get into Neovim. If you use Neovim and have a favourite distro please let ...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
RisingFromAshes
I’ve read in another post that it may be possible with a router helper - but I couldn’t find an appropriate one, and tbh, I’m still just ...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
jaysoifer
Is there a way to rollback a specific migration and only that one (“skipping” all the other ones)? Would mix ecto.rollback -v 200809061...
New

Latest on Elixir Forum

Elixir Forum

We're in Beta

About us Mission Statement