Looking for help translating the examples in Programming Machine Learning into Elixir

Hi,

I’m reading the book Programming Machine Learning and trying to translate examples in Elixir using Nx.

Here is the code I exported from my Livebook.

Linear Regression

Mix.install([
  {:nx, "~> 0.4.2"},
  {:axon, "~> 0.4.1"},
  {:explorer, "~> 0.5.0"},
  {:kino, "~> 0.8.0"},
  {:vega_lite, "~> 0.1.6"},
  {:kino_vega_lite, "~> 0.1.7"}
])

Data

csv = """
Reservations,Pizzas
13,33
2,16
14,32
23,51
13,27
1,16
18,34
10,17
26,29
3,15
3,15
21,32
7,22
22,37
2,13
27,44
6,16
10,21
18,37
15,30
9,26
26,34
8,23
15,39
10,27
21,37
5,17
6,18
13,25
13,23
"""

{:ok, data} =
  csv
  |> Explorer.DataFrame.load_csv()

reserv = data["Reservations"]
pizzas = data["Pizzas"]

Let’s plot the actual values and our first attempt to model.

weight_text = Kino.Input.text("Weight", default: "1")
alias VegaLite, as: Vl
{weight, _} = Float.parse(Kino.Input.read(weight_text))
model = Explorer.DataFrame.new(iter: 1..Explorer.Series.max(reserv))

chart1 =
  Vl.new()
  |> Vl.data_from_values(data, only: ["Reservations", "Pizzas"])
  |> Vl.mark(:point)
  |> Vl.encode_field(:x, "Reservations", type: :quantitative)
  |> Vl.encode_field(:y, "Pizzas", type: :quantitative)

chart2 =
  Vl.new()
  |> Vl.data_from_values(model, only: ["iter", "weights"])
  |> Vl.transform(calculate: "datum.iter * #{weight}", as: "weights")
  |> Vl.mark(:line)
  |> Vl.encode_field(:x, "iter", type: :quantitative)
  |> Vl.encode_field(:y, "weights", type: :quantitative)

combined =
  Vl.new(width: 400, height: 300)
  |> Vl.layers([chart1, chart2])

This is how VegaLite creates a linear regression.

Vl.new(width: 400, height: 300)
|> Vl.data_from_values(
  reservations: Explorer.Series.to_list(reserv),
  pizzas: Explorer.Series.to_list(pizzas)
)
|> Vl.layers([
  Vl.new()
  |> Vl.mark(:point, filled: true)
  |> Vl.encode_field(:x, "reservations", type: :quantitative)
  |> Vl.encode_field(:y, "pizzas", type: :quantitative),
  Vl.new()
  |> Vl.mark(:line, color: :firebrick)
  |> Vl.transform(regression: "pizzas", on: "reservations")
  |> Vl.encode_field(:x, "reservations", type: :quantitative)
  |> Vl.encode_field(:y, "pizzas", type: :quantitative)
])

Predict and Loss

defmodule MachineLearning do
  def predict(x, w, b) do
    x
    |> Nx.multiply(w)
    |> Nx.add(b)
  end

  def loss(x, y, w, b) do
    # basically mean squared error
    predict(x, w, b)
    |> Nx.subtract(y)
    |> Nx.power(2)
    |> Nx.mean()
  end

  def train(x, y, iter, lr) do
    # initialize weights and biases to 0
    Enum.reduce_while(1..iter, {0.0, 0.0}, fn i, {w, b} ->
      current_loss = loss(x, y, w, b)
      IO.puts("Iter ##{i} => Loss: #{inspect(current_loss)}")

      cond do
        loss(x, y, w + lr, b) < current_loss ->
          {:cont, {w + lr, b}}

        loss(x, y, w - lr, b) < current_loss ->
          {:cont, {w - lr, b}}

        loss(x, y, w, b + lr) < current_loss ->
          {:cont, {w, b + lr}}

        loss(x, y, w, b - lr) < current_loss ->
          {:cont, {w, b - lr}}

        true ->
          {:halt, {w, b}}
      end
    end)
  end
end

reserv_t = Nx.tensor(Explorer.Series.to_list(reserv))
pizzas_t = Nx.tensor(Explorer.Series.to_list(pizzas))

MachineLearning.train(reserv_t, pizzas_t, 50, 0.1)

I could not properly make it work on the training part, it always stops the training after a few iterations. Where do you think I get it wrong?

2 Likes

Because you’re using def and not defn, your loss function is returning an Nx.Tensor struct ane not a number. By using the standard comparison operators, you’re comparing numbers and structs, so that will fail fast because numbers are always evaluated as less than structs in Elixir

I strongly advise you to try and rewrite everything with defn, replacing the Enum.reduce_while with while.

Otherwise, look into using Nx.to_number so that your comparisons work properly.

3 Likes

I just want to add more context on this. This is the output that I ended up with.

Iter #1 => Loss: #Nx.Tensor<
  f32
  812.8666381835938
>
Iter #2 => Loss: #Nx.Tensor<
  f32
  895.734619140625
>

{-0.1, 0.0} # => weights and bias

As you can see, it stops at 2nd iteration.

And here is the python version of the code from the book

import numpy as np


def predict(X, w, b):
    return X * w + b


def loss(X, Y, w, b):
    return np.average((predict(X, w, b) - Y) ** 2)


def train(X, Y, iterations, lr):
    w = b = 0
    for i in range(iterations):
        current_loss = loss(X, Y, w, b)
        print("Iteration %4d => Loss: %.6f" % (i, current_loss))

        if loss(X, Y, w + lr, b) < current_loss:
            w += lr
        elif loss(X, Y, w - lr, b) < current_loss:
            w -= lr
        elif loss(X, Y, w, b + lr) < current_loss:
            b += lr
        elif loss(X, Y, w, b - lr) < current_loss:
            b -= lr
        else:
            return w, b

    raise Exception("Couldn't converge within %d iterations" % iterations)


# Import the dataset
X, Y = np.loadtxt("pizza.txt", skiprows=1, unpack=True)

# Train the system
w, b = train(X, Y, iterations=10000, lr=0.01)
print("\nw=%.3f, b=%.3f" % (w, b))

# Predict the number of pizzas
print("Prediction: x=%d => y=%.2f" % (20, predict(20, w, b)))

# Plot the chart
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
plt.plot(X, Y, "bo")
plt.xticks(fontsize=15)
plt.yticks(fontsize=15)
plt.xlabel("Reservations", fontsize=30)
plt.ylabel("Pizzas", fontsize=30)
x_edge, y_edge = 50, 50
plt.axis([0, x_edge, 0, y_edge])
plt.plot([0, x_edge], [b, predict(x_edge, w, b)], linewidth=1.0, color="g")
plt.show()

The main difference between Nx and Python is that Python has the concept of operator overloading that Elixir doesn’t. So when you have def loss returning a numpy array and compare it with < current_loss, you get numpy to do the comparison correctly.

In Nx, because your using def loss and def train, you’re actually comparing 2 Nx.Tensor structs with the default Elixir comparison operator. That will not work semantically, the same reason that comapring two DateTime structs doesn’t work properly.

You need to fix this by turning your code to defn, which replaces the default Kernel with Nx.Defn.Kernel.

1 Like

Hey @t12a :wave:
I’m writing you since I just published a collection of livebooks for the book you are reading (Programming Machine Learning by P. Perrotta) in case you are curious you can find them in this GH repository GitHub - nickgnd/programming-machine-learning-livebooks: Programming Machine Learning - Elixir Livebooks

Have a great day :wave:

2 Likes