djaouen

djaouen

Axon question: LSTM with Dropout

I would like to do something like the following:

input = Axon.input("input", shape: {prediction_days, 1})

model =
  input
  |> Axon.lstm(50)
  |> Axon.dropout(rate: 0.2)
  |> Axon.lstm(50)
  |> Axon.dropout(rate: 0.2)
  |> Axon.lstm(50)
  |> Axon.dropout(rate: 0.2)
  |> Axon.dense(1)

But when I try this out in LiveBook, I get the following error:

** (FunctionClauseError) no function clause matching in Axon.dropout/2    
    
    The following arguments were given to Axon.dropout/2:
    
        # 1
        {#Axon<
           inputs: %{"input" => {60, 1}}
           outputs: "lstm_1_output_sequence"
           nodes: 6
         >,
         {#Axon<
            inputs: %{"input" => {60, 1}}
            outputs: "lstm_1_c_hidden_state"
            nodes: 6
          >,
          #Axon<
            inputs: %{"input" => {60, 1}}
            outputs: "lstm_1_h_hidden_state"
            nodes: 6
          >}}
    
        # 2
        [rate: 0.2]
    
    Attempted function clauses (showing 1 out of 1):
    
        def dropout(%Axon{} = x, opts)

The Python code I am trying to replicate looks like this:

model = Sequential()

model.add(LSTM(units=50, return_sequences=True, input_shape=[x_train.shape[1],  1]))
model.add(Dropout(0.2))
model.add(LSTM(units=50, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(units=50))
model.add(Dropout(0.2))
model.add(Dense(units=1))

model.compile(optimizer='adam', loss='mean_squared_error')
model.fit(x_train, y_train, epochs=25, batch_size=32)

Any help? Thanks in advance!

Most Liked

seanmor5

seanmor5

Author of Genetic Algorithms in Elixir

Axon’s LSTM (and all other RNNs) return tuples of {seq, state}. If you want to apply dropout you need to extract the seq element and apply dropout to that from the tuple.

I would advise against this though, it’s usually discouraged to apply normal dropout to RNN outputs because it can hinder learning. You’d need to use recurrent dropout to achieve the regularization effect you’re looking for, but Axon does not support this.

seanmor5

seanmor5

Author of Genetic Algorithms in Elixir

No problem, and don’t feel bad for having questions! It’s how we all learn.

There are some RNN examples in the Axon repo if you want to check those out for some additional help!

Last Post!

djaouen

djaouen

Nevermind, I updated EXLA to use GitHub (instead of hex.pm) and the error went away on its own. Cheers!

Where Next?

Popular in Questions Top

hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New

Other popular topics Top

vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

We're in Beta

About us Mission Statement