An example

Objective: investigate the relationship between drug dosage and the efficacy

Terminology Alert🤯!!!

Activation functions at a glance

Main idea of NN

flip, twist, and/or stretch the activation functions

Update parameters

The more common framework

Scenarios

• Example 1: use Amazon review to predict customer behavior (buying/not buying)

Amazing, this box of cereal gave me a perfectly balanced breakfast, as all things should be. I only ate half of it but will definitely be buying again!

• Example 2: stock price prediction

  • company 1: $Da{y}_1$, $Da{y}_2$ , $Da{y}_3$ , $Da{y}_4$ ,... $Da{y}_n$;

  • company 2: $Da{y}_{1001}$, $Da{y}_{1002}$ , $Da{y}_{1003}$ , $Da{y}_{1004}$ ,... $Da{y}_{\mathrm{100...}}$

We see:

• Different amounts of input

• Based on historical information (correlation)

So, traditional NN does not work in these scenarios

Feedback loops

Information before is taken in and passed on to the next cell

An example

Given yesterday's and today's stock prices (i.e., time series data), noted as $x_{t-1}$ and $x_t$, we want to predict tomorrow's stock price. A little bit of formulation...

btw $h_t$ is called the hidden state

Properties of RNNs

$h_{t-1}$ = $tanh(w_1\ast x_{t-1}+b_1)$

$h_t$= $tanh(w_2\ast h_{t-1}$ + $w_1\ast x_t$ + $b_1$)

RNNs have the following properties:

• Weights and biases are shared across across every input

• The exploding/vanishing gradient problem: The more we unroll a RNN, the harder it is to train

• Not capable to handle long-term dependencies,only retains short term memory(proved by Hochreiter (1991) and Bengio, et al. (1994))

(Reminder: skip the following two pages if not interested)

The exploding/vanishing gradient problem

Taken the stock price prediction task, suppose we had 100 sequential days of stock price data

The gradient/step size: $\frac{d_{SSR}}{d_{w_1}}$ = $\frac{d_{SSR}}{d_{pred}}$ $\frac{d_{pred}}{d_h}$ $\frac{d_h}{d_z}$ $\frac{d_z}{d_{w_1}}$, where $z$ = $w_1$ $x_1$ + $b_1$

$h_1$ = $tanh(w_1\ast x_1+b_1)$;

$h_2$= $tanh(w_2\ast h_1$ + $w_1\ast x_2$ + $b_1)$

$h_3$ = $tanh(w_2\ast h_2$ + $w_1\ast x_3$ + $b_1$)

simplified: $h_3$ ~ $f[w_2\ast f[w_2\ast f(w_1\ast x_1)\left]\right]$

... ...

$h_{100}$ ~ $f[w_2\ast f[w_2\ast f[w_2\ast f[w_2\ast ....f[w_2\ast f(w_1\ast x1\left)\right]\left]\right]\left]\right]$ ~ $ffff...f(w_2^{99}\ast w_1\ast x_1)$

continued, see next page

The gradient: $\frac{d_{SSR}}{d_{w_1}}$ = $\frac{d_{SSR}}{d_{pred}}$ $\frac{d_{pred}}{d_h}$ $\frac{d_h}{d_z}$ $\frac{d_z}{d_{w1}}$, where $z$ = $w_1$ $x_1$ + $b_1$

$h_{100}$ ~ $f[w_2\ast f[w_2\ast f[w_2\ast f[w_2\ast ....f[w_2\ast f(w_1\ast x1\left)\right]\left]\right]\left]\right]$ ~ $ffff...f(w_2^{99}\ast w_1\ast x_1)$

$\frac{d_z}{d_{w1}}$ = $x_1\ast w_2^{99}$,so the gradient will be like: $\frac{d_{SSR}}{d_{w_1}}$ = something* $w_2^{99}$

$new$ $w$ = $old$ $w$ - $\alpha$ * derivative

If $w_2>1$, gradient explodes; if $w_2<1$, gradient vanishes

The GATES

• A way to optionally let information through (remove/add) to the cell state;
• composition: sigmoid layer + a point-wise multiplication operation;
• The sigmoid ∈(0,1), describing how much of each component should be let through.

Forget gate

Decides what information will be removed/thrown away in the cell state

Previous example: Amazing, this box of cereal gave me a perfectly balanced breakfast, as all things should be. I only ate half of it but will definitely be buying again!

Input gates

• part 1: input layer gate decides values to be updated
• part 2: input modulation gate create a new vector of candidate values Input Modulation Gate(g): It is often considered as a sub-part of the input gate and much literature on LSTM’s does not even mention it and assume it is inside the Input gate. It is used to modulate the information that the Input gate will write onto the Internal State Cell by adding non-linearity to the information and making the information Zero-mean. This is done to reduce the learning time as Zero-mean input has faster convergence. Although this gate’s actions are less important than the others and are often treated as a finesse-providing concept, it is good practice to include this gate in the structure of the LSTM unit.

Update the old cell state

• part 1: multiply old cell state by the forget gate
• part 2: add the input gate to keep the updated information

Output gate

output the cell state (with some conditions)

• part 1: a sigmoid layer to decide what parts of the cell state we're going to output
• part 2: push cell state with a tanh, and multiply it by the output gate

An example with python code

• Import necessary libs

import numpy as np
from keras.models import Sequential
from keras.layers import LSTM
from keras.layers import Dense, Dropout
import pandas as pd
from matplotlib import pyplot as plt
from sklearn.preprocessing import StandardScaler
import seaborn as sns

• Import data

df = pd.read_csv("data/GE.csv")
train_datesdates = pd.to_datetime(df['Date'])
cols = list(df)[1:6]
df_for_training = df[cols].astype(float)
## scale the data
scaler = StandardScaler()
scaler = scaler.fit(df_for_training)
df_for_training_scaled = scaler.transform(df_for_training)

## preview first five rows
df_for_training.head(5)

##          Open        High         Low       Close   Adj Close
## 0  101.290001  103.510002  101.059998  103.269997  102.884109
## 1  103.360001  105.129997  102.550003  104.699997  104.308762
## 2  104.459999  104.620003  102.839996  103.379997  102.993690
## 3  103.900002  106.150002  103.900002  106.089996  105.693565
## 4  106.330002  106.459999  104.800003  105.190002  104.796944