定番のoptimizerであるSGD, RMSProp, Adamに続いて新しい学習アルゴリズムが次々提案されています。そのうち、以下の3つはtensorflow.addonsを使えばtensorflow.kerasでも使うことができます。

• SGDW, AdamW
• cosine annealing (SGDR, AdamRなど)
• RAdam

Decoupled Weight Decay Regularization
https://arxiv.org/abs/1711.05101

On the Variance of the Adaptive Learning Rate and Beyond
https://arxiv.org/pdf/1908.03265.pdf

この記事の内容

本記事はとりあえず動くコードをペタペタ貼ってるだけです。

インストール

pipで入ります

terminal

pip install tensorflow-addons

1. 準備

python

import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
import tensorflow as tf
import tensorflow_addons as tfa

N = 60
x = np.linspace(0, 5, N)
y = np.sin(x) + 0.2 + np.random.normal(0, 0.1, N)

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.33, random_state=42)

plt.figure(figsize=(5, 4))
plt.scatter(x_train, y_train)
plt.scatter(x_test, y_test)
plt.xlabel('x')
plt.ylabel('y')
plt.show()

2. Adamで学習

python

def make_model(activation, optimizer):
    inputs = tf.keras.layers.Input(1)
    network = tf.keras.layers.Dense(20, activation=activation)(inputs)
    network = tf.keras.layers.Dense(20, activation=activation)(network)
    outputs = tf.keras.layers.Dense(1, activation='linear')(network)

    model = tf.keras.models.Model(inputs=inputs, outputs=outputs)
    model.compile(loss='mse', optimizer=optimizer)
    return model

def plot_history(hist):
    plt.plot(hist.history['loss'], label='loss')
    plt.plot(hist.history['val_loss'], label='val_loss')
    plt.yscale('log')
    plt.xlabel('epochs')
    plt.ylabel('loss')
    plt.legend()
    plt.show()    

def plot_result(x_train, y_train, x_test, y_test, x, y_pred):
    plt.figure(figsize=(5, 4))
    plt.scatter(x_train, y_train, label='train')
    plt.scatter(x_test, y_test, label='test')
    plt.plot(x, y_pred, label='pred')
    plt.xlabel('x')
    plt.ylabel('y')
    plt.show()

optimizer = tf.keras.optimizers.Adam(learning_rate=0.05)
activation = 'sigmoid'
model = make_model(activation, optimizer)

hist = model.fit(x_train, y_train, 
                 validation_data=(x_test, y_test), 
                 epochs=130, verbose=False)
plot_history(hist)

y_pred = model.predict(x).reshape(-1)
plot_result(x_train, y_train, x_test, y_test, x, y_pred)

3. SGDWで学習

tensorflow_addons document
https://www.tensorflow.org/addons/api_docs/python/tfa/optimizers/SGDW

python

optimizer = tfa.optimizers.SGDW(learning_rate=0.02,
                                weight_decay=0.0001)
activation = 'sigmoid'
model = make_model(activation, optimizer)

hist = model.fit(x_train, y_train, 
                 validation_data=(x_test, y_test), 
                 epochs=1000, verbose=False)
plot_history(hist)

y_pred = model.predict(x).reshape(-1)
plot_result(x_train, y_train, x_test, y_test, x, y_pred)

4. AdamWで学習

tensorflow_addons document
https://www.tensorflow.org/addons/api_docs/python/tfa/optimizers/AdamW

python

optimizer = tfa.optimizers.AdamW(learning_rate=0.05,
                                 weight_decay=0.001)
activation = 'sigmoid'
model = make_model(activation, optimizer)

hist = model.fit(x_train, y_train, 
                 validation_data=(x_test, y_test), 
                 epochs=200, verbose=False)
plot_history(hist)

y_pred = model.predict(x).reshape(-1)
plot_result(x_train, y_train, x_test, y_test, x, y_pred)

5. RAdamで学習

tensorflow_addons document
https://www.tensorflow.org/addons/api_docs/python/tfa/optimizers/RectifiedAdam

python

optimizer = tfa.optimizers.RectifiedAdam(learning_rate=0.1,
                                         weight_decay=0.001)
activation = 'sigmoid'
model = make_model(activation, optimizer)

hist = model.fit(x_train, y_train, 
                 validation_data=(x_test, y_test), 
                 epochs=300, verbose=False)
plot_history(hist)

y_pred = model.predict(x).reshape(-1)
plot_result(x_train, y_train, x_test, y_test, x, y_pred)

6. cosine annealing

cosine annealingはtensorflowのCosineDecayRestarts()がやってくれます。optimizerの引数にlearning_rateの代わりに与えるだけです。

条件の内訳は以下のようになっています。

learning_rate

最初のlearning rate

first_decay_steps

最初のdecayが終わるまでのepoch数

t_mul

次のdecayのepoch数が前のdecayのepoch数の何倍になるか

m_mul

learning rateを前のdecayの最初のlearning rate

alpha

learning rateの下限を初期値の何倍にするか

tensorflow document
https://www.tensorflow.org/api_docs/python/tf/keras/experimental/CosineDecayRestarts

python

def plot_learning_rate(learning_rate):
    plt.plot(epochs, learning_rate(epochs))
    plt.xlim(0, 500)
    plt.ylim(-0.005, 0.105)
    plt.xlabel('epochs')
    plt.ylabel('learning rate')
    plt.show()

learning_rate = 0.1
learning_rate = tf.keras.experimental.CosineDecayRestarts(learning_rate,
                                                          first_decay_steps=200,
                                                          t_mul=0.5,
                                                          m_mul=0.8,
                                                          alpha=0.1.)
plot_learning_rate(learning_rate)

条件によっては早めにlearning rateがなくなってweightがNaNになってしまうので、パラメーターチューニングするときは範囲に注意が必要です。

7. SGDWRで学習

learning_rate

python

learning_rate = 0.1
learning_rate = tf.keras.experimental.CosineDecayRestarts(learning_rate,
                                                          first_decay_steps=150,
                                                          t_mul=0.8,
                                                          m_mul=0.8,
                                                          alpha=0.)
optimizer = tfa.optimizers.SGDW(learning_rate=learning_rate,
                                weight_decay=0.0001)

activation = 'sigmoid'
model = make_model(activation, optimizer)

hist = model.fit(x_train, y_train, 
                 validation_data=(x_test, y_test), 
                 epochs=300, verbose=False)
plot_history(hist)

y_pred = model.predict(x).reshape(-1)
plot_result(x_train, y_train, x_test, y_test, x, y_pred)

8. AdamWRで学習

python

learning_rate = 0.1
learning_rate = tf.keras.experimental.CosineDecayRestarts(learning_rate,
                                                          first_decay_steps=160,
                                                          t_mul=0.9,
                                                          m_mul=0.8,
                                                          alpha=0.)
optimizer = tfa.optimizers.AdamW(learning_rate=learning_rate,
                                 weight_decay=0.001)

activation = 'sigmoid'
model = make_model(activation, optimizer)

hist = model.fit(x_train, y_train, 
                 validation_data=(x_test, y_test), 
                 epochs=400, verbose=False)
plot_history(hist)

y_pred = model.predict(x).reshape(-1)
plot_result(x_train, y_train, x_test, y_test, x, y_pred)

9. RAdamRで学習

python

learning_rate = 0.1
learning_rate = tf.keras.experimental.CosineDecayRestarts(learning_rate,
                                                          first_decay_steps=160,
                                                          t_mul=0.9,
                                                          m_mul=0.8,
                                                          alpha=0.)
optimizer = tfa.optimizers.RectifiedAdam(learning_rate=learning_rate,
                                         weight_decay=0.001)

activation = 'sigmoid'
model = make_model(activation, optimizer)

hist = model.fit(x_train, y_train, 
                 validation_data=(x_test, y_test), 
                 epochs=300, verbose=False)
plot_history(hist)

y_pred = model.predict(x).reshape(-1)
plot_result(x_train, y_train, x_test, y_test, x, y_pred)