Introduction

When developing deep learning models, you may find yourself investing countless hours or even days before noticing significant results. Often, you have to halt model training to adjust settings like the learning rate, log training metrics for later review, or observe the training process through TensorBoard. This reveals that completing even fundamental tasks can require considerable effort—this is where TensorFlow callbacks become essential.

This article will explore the intricacies, applications, and examples of TensorFlow callbacks. Here’s the structure we’ll follow:

• Understanding callback functions
• Events that trigger callbacks
• The available callbacks in TensorFlow 2.0
• Conclusion

Prerequisites

• Familiarity with Python programming and TensorFlow: A basic understanding of Python and experience with TensorFlow for constructing and training deep learning models.
• Knowledge of Deep Learning Concepts: Awareness of terms such as epochs, batches, training/validation loss, and accuracy.
• Previous Experience with TensorFlow Model Training: Experience using TensorFlow’s Model.fit() function, including the specification of training and validation datasets.
• Familiarity with Keras API: An understanding of Keras as a high-level API for TensorFlow, covering model creation, compiling, and training processes.
• TensorFlow Installation: Ensure TensorFlow is installed in your working environment (e.g., using pip install tensorflow).

What’s a Callback Function?

In essence, callbacks are specialized functions executed at specific stages during the training process. They can help mitigate overfitting, visualize training progress, debug code, save model checkpoints, generate logs, and facilitate the use of TensorBoard, among other tasks. TensorFlow provides an array of built-in callbacks, and you can utilize several of them simultaneously. We will examine various callbacks and their implementations.

When a Callback is Triggered

Callbacks are activated when specific events occur during training, including:

on_epoch_begin: Triggered when a new epoch commences.

on_epoch_end: Triggered when an epoch concludes.

on_batch_begin: Triggered at the commencement of a new training batch.

on_batch_end: Triggered upon the completion of a batch training.

on_train_begin: Triggered when training begins.

on_train_end: Triggered when training concludes.

To incorporate any callback in model training, simply pass the callback object within the model.fit call, like this:

model.fit(x, y, callbacks=list_of_callbacks)

Available Callbacks in TensorFlow 2.0

Let’s explore the callbacks offered under the tf.keras.callbacks module.

1. EarlyStopping

This commonly used callback monitors specific metrics and halts model training when no improvement is detected. For instance, if you wish to stop training when accuracy fails to improve by 0.05, this callback is ideal for the task. It helps to reduce the risk of model overfitting.

tf.keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0, patience=0, verbose=0, mode='auto', baseline=None, restore_best_weights=False)

monitor: Specifies the metrics we want to observe.

min_delta: Denotes the minimal expected improvement for each epoch.

patience: Number of epochs to wait before halting training.

verbose: Controls the display of additional output logs.

mode: Defines if metrics should increase, decrease, or are inferred from the name; allowable values are 'min', 'max', or 'auto'.

baseline: Sets benchmarks for observed metrics.

restore_best_weights: If True, the model retrieves weights from the most favorable epoch; otherwise, it retains the weights from the final epoch.

The EarlyStopping callback executes via the on_epoch_end trigger during training.

2. ModelCheckpoint

This callback offers a mechanism to save the model at regular intervals during training, which is particularly beneficial when training models that require substantial time. It keeps an eye on training progress and saves checkpoints based on specific metrics.

tf.keras.callbacks.ModelCheckpoint(filepath, monitor='val_loss', verbose=0, save_best_only=False, save_weights_only=False, mode='auto', save_freq='epoch')

filepath: The path to save the model. You can format the file name with options like model-{epoch:02d}-{val_loss:0.2f}; this way, the model gets saved with the specified values in the title.

monitor: Metrics to observe.

save_best_only: If True, it prevents overriding the best model.

mode: Dictates whether monitored metrics should increase, decrease, or are inferred from the name; acceptable values are 'min', 'max', or 'auto'.

save_weights_only: If True, only the model’s weights get saved. Otherwise, the entire model will be stored.

save_freq: If set to 'epoch', the model saves after each epoch. If an integer value is specified, it saves after the defined number of batches (not to be mixed with epochs).

The ModelCheckpoint callback is executed via the on_epoch_end trigger of the training phase.

3. TensorBoard

This callback is excellent for visualizing the training summary of your model. It generates logs for TensorBoard, which can then be launched for a comprehensive view of training progress—a detailed exploration of TensorBoard will be the subject of a separate article.

tf.keras.callbacks.TensorBoard(log_dir='logs', histogram_freq=0, write_graph=True, write_images=False, update_freq='epoch', profile_batch=2, embeddings_freq=0, embeddings_metadata=None, **kwargs)

For now, we will discuss the log_dir parameter, which specifies the folder path for storing logs. To launch TensorBoard, execute the command as follows:

tensorboard --logdir=path_to_your_logs

TensorBoard can be initiated before or after the training begins.

The TensorBoard callback also triggers at on_epoch_end.

4. LearningRateScheduler

This callback is useful when you’d like to change the learning rate as training progresses. For example, you may want to reduce the learning rate after a specific number of epochs. The LearningRateScheduler facilitates this action.

tf.keras.callbacks.LearningRateScheduler(schedule, verbose=0)

schedule: A function that accepts the epoch index and returns a new learning rate.

verbose: Controls logging output.

Below is an example that illustrates how to lower the learning rate after three epochs.

5. CSVLogger

This callback logs training details to a CSV file. The logged parameters include epoch, accuracy, loss, val_accuracy, and val_loss. Do remember to include accuracy as a metric during model compilation to avoid execution errors.

tf.keras.callbacks.CSVLogger(filename, separator=',', append=False)

The logger accepts parameters such as filename, separator, and append. The append option specifies whether to add to an existing file or write to a new file.

The CSVLogger callback is executed with the on_epoch_end trigger during training. Hence, once an epoch concludes, the logs are saved to the file.

6. LambdaCallback

This callback is essential when you need to execute a custom function at various events, and the standard callbacks may not fully meet your requirements. For example, you might wish to log data into a database.

tf.keras.callbacks.LambdaCallback(on_epoch_begin=None, on_epoch_end=None, on_batch_begin=None, on_batch_end=None, on_train_begin=None, on_train_end=None, **kwargs)

All parameters of this callback expect a function that processes the specified arguments:

on_epoch_begin and on_epoch_end: epoch, logs.

on_batch_begin and on_batch_end: batch, logs.

on_train_begin and on_train_end: logs.

Here’s an example:

This callback will log data into a file after a batch is processed, producing outputs similar to this in the file:

7. ReduceLROnPlateau

This callback allows for adjusting the learning rate when metrics show no improvement. Unlike LearningRateScheduler, it decreases the learning rate based on the metric rather than the epoch count.

tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=10, verbose=0, mode='auto', min_delta=0.0001, cooldown=0, min_lr=0, **kwargs)

<pMost parameters resemble those in the EarlyStopping callback, so let’s highlight the unique ones.

monitor, patience, verbose, mode, min_delta: similar to EarlyStopping.

factor: Specifies the factor by which the learning rate should reduce (new learning rate = old learning rate * factor).

cooldown: Denotes the number of epochs to wait before restarting metric monitoring.

min_lr: Sets the lowest possible value for the learning rate.

This callback is also triggered at on_epoch_end.

8. RemoteMonitor

This callback becomes useful when there’s a need to send logs to an API. Its functionality can also be replicated using LambdaCallback.

tf.keras.callbacks.RemoteMonitor(root='http://localhost:9000', path='/publish/epoch/end/', field='data', headers=None, send_as_json=False)

root: URL endpoint.

path: Endpoint name/path.

field: Key name for storing logs.

header: Required header for sending.

send_as_json: If set to True, data is sent as JSON.

To demonstrate this callback, you must have an endpoint active on localhost:8000. You can use Node.js for implementation. Save the server code in a file named server.js:

Start the server using the command node server.js (make sure node is installed). At the end of each epoch, logs will be visible in the node console. If the server isn’t running, you’ll receive a warning after each epoch.

This callback also triggers at the on_epoch_end event.

9. BaseLogger & History

These two callbacks are automatically applied to every Keras model. The history object returned by model.fit contains a data dictionary with average accuracy and loss throughout the epochs, along with a dictionary of training parameters like epochs, steps, and verbose. If a learning rate modification callback is present, it will also reflect in the history object.

BaseLogger maintains an average of your metrics over the epochs, consequently, the metrics shown at the end of any epoch indeed represent an average calculated from all batches.

10. TerminateOnNaN

This callback halts the training process if the loss value becomes NaN.

tf.keras.callbacks.TerminateOnNaN()

Conclusion

You can choose and utilize various callbacks as needed. Employing multiple callbacks simultaneously is often advantageous (or even essential), such as using TensorBoard for tracking progress, EarlyStopping or LearningRateScheduler to avoid overfitting, and ModelCheckpoint to secure training progress.

Keep in mind that the code for all callbacks available in tensorflow.keras can be executed. I hope this information proves beneficial in your model training journey.

Wishing you success in Deep Learning.

Thank you for engaging with the DigitalOcean Community. Explore our range of services including compute, storage, networking, and managed databases.

Learn more about our products.