Enhancing Data Encoding with Stacked AutoEncoders (SAEs)

Dive into our 2015 project on Stacked AutoEncoders (SAEs), a pioneering approach in deep learning for efficient data encoding, inspired by advancements in neural network-based instrument extraction from music. Discover how SAEs optimize encoding processes and structure through unique training methodologies.

In 2015, at the International Audio Laboratories, I embarked on an exploratory project into Machine Learning (ML) and Deep Learning (DL), focusing on Stacked AutoEncoders (SAEs). Inspired by groundbreaking research in Deep Neural Network Based Instrument Extraction From Music, this project aimed to leverage SAEs for advanced data encoding.

Understanding Autoencoders

Autoencoder Schema

Autoencoders, a cornerstone of neural network architecture, are designed to learn efficient data encodings automatically. By progressively reducing neuron count layer by layer, autoencoders distill input data into a concise, efficient encoding before attempting to reconstruct the original input. Learn more about Autoencoders.

The Innovation of Stacked Autoencoders

Our approach, as detailed in the referenced paper, introduces a novel training method for autoencoders. Starting with a single-layer network, we experiment with two initial weight settings: the Identity function and the least-squares method, focusing on optimizing the encoding of complex audio data.

Stacked Autoencoder Training Process


This methodical layering and training process, repeated until the network reaches a specified depth, preserves initial weights while adapting to new complexities, resulting in a finely tuned SAE.

Observing Training Loss Evolution

The hallmark of our stacked autoencoder’s architecture is its characteristic “staircase” pattern of training and validation loss. This pattern demonstrates the effectiveness of our training methodology, with each new layer contributing to a significant leap in performance and accuracy.

Training and Validation Loss


For an in-depth look at the development and application of this technology, including a step-by-step guide and example implementations, visit our GitHub repository and explore the notebook example.