A Recurrent Latent Variable Model for Sequential Data

TL;DR

We propose VRNN, which combines latent variables of a variational autoencoder with a recurrent neural network to effectively model variability in sequential data.

Problem

Existing RNNs with deterministic hidden states struggle to capture the diverse variability of complex sequential data such as natural speech. It is necessary to introduce stochastic elements to overcome this limitation.

Approach

At each time step of the RNN, a latent variable is sampled and combined with the hidden state to predict the next output. Variational inference approximates the posterior distribution of the latent variables, and learning is performed by maximizing the ELBO.

Results & Contribution

On four speech datasets and one handwriting dataset, VRNN outperformed existing models such as RNN, VAE, and SRNN. Empirical results confirm that latent variables play an important role in RNN dynamics.