Text Generation
Learn about seq2seq and LSTM neural networks commonly used in NLP work and how to implement them with TensorFlow for machine translation.
StartKey Concepts
Review core concepts you need to learn to master this subject
Generating text with seq2seq
One-hot vectors
Seq2Seq Timesteps
Teacher forcing for seq2seq
Deep Learning with TensorFlow
Improving seq2seq
Generating text with seq2seq
Generating text with seq2seq
The seq2seq (sequence to sequence) model is a type of encoder-decoder deep learning model commonly employed in natural language processing that uses recurrent neural networks like LSTM to generate output. seq2seq can generate output token by token or character by character. In machine translation, seq2seq networks have an encoder accepting language as input and outputting state vectors and a decoder accepting the encoder’s final state and outputting possible translations.
- 1LSTMs are pretty extraordinary, but they’re only the tip of the iceberg when it comes to actually setting up and running a neural language model for text generation. In fact, an LSTM is usually jus…
- 2If you’re feeling a bit nervous about building this all on your own, never fear. You don’t need to start from scratch — there are a few neural network libraries at your disposal. In our case, we’ll…
- 3For each sentence, Keras expects a NumPy matrix containing one-hot vectors for each token. What’s a one-hot vector? In a one-hot vector, every token in our set is represented by a 0 except for th…
- 4At this point we need to fill out the 1s in each vector. We can loop over each English-Spanish pair in our training sample using the features dictionaries to add a 1 for the token in question. For …
- 5It’s time for some deep learning! Deep learning models in Keras are built in layers, where each layer is a step in the model. Our encoder requires two layer types from Keras: - An input layer, w…
- 6The decoder looks a lot like the encoder (phew!), with an input layer and an LSTM layer that we use together: decoder_inputs = Input(shape=(None, num_decoder_tokens)) decoder_lstm = LSTM(100, retu…
- 7Alright! Let’s get model-building! First, we define the seq2seq model using the Model() function we imported from Keras. To make it a seq2seq model, we feed it the encoder and decoder inputs, as w…
- 8Now our model is ready for testing! Yay! However, to generate some original output text, we need to redefine the seq2seq architecture in pieces. Wait, didn’t we just define and train a model? Wel…
- 9Finally, we can get to testing our model! To do this, we need to build a function that: - accepts a NumPy matrix representing the test English sentence input - uses the encoder and decoder we’ve cr…
- 10At long last, it’s translation time. Inside the test function, we’ll decode the sentence word by word using the output state that we retrieved from the encoder (which becomes our decoder’s initial …
- 11Congrats! You’ve successfully built a machine translation program using deep learning with Tensorflow’s Keras API. While the translation output may not have been quite what you expected, this is j…
How you'll master it
Stress-test your knowledge with quizzes that help commit syntax to memory