Key Concepts

Review core concepts you need to learn to master this subject

Generating text with seq2seq

The seq2seq (sequence to sequence) model is a type of encoder-decoder deep learning model commonly employed in natural language processing that uses recurrent neural networks like LSTM to generate output. seq2seq can generate output token by token or character by character. In machine translation, seq2seq networks have an encoder accepting language as input and outputting state vectors and a decoder accepting the encoder’s final state and outputting possible translations.

Generating Text with Deep Learning
Lesson 1 of 1
  1. 1
    LSTMs are pretty extraordinary, but they’re only the tip of the iceberg when it comes to actually setting up and running a neural language model for text generation. In fact, an LSTM is usually jus…
  2. 2
    If you’re feeling a bit nervous about building this all on your own, never fear. You don’t need to start from scratch — there are a few neural network libraries at your disposal. In our case, we’ll…
  3. 3
    For each sentence, Keras expects a NumPy matrix containing one-hot vectors for each token. What’s a one-hot vector? In a one-hot vector, every token in our set is represented by a 0 except for th…
  4. 4
    At this point we need to fill out the 1s in each vector. We can loop over each English-Spanish pair in our training sample using the features dictionaries to add a 1 for the token in question. For …
  5. 5
    It’s time for some deep learning! Deep learning models in Keras are built in layers, where each layer is a step in the model. Our encoder requires two layer types from Keras: - An input layer, w…
  6. 6
    The decoder looks a lot like the encoder (phew!), with an input layer and an LSTM layer that we use together: decoder_inputs = Input(shape=(None, num_decoder_tokens)) decoder_lstm = LSTM(100, retu…
  7. 7
    Alright! Let’s get model-building! First, we define the seq2seq model using the Model() function we imported from Keras. To make it a seq2seq model, we feed it the encoder and decoder inputs, as w…
  8. 8
    Now our model is ready for testing! Yay! However, to generate some original output text, we need to redefine the seq2seq architecture in pieces. Wait, didn’t we just define and train a model? Wel…
  9. 9
    Finally, we can get to testing our model! To do this, we need to build a function that: - accepts a NumPy matrix representing the test English sentence input - uses the encoder and decoder we’ve cr…
  10. 10
    At long last, it’s translation time. Inside the test function, we’ll decode the sentence word by word using the output state that we retrieved from the encoder (which becomes our decoder’s initial …
  11. 11
    Congrats! You’ve successfully built a machine translation program using deep learning with Tensorflow’s Keras API. While the translation output may not have been quite what you expected, this is j…

How you'll master it

Stress-test your knowledge with quizzes that help commit syntax to memory

Pro Logo