· nlp · deep-learning · mxnet · open-source

GluonNLP — Deep Learning Toolkit for Natural Language Processing

Why we built GluonNLP to make NLP experiments easier to reproduce, maintain, and reuse.

The Reproduction Problem in NLP

Consider a hypothetical second-year PhD student named Alexander. He tries to reproduce the Transformer results from “Attention Is All You Need,” but the hyperparameters in Google’s Tensor2Tensor implementation differ from those reported in the paper.

Plan vs. Reality

His advisor tells him to “just tweak the parameters a bit, run and see, it should be fine.” Alexander spends the next three days running experiments on every available GPU, then opens a GitHub issue. Other users report the same problem, but no one has a solution.

GitHub issue: can not reproduce the result

After half a month, a maintainer replies that he will investigate.

Maintainer response

Three months after opening the issue, Alexander is still asking, “Is there any progress?”

Follow-up plea

After working on deep learning and NLP at AWS, I found that this was not an isolated example. Compared with computer vision experiments at the time, NLP experiments often involved more steps in the data pipeline and more moving parts in the model. A mismatch in any of the following could prevent reproduction:

  • String encoding/decoding and Unicode format
  • Parsing and tokenization
  • Text data from various languages with different grammatical rules
  • Left-to-right or right-to-left reading order
  • Word embedding
  • Input padding
  • Gradient clipping
  • Variable-length input data and states

Several MXNet contributors and I compared the problems we had encountered across NLP projects. Those discussions led us to build GluonNLP around four design goals.

Four Design Goals

Problem 1: Reproducibility

Published hyperparameters and public implementations often disagree. GitHub implementations vary widely in quality, and many are abandoned.

GluonNLP’s approach: Maintain reproduction code together with training scripts, the hyperparameters used, and runtime logs.

Problem 2: API Stability

Reproduction code is sensitive to API changes and thus has limited shelf-life. Code breaks when deep learning frameworks evolve.

GluonNLP’s approach: Run every training and evaluation script in continuous integration so API regressions are detected as the underlying framework changes.

Problem 3: Code Reusability

Researchers often copy code across projects when existing interfaces do not fit their use case.

While working on NLP services, I accumulated five Beam Search implementations that differed mainly in their score and step functions. GluonNLP extracts shared interfaces from such recurring use cases so researchers can extend one implementation instead of copying it for each project.

Problem 4: Resource Fragmentation

NLP resources are scattered. To complete one project you may have to depend on multiple packages.

GluonNLP’s approach: Provide one interface for downloading public pre-trained embeddings, language models, and benchmark datasets.

At the time, these resources were distributed across different frameworks, packages, and data formats. Comparing them in one project required installing several tools and converting their outputs into a common representation.

“Enough talking, show me the code!”

Loading GloVe embeddings and a pre-trained AWD LSTM language model, then comparing cosine similarity between the words “baby” and “infant”:

import mxnet as mx
import gluonnlp as nlp

# Load GloVe word embeddings
glove = nlp.embedding.create('glove', source='glove.6B.50d')

# Compute 'baby' and 'infant' word embeddings
baby_glove, infant_glove = glove['baby'], glove['infant']

# Load pre-trained AWD LSTM language model and get the embedding
lm_model, lm_vocab = nlp.model.get_model(name='awd_lstm_lm_1150',
                                         dataset_name='wikitext-2',
                                         pretrained=True)
baby_idx, infant_idx = lm_vocab['baby', 'infant']
lm_embedding = lm_model.embedding[0]

# Get the word embeddings of 'baby' and 'infant'
baby_lm, infant_lm = lm_embedding(mx.nd.array([baby_idx, infant_idx]))

# cosine similarity
def cos_similarity(vec1, vec2):
    return mx.nd.dot(vec1, vec2) / (vec1.norm() * vec2.norm())

print(cos_similarity(baby_glove, infant_glove))  # 0.74056691
print(cos_similarity(baby_lm, infant_lm))        # 0.3729561

For this word pair, the GloVe vectors have a higher cosine similarity (0.74) than the AWD-LSTM input embeddings (0.37). This single comparison illustrates the shared loading interface; it is not an evaluation of either representation overall.

GluonNLP v0.3.2

Features at the time of writing:

  • Over 300 pre-trained word embeddings (GloVe, FastText, Word2vec)
  • 5 language models (AWD, Cache, LSTM)
  • Neural Machine Translation training (Google NMT, Transformer)
  • Word2vec and FastText embedding training with subword interpolation
  • Flexible data pipeline tools and public datasets
  • NLP examples including sentiment analysis
pip install gluonnlp

Team

The GluonNLP team: