Section 10 of 11
Detailed Experimental Setup
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · about 3 minutes
¶B Detailed Experimental Setup B.1 Detailed Descriptions for the GLUE Benchmark Experiments.
¶The GLUE benchmark includes the following datasets, the descriptions of which were originally summarized in Wang et al. (2018a): MNLI Multi-Genre Natural Language Inference is a large-scale, crowdsourced entailment classification task (Williams et al., 2018). Given a pair of sentences, the goal is to predict whether the second sentence is an entailment, contradiction, or neutral with respect to the first one.
¶QQP Quora Question Pairs is a binary classification task where the goal is to determine if two questions asked on Quora are semantically equivalent (Chen et al., 2018).
¶QNLI Question Natural Language Inference is a version of the Stanford Question Answering Dataset (Rajpurkar et al., 2016) which has been converted to a binary classification task (Wang et al., 2018a). The positive examples are (question, sentence) pairs which do contain the correct answer, and the negative examples are (question, sentence) from the same paragraph which do not contain the answer.
¶SST-2 The Stanford Sentiment Treebank is a binary single-sentence classification task consisting of sentences extracted from movie reviews Class Label C T... T T T ’... T ’
1 N [SEP] 1 M
¶BERT E[CLS] E... E E E ’... E ’
1 N [SEP] 1 M
¶Tok Tok Tok Tok [CLS]... [SEP]...
1 N 1 M
¶Sentence 1 Sentence 2 Start/End Span C T... T T T ’... T ’
1 N [SEP] 1 M
¶BERT E[CLS] E... E E E ’... E ’
1 N [SEP] 1 M
¶Tok Tok Tok Tok [CLS]... [SEP]...
1 N 1 M
¶Question Paragraph Class Label C T T... T 1 2 N BERT E E E... E [CLS] 1 2 N [CLS][CLS] Tok 1Tok 1 Tok 2... Tok N Single Sentence ...
¶O B-PER O C T T... T 1 2 N BERT E E E... E [CLS] 1 2 N [CLS] Tok 1 Tok 2... Tok N Single Sentence Figure 4: Illustrations of Fine-tuning BERT on Different Tasks.
¶with human annotations of their sentiment (Socher et al., 2013).
¶CoLA The Corpus of Linguistic Acceptability is a binary single-sentence classification task, where the goal is to predict whether an English sentence is linguistically “acceptable” or not (Warstadt et al., 2018).
¶STS-B The Semantic Textual Similarity Benchmark is a collection of sentence pairs drawn from news headlines and other sources (Cer et al., 2017). They were annotated with a score from 1 to 5 denoting how similar the two sentences are in terms of semantic meaning.
¶MRPC Microsoft Research Paraphrase Corpus consists of sentence pairs automatically extracted from online news sources, with human annotations for whether the sentences in the pair are semantically equivalent (Dolan and Brockett, 2005).
¶RTE Recognizing Textual Entailment is a binary entailment task similar to MNLI, but with much less training data (Bentivogli et al., 2009).
¶WNLI Winograd NLI is a small natural language inference dataset (Levesque et al., 2011).
¶The GLUE webpage notes that there are issues with the construction of this dataset, and every trained system that’s been submitted to GLUE has performed worse than the 65.1 baseline accuracy of predicting the majority class. We therefore exclude this set to be fair to OpenAI GPT. For our GLUE submission, we always predicted the majority class.
¶Note that we only report single-task fine-tuning results in this paper. A multitask fine-tuning approach could potentially push the performance even further. For example, we did observe substantial improvements on RTE from multitask training with MNLI.