Experiments

4 Experiments

¶In this section, we present BERT fine-tuning results on 11 NLP tasks.

4.1 GLUE

¶The General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2018a) is a collection of diverse natural language understanding tasks. Detailed descriptions of GLUE datasets are included in Appendix B.1.

¶To fine-tune on GLUE, we represent the input sequence (for single sentence or sentence pairs) as described in Section 3, and use the final hid- H den vector C ∈ R corresponding to the first input token ([CLS]) as the aggregate representation. The only new parameters introduced during fine-tuning are classification layer weights W ∈ K×H R, where K is the number of labels. We compute a standard classification loss with C and W, T i.e., log(softmax(CW )).

¶For example, the BERT SQuAD model can be trained in around 30 minutes on a single Cloud TPU to achieve a Dev F1 score of 91.0%.

¶See (10) in https://gluebenchmark.com/faq.

¶System MNLI-(m/mm) QQP QNLI 392k 363k 108k Pre-OpenAI SOTA 80.6/80.1 66.1 82.3 BiLSTM+ELMo+Attn 76.4/76.1 64.8 79.8 OpenAI GPT 82.1/81.4 70.3 87.4 BERTBASE 84.6/83.4 71.2 90.5 BERTLARGE 86.7/85.9 72.1 92.7 SST-2 CoLA STS-B MRPC RTE Average 67k 8.5k 5.7k 3.5k 2.5k - 93.2 35.0 81.0 86.0 61.7 74.0 90.4 36.0 73.3 84.9 56.8 71.0 91.3 45.4 80.0 82.3 56.0 75.1 93.5 52.1 85.8 88.9 66.4 79.6 94.9 60.5 86.5 89.3 70.1 82.1 Table 1: GLUE Test results, scored by the evaluation server (https://gluebenchmark.com/leaderboard).

¶The number below each task denotes the number of training examples. The “Average” column is slightly different than the official GLUE score, since we exclude the problematic WNLI set. BERT and OpenAI GPT are singlemodel, single task. F1 scores are reported for QQP and MRPC, Spearman correlations are reported for STS-B, and accuracy scores are reported for the other tasks. We exclude entries that use BERT as one of their components.

¶We use a batch size of 32 and fine-tune for 3 epochs over the data for all GLUE tasks. For each task, we selected the best fine-tuning learning rate (among 5e-5, 4e-5, 3e-5, and 2e-5) on the Dev set.

¶Additionally, for BERTLARGE we found that finetuning was sometimes unstable on small datasets, so we ran several random restarts and selected the best model on the Dev set. With random restarts, we use the same pre-trained checkpoint but perform different fine-tuning data shuffling and clas- sifier layer initialization.

¶Results are presented in Table 1. Both BERTBASE and BERTLARGE outperform all systems on all tasks by a substantial margin, obtaining 4.5% and 7.0% respective average accuracy improvement over the prior state of the art. Note that BERTBASE and OpenAI GPT are nearly identical in terms of model architecture apart from the attention masking. For the largest and most widely reported GLUE task, MNLI, BERT obtains a 4.6% absolute accuracy improvement. On the official GLUE leaderboard, BERTLARGE obtains a score of 80.5, compared to OpenAI GPT, which obtains 72.8 as of the date of writing.

¶We find that BERTLARGE significantly outperforms BERTBASE across all tasks, especially those with very little training data. The effect of model size is explored more thoroughly in Section 5.2.

4.2 SQuAD v1.1

¶The Stanford Question Answering Dataset (SQuAD v1.1) is a collection of 100k crowdsourced question/answer pairs (Rajpurkar et al., 2016). Given a question and a passage from The GLUE data set distribution does not include the Test labels, and we only made a single GLUE evaluation server submission for each of BERTBASE and BERTLARGE.

¶https://gluebenchmark.com/leaderboard Wikipedia containing the answer, the task is to predict the answer text span in the passage.

¶As shown in Figure 1, in the question answering task, we represent the input question and passage as a single packed sequence, with the question using the A embedding and the passage using the B embedding. We only introduce a start vec- H H tor S ∈ R and an end vector E ∈ R during fine-tuning. The probability of word i being the start of the answer span is computed as a dot product between Ti and S followed by a softmax over S·Ti e all of the words in the paragraph: Pi = ∑ S·T.

¶e j j The analogous formula is used for the end of the answer span. The score of a candidate span from position i to position j is defined as S·Ti + E·Tj, and the maximum scoring span where j ≥ i is used as a prediction. The training objective is the sum of the log-likelihoods of the correct start and end positions. We fine-tune for 3 epochs with a learning rate of 5e-5 and a batch size of 32.

¶Table 2 shows top leaderboard entries as well as results from top published systems (Seo et al., 2017; Clark and Gardner, 2018; Peters et al., 2018a; Hu et al., 2018). The top results from the SQuAD leaderboard do not have up-to-date public system descriptions available, and are allowed to use any public data when training their systems.

¶We therefore use modest data augmentation in our system by first fine-tuning on TriviaQA (Joshi et al., 2017) befor fine-tuning on SQuAD.

¶Our best performing system outperforms the top leaderboard system by +1.5 F1 in ensembling and +1.3 F1 as a single system. In fact, our single BERT model outperforms the top ensemble system in terms of F1 score. Without TriviaQA fine- QANet is described in Yu et al. (2018), but the system has improved substantially after publication.

¶System Dev Test EM F1 EM F1 Top Leaderboard Systems (Dec 10th, 2018) Human - - 82.3 91.2 #1 Ensemble - nlnet - - 86.0 91.7 #2 Ensemble - QANet - - 84.5 90.5 Published BiDAF+ELMo (Single) - 85.6 - 85.8 R.M. Reader (Ensemble) 81.2 87.9 82.3 88.5 Ours BERTBASE (Single) 80.8 88.5 - - BERTLARGE (Single) 84.1 90.9 - - BERTLARGE (Ensemble) 85.8 91.8 - - BERTLARGE (Sgl.+TriviaQA) 84.2 91.1 85.1 91.8 BERTLARGE (Ens.+TriviaQA) 86.2 92.2 87.4 93.2 Table 2: SQuAD 1.1 results. The BERT ensemble is 7x systems which use different pre-training checkpoints and fine-tuning seeds.

¶System Dev Test EM F1 EM F1 Top Leaderboard Systems (Dec 10th, 2018) Human 86.3 89.0 86.9 89.5 #1 Single - MIR-MRC (F-Net) - - 74.8 78.0 #2 Single - nlnet - - 74.2 77.1 Published unet (Ensemble) - - 71.4 74.9 SLQA+ (Single) - 71.4 74.4 Ours BERTLARGE (Single) 78.7 81.9 80.0 83.1 Table 3: SQuAD 2.0 results. We exclude entries that use BERT as one of their components.

¶tuning data, we only lose 0.1-0.4 F1, still outper- forming all existing systems by a wide margin.

4.3 SQuAD v2.0

¶The SQuAD 2.0 task extends the SQuAD 1.1 problem definition by allowing for the possibility that no short answer exists in the provided paragraph, making the problem more realistic.

¶We use a simple approach to extend the SQuAD v1.1 BERT model for this task. We treat questions that do not have an answer as having an answer span with start and end at the [CLS] token. The probability space for the start and end answer span positions is extended to include the position of the [CLS] token. For prediction, we compare the score of the no-answer span: snull = S·C + E·C to the score of the best non-null span The TriviaQA data we used consists of paragraphs from TriviaQA-Wiki formed of the first 400 tokens in documents, that contain at least one of the provided possible answers.

¶System Dev Test ESIM+GloVe 51.9 52.7 ESIM+ELMo 59.1 59.2 OpenAI GPT - 78.0 BERTBASE 81.6 - BERTLARGE 86.6 86.3 † Human (expert) - 85.0 † Human (5 annotations) - 88.0 † Table 4: SWAG Dev and Test accuracies. Human performance is measured with 100 samples, as reported in the SWAG paper.

¶sˆi,j = maxj≥iS·Ti + E·Tj. We predict a non-null answer when sˆi,j > snull + τ, where the threshold τ is selected on the dev set to maximize F1.

¶We did not use TriviaQA data for this model. We fine-tuned for 2 epochs with a learning rate of 5e-5 and a batch size of 48.

¶The results compared to prior leaderboard entries and top published work (Sun et al., 2018; Wang et al., 2018b) are shown in Table 3, excluding systems that use BERT as one of their components. We observe a +5.1 F1 improvement over the previous best system.

4.4 SWAG

¶The Situations With Adversarial Generations (SWAG) dataset contains 113k sentence-pair completion examples that evaluate grounded commonsense inference (Zellers et al., 2018). Given a sentence, the task is to choose the most plausible continuation among four choices.

¶When fine-tuning on the SWAG dataset, we construct four input sequences, each containing the concatenation of the given sentence (sentence A) and a possible continuation (sentence B). The only task-specific parameters introduced is a vector whose dot product with the [CLS] token representation C denotes a score for each choice which is normalized with a softmax layer.

¶We fine-tune the model for 3 epochs with a learning rate of 2e-5 and a batch size of 16. Results are presented in Table 4. BERTLARGE outperforms the authors’ baseline ESIM+ELMo system by +27.1% and OpenAI GPT by 8.3%.

¶

¶

¶

¶