SkillCompiler/data/format-preference/natural-instructions/tasks/task462_qasper_classification.json

{
    "Contributors": [
        "Yeganeh Kordi"
    ],
    "Source": [
        "qasper"
    ],
    "URL": [
        "https://allenai.org/project/qasper/home"
    ],
    "Categories": [
        "Question Understanding"
    ],
    "Reasoning": [],
    "Definition": [
        "In this task, you will be presented with a context from an academic paper and a question based on the context. You have to classify the questions into \"Extractive\", \"Abstractive\", or \"Yes-no\" questions. Extractive questions can be answered by concatenating extracts taken from a context into a summary while answering abstractive questions involves paraphrasing the context using novel sentences. Yes-no question is a question whose expected answer is one of two choices, one that affirms the question and one that denies the question. Typically, the choices are either yes or no."
    ],
    "Input_language": [
        "English"
    ],
    "Output_language": [
        "English"
    ],
    "Instruction_language": [
        "English"
    ],
    "Domains": [
        "Scientific Research Papers"
    ],
    "Positive Examples": [
        {
            "input": "We build a dataset of Twitter accounts based on two lists annotated in previous works. For the non-factual accounts, we rely on a list of 180 Twitter accounts from BIBREF1. On the other hand, for the factual accounts, we use a list with another 32 Twitter accounts from BIBREF19 that are considered trustworthy by independent third parties. \n Question: How did they obtain the dataset?",
            "output": "Extractive",
            "explanation": "The answer to this question has been explicitly mentioned in the context, so the question is extractive."
        },
        {
            "input": "We evaluate our pointer-generator performance using BLEU score. The baseline language model is trained using RNNLM BIBREF23 . Perplexity measure is used in the evaluation.\n\n \n Question: Did they use other evaluation metrics?",
            "output": "Yes-no",
            "explanation": "This is a good example, and the given question is a yes-no question."
        },
        {
            "input": "As in the example above, we pre-process documents by removing all numbers and interjections. \n Question: what processing was done on the speeches before being parsed?",
            "output": "Abstractive",
            "explanation": "Answering this question needs paraphrasing the context, so this question is abstractive."
        }
    ],
    "Negative Examples": [
        {
            "input": "For the English version, we performed both a thorough manual analysis and automatic evaluation across three commonly used TS datasets from two different domains  For the English version, we performed both a thorough manual analysis and automatic evaluation across three commonly used TS datasets from two different domains  The evaluation of the German version is in progress. \n Question: What are the corpora used for the task?",
            "output": "Yes-no",
            "explanation": "This question is extractive, and it's not a yes-no question."
        },
        {
            "input": "For Turkish, as the first dataset, we utilised the movie reviews which are collected from a popular website. The number of reviews in this movie corpus is 20,244 and the average number of words in reviews is 39. Each of these reviews has a star-rating score which is indicative of sentiment. These polarity scores are between the values 0.5 and 5, at intervals of 0.5. We consider a review to be negative it the score is equal to or lower than 2.5. On the other hand, if it is equal to or higher than 4, it is assumed to be positive. We have randomly selected 7,020 negative and 7,020 positive reviews and processed only them. \n Question: What details are given about the movie domain dataset?",
            "output": "Extractive",
            "explanation": "This question is Abstractive because we need to paraphrase the context to get to the answer."
        }
    ],
    "Instances": [
        {
            "id": "task462-1e1cddb7ec8c42fea8c2aa024d95955e",
            "input": "We collected tweets associated to a dozen US mainstream news websites, i.e. most trusted sources described in BIBREF18, with the Streaming API, and we referred to Hoaxy API BIBREF16 for what concerns tweets containing links to 100+ US disinformation outlets. We filtered out articles associated to less than 50 tweets. The resulting dataset contains overall $\\sim $1.7 million tweets for mainstream news, collected in a period of three weeks (February 25th, 2019-March 18th, 2019), which are associated to 6,978 news articles, and $\\sim $1.6 million tweets for disinformation, collected in a period of three months (January 1st, 2019-March 18th, 2019) for sake of balance of the two classes, which hold 5,775 distinct articles. Diffusion censoring effects BIBREF14 were correctly taken into account in both collection procedures. We provide in Figure FIGREF4 the distribution of articles by source and political bias for both news domains. For what concerns the Italian scenario we first collected tweets with the Streaming API in a 3-week period (April 19th, 2019-May 5th, 2019), filtering those containing URLs pointing to Italian official newspapers websites as described in BIBREF22; these correspond to the list provided by the association for the verification of newspaper circulation in Italy (Accertamenti Diffusione Stampa). We instead referred to the dataset provided by BIBREF23 to obtain a set of tweets, collected continuously since January 2019 using the same Twitter endpoint, which contain URLs to 60+ Italian disinformation websites. In order to get balanced classes (April 5th, 2019-May 5th, 2019), we retained data collected in a longer period w.r.t to mainstream news. In both cases we filtered out articles with less than 50 tweets; overall this dataset contains $\\sim $160k mainstream tweets, corresponding to 227 news articles, and $\\sim $100k disinformation tweets, corresponding to 237 news articles. We provide in Figure FIGREF5 the distribution of articles according to distinct sources for both news domains.  \n Question: Which two news domains are country-independent?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-7fd3f50574ad4c8a8f62d96c035d6a5b",
            "input": "Given raw audio samples INLINEFORM0 , we apply the encoder network INLINEFORM1 which we parameterize as a five-layer convolutional network similar to BIBREF15 . Next, we apply the context network INLINEFORM0 to the output of the encoder network to mix multiple latent representations INLINEFORM1 into a single contextualized tensor INLINEFORM2 for a receptive field size INLINEFORM3 . The context network has seven layers and each layer has kernel size three and stride one.  \n Question: How many convolutional layers does their model have?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-4c831be9d566400894b0dcce4a1161e4",
            "input": "Therefore, we implement reinforcement learning and elaborately design two rewards to describe the irony accuracy and sentiment preservation, respectively. \n Question: What is the combination of rewards for reinforcement learning?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-d9ed2f90408c4ebf89c203a30e3e887b",
            "input": "A few sample advice rules in English (these are converted to first-order logic format and given as input to our algorithm) are presented in Table TABREF11 .  We modified the work of Odom et al. odomAIME15,odomAAAI15 to learn RDNs in the presence of advice. The key idea is to explicitly represent advice in calculating gradients. \n Question: How do they incorporate human advice?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-1bf3490d9040499fb312e8bf746d3d06",
            "input": "We designed 3 evaluation sets: (1) Base Set (1,264 samples) held out from the simulated data. (2) Augmented Set (1,280 samples) built by adding two out-of-distribution symptoms, with corresponding dialogue contents and queries, to the Base Set (“bleeding” and “cold”, which never appeared in training data). (3) Real-World Set (944 samples) manually delineated from the the symptom checking portions (approximately 4 hours) of real-world dialogues, and annotated as evaluation samples. \n Question: How do they select instances to their hold-out test set?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-9b43c0ca2319465b8db38d1a1e61f72b",
            "input": "We initiated crawling with 100 questions randomly selected from different topics so that different genre of questions can be covered. The crawling of the questions follow a BFS pattern through the related question links. We obtained 822,040 unique questions across 80,253 different topics with a total of 1,833,125 answers to these questions. \n Question: Does the experiments focus on a specific domain?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-b6d6983c2b254d95a012ace0d79db179",
            "input": " In order to explore the problem space, we thus specifically target five binary classification problems addressed in BIBREF17 , BIBREF18 , i.e presence/absence of consonants, phonemic nasal, bilabial, high-front vowels and high-back vowels. \n Question: What are the five different binary classification tasks?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-ae83cc6f79ef456293f232f6640d6b2b",
            "input": "We evaluate our cross-lingual pre-training based transfer approach against several strong baselines on two public datatsets, Europarl BIBREF31 and MultiUN BIBREF32, which contain multi-parallel evaluation data to assess the zero-shot performance. \n Question: which datasets did they experiment with?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-349289902cee409c9a930c9e528364bf",
            "input": " Conditional Random Fields\nConditional Random Fields (CRF) BIBREF10 are a standard approach when dealing with sequential data in the context of sequence labeling. BiLSTM-CRF\nPrior to the emergence of deep neural language models, BiLSTM-CRF models BIBREF11 had achieved state-of-the-art results for the task of sequence labeling. Multi-Task Learning\nMulti-Task Learning (MTL) BIBREF15 has become popular with the progress in deep learning. BioBERT\nDeep neural language models have recently evolved to a successful method for representing text. In particular, Bidirectional Encoder Representations from Transformers (BERT) outperformed previous state-of-the-art methods by a large margin on various NLP tasks BIBREF17. \n Question: What baselines do they introduce?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-202a462c9d284981b010d97e36340323",
            "input": "We find that dynamic communities, such as Seahawks or starcraft, have substantially higher rates of monthly user retention than more stable communities (Spearman's INLINEFORM0 = 0.70, INLINEFORM1 0.001, computed with community points averaged over months; Figure FIGREF11 .A, left). Similarly, more distinctive communities, like Cooking and Naruto, exhibit moderately higher monthly retention rates than more generic communities (Spearman's INLINEFORM2 = 0.33, INLINEFORM3 0.001; Figure FIGREF11 .A, right). As with monthly retention, we find a strong positive relationship between a community's dynamicity and the average number of months that a user will stay in that community (Spearman's INLINEFORM0 = 0.41, INLINEFORM1 0.001, computed over all community points; Figure FIGREF11 .B, left). This verifies that the short-term trend observed for monthly retention translates into longer-term engagement and suggests that long-term user retention might be strongly driven by the extent to which a community continually provides novel content. Interestingly, there is no significant relationship between distinctiveness and long-term engagement (Spearman's INLINEFORM2 = 0.03, INLINEFORM3 0.77; Figure FIGREF11 .B, right). Thus, while highly distinctive communities like RandomActsOfMakeup may generate focused commitment from users over a short period of time, such communities are unlikely to retain long-term users unless they also have sufficiently dynamic content. \n Question: How do the various social phenomena examined manifest in different types of communities?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-77848830cfb04e5397e586199f750ec6",
            "input": "The high nearest neighbours accuracy indicates that syntax information was successfully captured by the embeddings. \n Question: Do they do quantitative quality analysis of learned embeddings?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-fd429218caba49e88b1daea689480e75",
            "input": "We computed Fliess' INLINEFORM0 on the trial set for the five annotators on 21 of the tweets. INLINEFORM1 is .83 for Layer A (OFF vs NOT) indicating high agreement. As to normalization and anonymization, no user metadata or Twitter IDs have been stored, and URLs and Twitter mentions have been substituted to placeholders.  \n Question: How many annotators participated?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-d98f7445767d4b60b62f6f15837d8418",
            "input": "We assess our dataset using traditional and deep learning methods. Our simplest model is a linear SVM trained on word unigrams. SVMs have produced state-of-the-art results for many text classification tasks BIBREF13 . We also train a bidirectional Long Short-Term-Memory (BiLSTM) model, which we adapted from the sentiment analysis system of sentimentSystem,rasooli2018cross and altered to predict offensive labels instead. It consists of (1) an input embedding layer, (2) a bidirectional LSTM layer, (3) an average pooling layer of input features. The concatenation of the LSTM's and average pool layer is passed through a dense layer and the output is passed through a softmax function. We set two input channels for the input embedding layers: pre-trained FastText embeddings BIBREF14 , as well as updatable embeddings learned by the model during training. Finally, we also apply a Convolutional Neural Network (CNN) model based on the architecture of BIBREF15 , using the same multi-channel inputs as the above BiLSTM. \n Question: What models are used in the experiment?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-b9ffb92400e0417584f06d6453bf76ea",
            "input": "o sufficiently utilize the large dataset $\\mathcal {A}$ and $\\mathcal {M}$, the model is pre-trained on CTC-based ASR task and MT task in the pre-training stage. \n Question: What is the attention module pretrained on?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-c547e944b15c4c61b990d9e96643acfb",
            "input": "In this paper we focus on the skipgram approach with random negative examples proposed in BIBREF0 .  \n Question: Do they use skipgram version of word2vec?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-d59bbbf7f1354ad29e8c9f8e0e559131",
            "input": "The contributions of this work are: \n Question: Do they look for inconsistencies between different UD treebanks?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-c94e74c73b004c709136795e90b99a99",
            "input": "For a word belonging to any one of the word-groups representing these concepts, the modified cost term favors an increase for the value of this word's embedding vector dimension corresponding to the concept that the particular word belongs to.  \n Question: Along which dimension do the semantically related words take larger values?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-5540f2d940cc439ca63e2f0221095ac8",
            "input": "The informal setting/environment of social media often encourage multilingual speakers to switch back and forth between languages when speaking or writing. These all resulted in code-mixing and code-switching. Code-mixing refers to the use of linguistic units from different languages in a single utterance or sentence, whereas code-switching refers to the co-occurrence of speech extracts belonging to two different grammatical systemsBIBREF3. This language interchange makes the grammar more complex and thus it becomes tough to handle it by traditional algorithms. Thus the presence of high percentage of code-mixed content in social media text has increased the complexity of the aggression detection task. For example, the dataset provided by the organizers of TRAC-2018 BIBREF0, BIBREF2 is actually a code-mixed dataset. \n Question: What data/studies do the authors provide to support the assertion that the majority of aggressive conversations contain code-mixed languages?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-9fa62162e0684d57b5ce2728daa8cbb4",
            "input": "We have downloaded 1,873 Twitter conversation threads, roughly 14k tweets, from a publicly available resource that were previously pre-processed and have conversation threads extracted. \n Question: How large is the Twitter dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-49524567f74f457c8af6c73be167e0dd",
            "input": "N-GrAM ranked first in all cases except for the language variety task. In this case, the baseline was the top-ranked system, and ours was second by a small margin. Our system significantly out-performed the baseline on the joint task, as the baseline scored significantly lower for the gender task than for the variety task.\n\n \n Question: On which task does do model do worst?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-c4b9e80c4fb5465cb468a4199f27efa1",
            "input": "For each variable in encoding set, learn the new embeddings using the embeddings train set . Since there is stochasticity in the embeddings training model, we will repeat the above multiple times, for the different experiments in the paper (and report the respective mean and standard deviation statistics). \n Question: How do their train their embeddings?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-ebb961af24464a20961a4d04d56db482",
            "input": "Meanwhile, attending colleges in the Northeast, West and South regions increases the possibility of posting about sexual harassment (positive coefficients), over the Midwest region.  \n Question: Which geographical regions correlate to the trend?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-a0a13f4db382431a8229adc7a4cba10a",
            "input": "To assess the predictive capability of this and other models, we require some method by which we can compare the models. For that purpose, we use receiver operating characteristic (ROC) curves as a visual representation of predictive effectiveness. ROC curves compare the true positive rate (TPR) and false positive rate (FPR) of a model's predictions at different threshold levels. The area under the curve (AUC) (between 0 and 1) is a numerical measure, where the higher the AUC is, the better the model performs. We cross-validate our model by first randomly splitting the corpus into a training set (95% of the corpus) and test set (5% of the corpus). We then fit the model to the training set, and use it to predict the response of the documents in the test set. We repeat this process 100 times. The threshold-averaged ROC curve BIBREF13 is found from these predictions, and shown in Figure 3 . Table 1 shows the AUC for each model considered. \n Question: How is performance measured?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-5719a98a1a514ed3b9ceebb435befcb3",
            "input": "Instead of providing an analytic approximation, AEM uses a discriminator network to discriminate between the reconstructed documents from latent events and the original input documents. This essentially helps the generator to construct a more realistic document from a random noise drawn from a Dirichlet distribution. Due to the flexibility of neural networks, the generator is capable of learning complicated nonlinear distributions. And the supervision signal provided by the discriminator will help generator to capture the event-related patterns. \n Question: How does this model overcome the assumption that all words in a document are generated from a single event?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-41136407c4a048a8be4e7ad41759e70f",
            "input": "Results show that the CJFA encoder obtains significantly better phone classification accuracy than the VAE baseline and also than the CJFS encoder. These results are replicated for speaker recognition tasks. \n Question: Which approach out of two proposed in the paper performed better in experiments?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-d55d0d80b94641bca26024f33051bf9f",
            "input": "We found interesting patterns which are learned by our model and help understand these monolingual gains. For example, a recurring pattern is that words in English which are translated to the same word, or to semantically close words, in the target language end up closer together after our transformation. For example, in the case of English-Spanish the following pairs were among the pairs whose similarity increased the most by applying our transformation: cellphone-telephone, movie-film, book-manuscript or rhythm-cadence, which are either translated to the same word in Spanish (i.e., teléfono and película in the first two cases) or are already very close in the Spanish space. More generally, we found that word pairs which move together the most tend to be semantically very similar and belong to the same domain, e.g., car-bicycle, opera-cinema, or snow-ice.\n\n \n Question: Why does the model improve in monolingual spaces as well? ",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-0d2e0aef888849999bf8cd3975d50074",
            "input": "Secondly, texts go through a cascade of annotation tools, enriching it with the following information:\n\nMorphosyntactic interpretations (sets of tags), using Morfeusz 0.82 BIBREF25 ,\n\nTagging (selection of the most probable interpretation), using a transformation-based learning tagger, PANTERA 0.9.1 BIBREF26 ,\n\nSyntactic groups (possibly nested) with syntactic and semantic heads, using a rule-based shallow parser Spejd 1.3.7 BIBREF27 with a Polish grammar, including improved version of modifications by BIBREF28 , enabling lemmatisation of nominal syntactic groups,\n\nNamed entities, using two available tools: NERF 0.1 BIBREF29 and Liner2 2.3 BIBREF30 .\n\n \n Question: How is the data in RAFAEL labelled?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-dcb3bcf5e7c84131838043a6a3df2cc9",
            "input": "To ground moral sentiment in text, we leverage the Moral Foundations Dictionary BIBREF27. The MFD is a psycholinguistic resource that associates each MFT category with a set of seed words, which are words that provide evidence for the corresponding moral category in text. To implement the first tier of our framework and detect moral relevance, we complement our morally relevant seed words with a corresponding set of seed words approximating moral irrelevance based on the notion of valence, i.e., the degree of pleasantness or unpleasantness of a stimulus. We refer to the emotional valence ratings collected by BIBREF28 for approximately 14,000 English words, and choose the words with most neutral valence rating that do not occur in the MFD as our set of morally irrelevant seed words, for an equal total number of morally relevant and morally irrelevant words. We divide historical time into decade-long bins, and use two sets of embeddings provided by BIBREF30, each trained on a different historical corpus of English:\n\nGoogle N-grams BIBREF31: a corpus of $8.5 \\times 10^{11}$ tokens collected from the English literature (Google Books, all-genres) spanning the period 1800-1999.\n\nCOHA BIBREF32: a smaller corpus of $4.1 \\times 10^8$ tokens from works selected so as to be genre-balanced and representative of American English in the period 1810-2009. \n Question: Which datasets are used in the paper?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-fcba5b68300743f184d6f49ebb7c790c",
            "input": "Since the training data consists only of utterance-denotation pairs, the ranker is trained to maximize the log-likelihood of the correct answer $z$ by treating logical forms as a latent variable The role of the ranker is to score the candidate logical forms generated by the parser; at test time, the logical form receiving the highest score will be used for execution. The ranker is a discriminative log-linear model over logical form $y$ given utterance $x$ :  \n Question: How does the model compute the likelihood of executing to the correction semantic denotation?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-1c3faeafcc3442beaf4f6f82343d670d",
            "input": "Since, we there was no publicly available standard Nepali NER dataset and did not receive any dataset from the previous researchers, we had to create our own dataset. This dataset contains the sentences collected from daily newspaper of the year 2015-2016.  \n Question: What is the source of their dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-9c0690d3c71a4e3c85b9bea1dc9782bd",
            "input": "In this work, the metrics detailed below are proposed and we evaluate their quality through a human evaluation in subsection SECREF32. In addition to the automatic metrics, we proceeded to a human evaluation. We chose to use the data from our SQuAD-based experiments in order to also to measure the effectiveness of the proposed approach to derive Curiosity-driven QG data from a standard, non-conversational, QA dataset. We randomly sampled 50 samples from the test set. Three professional English speakers were asked to evaluate the questions generated by: humans (i.e. the reference questions), and models trained using pre-training (PT) or (RL), and all combinations of those methods.\n\nBefore submitting the samples for human evaluation, the questions were shuffled. Ratings were collected on a 1-to-5 likert scale, to measure to what extent the generated questions were: answerable by looking at their context; grammatically correct; how much external knowledge is required to answer; relevant to their context; and, semantically sound. The results of the human evaluation are reported in Table TABREF33. \n Question: How they evaluate quality of generated output?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-f1348c372db24a1689c3bbca9bf442d5",
            "input": "Word Lattice\nAs shown in Figure FIGREF4 , a word lattice is a directed graph INLINEFORM0 , where INLINEFORM1 represents a node set and INLINEFORM2 represents a edge set. For a sentence in Chinese, which is a sequence of Chinese characters INLINEFORM3 , all of its possible substrings that can be considered as words are treated as vertexes, i.e. INLINEFORM4 . Then, all neighbor words are connected by directed edges according to their positions in the original sentence, i.e. INLINEFORM5 . \n Question: How do they obtain word lattices from words?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-8642debc81f74e6e9a929717b9c399cd",
            "input": " In the pyramid scoring, the content units in the gold human written summaries are organized in a pyramid. In this pyramid, the content units are organized in tiers and higher tiers of the pyramid indicate higher importance.  \n Question: What manual Pyramid scores are used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-c5a44263a62b4e22ae1c7d9f8850147f",
            "input": "These datasets are part of SemEval-2016 Challenge Task 5 BIBREF27 , BIBREF28 . Table TABREF7 shows the number of observations in each test corpus. \n Question: what datasets were used in evaluation?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-ce6719e7011042c8a4836767f1620257",
            "input": "The method is evaluated on the BERTbase model, which has 12 layers, 12 self-attention heads with a hidden size of 768.\n\nTo accelerate the training speed, two-phase training BIBREF1 is adopted. The first phase uses a maximal sentence length of 128, and 512 for the second phase. The numbers of training steps of two phases are 50K and 40K for the BERTBase model. We used AdamW BIBREF13 optimizer with a learning rate of 1e-4, a $\\beta _1$ of 0.9, a $\\beta _2$ of 0.999 and a L2 weight decay rate of $0.01$. The first 10% of the total steps are used for learning rate warming up, followed by the linear decay schema. We used a dropout probability of 0.1 on all layers. The data used for pre-training is the same as BERT, i.e., English Wikipedia (2500M words) and BookCorpus (800M words) BIBREF14. \n Question: Do they train their model starting from a checkpoint?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-e56d1ae84cb445b285c50618e6a82109",
            "input": "In this paper, we manually annotate the predicate-argument structures for the 600 L2-L1 pairs as the basis for the semantic analysis of learner Chinese. \n Question: Who manually annotated the semantic roles for the set of learner texts?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-6f3b9436e3d0475eb55e6415b994144e",
            "input": " The method is evaluated on the BERTbase model, which has 12 layers, 12 self-attention heads with a hidden size of 768. \n Question: What BERT model do they test?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-e494a70279914546abae837b2a3a3b53",
            "input": "We evaluate our method on three tagging tasks: POS tagging (Pos), morphological tagging (Morph) and supertagging (Stag). We select these tasks as examples for tagging applications because they differ strongly in tag set sizes. The test results for the three tasks are shown in Table TABREF17 in three groups. The first group of seven columns are the results for Pos, where both LSTM and CNN have three variations of input features: word only ( INLINEFORM0 ), character only ( INLINEFORM1 ) and both ( INLINEFORM2 ). For Morph and Stag, we only use the INLINEFORM3 setting for both LSTM and CNN. \n Question: Do they jointly tackle multiple tagging problems?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-5339d53aa9b8417c991ac29613899891",
            "input": "We characterize the evaluation of Emotionally-Aware Chatbot into two different parts, qualitative and quantitative assessment. Qualitative assessment will focus on assessing the functionality of the software, while quantitative more focus on measure the chatbots' performance with a number. Based on our investigation of several previous studies, we found that most of the works utilized ISO 9241 to assess chatbots' quality by focusing on the usability aspect. This aspect can be grouped into three focuses, including efficiency, effectiveness, and satisfaction, concerning systems' performance to achieve the specified goals. In automatic evaluation, some studies focus on evaluating the system at emotion level BIBREF15 , BIBREF28 . Therefore, some common metrics such as precision, recall, and accuracy are used to measure system performance, compared to the gold label. This evaluation is similar to emotion classification tasks such as previous SemEval 2018 BIBREF32 and SemEval 2019 . Other studies also proposed to use perplexity to evaluate the model at the content level (to determine whether the content is relevant and grammatical) BIBREF14 , BIBREF39 , BIBREF28 . This evaluation metric is widely used to evaluate dialogue-based systems which rely on probabilistic approach BIBREF61 . Another work by BIBREF14 used BLEU to evaluate the machine response and compare against the gold response (the actual response), although using BLEU to measure conversation generation task is not recommended by BIBREF62 due to its low correlation with human judgment. This evaluation involves human judgement to measure the chatbots' performance, based on several criteria. BIBREF15 used three annotators to rate chatbots' response in two criteria, content (scale 0,1,2) and emotion (scale 0,1). Content is focused on measuring whether the response is natural acceptable and could plausible produced by a human. This metric measurement is already adopted and recommended by researchers and conversation challenging tasks, as proposed in BIBREF38 .  \n Question: How are EAC evaluated?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-156d1ad51ec24d478c417480d0e3ac7d",
            "input": "Internally, the ASR system maintains a rich hypothesis space in the form of speech lattices or confusion networks (cnets). \n Question: What is a word confusion network?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-d2c807c6f5954ad3ac9bdee38928e0fe",
            "input": "As we only extracted references to other judicial decisions, we obtained 471,319 references from Supreme Court decisions, 167,237 references from Supreme Administrative Court decisions and 264,463 references from Constitutional Court Decisions. These are numbers of text spans identified as references prior the further processing described in Section SECREF3. \n Question: How big is the dataset?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-651aaeb4f766457d88fc850a5165eea2",
            "input": "Causal documents were chosen to contain one occurrence only of the exact unigrams: `caused', `causing', or `causes'. \n Question: What are the selection criteria for \"causal statements\"?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-2d6593cd081c45959ad3dc7cc1de4fe4",
            "input": "In this section, we first describe a baseline method inspired by the “align to segment” of BIBREF12, BIBREF13. We then propose two extensions providing the model with a signal relevant to the segmentation process, so as to move towards a joint learning of segmentation and alignment. \n Question: Does the paper report any alignment-only baseline?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-cd56dd6ee5d94a27a37e2d54036b3989",
            "input": "We consider pre-training on the audio data (without labels) of WSJ, part of clean Librispeech (about 80h) and full Librispeech as well as a combination of all datasets (§ SECREF7 ).  Our experimental results on the WSJ benchmark demonstrate that pre-trained representations estimated on about 1,000 hours of unlabeled speech can substantially improve a character-based ASR system and outperform the best character-based result in the literature, Deep Speech 2.  \n Question: Which unlabeled data do they pretrain with?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-1509e989296049d2b163ba27e5abfe7a",
            "input": "As a starting point, we used the DIP corpus BIBREF37 , a collection of 49 clusters of 100 web pages on educational topics (e.g. bullying, homeschooling, drugs) with a short description of each topic. \n Question: Which collections of web documents are included in the corpus?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-b0bce1f831d84f1d95d03d1f384ee06e",
            "input": "We first evaluate our converted embedding models on multi-language lexical similarity and relatedness tasks, as a sanity check, to make sure the word sense induction process did not hurt the general performance of the embeddings. Then, we test the sense embeddings on WSD task. \n Question: Was any extrinsic evaluation carried out?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-8defea09c3a14d279c283f3a7a022acf",
            "input": "we follow Lapata2005Automatic to measure coherence as sentence similarity \n Question: Did the authors evaluate their system output for coherence?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-76707e3ccdbd4c87a7527223c051ca72",
            "input": "n this section, we first describe the phenomenon of mention of political parties names in the profile attributes of users. This is followed by the analysis of profiles that make specific mentions of political handles in their profile attributes. Both of these constitute an organic way of showing support to a party and does not involve any direct campaigning by the parties We opine that the whole campaign of adding Chowkidar to the profile attributes show an inorganic behavior, with political leaders acting as the catalyst. We believe, the effect of changing the profile attribute in accordance with Prime Minister's campaign is an example of inorganic behavior contagion BIBREF6, BIBREF9.   However, we argue that the addition of election campaign related keywords to the profile is a form of inorganic behavior.  \n Question: What are the organic and inorganic ways to show political affiliation through profile changes?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-e34d0e60955541d29bc9c9acd61201e8",
            "input": "We evaluate a multi-task approach and three algorithms that explicitly model the task dependencies. \n Question: Will these findings be robust through different datasets and different question answering algorithms?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-2e04897259ab493a9a28751b59ffb068",
            "input": "We use three datasets for our experiments: IWSLT14 German-English, Turkish-English, and WMT14 English-German. \n Question: what datasets were used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-68b53420ade84cc096dba2a93150d789",
            "input": ". Grammatically correct text is needed as the starting point for inserting artificial errors, and we used two different sources: 1) the corrected version of the same FCE training set on which the system is trained (450K tokens), and 2) example sentences extracted from the English Vocabulary Profile (270K tokens). \n Question: Which languages are explored in this paper?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-7b042e77193040bb8f3f90f85211e256",
            "input": "Experimental Setup \n Question: What domains are detected in this paper?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-e0a2304026c243e09b7c0488465eece2",
            "input": "For reliable multi-hop reasoning using OpenIE tuples, we can add inter-tuple connections to the support graph search, controlled by a small number of rules over the OpenIE predicates. Learning such rules for the Science domain is an open problem and potential avenue of future work. \n Question: Can the method answer multi-hop questions?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-9ec1279635bc4ce19135ebfa4cddaabb",
            "input": "Experimental Setup \n Question: which chinese datasets were used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-37ad3cb1a552466eab3bdd6dd1a6e863",
            "input": "Experiment 1: scene variation in modified referring expressions We recruited 58 pairs of participants (116 participants total) over Amazon's Mechanical Turk who were each paid $1.75 for their participation. \n Question: Does the paper describe experiments with real humans?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-9e1ef07f7bac49039e2dd75a93f7a8c4",
            "input": "As it is reported that conservatives and liberals exhibit different behaviors on online social platforms BIBREF19BIBREF20BIBREF21, we further assigned a political bias label to different US outlets (and therefore news articles) following the procedure described in BIBREF2. In order to assess the robustness of our method, we performed classification experiments by training only on left-biased (or right-biased) outlets of both disinformation and mainstream domains and testing on the entire set of sources, as well as excluding particular sources that outweigh the others in terms of samples to avoid over-fitting. \n Question: How is the political bias of different sources included in the model?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-35abb5405c9d463aa2fc60f9bec02be8",
            "input": "The parameters of the entire MSD (auxiliary-task) decoder are shared across languages.\n\nSince a grouping of the languages based on language family would have left several languages in single-member groups (e.g. Russian is the sole representative of the Slavic family), we experiment with random groupings of two to three languages. Multilingual training is performed by randomly alternating between languages for every new minibatch. \n Question: How do they perform multilingual training?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-3098ff7c1fb84efaa714954eb7ca1541",
            "input": "The seed lexicon consists of positive and negative predicates. If the predicate of an extracted event is in the seed lexicon and does not involve complex phenomena like negation, we assign the corresponding polarity score ($+1$ for positive events and $-1$ for negative events) to the event. It is a  \n Question: What is the seed lexicon?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-3c4af27dbeb744fcbd3e586dd27d2777",
            "input": "Four factual attributes pertaining to the 200 colleges are extracted from the U.S. News Statistics, which consists of Undergraduate Enrollment, Male/Female Ratio, Private/Public, and Region (Northeast, South, West, and Midwest).  \n Question: Which major geographical regions are studied?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-7771f537f05841bc85aeea30a17fdb6c",
            "input": "Table TABREF14 shows the effectiveness of our model (multi-task transformer) over the baseline transformer BIBREF8 .  \n Question: what are the baselines?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-42f2a04286144f92941f6595d640f99d",
            "input": "Three baseline classification methods: Support Vector Machines (SVM), Adaboost, and Random Forests are adopted to evaluate our extracted features. On each dataset, the employed classifiers are trained with individual feature first, and then with the combination of the two features. \n Question: LDA is an unsupervised method; is this paper introducing an unsupervised approach to spam detection?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-a7cdb23567754ee9a9ab92a8eb575c27",
            "input": "We use two different unsupervised approaches for word sense disambiguation.  The second, called `dense model', represents synsets and contexts in a dense, low-dimensional space by averaging word embeddings. In the synset embeddings model approach, we follow SenseGram BIBREF14 and apply it to the synsets induced from a graph of synonyms.  We observe that the SenseGram-based approach for word sense disambiguation yields substantially better results in every case (Table TABREF25 ). The primary reason for that is the implicit handling of similar words due to the averaging of dense word vectors for semantically related words.  \n Question: Do the authors offer any hypothesis about why the dense mode outperformed the sparse one?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-cb949162927149868aefebe9ea5c4016",
            "input": "The result on original WikiQA indicates that all three transfer learning methods not only do not improve the results but also hurt the F1-score. These are because other datasets could not add new information to the original dataset or they apparently add some redundant information which are dissimilar to the target dataset. \n Question: Do transferring hurt the performance is the corpora are not related?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-f391e248aa654a63b5a7b32f0ad2ee5d",
            "input": "By choosing English (En) as the pivot language, we perform pivot alignments for identical English segments on Europarl Fr-En and En-De parallel corpora BIBREF18 , constructing a multi-parallel corpus of Fr-En-De. Then each of the Fr*-De and Fr-De* pseudo parallel corpora is established from the multi-parallel data by applying the pivot language-based translation described in the previous section. For automatic translation, we utilize a pre-trained and publicly released NMT model for En $\\rightarrow $ De and train another NMT model for En $\\rightarrow $ Fr using the WMT'15 En-Fr parallel corpus BIBREF19 . \n Question: Where do they collect the synthetic data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-3a8cc1d2688f4f469816f3234965bc06",
            "input": "As discussed in Section SECREF2 , a variety of features that measure salience of an entity in text are available from the NLP community. We reimplemented the ones in Dunietz and Gillick BIBREF11 . This includes a variety of features, e.g. positional features, occurrence frequency and the internal POS structure of the entity and the sentence it occurs in.  \n Question: What features are used to represent the salience and relative authority of entities?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-fb06feeb2ddb48628b15cfb27495e415",
            "input": "Given a multiple-choice question $qa$ with question text $q$ and answer choices A= $\\lbrace a_i\\rbrace $ , we select the most relevant tuples from $T$ and $S$ as follows. Selecting from Tuple KB: We use an inverted index to find the 1,000 tuples that have the most overlapping tokens with question tokens $tok(qa).$ . \n Question: Is an entity linking process used?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-eb17b4e959e247efa140659b0bcc737b",
            "input": "The focus of this section is on recently published datasets and LID research applicable to the South African context. An in depth survey of algorithms, features, datasets, shared tasks and evaluation methods may be found in BIBREF0.\n\nThe datasets for the DSL 2015 & DSL 2017 shared tasks BIBREF1 are often used in LID benchmarks and also available on Kaggle . The recently published JW300 parallel corpus BIBREF2 covers over 300 languages with around 100 thousand parallel sentences per language pair on average. The WiLI-2018 benchmark dataset BIBREF4 for monolingual written natural language identification includes around 1000 paragraphs of 235 languages. The NCHLT text corpora BIBREF7 is likely a good starting point for a shared LID task dataset for the South African languages BIBREF8. \n Question: Which datasets are employed for South African languages LID?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-1306fa880f454d5cba9ec9d667fadd9f",
            "input": "Here I would use Informap algorithm BIBREF12. To make a comparison to this method, I am using CCM and SCA for distance measurement in this experiment, too. UPGMA algorithm would be used accordingly in these two cases. \n Question: Is the proposed method compared to previous methods?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-78d1fc313ff0478cb76d6271fbd6dc89",
            "input": "Similar equations were discovered using Euclidean distance computed between the context vector representations of the equations ( INLINEFORM2 ). We give additional example results in Appendix B. \n Question: How do they define similar equations?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-87dc1901f32f466f8a7b9897ea1fa0d5",
            "input": "Table TABREF20 shows that, on WoZ and DSTC2 datasets, SIM model has the same number of parameters, which is only 23% and 19% of that in GLAD model. \n Question: How do they measure model size?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-1bf27c12465c498b997688270a8e2db8",
            "input": "We carried out two human evaluations using Mechanical Turk to compare the performance of our model and the baseline. \n Question: Is there any human evaluation involved in evaluating this famework?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-8b7066e7659d4a11b5d5adc244aa5cc5",
            "input": " 29,794 The Wikipedia dataset consists of articles from English Wikipedia, with quality class labels assigned by the Wikipedia community. Wikipedia articles are labelled with one of six quality classes, in descending order of quality: Featured Article (“FA”), Good Article (“GA”), B-class Article (“B”), C-class Article (“C”), Start Article (“Start”), and Stub Article (“Stub”). We randomly sampled 5,000 articles from each quality class and removed all redirect pages, resulting in a dataset of 29,794 articles. \n Question: How large is their data set?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-b5974b42f1a5412e92fb42dfd7374b9c",
            "input": "We call the model as a modified Toulmin's model. It contains five argument components, namely claim, premise, backing, rebuttal, and refutation. When annotating a document, any arbitrary token span can be labeled with an argument component; the components do not overlap.  \n Question: What argument components do the ML methods aim to identify?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-e82fb9a6bb434ab5b14957dc25051e56",
            "input": "We see that gated architectures almost always outperform recurrent, attention and linear models BoW, TFIDF, PV. This is largely because while training and testing on same domains, these models especially recurrent and attention based may perform better. However, for Domain Adaptation, as they lack gated structure which is trained in parallel to learn importance, their performance on target domain is poor as compared to gated architectures. As gated architectures are based on convolutions, they exploit parallelization to give significant boost in time complexity as compared to other models. The effectiveness of gated architectures rely on the idea of training a gate with sole purpose of identifying a weightage. In the task of sentiment analysis this weightage corresponds to what weights will lead to a decrement in final loss or in other words, most accurate prediction of sentiment. In doing so, the gate architecture learns which words or n-grams contribute to the sentiment the most, these words or n-grams often co-relate with domain independent words. On the other hand the gate gives less weightage to n-grams which are largely either specific to domain or function word chunks which contribute negligible to the overall sentiment. This is what makes gated architectures effective at Domain Adaptation. \n Question: Does the fact that GCNs can perform well on this tell us that the task is simpler than previously thought?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-9aa5dd44c7434ce6bccdfda3de0c4b9a",
            "input": "Questions are gathered from anonymized, aggregated queries to the Google search engine. Queries that are likely to be yes/no questions are heuristically identified: we found selecting queries where the first word is in a manually constructed set of indicator words and are of sufficient length, to be effective.\n\nQuestions are only kept if a Wikipedia page is returned as one of the first five results, in which case the question and Wikipedia page are given to a human annotator for further processing.\n\nAnnotators label question/article pairs in a three-step process. First, they decide if the question is good, meaning it is comprehensible, unambiguous, and requesting factual information. This judgment is made before the annotator sees the Wikipedia page. Next, for good questions, annotators find a passage within the document that contains enough information to answer the question. Annotators can mark questions as “not answerable\" if the Wikipedia article does not contain the requested information. Finally, annotators mark whether the question's answer is “yes\" or “no\". Annotating data in this manner is quite expensive since annotators need to search entire Wikipedia documents for relevant evidence and read the text carefully. \n Question: how was the dataset built?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-d90c5a10d6694053b5b1bbb942885e09",
            "input": "We also applied our model to MCTest dataset which requires machines to answer multiple-choice reading comprehension questions about fictional stories.  \n Question: Do they experiment with their proposed model on any other dataset other than MovieQA?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-52f2ee969f21466b853d1d2d972788b0",
            "input": "For our study, we consider that a tweet went viral if it was retweeted more than 1000 times. \n Question: What is their definition of tweets going viral?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-4d529a46e4cf4a128bde01dfb67d0927",
            "input": "We further explore the effect of INLINEFORM0 -gram length in our model (i.e., the filter size for the covolutional layers used by the attention parsing module). In Figure FIGREF39 we plot the AUC scores for link prediction on the Cora dataset against varying INLINEFORM1 -gram length.  \n Question: Do they measure how well they perform on longer sequences specifically?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-88b5fa4cd0f94fcb8a7d04729bae4d65",
            "input": "While the mixing strategy compensates for most of the gap between the Fr-De* and the Fr*-De (3.01 $\\rightarrow $ 0.17) in the De $\\rightarrow $ Fr case, the resulting PSEUDOmix still shows lower BLEU than the target-originated Fr-De* corpus. We thus enhance the quality of the synthetic examples of the source-originated Fr*-De data by further training its mother translation model (En $\\rightarrow $ Fr). As illustrated in Figure 2 , with the target-originated Fr-De* corpus being fixed, the quality of the models trained with the source-originated Fr*-De data and PSEUDOmix increases in proportion to the quality of the mother model for the Fr*-De corpus. Eventually, PSEUDOmix shows the highest BLEU, outperforming both Fr*-De and Fr-De* data.  As presented in Table 6 , we observe that fine-tuning using ground truth parallel data brings substantial improvements in the translation qualities of all NMT models. Among all fine-tuned models, PSEUDOmix shows the best performance in all experiments. This is particularly encouraging for the case of De $\\rightarrow $ Fr, where PSEUDOmix reported lower BLEU than the Fr-De* data before it was fine-tuned. Even in the case where PSEUDOmix shows comparable results with other synthetic corpora in the Pseudo Only scenario, it shows higher improvements in the translation quality when fine-tuned with the real parallel data.  \n Question: How many improvements on the French-German translation benchmark?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-32045351ebf14022b0d979fd257ff02f",
            "input": "Given a pair of ultrasound and audio segments we can calculate the distance between them using our model. To predict the synchronisation offset for an utterance, we consider a discretised set of candidate offsets, calculate the average distance for each across utterance segments, and select the one with the minimum average distance.  \n Question: Does their neural network predict a single offset in a recording?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-4330b3795e6046daa887ecdcd9a014db",
            "input": "In this study, we investigate the efficacy of bias reduction during training by introducing a new loss function which encourages the language model to equalize the probabilities of predicting gendered word pairs like he and she.  \n Question: what kinds of male and female words are looked at?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-139c0d19246c4268a794d8e749d87c3d",
            "input": "The second series of tests consisted of using all the documents of the subcorpus “specialist” $E$, because the documents of the subcorpus of Annodis are not identical.  As far as system evaluations are concerned, we use the 78 $E$ documents as reference.  We calculate the precision $P$, the recall $R$ and the $F$-score on the text corpus used in our tests, as follow: \n Question: How is segmentation quality evaluated?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-20edcf6caa27493b8d8fb16503c53804",
            "input": "To systematically evaluate the importance of the clinical sentiment values extracted from the free text in EHRs, we first build a baseline model using the structured features, which are similar to prior studies on readmission risk prediction BIBREF6. \n Question: Do they compare to previous models?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-af3319ee1b504fa5859c7828a482f3b8",
            "input": "Since the ground-truth reference captions in ShapeWorld are randomly sampled, we take the sampled captions accompanying the test images as a proxy for optimal caption diversity, and compare it with the empirical output diversity of the evaluated model on these test images. Practically, we look at language constructions used and compute the corresponding diversity score as the ratio of observed number versus optimal number \n Question: How is diversity measured?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-ecd1c18a739e4068806d0ce40f303b13",
            "input": "This paper used the real-time method to randomly collect 10% of publicly available English tweets using several pre-defined DDEO-related queries (Table TABREF6 ) within a specific time frame.  \n Question: Do they evaluate only on English data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-a4ab90b2bd35417591890c0432f1fa1a",
            "input": "Our system learns Perceptron models BIBREF37 using the Machine Learning machinery provided by the Apache OpenNLP project with our own customized (local and clustering) features.  The local features constitute our baseline system on top of which the clustering features are added. \n Question: what are the baselines?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-5f088dbb513442e28c3ab4226a50b170",
            "input": "Our systems are attentional encoder-decoder networks BIBREF0 . We base our implementation on the dl4mt-tutorial, which we enhanced with new features such as ensemble decoding and pervasive dropout. \n Question: what are the baseline systems?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-b45c8de5eb764948912860a06322320f",
            "input": "Our full dataset consists of all subreddits on Reddit from January 2013 to December 2014, for which there are at least 500 words in the vocabulary used to estimate our measures, in at least 4 months of the subreddit's history. We compute our measures over the comments written by users in a community in time windows of months, for each sufficiently active month, and manually remove communities where the bulk of the contributions are in a foreign language.  \n Question: How did the select the 300 Reddit communities for comparison?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-5e1e11c02ab940b288d6a1fdca0cec52",
            "input": "The word intrusion test is expensive to apply since it requires manual evaluations by human observers separately for each embedding dimension. Furthermore, the word intrusion test does not quantify the interpretability levels of the embedding dimensions, instead it yields a binary decision as to whether a dimension is interpretable or not. However, using continuous values is more adequate than making binary evaluations since interpretability levels may vary gradually across dimensions. \n Question: What advantages does their proposed method of quantifying interpretability have over the human-in-the-loop evaluation they compare to?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-a8511e7bee864fcb9d1387044fcc8ff8",
            "input": " In this work, we develop a technique to rapidly transfer an existing pre-trained model from English to other languages in an energy efficient way BIBREF8. As the first step, we focus on building a bilingual language model (LM) of English and a target language. Starting from a pre-trained English LM, we learn the target language specific parameters (i.e., word embeddings), while keeping the encoder layers of the pre-trained English LM fixed. We then fine-tune both English and target model to obtain the bilingual LM.  \n Question: How is the model transferred to other languages?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-f15a90e2d17e4a88ade62f5b3a003800",
            "input": "We chose Conditional Copy (CC) model as our baseline, which is the best model in Wiseman.  \n Question: What is the strong baseline?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-837cca8b10d84faa90c251c41ca59b6b",
            "input": "In this work, we make use of the widely-recognized state of the art entailment technique - BERT BIBREF18, and train it on three mainstream entailment datasets: MNLI BIBREF19, GLUE RTE BIBREF20, BIBREF21 and FEVER BIBREF22, respectively. We convert all datasets into binary case: “entailment” vs. “non-entailment”, by changing the label “neutral” (if exist in some datasets) into “non-entailment”.\n\nFor our label-fully-unseen setup, we directly apply this pretrained entailment model on the test sets of all $\\textsc {0shot-tc}$ aspects. For label-partially-unseen setup in which we intentionally provide annotated data, we first pretrain BERT on the MNLI/FEVER/RTE, then fine-tune on the provided training data. \n Question: Do they use pretrained models?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-504181ea19ea4c768012bd7b56dc9907",
            "input": " BIBREF3 also use bilingual crowd workers, but the studies supporting the use of crowdsourcing for MT evaluation were performed with older MT systems, and their findings may not carry over to the evaluation of contemporary higher-quality neural machine translation (NMT) systems. In addition, the MT developers to which crowd workers were compared are usually not professional translators.  We hypothesise that an evaluation of sentences in isolation, as applied by BIBREF3, precludes raters from detecting translation errors that become apparent only when inter-sentential context is available, and that they will judge MT quality less favourably when evaluating full documents. BIBREF3 used all source texts of the WMT 2017 Chinese-English test set for their experiments, of which only half were originally written in Chinese;  \n Question: What was the weakness in Hassan et al's evaluation design?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-0dfa2dcdbb964f4eb98148ec1c3c0157",
            "input": "Following titovcrosslingual, we run our experiments on the English (EN) and German (DE) sections of the CoNLL 2009 corpus BIBREF13 , and EN-DE section of the Europarl corpus BIBREF14 . \n Question: Which parallel corpora are used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-fa8aaf8c775e4e598e4e4b55ee72d7de",
            "input": "As a first baseline we use Bag-of-Words, a well-known and robust text representations used in various domains BIBREF21 , combined with a standard shallow classifier, namely, a Support Vector Machine with linear kernel. We used LIBSVM implementation of SVM. \n Question: Which shallow approaches did they experiment with?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-d4aba5d3b9e74cf190ef6466a0fac75b",
            "input": "Probing Methodology and Modeling ::: Task Definition and Modeling ::: Baselines and Sanity Checks.\nWhen creating synthetic datasets, it is important to ensure that systematic biases, or annotation artifacts BIBREF41, are not introduced into the resulting probes and that the target datasets are sufficiently challenging (or good, in the sense of BIBREF42). To test for this, we use several of the MCQA baseline models first introduced in BIBREF0, which take inspiration from the LSTM-based models used in BIBREF43 for NLI and various partial-input baselines based on these models. \n Question: How do they control for annotation artificats?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-1cd3af5ee6f5422da4997bb7f213bccd",
            "input": "In this paper, we propose the first large-scale dataset for QA over social media data.   Table TABREF3 gives an example from our TweetQA dataset. The dataset now includes 10,898 articles, 17,794 tweets, and 13,757 crowdsourced question-answer pairs \n Question: What is the size of this dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-0652d0b0951947619efb0b12cbb20864",
            "input": " We use hyperopt library to select the hyper-parameters on the following values: LSTM layer size (16, 32, 64), dropout ($0.0-0.9$), activation function ($relu$, $selu$, $tanh$), optimizer ($sgd$, $adam$, $rmsprop$) with varying the value of the learning rate (1e-1,..,-5), and batch size (4, 8, 16) \n Question: What activation function do they use in their model?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-a211285190b741e38de2a3351c6b3a8e",
            "input": "For the sake of interpretability, we will also project back coefficients from the embeddings as well as PCA models into the dummy variable space. \n Question: How do their interpret the coefficients?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-49d4e7e7d76342c9804c1e06efef7eed",
            "input": "Since we were also interested in whether argumentation differs across registers, we included four different registers — namely (1) user comments to newswire articles or to blog posts, (2) posts in discussion forums (forum posts), (3) blog posts, and (4) newswire articles.  \n Question: How is the data in the new corpus come sourced?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-52b488289ac74067b66a80708ca0e985",
            "input": "For reliable multi-hop reasoning using OpenIE tuples, we can add inter-tuple connections to the support graph search, controlled by a small number of rules over the OpenIE predicates. Learning such rules for the Science domain is an open problem and potential avenue of future work. \n Question: Is their method capable of multi-hop reasoning?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-acf17f8699f441938d96fab4dbfb8b63",
            "input": "We also observe that the best combination seems to consist in training our model on the original out-of-context dataset and testing it on the in-context pairs. In this configuration we reach an F-score (0.72) only slightly lower than the one reported in BIBREF3 (0.74), and we record the highest Pearson correlation, 0.3 (which is still not strong, compared to BIBREF3 's best run, 0.75).  \n Question: What were the results of the first experiment?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-b94ad00d623044b098f4d81a2d927cad",
            "input": "VQA research began in earnest in late 2014 when the DAQUAR dataset was released BIBREF0 \n Question: From when are many VQA datasets collected?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-71fb1e5eaa1a452688cb93d18bc8c529",
            "input": "As far as we know, all existing systems treat this task as a pipeline of two separate subtasks, i.e., event extraction and temporal relation classification, and they also assume that gold events are given when training the relation classifier BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5. Specifically, they built end-to-end systems that extract events first and then predict temporal relations between them (Fig. FIGREF1). In these pipeline models, event extraction errors will propagate to the relation classification step and cannot be corrected afterwards. Our first contribution is the proposal of a joint model that extracts both events and temporal relations simultaneously (see Fig. FIGREF1). \n Question: Is this the first paper to propose a joint model for event and temporal relation extraction?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-b6e2b8e81ea543b2ae10a6ca4ec038fb",
            "input": "A base model, INLINEFORM0 , is trained on 190 hours ( INLINEFORM1 100K instances) of transcribed speech data. Then, it selects a subset of a 1,700-hour ( INLINEFORM2 1.1M instances) unlabeled dataset. We query labels for the selected subset and incorporate them into training. Learning rates are tuned on a small validation set of 2048 instances. The trained model is then tested on a 156-hour ( INLINEFORM3 100K instances) test set and we report CTC loss, Character Error Rate (CER) and Word Error Rate (WER). \n Question: Which dataset do they use?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-7bcdba37c9f04a68830429e691b60d3e",
            "input": "As a further application of our work, we have carried out a supervised classification task aimed at predicting the degree of harm of an incident directly from the text and the hand-coded features (e.g., external category, medical specialty, location).  We also checked if using our unsupervised content-driven cluster labels as additional features can improve the performance of the supervised classification. \n Question: How are content clusters used to improve the prediction of incident severity?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-c252021648cf410887eb2dab6bb2cf54",
            "input": "The first dataset is an expanded version of the annotated Wikipedia conversations dataset from BIBREF9. This dataset uses carefully-controlled crowdsourced labels, strictly filtered to ensure the conversations are civil up to the moment of a personal attack. This is a useful property for the purposes of model analysis, and hence we focus on this as our primary dataset. However, we are conscious of the possibility that these strict labels may not fully capture the kind of behavior that moderators care about in practice. We therefore introduce a secondary dataset, constructed from the subreddit ChangeMyView (CMV) that does not use post-hoc annotations. Instead, the prediction task is to forecast whether the conversation will be subject to moderator action in the future. Wikipedia data. BIBREF9's `Conversations Gone Awry' dataset consists of 1,270 conversations that took place between Wikipedia editors on publicly accessible talk pages. The conversations are sourced from the WikiConv dataset BIBREF59 and labeled by crowdworkers as either containing a personal attack from within (i.e., hostile behavior by one user in the conversation directed towards another) or remaining civil throughout. Reddit CMV data. The CMV dataset is constructed from conversations collected via the Reddit API. In contrast to the Wikipedia-based dataset, we explicitly avoid the use of post-hoc annotation. Instead, we use as our label whether a conversation eventually had a comment removed by a moderator for violation of Rule 2: “Don't be rude or hostile to other users”. \n Question: What labels for antisocial events are available in datasets?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-0be980c10a404794a532f009209d84ca",
            "input": "We evaluated our model on three benchmark datasets, namely the CNN/DailyMail news highlights dataset BIBREF24, the New York Times Annotated Corpus (NYT; BIBREF25), and XSum BIBREF22. \n Question: What are the datasets used for evaluation?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-763a758bc6954d42b9d3088f2abae328",
            "input": "Automation of cQA forums can be divided into three tasks: question-comment relevance (Task A), question-question relevance (Task B), and question-external comment relevance (Task C). In our cQA tasks, the pair of objects are (question, question) or (question, comment), and the relationship is relevant/irrelevant. For task C, in addition to an original question (oriQ) and an external comment (relC), the question which relC commented on is also given (relQ). To incorporate this extra information, we consider a multitask learning framework which jointly learns to predict the relationships of the three pairs (oriQ/relQ, oriQ/relC, relQ/relC). \n Question: What supplemental tasks are used for multitask learning?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-6bbc947c722a41c0a23a437c1cb3eb7c",
            "input": "We assess the optimal number of topics that need to be specified for the STM analysis. We follow the recommendations of the original STM paper and focus on exclusivity and semantic coherence measures. Highly frequent words in a given topic that do not appear very often in other topics are viewed as making that topic exclusive.  Following BIBREF8 we generate a set of candidate models ranging between 3 and 50 topics. We then plot the exclusivity and semantic coherence (numbers closer to 0 indicate higher coherence), with a linear regression overlaid (Figure FIGREF3 ). Models above the regression line have a “better” exclusivity-semantic coherence trade off. We select the 16-topic model, which has the largest positive residual in the regression fit, and provides higher exclusivity at the same level of semantic coherence. The topic quality is usually evaluated by highest probability words, which is presented in Figure FIGREF4 . \n Question: How are the main international development topics that states raise identified?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-099ebd3631c54a1cb7ac6952a26341d4",
            "input": "The Software Ontology (SWO) BIBREF5 is included because its set of CQs is of substantial size and it was part of Ren et al.'s set of analysed CQs. The CQ sets of Dem@Care BIBREF8 and OntoDT BIBREF9 were included because they were available. CQs for the Stuff BIBREF6 and African Wildlife (AWO) BIBREF7 ontologies were added to the set, because the ontologies were developed by one of the authors (therewith facilitating in-depth domain analysis, if needed), they cover other topics, and are of a different `type' (a tutorial ontology (AWO) and a core ontology (Stuff)), thus contributing to maximising diversity in source selection. \n Question: How many domains of ontologies do they gather data from?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-d99b070428ae4861b817628e6b6100ad",
            "input": "We evaluate the proposed architecture on two publicly available datasets: the Adverse Drug Events (ADE) dataset BIBREF6 and the CoNLL04 dataset BIBREF7. We show that our architecture is able to outperform the current state-of-the-art (SOTA) results on both the NER and RE tasks in the case of ADE. In the case of CoNLL04, our proposed architecture achieves SOTA performance on the NER task and achieves near SOTA performance on the RE task. On both datasets, our results are SOTA when averaging performance across both tasks. \n Question: Do they repot results only on English data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-42604b7d7d6f4dc59d87c72264d7cabe",
            "input": "We evaluate our approach by training an autocomplete system on 500K randomly sampled sentences from Yelp reviews BIBREF6 (see Appendix for details). We quantify the efficiency of a communication scheme $(q_{\\alpha },p_{\\beta })$ by the retention rate of tokens, which is measured as the fraction of tokens that are kept in the keywords. The accuracy of a scheme is measured as the fraction of sentences generated by greedily decoding the model that exactly matches the target sentence. \n Question: How are models evaluated in this human-machine communication game?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-07be0c6219ab4a5492424ab28be5ec37",
            "input": "GPT-2 achieves the highest score and the $n$-gram the lowest. Transformer-XL and the LSTM LM perform in the middle, and at roughly the same level as each other. We report the 12-category accuracy results for all models and human evaluation in Table TABREF14. \n Question: What is the performance of the models on the tasks?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-0faa22e03c4a4bf28600681e5b4d4da4",
            "input": "Adding word alignments in parallel sentences results in small, non significant improvements, even if there is some labeled data available in the source language. This difficulty in showing the usefulness of parallel corpora for SRI may be due to the current assumptions about role alignments, which mean that only a small percentage of roles are aligned. Further analyses reveals that annotating small amounts of data can easily outperform the performance gains obtained by adding large unlabeled dataset as well as adding parallel corpora. \n Question: Overall, does having parallel data improve semantic role induction across multiple languages?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-2e9146ed40f74f168dab09dfec107b8e",
            "input": "We evaluate the sequence-to-sequence QRNN architecture described in SECREF5 on a challenging neural machine translation task, IWSLT German-English spoken-domain translation, applying fully character-level segmentation.  \n Question: What languages pairs are used in machine translation?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-f96ea0398aab4fa198b53e6bdcd3c65a",
            "input": "The dataset MSParS is published by NLPCC 2019 evaluation task. The whole dataset consists of 81,826 samples annotated by native English speakers. 80% of them are used as training set. 10% of them are used as validation set while the rest is used as test set. 3000 hard samples are selected from the test set. Metric for this dataset is the exactly matching accuracy on both full test set and hard test subset. Each sample is composed of the question, the logical form, the parameters(entity/value/type) and question type as the Table TABREF3 demonstrates. \n Question: Does the training dataset provide logical form supervision?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-c064083381cc4ee4b23f8b1953f39e53",
            "input": "Multi-task Pairwise Neural Ranking\nWe propose a multi-task pairwise neural ranking approach to better incorporate and distinguish the relative order between the candidate segmentations of a given hashtag. Our model adapts to address single- and multi-token hashtags differently via a multi-task learning strategy without requiring additional annotations. In this section, we describe the task setup and three variants of pairwise neural ranking models (Figure FIGREF11 ). Pairwise Neural Ranking Model Margin Ranking (MR) Loss Adaptive Multi-task Learning \n Question: What set of approaches to hashtag segmentation are proposed?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-23daa65b033c4138aaf31637e628dce5",
            "input": " We defined a sequence labeling task to extract custom entities from user input. We assumed seven (7) possible entities (see Table TABREF43) to be recognized by the model: topic, subtopic, examination mode and level, question number, intent, as well as the entity other for remaining words in the utterance.   We defined a classification problem to predict the system's next action according to the given user input. We assumed 13 custom actions (see Table TABREF42) that we considered being our labels. In the conversational dataset, each input was automatically labeled by the rule-based system with the corresponding next action and the dialogue-id. Thus, no additional post-labeling was required.  \n Question: How does the IPA label data after interacting with users?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-a81d16401ab44c50b99d5e7d4db50c5c",
            "input": "The data stems from a joint ADAPT-Microsoft project.  \n Question: what dataset was used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-8fec1bcc4bab4550be0c30277e319432",
            "input": "We compared our model with MLE, RL and GAN baselines. Since COCO and EMNLP2017 WMT don't have input while WeiboDial regards posts as input, we chose the following baselines respectively:\n\nMLE: a RNN model trained with MLE objective BIBREF4 . Its extension, Seq2Seq, can work on the dialogue dataset BIBREF2 .\n\nSeqGAN: The first text GAN model that updates the generator with policy gradient based on the rewards from the discriminator BIBREF7 .\n\nLeakGAN: A variant of SeqGAN that provides rewards based on the leaked information of the discriminator for the generator BIBREF11 .\n\nMaliGAN: A variant of SeqGAN that optimizes the generator with a normalized maximum likelihood objective BIBREF8 .\n\nIRL: This inverse reinforcement learning method replaces the discriminator with a reward approximator to provide dense rewards BIBREF12 .\n\nRAML: A RL approach to incorporate MLE objective into RL training framework, which regards BLEU as rewards BIBREF17 .\n\nDialogGAN: An extension of SeqGAN tuned to dialogue generation task with MLE objective added to the adversarial objective BIBREF16 .\n\nDPGAN: A variant of DialogGAN which uses a language model based discriminator and regards cross-entropy as rewards BIBREF13 . \n Question: What GAN models were used as baselines to compare against?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-a4eb7c09c9294c57bf8d53585bdf678b",
            "input": "We first pre-train our generator on the Gutenberg dataset BIBREF24 for 20 epochs and then fine-tune BIBREF19 them to our target datasets with a language modeling objective. \n Question: What objective function is used in the GAN?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-5186c53a48424709b70272426e9b9fc4",
            "input": "Wikilinks can be seen as a large-scale, naturally-occurring, crowd-sourced dataset where thousands of human annotators provide ground truths for mentions of interest. This means that the dataset contains various kinds of noise, especially due to incoherent contexts. We prepare our dataset from the local-context version of Wikilinks, and resolve ground-truth links using a Wikipedia dump from April 2016. We use the page and redirect tables for resolution, and keep the database pageid column as a unique identifier for Wikipedia entities. We discard mentions where the ground-truth could not be resolved (only 3% of mentions). \n Question: How was a quality control performed so that the text is noisy but the annotations are accurate?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-ad74ff5fe3e444b0828cabc0f53884c4",
            "input": "Finally, we transform text to be classified into scalars representing their distance from the constructed hate vector and use these as input to a Random Forest classifier. \n Question: What classifier did they use?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-c988ef8003d24d0eb3ed2f4689a1fac6",
            "input": "To show the effectiveness of our approach, we show results on the SICK dataset BIBREF1, a common benchmark for logic-based NLI, and find MonaLog to be competitive with more complicated logic-based approaches (many of which require full semantic parsing and more complex logical machinery). \n Question: Do they beat current state-of-the-art on SICK?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-8630fa0ce3da4d5582c1df7021f1a6bb",
            "input": "Below, we explore different ways to back off when the ScRNN predicts UNK (a frequent outcome for rare and unseen words):\n\nPass-through: word-recognizer passes on the (possibly misspelled) word as is.\n\nBackoff to neutral word: Alternatively, noting that passing $\\colorbox {gray!20}{\\texttt {UNK}}$ -predicted words through unchanged exposes the downstream model to potentially corrupted text, we consider backing off to a neutral word like `a', which has a similar distribution across classes.\n\nBackoff to background model: We also consider falling back upon a more generic word recognition model trained upon a larger, less-specialized corpus whenever the foreground word recognition model predicts UNK. Figure 1 depicts this scenario pictorially. \n Question: How do the backoff strategies work?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-e5fe3a17b32346c69e7e6cfc0d1622ad",
            "input": "In a recent work on sentiment analysis in Turkish BIBREF10, they learn embeddings using Turkish social media. They use the word2vec algorithm, create several unsupervised hand-crafted features, generate document vectors and feed them as input into the support vector machines (SVM) approach. We outperform this baseline approach using more effective word embeddings and supervised hand-crafted features. \n Question: What baseline method is used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-0d2e46c2f230453c8f278fc40ca2ec4d",
            "input": "Our dataset is annotated based on Chinese pathology reports provided by the Department of Gastrointestinal Surgery, Ruijin Hospital. It contains 17,833 sentences, 826,987 characters and 2,714 question-answer pairs. \n Question: How many questions are in the dataset?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-8046ff913c3e404c85b93cba21405a31",
            "input": "Specifically, we group the models based on the objective function it optimizes. We believe this work can aid the understanding of the existing literature.  We believe that the performance of these models is highly dependent on the objective function it optimizes - predicting adjacent word (within-tweet relationships), adjacent tweet (inter-tweet relationships), the tweet itself (autoencoder), modeling from structured resources like paraphrase databases and weak supervision.  \n Question: How do they encourage understanding of literature as part of their objective function?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-a720c1947b414e998b3f9fbd65cd5310",
            "input": "In ConvE, only $v_h$ and $v_r$ are reshaped and then concatenated into an input matrix which is fed to the convolution layer \n Question: Did the authors try stacking multiple convolutional layers?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-f2a4c22ee2e34dc49f0afc539c39bb9b",
            "input": "The policies trained with the NUS achieved an average success rate (SR) of 94.0% and of 96.6% when tested on the ABUS and the NUS, respectively. \n Question: by how much did nus outperform abus?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-25586b3475254234979065fcbc55e2ab",
            "input": "More than 2,100 texts were paired with 15 questions each, resulting in a total number of approx. 32,000 annotated questions. For 13% of the questions, the workers did not agree on one of the 4 categories with a 3 out of 5 majority, so we did not include these questions in our dataset.\n\nThe distribution of category labels on the remaining 87% is shown in Table TABREF10 . 14,074 (52%) questions could be answered. Out of the answerable questions, 10,160 could be answered from the text directly (text-based) and 3,914 questions required the use of commonsense knowledge (script-based). After removing 135 questions during the validation, the final dataset comprises 13,939 questions, 3,827 of which require commonsense knowledge (i.e. 27.4%). This ratio was manually verified based on a random sample of questions. \n Question: what dataset statistics are provided?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-769b78a3a3614037b06ba6ac182bd646",
            "input": "As indicated in its name, Recurrent Deep Stacking Network stacks and concatenates the outputs of previous frames into the input features of the current frame. \n Question: What does recurrent deep stacking network do?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-db3303370a1a4c3abca1e169010edffc",
            "input": "Subsequently, an LSTM decoder generates the characters in the output word form using encoder states and an attention mechanism. \n Question: What architecture does the decoder have?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-db5c9ddd30694ba2b760d6629a5fae74",
            "input": "Our unsupervised ranking model outperforms the supervised IMS system by 1.02% on the CoNLL F1 score, and achieves competitive performance with the latent tree model. Moreover, our approach considerably narrows the gap to other supervised systems listed in Table 3 . \n Question: Is the model presented in the paper state of the art?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-fbd403d42b7f411181fc9f82d3da60cf",
            "input": "Attention-base translation model We use the system of BIBREF6 , a convolutional sequence to sequence model. \n Question: Which translation system do they use to translate to English?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-deee6f5bc5ae47fdb6384dabae4eaa5c",
            "input": "In order to achieve these configuration the TIMIT data was split. Fig. FIGREF12 illustrates the split of the data into 8 subsets (A-H). The TIMIT dataset contains speech from 462 speakers in training and 168 speakers in the test set, with 8 utterances for each speaker. The TIMIT training and test set are split into 8 blocks, where each block contains 2 utterances per speaker, randomly chosen. Thus each block A,B,C,D contains data from 462 speakers with 924 utterances taken from the training sets, and each block E,F,G,H contains speech from 168 test set speakers with 336 utterances.\n\nFor Task a training of embeddings and the classifier is identical, namely consisting of data from blocks (A+B+C+E+F+G). The test data is the remainder, namely blocks (D+H). For Task b the training of embeddings and classifiers uses (A+B+E+F) and (C+G) respectively, while again using (D+H) for test. Task c keeps both separate: embeddings are trained on (A+B+C+D), classifiers on (E+G) and tests are conducted on (F+H). Note that H is part of all tasks, and that Task c is considerably easier as the number of speakers to separate is only 168, although training conditions are more difficult. \n Question: What TIMIT datasets are used for testing?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-207cc81ce6f742d386ad739eb698dd3f",
            "input": "machine comprehension  Nelufar  \n Question: What MC abbreviate for?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-8a668a6d54a94c5faa2d95b7e73b6b8c",
            "input": "We compare classification and regression approaches and show that classification produces better results than regression but the quality of the results depends on the approach followed to annotate the data labels. \n Question: Did classification models perform better than previous regression one?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-58c99558a9d645139389d95ba3244a44",
            "input": "In this paper, we introduce automatic `drunk-texting prediction' as a computational task. Given a tweet, the goal is to automatically identify if it was written by a drunk user.  \n Question: Do the authors equate drunk tweeting with drunk texting? ",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-84d73a526c634044afaa48dbc0e4bb7f",
            "input": "We present two use cases on which Seshat was developped: clinical interviews, and daylong child-centered recordings. \n Question: Did they experiment with the tool?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-0a422636cd55401789637d93b4e24d54",
            "input": "Results of the DQN-based agent are presented in fig: scenario comparison. Each plot depicts the average reward (across 5 seeds) of all representations methods. It can be seen that the NLP representation outperforms the other methods.  NLP representations remain robust to changes in the environment as well as task-nuisances in the state.  \n Question: What result from experiments suggest that natural language based agents are more robust?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-39d7a83f9909416584617a4275e3b12d",
            "input": "We compile three Chinese text corpora from online data for three domains, namely, “hotel\", “mobile phone (mobile)\", and “travel\". All texts are about user reviews.  \n Question: What are the sources of the data?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-3556b9c67a144210ba23f2aec4bb75f2",
            "input": "Posts in FBFans dataset are used for this analysis. We calculate the like statistics of each distinct author from these 32,595 posts. \n Question: What is the size of the Chinese data?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-b12cde0e8ea54f6bbf79f0f18c80ba24",
            "input": "In Figure FIGREF32 , we visualize the syntactic distance estimated by the Parsing Network, while reading three different sequences from the PTB test set. We observe that the syntactic distance tends to be higher between the last character of a word and a space, which is a reasonable breakpoint to separate between words.  The model autonomously discovered to avoid inter-word attention connection, and use the hidden states of space (separator) tokens to summarize previous information. This is strong proof that the model can understand the latent structure of data. \n Question: How do they show their model discovers underlying syntactic structure?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-f7b56e8d3c6f446f88a70a49eabaa431",
            "input": "We first use state-of-the-art PDTB taggers for our baseline BIBREF13 , BIBREF12 for the evaluation of the causality prediction of our models ( BIBREF12 requires sentences extracted from the text as its input, so we used our parser to extract sentences from the message). \n Question: What baselines did they consider?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-3d644004cfa045d19c87edd03981f424",
            "input": "The annotator carried out all annotation. \n Question: How many annotators tagged each tweet?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-1b520d1294314e589802d5cfb4c72205",
            "input": "In future work we also intend to add these types of studies to the ERP predictions.\n\nDiscussion \n Question: Which two pairs of ERPs from the literature benefit from joint training?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-5e03eb82c6e349d1a6005789befe2c4b",
            "input": "We introduce our proposed diversity, density, and homogeneity metrics with their detailed formulations and key intuitions. \n Question: Did they propose other metrics?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-329398820c05488a8945c3c3d40c26ca",
            "input": "Looking to other target tasks, the grammar-related CoLA task benefits dramatically from ELMo pretraining: The best result without language model pretraining is less than half the result achieved with such pretraining. In contrast, the meaning-oriented textual similarity benchmark STS sees good results with several kinds of pretraining, but does not benefit substantially from the use of ELMo. \n Question: Do some pretraining objectives perform better than others for sentence level understanding tasks?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-bf264ce72b294d6b8a0cffc786bf8b04",
            "input": "The crowdworkers were located in the US and hired on the BIBREF22 platform. \n Question: Who are the crowdworkers?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-25b29e0c89d1487988334565fe65a6bf",
            "input": "As shown in Table TABREF3 , we collect Amazon review keywords for 2,896 e-books (publishers: Kiwi, Rowohlt, Fischer, and Droemer), which leads to 33,663 distinct review keywords and on average 30 keyword assignments per e-book. \n Question: how large is the vocabulary?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-456fc16730db42c7b2715b201ac94c1f",
            "input": "We collected Japanese-Vietnamese parallel data from TED talks extracted from WIT3's corpus BIBREF15 .  \n Question: did they collect their own data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-5f0bb594e009487599a5cd9c79db86b2",
            "input": "We describe our rules for WikiSQL here. Our rule for KBQA is simple without using a curated mapping dictionary. The pipeline of rules in SequentialQA is similar to that of WikiSQL. \n Question: Are the rules dataset specific?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-568e311093394b3fa20c794c5365854e",
            "input": "Following previous studies BIBREF1, we collect event-related microposts from Twitter using 11 and 8 seed events (see Section SECREF2) for CyberAttack and PoliticianDeath, respectively. Unlabeled microposts are collected by using the keyword `hack' for CyberAttack, while for PoliticianDeath, we use a set of keywords related to `politician' and `death' (such as `bureaucrat', `dead' etc.) \n Question: Do they report results only on English data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-54cb4ec6e5f24f939d3783e8800d9544",
            "input": "On the same architecture, our RCRN outperforms ablative baselines BiLSTM by INLINEFORM2 and 3L-BiLSTM by INLINEFORM3 on average across 16 datasets. \n Question: By how much do they outperform BiLSTMs in Sentiment Analysis?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-faa649dc70314368844833c38bd55aeb",
            "input": "For annotation purposes, we created snippets of conversations as the ones shown in Example 1 and Example 2 consisting of the parent of the suspected trolling event, the suspected trolling event comment, and all of the direct responses to the suspected trolling event. However, the trade off is that snippets like this allow us to make use of Amazon Mechanical Turk (AMT) to have the dataset annotated, because it is not a big burden for a “turker” to work on an individual snippet in exchange for a small pay, and expedites the annotation process by distributing it over dozens of people. Specifically, for each snippet, we requested three annotators to label the four aspects previously described. \n Question: how was annotation done?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-370364aae5234e5aa6c6c60b03f332eb",
            "input": "Los resultados de la evaluación se presentan en la Tabla TABREF42, en la forma de promedios normalizados entre [0,1] y de su desviación estándar $\\sigma $. \n Question: What evaluation metrics did they look at?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-f5538a6ea91e46cc97f6f30543ebdf51",
            "input": "Data was collected from a 10% uniform sample of Twitter posts made during 2013, specifically the Gardenhose API. Control documents were also selected. These documents did not contain any of `caused', `causing', or `causes', nor any bidirectional words, and are further matched temporally to obtain the same number of control documents as causal documents in each fifteen-minute period during 2013. Control documents were otherwise selected randomly; causal synonyms may be present.  \n Question: How do they collect the control corpus?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-a74cd5dcb52d4808878e2232c06e7e8b",
            "input": "In order to evaluate the models trained on generated data, we manually annotated a named entities dataset comprising 53453 tokens and 2566 sentences selected from over 250 news texts from ilur.am. During annotation, we generally relied on categories and guidelines assembled by BBN Technologies for TREC 2002 question answering track. \n Question: did they use a crowdsourcing platform for manual annotations?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-bc946352cedd476aae2aee553ee457cf",
            "input": "The Gallup survey asked for opinions on legality of “homosexual relations\" until 2008, but then changed the wording to “gay and lesbian relations\". This was likely because many people who identify as gay and lesbian find the word homosexual to be outdated and derogatory. \n Question: Do they analyze specific derogatory words?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-87929085eee34d2898f24f35624ce2f6",
            "input": "For each qualifying diagnosis tweet we retrieve the timeline of the corresponding Twitter user using the Twitter user_timeline API endpoint . Subsequently, we remove all non-English tweets (Twitter API machine-detected“lang” field), all retweets, and tweets that contain “diagnos*” or “depress*”, but not a valid diagnosis statement. The resulting Depressed cohort contains 1,207 individuals and 1,759,644 tweets ranging from from May 2008 to September 2018. \n Question: Do they report results only on English datasets?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-eebb89a0c4ce424aa48390530ec0897e",
            "input": "Three different datasets have been used to train our models: the Toronto book corpus, Wikipedia sentences and tweets.  Our Sent2Vec models also on average outperform or are at par with the C-PHRASE model, despite significantly lagging behind on the STS 2014 WordNet and News subtasks. This observation can be attributed to the fact that a big chunk of the data that the C-PHRASE model is trained on comes from English Wikipedia, helping it to perform well on datasets involving definition and news items.  \n Question: Do they report results only on English data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-2c65f3c84f43497d97844314692f6f88",
            "input": "Secondly, a well-known problem of uneven yet unknown distribution of word senses is alleviated by modifying a naïve Bayesian classifier. Thanks to this correction, the classifier is no longer biased towards senses that have more training data. Finally, the aggregated feature values corresponding to target word senses are used to build a naïve Bayesian classifier adjusted to a situation of unknown a priori probabilities. \n Question: How do they deal with unknown distribution senses?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-fae4853c166b4d0095e773230ff25ada",
            "input": "The question is encoded with a 1024-dimensional LSTM model that takes in a one-hot descriptor of each word in the question. The image is described with the 4096-dimensional output from the last fully connected layer of the Convolutional Neural Network (CNN), VGG16 BIBREF25 . The system performs an element-wise multiplication of the image and question features, after linearly transforming the image descriptor to 1024 dimensions. The final layer of the architecture is a softmax layer. \n Question: What is the model architecture used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-52b987fb62c74546b78b5aa72c3f8853",
            "input": "We asked for two distinct paraphrases of each sentence because we believe that a good sentence embedding should put paraphrases close together in vector space.\n\nSeveral modification types were specifically selected to constitute a thorough test of embeddings. In different meaning, the annotators should create a sentence with some other meaning using the same words as the original sentence. Other transformations which should be difficult for embeddings include minimal change, in which the sentence meaning should be significantly changed by using only very small modification, or nonsense, in which words of the source sentence should be shuffled so that it is grammatically correct, but without any sense. \n Question: What annotations are available in the dataset?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-4fbbd86b5a2541acadb44bdf6d010c0c",
            "input": "The overall Total Accuracy score reported in table TABREF19 using the entire feature set is 549.  \n Question: Do they experiment with the dataset?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-314984f81d8b4050ba9a69d3098aba34",
            "input": "While it is obvious that our embeddings can be used as features for new predictive models, it is also very easy to incorporate our learned Dolores embeddings into existing predictive models on knowledge graphs. The only requirement is that the model accepts as input, an embedding layer (for entities and relations). If a model fulfills this requirement (which a large number of neural models on knowledge graphs do), we can just use Dolores embeddings as a drop-in replacement. We just initialize the corresponding embedding layer with Dolores embeddings. In our evaluation below, we show how to improve several state-of-the-art models on various tasks simply by incorporating Dolores as a drop-in replacement to the original embedding layer. \n Question: How are meaningful chains in the graph selected?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-f2854f74c6284f4caccfee157d04977e",
            "input": "BLEU: We use the Bilingual Evaluation Understudy (BLEU) BIBREF34 metric which is commonly used in machine translation tasks. The BLEU metric can be used to evaluate dialogue generation models as in BIBREF5, BIBREF35. The BLEU metric is a word-overlap metric which computes the co-occurrence of N-grams in the reference and the generated response and also applies the brevity penalty which tries to penalize far too short responses which are usually not desired in task-oriented chatbots. We compute the BLEU score using all generated responses of our systems.\n\nPer-turn Accuracy: Per-turn accuracy measures the similarity of the system generated response versus the target response. Eric and Manning eric2017copy used this metric to evaluate their systems in which they considered their response to be correct if all tokens in the system generated response matched the corresponding token in the target response. This metric is a little bit harsh, and the results may be low since all the tokens in the generated response have to be exactly in the same position as in the target response.\n\nPer-Dialogue Accuracy: We calculate per-dialogue accuracy as used in BIBREF8, BIBREF5. For this metric, we consider all the system generated responses and compare them to the target responses. A dialogue is considered to be true if all the turns in the system generated responses match the corresponding turns in the target responses. Note that this is a very strict metric in which all the utterances in the dialogue should be the same as the target and in the right order.\n\nF1-Entity Score: Datasets used in task-oriented chores have a set of entities which represent user preferences. For example, in the restaurant domain chatbots common entities are meal, restaurant name, date, time and the number of people (these are usually the required entities which are crucial for making reservations, but there could be optional entities such as location or rating). Each target response has a set of entities which the system asks or informs the user about. Our models have to be able to discern these specific entities and inject them into the generated response. To evaluate our models we could use named-entity recognition evaluation metrics BIBREF36. The F1 score is the most commonly used metric used for the evaluation of named-entity recognition models which is the harmonic average of precision and recall of the model. We calculate this metric by micro-averaging over all the system generated responses. \n Question: Is human evaluation performed?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-d7e5f90f5cfd45e7a252733577868661",
            "input": "In addition, in order to estimate annotation reliability and provide for better evaluation, every question in the test set is answered by at least two additional experts. \n Question: Are the answers double (and not triple) annotated?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-6bc1dec7fe2c48fdb687da1763067c2d",
            "input": "We carried out our experiments on the Universal Dependencies v1.2 treebanks BIBREF21 , hereafter UD1.2, from which morphosyntactically annotated corpora can be trivially extracted.  We considered UD1.2 corpora for the following 16 languages: Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish and Swedish.  \n Question: which datasets did they experiment with?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-a4c84da4eeb045a79447dd7227393a99",
            "input": "The main reason is that when the controllability is strong, the change of selection will directly affect the text realization so that a tiny error of content selection might lead to unrealistic text. If the selector is not perfectly trained, the fluency will inevitably be influenced. When the controllability is weaker, like in RS, the fluency is more stable because it will not be affected much by the selection mask. \n Question: Does the performance necessarily drop when more control is desired?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-94483d7def7f452fa3f4a6a4ec0c3543",
            "input": " We use standard Precision, Recall and F1 at mention level (Micro) and at the document level (Macro) as measurements. We can see the average linking precision (Micro) of WW is lower than that of TAC2010, and NCEL outperforms all baseline methods in both easy and hard cases. \n Question: How do they verify generalization ability?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-5263f8f709ea49709354ab03860bc725",
            "input": "Practical evaluation of GTD is currently only possible on synthetic data. We construct a range of datasets designed for image captioning evaluation. We call this diagnostic evaluation benchmark ShapeWorldICE (ShapeWorld for Image Captioning Evaluation). We illustrate the evaluation of specific image captioning models on ShapeWorldICE. \n Question: Are the images from a specific domain?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-9acc4fd137fa4fafb411cc20b1bcd788",
            "input": "Further, FSDM has a new module called response slot binary classifier that adds extra supervision to generate the slots that will be present in the response more precisely before generating the final textual agent response (see Section \"Methodology\" for details). \n Question: How do slot binary classifiers improve performance?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-f65d1ea7a1ac4f409cff7ce39b7d0c86",
            "input": "Given a news Twitter account, we read its tweets from the account's timeline. Then we sort the tweets by the posting date in ascending way and we split them into $N$ chunks. Each chunk consists of a sorted sequence of tweets labeled by the label of its corresponding account. \n Question: Was the approach used in this work to detect fake news fully supervised?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-c2717930735f4f4590ac662cfa4c594d",
            "input": "For each model, we examined word-level perplexity, R@3 in next-word prediction, latency (ms/q), and energy usage (mJ/q). To explore the perplexity-recall relationship, we collected individual perplexity and recall statistics for each sentence in the test set. \n Question: What aspects have been compared between various language models?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-7a40d6ac207c4b7ba83ea4b55e902bf5",
            "input": "Table TABREF15 shows a comparison of the results on SimCluster versus K-means algorithm. Here our SimCluster algorithm improves the F1-scores from 0.412 and 0.417 in the two domains to 0.442 and 0.441. The ARI scores also improve from 0.176 and 0.180 to 0.203 and 0.204. \n Question: Do they use the same distance metric for both the SimCluster and K-means algorithm?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-66a65bab463943cca54a1235a81ef216",
            "input": "We consider two types of architectures: a bi-directional language model to augment the sequence to sequence encoder and a uni-directional model to augment the decoder. Both use self-attention BIBREF16 and the uni-directional model contains INLINEFORM0 transformer blocks, followed by a word classifier to predict the next word on the right. bi-directional language model to augment the sequence to sequence encoder \n Question: What language model architectures are used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-371fefbbca1a47f181cc6f615082cd9d",
            "input": "We extracted 200 sentence pairs from BIBREF3 's dataset and provided each pair with a document context consisting of a preceding and a following sentence, as in the following example. \n Question: What document context was added?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-8c008260bc6442f69759259399cd2ef4",
            "input": "The GENIA Corpus BIBREF3 contains biomedical abstracts from the PubMed database. We use GENIA technical term annotations 3.02, which cover linguistic expressions to entities of interest in molecular biology, e.g. proteins, genes and cells. CoNLL2003 BIBREF14 is a standard NER dataset based on the Reuters RCV-1 news corpus. It covers named entities of type person, location, organization and misc.\n\nFor testing the overall annotation performance, we utilize CoNLL2003-testA and a 50 document split from GENIA.  \n Question: what standard dataset were used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-d4061f09520a44089269ac4b9b983531",
            "input": "The incomplete dataset used for training is composed of lower-cased incomplete data obtained by manipulating the original corpora. \n Question: Do they test their approach on a dataset without incomplete data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-46bfc01d837b43e5a63ef04a1b2d1a4f",
            "input": "These actions consist of the following components:\n\n[leftmargin=*]\n\nCo-Reference Resolution: To support multi-turn interactions, it is sometimes necessary to use co-reference resolution techniques for effective retrieval. Query Generation: This component generates a query based on the past user-system interactions. Retrieval Model: This is the core ranking component that retrieves documents or passages from a large collection. Result Generation: The retrieved documents can be too long to be presented using some interfaces. \n Question: What are the different modules in Macaw?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-701e871de0ba40aa901cc7a1448d78a6",
            "input": "We evaluated our models on 3 different datasets:\n\nCSAT dataset for CSAT prediction, consisting of spoken transcripts (automatic via ASR).\n\n20 newsgroups for topic identification task, consisting of written text;\n\nFisher Phase 1 corpus for topic identification task, consisting of spoken transcripts (manual); \n Question: What datasets did they use for evaluation?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-407f91f421f14b37a961b9fac7ffc26c",
            "input": "The straight line in figures FIGREF39, FIGREF43 and FIGREF51 is the result of a supervised LDA algorithm which is used as a baseline.  \n Question: Was performance of the weakly-supervised model compared to the performance of a supervised model?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-2035117ccdb34af089b74cdbeb40cb58",
            "input": "We trained word embeddings using either GloVe BIBREF11 or SGNS BIBREF12 on a small or a large corpus. \n Question: What types of word representations are they evaluating?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-5d99b5b172b84f18a94b0a3d9c9fdf5b",
            "input": "The basic concept is to use a phone-discriminative model to produce frame-level phonetic features, and then use these features to enhance RNN LID systems that were originally built with raw acoustic features. The initial step is therefore feature combination, with the phonetic feature used as auxiliary information to assist acoustic RNN LID. This is improved further, as additional research identified that a simpler model using only the phonetic feature as the RNN LID input provides even better performance. We call this RNN model based on phonetic features the phonetic temporal neural LID approach, or PTN LID. As well as having a simplified model structure, the PTN offers deeper insight into the LID task by rediscovering the value of the phonetic temporal property in language discrimination.  \n Question: What is the main contribution of the paper? ",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-688fe7208a1a47ccbba1915b46475130",
            "input": "SIGHAN Bakeoff defines two types of evaluation settings, closed test limits all the data for learning should not be beyond the given training set, while open test does not take this limitation BIBREF21. \n Question: What is meant by closed test setting?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-90e623a8031d46e4a0d97db4ea0eac8e",
            "input": "MTMSN BIBREF4 is the first, and only model so far, that specifically tried to tackle the multi-span questions of DROP. Their approach consisted of two parts. The first was to train a dedicated categorical variable to predict the number of spans to extract. The second was to generalize the single-span head method of extracting a span, by utilizing the non-maximum suppression (NMS) algorithm BIBREF7 to find the most probable set of non-overlapping spans. The number of spans to extract was determined by the aforementioned categorical variable \n Question: What approach did previous models use for multi-span questions?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-3a621ce513dd4dd3bd7224f46182f908",
            "input": "We collect more human-written scenes for each concept-set in dev and test set through crowd-sourcing via the Amazon Mechanical Turk platform. Each input concept-set is annotated by at least three different humans. The annotators are also required to give sentences as the rationales, which further encourage them to use common sense in creating their scenes. \n Question: Are the rationales generated after the sentences were written?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-07aaf4b289ad4b7287e3135c5f40e698",
            "input": "We also run a manual evaluation which shows for the En-It task a slight quality degradation in exchange of a statistically significant reduction in the average length ratio, from 1.05 to 1.01. \n Question: Do they conduct any human evaluation?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-db742666b7c14ff49aa28c3c75abdf48",
            "input": "We employ BIBREF73 's weighted lexicon of terms that uses the dataset of 75,394 Facebook users who shared their status, age and gender. Utilizing these two weighted lexicon of terms, we are predicting the demographic information (age or gender) of INLINEFORM0 (denoted by INLINEFORM1 ) using following equation: INLINEFORM2\n\nwhere INLINEFORM0 is the lexicon weight of the term, and INLINEFORM1 represents the frequency of the term in the user generated INLINEFORM2 , and INLINEFORM3 measures total word count in INLINEFORM4 . \n Question: How do this framework facilitate demographic inference from social media?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-7a82c91a2cd74097b60c1fff39126994",
            "input": "In natural language processing (NLP), taking into account the morphological complexity inherent to each language could be important for improving or adapting the existing methods, since the amount of semantic and grammatical information encoded at the word level, may vary significantly from language to language. This information could be useful for linguistic analysis and for measuring the impact of different word form normalization tools depending of the language. \n Question: what is the practical application for this paper?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-cad9c1266b8b4498b8d39c5b21124512",
            "input": "Moreover, to be able to make use of multiple answer styles within a single system, our model introduces an artificial token corresponding to the target style at the beginning of the answer sentence ( $y_1$ ), like BIBREF14 . At test time, the user can specify the first token to control the answer styles. \n Question: Does their model also take the expected answer style as input?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-5a492e66d122474585fd872e4d28eb6b",
            "input": "Our approach improves LR by 5.17% (Accuracy) and 18.38% (AUC), and MLP by 10.71% (Accuracy) and 30.27% (AUC) on average. Such significant improvements clearly demonstrate that our approach is effective at improving model performance. \n Question: How are the accuracy merits of the approach demonstrated?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-6d24950b1a374a84aa3c402b6a893780",
            "input": "While confirming the effectiveness of BT, our study also proposed significantly cheaper ways to improve the baseline performance, using a slightly modified copy of the target, instead of its full BT.  \n Question: why are their techniques cheaper to implement?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-e7a9af8225a445bc83905ce2e76b0532",
            "input": "First, we introduce a two-stage labeling strategy for sentiment texts. In the first stage, annotators are invited to label a large number of short texts with relatively pure sentiment orientations. Each sample is labeled by only one annotator. In the second stage, a relatively small number of text samples with mixed sentiment orientations are annotated, and each sample is labeled by multiple annotators.  \n Question: What is the new labeling strategy?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-a442c9b7e5f04d5ba49b9d201b1a4119",
            "input": "In this work, we explore the applicability of our method to three popular architectures of this layer: the LSTM-based, the CNN-based, and the transformer-based. Table TABREF46 shows performance of our method with different sequence modeling architectures. From the table, we can first see that the LSTM-based architecture performed better than the CNN- and transformer- based architectures.  \n Question: Which are the sequence model architectures this method can be transferred across?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-d64746b7851c4429a49f245be95ba39c",
            "input": "We identified some limitations during the process, which we describe in this section.\n\nWhen deciding publisher partisanship, the number of people from whom we computed the score was small. For example, de Stentor is estimated to reach 275K readers each day on its official website. Deciding the audience leaning from 55 samples was subject to sampling bias. Besides, the scores differ very little between publishers. None of the publishers had an absolute score higher than 1, meaning that even the most partisan publisher was only slightly partisan. Deciding which publishers we consider as partisan and which not is thus not very reliable.\n\nThe article-level annotation task was not as well-defined as on a crowdsourcing platform. We included the questions as part of an existing survey and didn't want to create much burden to the annotators. Therefore, we did not provide long descriptive text that explained how a person should annotate an article. We thus run under the risk of annotator bias. This is one of the reasons for a low inter-rater agreement. \n Question: What limitations are mentioned?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-7ae9391a93cb4361a87f36515c0ac86f",
            "input": "Once the S-V-O is generated, Text2Visual provides users with visual components that convey the S-V-O text meanings. \n Question: Does their solution involve connecting images and text?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-fc4842f2045d4d579d692b9bf00dcdfe",
            "input": "MonaLog utilizes two auxiliary sets. First, a knowledge base ${K}$ that stores the world knowledge needed for inference, e.g., semanticist $\\le $ linguist and swim $\\le $ move, which captures the facts that $[\\![\\mbox{\\em semanticist}]\\!]$ denotes a subset of $[\\![\\mbox{\\em linguist}]\\!]$, and that $[\\![\\mbox{\\em swim}]\\!]$ denotes a subset of $[\\![\\mbox{\\em move}]\\!]$, respectively. Such world knowledge can be created manually for the problem at hand, or derived easily from existing resources such as WordNet BIBREF22. Note that we do not blindly add all relations from WordNet to our knowledge base, since this would hinge heavily on word sense disambiguation (we need to know whether the “bank” is a financial institution or a river bank to extract its relations correctly). In the current implementation, we avoid this by adding x $\\le $ y or x $\\perp $ y relations only if both x and y are words in the premise-hypothesis pair. Additionally, some relations that involve quantifiers and prepositions need to be hard-coded, since WordNet does not include them: every $=$ all $=$ each $\\le $ most $\\le $ many $\\le $ a few $=$ several $\\le $ some $=$ a; the $\\le $ some $=$ a; on $\\perp $ off; up $\\perp $ down; etc. \n Question: How do they select monotonicity facts?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-b33284db11994c46a1b09a61f0aa88d1",
            "input": "We compare our best F1 score with other state-of-the-art approaches in table TABREF39 , which shows our model has competitive advantage in dealing with drug-drug interaction extraction. \n Question: By how much does their model outperform existing methods?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-31f448e7550346ec985e32de0434f3d5",
            "input": "Fourteen such feature extractors have been implemented which can be clubbed into 3 major categories:\n\n[noitemsep]\n\nLexicon Features\n\nWord Vectors\n\nSyntax Features \n Question: how many total combined features were there?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-9b94de81c5134597b2d818d890cd1b70",
            "input": "The NLG model is a seq2seq model with attention as described in section SECREF2. \n Question: Do they use attention?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-5445e1d7867a4020a64f13e080edf443",
            "input": "For example, it can be used to extract features for usage with other machine learning tools, or to evaluate given features with the existing classifiers or regressors. Figure FIGREF19 demonstrates a simple feature extractor that retrieves the sentence length. \n Question: Do they show an example of usage for INFODENS?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-13bd3e76bb674e8fa9d96d16890a749c",
            "input": "We present a comparison of the proposed models to existing word embeddings approaches. These are: the Bernoulli embeddings (b-emb) BIBREF1 , continuous bag-of-words (CBOW) BIBREF5 , Distributed Memory version of Paragraph Vector (PV-DM) BIBREF11 and the Global Vectors (GloVe) BIBREF6 model. \n Question: What word embeddings do they test?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-634e9b2958564e99952e5c1276b9447f",
            "input": "Manual inspection of others examples also supports our claim. \n Question: Do they perform manual evaluation?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-bb0c6c26101c42e293e535f6b9112298",
            "input": "Results in Table TABREF13 show ten probes. Again, we see the performance of baseline ELMo-transformer and mSynC are similar, with mSynC doing slightly worse on 7 out of 9 tasks. As we would expect, on the probe for predicting chunk tags, mSynC achieves 96.9 $F_1$ vs. 92.2 $F_1$ for ELMo-transformer, indicating that mSynC is indeed encoding shallow syntax. Overall, the results further confirm that explicit shallow syntax does not offer any benefits over ELMo-transformer. \n Question: What are improvements for these two approaches relative to ELMo-only baselines?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-bef56edbd1a841c89634b3a8392edca3",
            "input": "Compared to single-task baselines, performance improved on the low-resource En-De task and was comparable on high-resource En-Fr task. \n Question: How big are negative effects of proposed techniques on high-resource tasks?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-0611e14347b14151acc8ddfd04d3e873",
            "input": "Sentence classification combines natural language processing (NLP) with machine learning to identify trends in sentence structure, BIBREF14 , BIBREF15 . Each tweet is converted to a numeric word vector in order to identify distinguishing features by training an NLP classifier on a validated set of relevant tweets. The classifier acts as a tool to sift through ads, news, and comments not related to patients. Our scheme combines a logistic regression classifier, BIBREF16 , with a Convolutional Neural Network (CNN), BIBREF17 , BIBREF18 , to identify self-reported diagnostic tweets.\n\nIt is important to be wary of automated accounts (e.g. bots, spam) whose large output of tweets pollute relevant organic content, BIBREF19 , and can distort sentiment analyses, BIBREF20 . Prior to applying sentence classification, we removed tweets containing hyperlinks to remove automated content (some organic content is necessarily lost with this strict constraint). Twitter allows users to spread content from other users via `retweets'. We also removed these posts prior to classification to isolate tweets authored by patients. We also accounted for non-relevant astrological content by removing all tweets containing any of the following horoscope indicators: `astrology',`zodiac',`astronomy',`horoscope',`aquarius',`pisces',`aries',`taurus',`leo',`virgo',`libra', and `scorpio'. We preprocessed tweets by lowercasing and removing punctuation. We also only analyzed tweets for which Twitter had identified `en' for the language English. \n Question: What machine learning and NLP methods were used to sift tweets relevant to breast cancer experiences?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-ab7aeb019afd443f9f918e509c9bbb05",
            "input": "In order to train a neural g2p system, one needs a large quantity of pronunciation data. A standard dataset for g2p is the Carnegie Mellon Pronouncing Dictionary BIBREF12 . However, that is a monolingual English resource, so it is unsuitable for our multilingual task. Instead, we use the multilingual pronunciation corpus collected by deri2016grapheme for all experiments. This corpus consists of spelling-pronunciation pairs extracted from Wiktionary. It is already partitioned into training and test sets. Corpus statistics are presented in Table TABREF10 In addition to the raw IPA transcriptions extracted from Wiktionary, the corpus provides an automatically cleaned version of transcriptions. Cleaning is a necessary step because web-scraped data is often noisy and may be transcribed at an inconsistent level of detail. The data cleaning used here attempts to make the transcriptions consistent with the phonemic inventories used in Phoible BIBREF4 .  \n Question: what datasets did they use?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-0cadd4f6f26f4fa895ad970d282e1be9",
            "input": "we apply our domain adaptation method to a neural captioning model and show performance improvement over other standard methods on several datasets and metrics.  \n Question: Did they only experiment with captioning task?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-b9ab8a4faf844592a12262d58c547cd9",
            "input": "Interesting prior work on quantifying social norm violation has taken a heavily data-driven focus BIBREF8 , BIBREF9 .  \n Question: Does this paper propose a new task that others can try to improve performance on?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-efe9b99b526a4e78ae30495c4cc9eee5",
            "input": "However, the coherence of the topics produced by LDA is poorer than expected.\n\nTo address this lack of coherence, we applied non-negative matrix factorization (NMF). \n Question: How are prominent topics idenified in Dabiq and Rumiyah?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-bc28d8ccab6841749724211987dfa135",
            "input": "$\\mathit {\\texttt {+}PPMI}$ only falters on the RW and analogy tasks, and we hypothesize this is where $\\mathit {\\texttt {-}PMI}$ is useful: in the absence of positive information, negative information can be used to improve rare word representations and word analogies. \n Question: What are the disadvantages to clipping negative PMI?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-95dbee39b953470e8d094a754c9efdfc",
            "input": "In order to understand the latent topics of those #MeToo tweets for college followers, we first utilize Latent Dirichlet Allocation (LDA) to label universal topics demonstrated by the users. Since certain words frequently appear in those #MeToo tweets (e.g., sexual harassment, men, women, story, etc.), we transform our corpus using TF-IDF, a term-weighting scheme that discounts the influence of common terms. \n Question: How are the topics embedded in the #MeToo tweets extracted?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-c545eb02e2ea43d697fcd20ec29c9e28",
            "input": "We employ two sources of e-book annotation data: (i) editor tags, and (ii) Amazon search terms. For editor tags, we collect data of 48,705 e-books from 13 publishers, namely Kunstmann, Delius-Klasnig, VUR, HJR, Diogenes, Campus, Kiwi, Beltz, Chbeck, Rowohlt, Droemer, Fischer and Neopubli.  For the Amazon search terms, we collect search query logs of 21,243 e-books for 12 months (i.e., November 2017 to October 2018).  \n Question: what dataset was used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-2bf67216fe4f494b9c1623c950a50c5c",
            "input": "As a framework, we use the sentence classifier configuration of FLAIR BIBREF46 with a biLSTM encoder/classifier architecture fed by character and word level representations composed of a concatenation of fixed 300 dimensional GloVe embeddings BIBREF47, pre-trained contextualized FLAIR word embeddings, and pre-trained contextualized character embeddings from AllenNLP BIBREF48 with FLAIR's default hyperparameters. \n Question: Are some models evaluated using this metric, what are the findings?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-8ee2db7079cd48b8b562d9aa03df9670",
            "input": "To evaluate the proposed LRC framework and the AutoJudge model, we carry out a series of experiments on the divorce proceedings, a typical yet complex field of civil cases.  \n Question: what civil field is the dataset about?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-90b75dfac9a34cbbab365c8128e9af3a",
            "input": "For the CNN-Dailymail dataset, the Lead-3 model is considered a strong baseline; both the abstractive BIBREF16 and extractive BIBREF14 state-of-the art methods on this dataset beat this baseline only marginally. For the proxy report section of the AMR bank, we consider the Lead-1-AMR model as the baseline. \n Question: Which other methods do they compare with?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-8ef6c18d275f4d969cc7eba65218b464",
            "input": "We conduct our annotation study on Amazon Mechanical Turk, presenting Turkers with Human Intelligence Tasks (henceforth, HITs) consisting of a single conversation between a customer and an agent. In each HIT, we present Turkers with a definition of each dialogue act, as well as a sample annotated dialogue for reference. For each turn in the conversation, we allow Turkers to select as many labels from our taxonomy as required to fully characterize the intent of the turn. Additionally, annotators are asked three questions at the end of each conversation HIT, to which they could respond that they agreed, disagreed, or could not tell: \n Question: How are customer satisfaction, customer frustration and overall problem resolution data collected?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-6fd9fefdd5e04f119746603eff599e03",
            "input": " In this task English tweets from conversation threads, each associated to a newsworthy event and the rumours around it, are provided as data. The goal is to determine whether a tweet in the thread is supporting, denying, querying, or commenting the original rumour which started the conversation.  \n Question: Is this an English-language dataset?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-3597159b74334dd89c81e2ef1e958804",
            "input": "In Figure FIGREF15 , we plot the zero-resource German and Japanese test set accuracy as a function of the number of steps taken, with and without adversarial training. The plot shows that the variation in the test accuracy is reduced with adversarial training, which suggests that the cross-lingual performance is more consistent when adversarial training is applied. \n Question: Do any of the evaluations show that adversarial learning improves performance in at least two different language families?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-ee7441fea13f441ba892f8e5c0ea287c",
            "input": "d  Experiments on two benchmark data sets, the Stanford Sentiment Treebank BIBREF7 and the AG English news corpus BIBREF3 , show that 1) our method achieves very competitive accuracy, 2) some views distinguish themselves from others by better categorizing specific classes, and 3) when our base bag-of-words feature set is augmented with convolutional features, the method establishes a new state-of-the-art for both data sets.  Stanford Sentiment Treebank  AG English news corpus  \n Question: which benchmark tasks did they experiment on?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-01bf08dfb7ce4b7e8cb69ce0828950df",
            "input": "We perform our experiments using ActivityNet Captions dataset BIBREF2 that is considered as the standard benchmark for dense video captioning task. The dataset contains approximately 20k videos from YouTube and split into 50/25/25 % parts for training, validation, and testing, respectively.  \n Question: What domain does the dataset fall into?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-888d2662cea046fbbca4aa34cedef247",
            "input": "In this work, we develop a technique to rapidly transfer an existing pre-trained model from English to other languages in an energy efficient way BIBREF8.  \n Question: How much training data from the non-English language is used by the system?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-e56f4b83853a483f859c4fe6e229bfd6",
            "input": "Spoken-SQuAD is chosen as the target domain data for training and testing. Spoken-SQuAD BIBREF5 is an automatically generated corpus in which the document is in spoken form and the question is in text form. The reference transcriptions are from SQuAD BIBREF1 . There are 37,111 and 5,351 question answer pairs in the training and testing sets respectively, and the word error rate (WER) of both sets is around 22.7%.\n\nThe original SQuAD, Text-SQuAD, is chosen as the source domain data, where only question answering pairs appearing in Spoken-SQuAD are utilized. In our task setting, during training we train the proposed QA model on both Text-SQuAD and Spoken-SQuAD training sets. While in the testing stage, we evaluate the performance on Spoken-SQuAD testing set. \n Question: Which datasets did they use for evaluation?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-13e18a32a3db4ebda3ee083673322191",
            "input": "When we annotate dialogues, we should read dialogues from begin to the end. For each utterance, we should find its one parent node at least from all its previous utterances. We assume that the discourse structure is a connected graph and no utterance is isolated. We propose three questions for eache dialogue and annotate the span of answers in the input dialogue. As we know, our dataset is the first corpus for multi-party dialogues reading comprehension.\n\nWe construct following questions and answers for the dialogue in Example 1:\n\nQ1: When does Bdale leave?\n\nA1: Fri morning\n\nQ2: How to get people love Mark in Mjg59's opinion.\n\nA2: Hire people to work on reverse-engineering closed drivers.\n\nOn the other hand, to improve the difficulty of the task, we propose $ \\frac{1}{6}$ to $ \\frac{1}{3}$ unanswerable questions in our dataset. We annotate unanswerable questions and their plausible answers (PA). Each plausible answer comes from the input dialogue, but is not the answer for the plausible question.\n\nQ1: Whis is the email of daniels?\n\nPA: +61 403 505 896 \n Question: Is annotation done manually?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-6771ca4ba0ea455987a63e9a229fa89b",
            "input": "Results in Table TABREF13 show ten probes. Again, we see the performance of baseline ELMo-transformer and mSynC are similar, with mSynC doing slightly worse on 7 out of 9 tasks. \n Question: For how many probe tasks the shallow-syntax-aware contextual embedding perform better than ELMo’s embedding?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-8e515a73032546c99fb12864ea88a6bf",
            "input": "Following these methods, we evaluate NCEL on the following five datasets: (1) CoNLL-YAGO BIBREF22 : the CoNLL 2003 shared task including testa of 4791 mentions in 216 documents, and testb of 4485 mentions in 213 documents. (2) TAC2010 BIBREF39 : constructed for the Text Analysis Conference that comprises 676 mentions in 352 documents for testing. (3) ACE2004 BIBREF23 : a subset of ACE2004 co-reference documents including 248 mentions in 35 documents, which is annotated by Amazon Mechanical Turk. (4) AQUAINT BIBREF40 : 50 news articles including 699 mentions from three different news agencies. (5) WW BIBREF19 : a new benchmark with balanced prior distributions of mentions, leading to a hard case of disambiguation. It has 6374 mentions in 310 documents automatically extracted from Wikipedia. \n Question: Which datasets do they use?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-8631aa3b7c154f8a94b81a366bcac6e0",
            "input": "GP-GNNs first build a fully-connected graph $\\mathcal {G} = (\\mathcal {V}, \\mathcal {E})$ , where $\\mathcal {V}$ is the set of entities, and each edge $(v_i, v_j) \\in \\mathcal {E}, v_i, v_j \\in \\mathcal {V}$ corresponds to a sequence $s = x_0^{i,j}, x_1^{i,j}, \\dots , x_{l-1}^{i,j}$ extracted from the text.  \n Question: So this paper turns unstructured text inputs to parameters that GNNs can read?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-49f0c50f650441688cc0e85231b1ef42",
            "input": "The lexicon is built over all the data and therefore includes the vocabulary from both the training and testing sets. \n Question: Is the lexicon the same for all languages?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-1b54a58dbf8040e784b4f3929121d93d",
            "input": "  In a second step, we will train a neural monolingual translation system, that translates from the output of the PBMT system INLINEFORM0 to a better target sentence INLINEFORM1 . \n Question: Do they train the NMT model on PBMT outputs?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-9becb49c39dc4c5c86b529fef1f8a501",
            "input": "We train our models on Sentiment140 and Amazon product reviews. Both of these datasets concentrates on sentiment represented by a short text.  For training the softmax model, we divide the text sentiment to two kinds of emotion, positive and negative. And for training the tanh model, we convert the positive and negative emotion to [-1.0, 1.0] continuous sentiment score, while 1.0 means positive and vice versa.  \n Question: Was the introduced LSTM+CNN model trained on annotated data in a supervised fashion?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-235943967a4441aeb13e5694903610e0",
            "input": "Our analysis shows that attention models traditional alignment in some cases more closely while it captures information beyond alignment in others. For instance, attention agrees with traditional alignments to a high degree in the case of nouns. However, it captures other information rather than only the translational equivalent in the case of verbs. One can see that attention loss varies substantially across different POS tags. In particular, we focus on the cases of NOUN and VERB which are the most frequent POS tags in the dataset. As shown, the attention of NOUN is the closest to alignments on average. But the average attention loss for VERB is almost two times larger than the loss for NOUN. \n Question: What useful information does attention capture?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-bdfda1699b0548c1b640c7af686a3d66",
            "input": "Methods ::: Length Token Method\nOur first approach to control the length is inspired by target forcing in multilingual NMT BIBREF15, BIBREF16. We first split the training sentence pairs into three groups according to the target/source length ratio (in terms of characters). Ideally, we want a group where the target is shorter than the source (short), one where they are equally-sized (normal) and a last group where the target is longer than the source (long). In practice, we select two thresholds $t_\\text{min}$ and $t_\\text{max}$ according to the length ratio distribution. All the sentence pairs with length ratio between $t_\\text{min}$ and $t_\\text{max}$ are in the normal group, the ones with ratio below $t_\\text{min}$ in short and the remaining in long. At training time we prepend a length token to each source sentence according to its group ($<$short$>$, $<$normal$>$, or $<$long$>$), in order to let a single network to discriminate between the groups (see Figure FIGREF2). At inference time, the length token is used to bias the network to generate a translation that belongs to the desired length group. \n Question: How do they condition the output to a given target-source class?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-cda76ec6b8104aab860731d0eee39aaf",
            "input": "Our resulting model obtains state of the art results on both the VQA dataset and the bAbI-10k text question-answering dataset, proving the framework can be generalized across input domains. \n Question: Does the DMN+ model establish state-of-the-art ?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-2de05041b6004709a9bef3fec4c3082e",
            "input": "Most of the above-discussed systems either shows high performance on (a) Twitter dataset or (b) Facebook dataset (given in the TRAC-2018), but not on both English code-mixed datasets. This may be due to the text style or level of complexities of both datasets. \n Question: How have the differences in communication styles between Twitter and Facebook increase the complexity of the problem?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-87845b7c18744a598b3be514d3750cd7",
            "input": "The average classification accuracy results are summarised in Table TABREF9. \n Question: What evaluation metric is used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-b5fe3e95716b4643819c32edb1d215be",
            "input": "The Word2Vec architecture has inspired a great deal of research in the bio/cheminformatics domains. The Word2Vec algorithm has been successfully applied for determining protein classes BIBREF44 and protein-protein interactions (PPI) BIBREF56. \n Question: Is there any concrete example in the paper that shows that this approach had huge impact on drug discovery?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-e7cd528ebd5544279ea4baf81ba243cc",
            "input": "Besides, we also use two of its adversarial sets, namely AddSent and AddOneSent BIBREF6 , to evaluate the robustness to noise of MRC models. The passages in the adversarial sets contain misleading sentences, which are aimed at distracting MRC models.  Specifically, we not only evaluate the performance of KAR on the development set and the test set, but also do this on the adversarial sets.  \n Question: How do the authors examine whether a model is robust to noise or not?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-eb762674c62f4ffdb6c15abf6715ff23",
            "input": "For sentence encoding models, we chose a simple one-layer bidirectional LSTM with max pooling (BiLSTM-max) with the hidden size of 600D per direction, used e.g. in InferSent BIBREF17 , and HBMP BIBREF18 . For the other models, we have chosen ESIM BIBREF19 , which includes cross-sentence attention, and KIM BIBREF2 , which has cross-sentence attention and utilizes external knowledge. We also selected two model involving a pre-trained language model, namely ESIM + ELMo BIBREF20 and BERT BIBREF0 . \n Question: Which models were compared?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-4dee25a11deb4d44a149d88cad3575bb",
            "input": "The focus of this publication is on representing information that is valuable for these tasks but that hitherto has largely been ignored in machine learning approaches centering around simplified language, specifically, text structure (e.g., paragraphs, lines), typography (e.g., font type, font style), and image (content, position, and dimensions) information. Information on physical page segmentation (for PDFs only), paragraph segmentation, and line segmentation was added as part of a textspan element in the textstructure layer \n Question: Which information about text structure is included in the corpus?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-b8e947a7a94a4f949764b414083990fb",
            "input": "to answer how offensive they thought the tweet was on a 6-point Likert scale from 1 (Not offensive at all) to 6 (Very offensive). If they answered 4 or higher, the participants had the option to state which particular words they found offensive. \n Question: Was the degree of offensiveness taken as how generally offensive the text was, or how personally offensive it was to the annotator?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-55d6ff2adaea4be0ad947c96b5090fe3",
            "input": "At the end of the annotation process, we have annotated 700 tweets, where 175 tweets are in favor of and 175 tweets are against Target-1, and similarly 175 tweets are in favor of and 175 are against Target-2.  \n Question: How many tweets did they collect?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-3dd18b946e374f2796fce98621fa283f",
            "input": "Most lexical resources for sentiment analysis are in English. To still be able to benefit from these sources, the lexicons in the AffectiveTweets package were translated to Spanish, using the machine translation platform Apertium BIBREF5  \n Question: How was the training data translated?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-70b55b618daf4caaa6cb92b7a5e5676b",
            "input": "We evaluated our attention transformations on three language pairs. We use the IWSLT 2014 corpus for De-En, the KFTT corpus for Ja-En BIBREF19 , and the WMT 2016 dataset for Ro-En. \n Question: What are the language pairs explored in this paper?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-30dbd85a4dd647e4881c6e66ea2882b0",
            "input": "Since our objective is to compare character-based and word-based approaches, we have also implemented a simple word-level encoder for tweets. The input tweet is first split into tokens along white-spaces. A more sophisticated tokenizer may be used, but for a fair comparison we wanted to keep language specific preprocessing to a minimum. The encoder is essentially the same as tweet2vec, with the input as words instead of characters. A lookup table stores word vectors for the $V$ (20K here) most common words, and the rest are grouped together under the `UNK' token. \n Question: what is the word level baseline they compare to?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-bf7ca479c2334ee4a6fb6fc53fe5886a",
            "input": "We have observed that the conflict model is very sensitive to even minor differences and compensates in such cases where attention poses high bias towards similarities already there in the sequences.\n\nSequence 1: What are the best ways to learn French ?\n\nSequence 2: How do I learn french genders ?\n\nAttention only: 1\n\nAttention+Conflict: 0\n\nGround Truth: 0\n\nSequence 1: How do I prevent breast cancer ?\n\nSequence 2: Is breast cancer preventable ?\n\nAttention only: 1\n\nAttention+Conflict: 0\n\nGround Truth: 0\n\nWe provide two examples with predictions from the models with only attention and combination of attention and conflict. Each example is accompanied by the ground truth in our data. \n Question: Do they show on which examples how conflict works better than attention?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-c516661c3065489c9f96c95296b2ee83",
            "input": "Our submissions ranked second (EI-Reg), second (EI-Oc), fourth (V-Reg) and fifth (V-Oc), demonstrating that the proposed method is accurate in automatically determining the intensity of emotions and sentiment of Spanish tweets. \n Question: What subtasks did they participate in?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-47e377a9725c4d42a38ecef08288aa19",
            "input": "Analysis of challenges from UTD\nOur system relies on the pseudotext produced by ZRTools (the only freely available UTD system we are aware of), which presents several challenges for MT.  Assigning wrong words to a cluster\nSince UTD is unsupervised, the discovered clusters are noisy.  Splitting words across different clusters\nAlthough most UTD matches are across speakers, recall of cross-speaker matches is lower than for same-speaker matches. As a result, the same word from different speakers often appears in multiple clusters, preventing the model from learning good translations. UTD is sparse, giving low coverage We found that the patterns discovered by ZRTools match only 28% of the audio. This low coverage reduces training data size, affects alignment quality, and adversely affects translation, which is only possible when pseudoterms are present. \n Question: what challenges are identified?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-6ce4a85ee6c843beaaeab4baa02bc3e0",
            "input": "Ethnicity/race\nOne interesting pattern is that the ethnicity/race of babies doesn't seem to be mentioned unless the baby is black or asian. In other words: white seems to be the default, and others seem to be marked. How can we tell whether or not the data is actually biased?\n\n The numbers in Table TABREF22 are striking: there seems to be a real, systematic difference in ethnicity marking between the groups. We can take one step further and look at all the 697 pictures with the word `baby' in it. If there turn out to be disproportionately many white babies, this strengthens the conclusion that the dataset is biased. \n Question: What biases are found in the dataset?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-9c680253cd9c40aa92283ff9bf3b6130",
            "input": "As explained in Section SECREF15 , the corruption introduced in Doc2VecC acts as a data-dependent regularization that suppresses the embeddings of frequent but uninformative words. In contrast, Doc2VecC manages to clamp down the representation of words frequently appear in the training set, but are uninformative, such as symbols and stop words. \n Question: How do they determine which words are informative?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-d121cfa906db4e9f8aacdf1c2b87adc9",
            "input": "For all the tasks in our experimental study, we use 36 millions English tweets collected between August and September 2017.  \n Question: Do they report results only on English datasets?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-71744416c37240e18e7e11e8a8d67914",
            "input": "This layer consists of two submodules: the Multi-Head Attention (MHA) and the Point-wise Convolution Transformation (PCT). \n Question: Do they use multi-attention heads?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-09b3e96f633e4e28a1ac9cf0d9240c3b",
            "input": "Two datasets are exploited in this article. Both datasets consist of plain text containing clinical narrative written in Spanish, and their respective manual annotations of sensitive information in BRAT BIBREF13 standoff format. NUBes BIBREF4 is a corpus of around 7,000 real medical reports written in Spanish and annotated with negation and uncertainty information. In order to avoid confusion between the two corpus versions, we henceforth refer to the version relevant in this paper as NUBes-PHI (from `NUBes with Personal Health Information'). The organisers of the MEDDOCAN shared task BIBREF3 curated a synthetic corpus of clinical cases enriched with sensitive information by health documentalists \n Question: What are the clinical datasets used in the paper?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-3d50645b6d994bd0a7a3a4d7b59b759e",
            "input": "The choice of two methods for our empirical study is motivated by the best performance achieved by Logistic Regression in question-question similarity at SemEval 2017 (best system BIBREF37 and second best system BIBREF38 ), and the high performance achieved by neural networks on larger datasets such as SNLI BIBREF13 , BIBREF39 , BIBREF40 , BIBREF41  \n Question: What machine learning and deep learning methods are used for RQE?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-41e354cf59ff44c590517d1fef252551",
            "input": "The main reason is that the datasets only contain a small portion of multi-aspect sentences with different polarities. The distraction of attention will not impact the sentiment prediction much in single-aspect sentences or multi-aspect sentences with the same polarities. \n Question: Is the model evaluated against the baseline also on single-aspect sentences?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-fa592661e5e447f3b0726a0c706c26e8",
            "input": "We use the Bayesian model of garg2012unsupervised as our base monolingual model. The semantic roles are predicate-specific. \n Question: What does an individual model consist of?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-dd034106ca7e4f3ab5ed039042e90b45",
            "input": "In contrast to other work, we do not show the documents to the workers at all, but provide only a description of the document cluster's topic along with the propositions. This ensures that tasks are small, simple and can be done quickly (see Figure FIGREF4 ). \n Question: How were crowd workers instructed to identify important elements in large document collections?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-86ee39749678489dbf0725c76dadee4e",
            "input": "Ternary and fine-grained sentiment classification were part of the SemEval-2016 “Sentiment Analysis in Twitter” task BIBREF16 . We use the high-quality datasets the challenge organizers released. \n Question: What dataset did they use?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-e4f672cc834a4e988f506c13f2e747aa",
            "input": "We evaluate sentence-level semantics using averaged bag of vectors (BoV) representations on the Semantic Textual Similarity (STSB) task BIBREF21 and Word Content (WC) probing task (identify from a list of words which is contained in the sentence representation) from SentEval BIBREF22. Syntax: Similarly, we use the Google Syntactic analogies (GSyn) BIBREF9 to evaluate word-level syntactic information, and Depth (Dep) and Top Constituent (TopC) (of the input sentence's constituent parse tree) probing tasks from SentEval BIBREF22 for sentence-level syntax. \n Question: What semantic and syntactic tasks are used as probes?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-e0375f6c0e7144d1b1e4b028380d9bf3",
            "input": "Our Recurrent +ELMo model uses the language model from BIBREF9 to provide contextualized embeddings to the baseline model outlined above, as recommended by the authors.\n\nOur OpenAI GPT model fine-tunes the 12 layer 768 dimensional uni-directional transformer from BIBREF27 , which has been pre-trained as a language model on the Books corpus BIBREF36 . \n Question: did they use other pretrained language models besides bert?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-43ce18a1ef344798bbf70bca03d69778",
            "input": "Given that words from texts of the same class belong to the same context, it is possible to model word vectors of each class as word subspaces and efficiently compare them in terms of similarity by using canonical angles between the word subspaces.  \n Question: What can word subspace represent?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-386d652171d44a4d9164d7c79176249c",
            "input": "We carried out a reliability study for the proposed scheme using two pairs of expert annotators, P1 and P2.  \n Question: do they use a crowdsourcing platform?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-3c6a13ac7240426a92fd560ed25f42fb",
            "input": "We introduce a list of 8 different competencies that a reading system should master in order to process reviews and text documents in general. These 8 tasks require different competencies and a different level of understanding of the document to be well answered. For instance, detecting if an aspect is mentioned in a review will require less understanding of the review than predicting explicitly the rating of this aspect. Table TABREF10 presents the 8 tasks we have introduced in this dataset with an example of a question that corresponds to each task. \n Question: What kind of questions are present in the dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-016e9c78f0e64145933d9ba07a4dadb4",
            "input": "We recruited 100 crowdworkers on Amazon Mechanical Turk (AMT) and measured completion times and accuracies for typing randomly sampled sentences from the Yelp corpus.  \n Question: What user variations have been tested?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-70b6c7a8300142cd88d788fb911b2492",
            "input": "Currently, the following WSD models induced from a text corpus are available: Word senses based on cluster word features. This model uses the cluster words from the induced word sense inventory as sparse features that represent the sense.\n\nWord senses based on context word features. This representation is based on a sum of word vectors of all cluster words in the induced sense inventory weighted by distributional similarity scores.\n\nSuper senses based on cluster word features. To build this model, induced word senses are first globally clustered using the Chinese Whispers graph clustering algorithm BIBREF9 . The edges in this sense graph are established by disambiguation of the related words BIBREF11 , BIBREF12 . The resulting clusters represent semantic classes grouping words sharing a common hypernym, e.g. “animal”. This set of semantic classes is used as an automatically learned inventory of super senses: There is only one global sense inventory shared among all words in contrast to the two previous traditional “per word” models. Each semantic class is labeled with hypernyms. This model uses words belonging to the semantic class as features.\n\nSuper senses based on context word features. This model relies on the same semantic classes as the previous one but, instead, sense representations are obtained by averaging vectors of words sharing the same class. \n Question: Do they use a neural model for their task?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-82d4483da1e44073a483af1bbc021504",
            "input": "To discover topics from the collected tweets, we used a topic modeling approach that fuzzy clusters the semantically related words such as assigning “diabetes\", “cancer\", and “influenza\" into a topic that has an overall “disease\" theme BIBREF44 , BIBREF45 . Among topic models, Latent Dirichlet Allocation (LDA) BIBREF49 is the most popular effective model BIBREF50 , BIBREF19 as studies have shown that LDA is an effective computational linguistics model for discovering topics in a corpus BIBREF51 , BIBREF52 . We used the Mallet implementation of LDA BIBREF49 , BIBREF56 with its default settings to explore opinions in the tweets. \n Question: How were topics of interest about DDEO identified?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-27649f705cff4b2ea26b032d2082378b",
            "input": "Although the competition proposes two different scenarios, in fact, both are guided by the snomed ct ontology —for subtask 1, entities must be identified with offsets and mapped to a predefined set of four classes (PROTEINAS, NORMALIZABLES, NO_NORMALIZABLES and UNCLEAR); for subtask 2, a list of all snomed ct ids (sctid) for entities occurring in the text must be given, which has been called concept indexing by the shared task organizers. \n Question: What are the two PharmaCoNER subtasks?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-db21a27453d448cdb641fe6ff6ab20ce",
            "input": " While domain transfer is not new, compared to prior summarization studies BIBREF6, BIBREF7, our training (news) and tuning (student reflection) domains are quite dissimilar, and the in-domain data is small. \n Question: Is the student reflection data very different from the newspaper data?  ",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-6ac4a0262c404ad4b3dd5b87d9168661",
            "input": "In our final system, after pre-training the forward and backward LMs separately, we remove the top layer softmax and concatenate the forward and backward LM embeddings to form bidirectional LM embeddings, i.e., INLINEFORM0 .  \n Question: how are the bidirectional lms obtained?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-aef2afd91603486ca7ce068ac0f2ff76",
            "input": "For the Russian language, with its rich morphology, lemmatizing the training and testing data for ELMo representations yields small but consistent improvements in the WSD task.  \n Question: What other examples of morphologically-rich languages do the authors give?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-a74a3b576a634f44bb9590631a627bda",
            "input": "The improved performance of our attention models that actively select their optimal context, over a model with the complete thread as context, hLSTM, shows that the context inference improves intervention prediction over using the default full context. \n Question: What aspects of discussion are relevant to instructor intervention, according to the attention mechanism?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-73fc070fd4364b0c830449cc8302678a",
            "input": "We also explored clustering 12742 STRENGTH sentences directly using CLUTO BIBREF19 and Carrot2 Lingo BIBREF20 clustering algorithms.  \n Question: What clustering algorithms were used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-a6bdaeb54561468cb75eb33bb422c267",
            "input": "Experimental results on public datasets show that our CRU model could substantially outperform various systems by a large margin, and set up new state-of-the-art performances on related datasets. \n Question: Are there some results better than state of the art on these tasks?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-33fbe7d270b8456384df70bc0748eba8",
            "input": "the basic model yields good performance for recognizing explicit discourse relations as well, which is comparable with previous best result (92.05% macro F1-score and 93.09% accuracy as reported in BIBREF11 ). In summary, our full paragraph-level neural network model achieves the best macro-average F1-score of 48.82% in predicting implicit discourse relations, which outperforms previous neural tensor network models (e.g., BIBREF18 ) by more than 2 percents and outperforms the best previous system BIBREF19 by 1 percent. Then we also created ensemble models by applying majority voting to combine results of ten runs. From table 5 , each ensemble model obtains performance improvements compared with single model. The full model achieves performance boosting of (51.84 - 48.82 = 3.02) and (94.17 - 93.21 = 0.96) in macro F1-scores for predicting implicit and explicit discourse relations respectively.  \n Question: How much does this model improve state-of-the-art?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-c49fa481fc684a05a522466fd447d01f",
            "input": "The English online magazine of ISIS was named Dabiq and first appeared on the dark web on July 2014 and continued publishing for 15 issues. This publication was followed by Rumiyah which produced 13 English language issues through September 2017. Looking through both Dabiq and Rumiyah, many issues of the magazines contain articles specifically addressing women, usually with “ to our sisters ” incorporated into the title. \n Question: Do they report results only on English data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-8679a2cc243045a0bee3042149f7b26b",
            "input": "Because our approach is specifically intended to yield sentences that are grammatical, we additionally consider the following log ratio (i.e., the grammatical phrase over the ungrammatical phrase): \n Question: How do they measure grammaticality?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-003e160ec0ed407bb572455ec8c51a63",
            "input": "INLINEFORM0 in the subsampled BIBREF2 training corpus and incrementing cell INLINEFORM1 for every context word INLINEFORM2 appearing within this window (forming a INLINEFORM3 pair). LexVec adjusts the PPMI matrix using context distribution smoothing BIBREF3 . We compare 1) the use of n-gram subwords, like fastText, and 2) unsupervised morphemes identified using Morfessor BIBREF11 to learn whether more linguistically motivated subwords offer any advantage over simple n-grams. \n Question: What types of subwords do they incorporate in their model?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-f91591b7eaa948c3978f39aba0c77d78",
            "input": "We illustrate the concept by discussing some instantiations that we have recently experimented with. The design also provides for a clear “relevance place” at which an opportunity arises for semantic negotiation, namely, before the final decision is made. An example of this is shown in the example below.   As SECREF12 shows, even in cases where a decision can be reached quickly, there can be an explicit mutual confirmation step, before the (silent) decision signal is sent. A third setting that we have explored BIBREF19 brings conceptual negotiation more clearly into the foreground. In that game, the players are presented with images of birds of particular species and are tasked with coming up with a description of common properties. Again, the final answer has to be approved by both participants. As SECREF13 shows, this can lead to an explicit negotiation of conceptual content. \n Question: Do the authors perform experiments using their proposed method?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-c491a2f3dc704f32b09ec67a96a9aee4",
            "input": " We refer the reader to BIBREF10 for more details on the propaganda techniques; below we report the list of techniques:\n\nPropaganda Techniques ::: 1. Loaded language.\nUsing words/phrases with strong emotional implications (positive or negative) to influence an audience BIBREF11.\n\nPropaganda Techniques ::: 2. Name calling or labeling.\nLabeling the object of the propaganda as something the target audience fears, hates, finds undesirable or otherwise loves or praises BIBREF12.\n\nPropaganda Techniques ::: 3. Repetition.\nRepeating the same message over and over again, so that the audience will eventually accept it BIBREF13, BIBREF12.\n\nPropaganda Techniques ::: 4. Exaggeration or minimization.\nEither representing something in an excessive manner: making things larger, better, worse, or making something seem less important or smaller than it actually is BIBREF14, e.g., saying that an insult was just a joke.\n\nPropaganda Techniques ::: 5. Doubt.\nQuestioning the credibility of someone or something.\n\nPropaganda Techniques ::: 6. Appeal to fear/prejudice.\nSeeking to build support for an idea by instilling anxiety and/or panic in the population towards an alternative, possibly based on preconceived judgments.\n\nPropaganda Techniques ::: 7. Flag-waving.\nPlaying on strong national feeling (or with respect to a group, e.g., race, gender, political preference) to justify or promote an action or idea BIBREF15.\n\nPropaganda Techniques ::: 8. Causal oversimplification.\nAssuming one cause when there are multiple causes behind an issue. We include scapegoating as well: the transfer of the blame to one person or group of people without investigating the complexities of an issue.\n\nPropaganda Techniques ::: 9. Slogans.\nA brief and striking phrase that may include labeling and stereotyping. Slogans tend to act as emotional appeals BIBREF16.\n\nPropaganda Techniques ::: 10. Appeal to authority.\nStating that a claim is true simply because a valid authority/expert on the issue supports it, without any other supporting evidence BIBREF17. We include the special case where the reference is not an authority/expert, although it is referred to as testimonial in the literature BIBREF14.\n\nPropaganda Techniques ::: 11. Black-and-white fallacy, dictatorship.\nPresenting two alternative options as the only possibilities, when in fact more possibilities exist BIBREF13. As an extreme case, telling the audience exactly what actions to take, eliminating any other possible choice (dictatorship).\n\nPropaganda Techniques ::: 12. Thought-terminating cliché.\nWords or phrases that discourage critical thought and meaningful discussion about a given topic. They are typically short and generic sentences that offer seemingly simple answers to complex questions or that distract attention away from other lines of thought BIBREF18.\n\nPropaganda Techniques ::: 13. Whataboutism.\nDiscredit an opponent's position by charging them with hypocrisy without directly disproving their argument BIBREF19.\n\nPropaganda Techniques ::: 14. Reductio ad Hitlerum.\nPersuading an audience to disapprove an action or idea by suggesting that the idea is popular with groups hated in contempt by the target audience. It can refer to any person or concept with a negative connotation BIBREF20.\n\nPropaganda Techniques ::: 15. Red herring.\nIntroducing irrelevant material to the issue being discussed, so that everyone's attention is diverted away from the points made BIBREF11. Those subjected to a red herring argument are led away from the issue that had been the focus of the discussion and urged to follow an observation or claim that may be associated with the original claim, but is not highly relevant to the issue in dispute BIBREF20.\n\nPropaganda Techniques ::: 16. Bandwagon.\nAttempting to persuade the target audience to join in and take the course of action because “everyone else is taking the same action” BIBREF15.\n\nPropaganda Techniques ::: 17. Obfuscation, intentional vagueness, confusion.\nUsing deliberately unclear words, to let the audience have its own interpretation BIBREF21, BIBREF11. For instance, when an unclear phrase with multiple possible meanings is used within the argument and, therefore, it does not really support the conclusion.\n\nPropaganda Techniques ::: 18. Straw man.\nWhen an opponent's proposition is substituted with a similar one which is then refuted in place of the original BIBREF22. \n Question: What are the 18 propaganda techniques?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-9cbf4d6535614e9caa18ae152a7fd19e",
            "input": "We have evaluated our models considering the F1 Score, which is the harmonic mean of precision and recall. We have run ten times the experiment for each model and considered the average F1 Score. The results are mentioned in Table TABREF11. Considering F1 Macro the models that include the multi-attention mechanism outperform the others and particularly the one with the Projected Layer has the highest performance. In three out of four pairs of models, the ones with the Projected Layer achieved better performance, so in most cases the addition of the Projected Layer had a significant enhancement.\n\n \n Question: Which variation provides the best results on this dataset?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-f815182320f4482295fc9f2a3056550e",
            "input": "We first run our experiment on BiLSTM, BiLSTM-CNN, BiLSTM-CRF BiLSTM-CNN-CRF using the hyper-parameters mentioned in Table TABREF30.  \n Question: Which machine learning models do they explore?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-f59cf813b85c49ceac4b8a3dfe5c08e9",
            "input": "Our evaluation comprises of a rich set of 19 different algorithms to recommend tags for e-books, which we group into (i) popularity-based, (ii) similarity-based (i.e., using content information), and (iii) hybrid approaches. \n Question: what algorithms did they use?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-0c10709caca04632914a2fed8a3e4049",
            "input": "Our system was ranked second in the competition only 0.3 BLEU points behind the winning team UPC-TALP. The relative low BLEU and high TER scores obtained by all teams are due to out-of-domain data provided in the competition which made the task equally challenging to all participants. The fact that out-of-domain data was provided by the organizers resulted in a challenging but interesting scenario for all participants. \n Question: Does the use of out-of-domain data improve the performance of the method?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-e59ca7c4fc0748d2b5cdd4656a743086",
            "input": "We use a standard set of supervised as well as unsupervised benchmark tasks from the literature to evaluate our trained models, following BIBREF16 Sentence embeddings are evaluated for various supervised classification tasks as follows. The predefined training split is used to tune the L2 penalty parameter using cross-validation and the accuracy and F1 scores are computed on the test set.  We perform unsupervised evaluation of the learnt sentence embeddings using the sentence cosine similarity, on the STS 2014 BIBREF31 and SICK 2014 BIBREF32 datasets. These similarity scores are compared to the gold-standard human judgements using Pearson's INLINEFORM0 BIBREF33 and Spearman's INLINEFORM1 BIBREF34 correlation scores.  \n Question: What metric is used to measure performance?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-20ac039adc6a4dc0a5a96d55469bfcf8",
            "input": "The stance towards vaccination was categorized into `Negative’, `Neutral’, `Positive’ and `Not clear’. \n Question: Do they allow for messages with vaccination-related key terms to be of neutral stance?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-9ec69cb884c14d85b7242a5b4bff37cf",
            "input": "We consider 5 major profile attributes for further analysis. These attributes are username, display name, profile image, location and description respectively. \n Question: What profile metadata is used for this analysis?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-344a54279e284f3d8b44a47a7d2301c3",
            "input": "We thus adapt an off-the-shelf reward learning algorithm BIBREF7 to the supervised setting for automated data manipulation. \n Question: What off-the-shelf reward learning algorithm from RL for joint data manipulation learning and model training is adapted?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-28e643e4bdbf42c2835340b21820bcb1",
            "input": "In general, the baseline model CAEVO is outperformed by both NN models, and NN model with BERT embedding achieves the greatest performance. \n Question: Do the BERT-based embeddings improve results?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-e6202734a75f4b7fbc4557969cdb233d",
            "input": "As shown in Figure FIGREF7 , KAR is an end-to-end MRC model consisting of five layers:\n\nLexicon Embedding Layer. This layer maps the words to the lexicon embeddings. Context Embedding Layer. This layer maps the lexicon embeddings to the context embeddings. Coarse Memory Layer. This layer maps the context embeddings to the coarse memories. Refined Memory Layer. This layer maps the coarse memories to the refined memories. Answer Span Prediction Layer. This layer predicts the answer start position and the answer end position based on the above layers. \n Question: What type of model is KAR?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-68005bc694154c9a8fa976bb470966a4",
            "input": "In this paper we proposed a methodology to identify words that could lead to confusion at any given node of a speech recognition based system. We used edit distance as the metric to identifying the possible confusion between the active words.  There is a significant saving in terms of being able to identify recognition bottlenecks in a menu based speech solution through this analysis because it does not require actual people testing the system.  Actual use of this analysis was carried out for a speech solution developed for Indian Railway Inquiry System to identify bottlenecks in the system before its actual launch. \n Question: what bottlenecks were identified?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-b259a5ce26584f1b9fb8e1b9eadfb290",
            "input": "This paper experimented with two different models and compared them against each other.  \n Question: what was the baseline?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-28765f93035e45b8b4f0b8d3f08b02ed",
            "input": "In this experiment, we compute the correlation of the proposed NED measure with the patient-perceived emotional bond ratings. Since the proposed measure is asymmetric in nature, we compute the measures for both patient-to-therapist and therapist-to-patient entrainment. We report Pearson's correlation coefficients ( INLINEFORM0 ) for this experiment in Table TABREF26 along with their INLINEFORM1 -values.  \n Question: How do they correlate NED with emotional bond levels?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-9fc7ab7be3b34b269bdc884b31f1c23f",
            "input": "With a single text, we were already able to predict the electricity consumption with a relative error of less than 5% for both data sets. \n Question: How accurate is model trained on text exclusively?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-4334c1df7f444b77aaa3e5f205414bdc",
            "input": "We train an LSTM with and without attention on the training set. After training, we take the best model in terms of BLEU score BIBREF16 on the development set and calculate the BLEU score on the test set.  \n Question: How is the generative model evaluated?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-b0bcf750ed76488a877d1adf94421bf4",
            "input": "the attention heads in the proposed adaptively sparse Transformer can specialize more and with higher confidence In particular, we analyzed how the attention heads in the proposed adaptively sparse Transformer can specialize more and with higher confidence. \n Question: How does their model improve interpretability compared to softmax transformers?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-a5e639e39a2a43629bd1d1187c2b3262",
            "input": "It thus seems feasible that by using a similar model, we can produce text samples that are conditioned upon a set of user-specified keywords. To illustrate by example, say a user desires the generated output to contain the keywords, $\\lbrace subway, manhattan\\rbrace $ .  \n Question: What are the user-defined keywords?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-7bc8b8f29ae843d0abacf71d18a53df2",
            "input": "To counter that, we use a left-to-right attention mask, similar to the one employed in the original Transformer decoder BIBREF1. For the input tokens in $X$, we apply such mask to all the target tokens $Y$ that were concatenated to $X$, so that input tokens can only attend to the other input tokens. Conversely, for target tokens $y_t$, we put an attention mask on all tokens $y_{>t}$, allowing target tokens $y_t$ to attend only to the input tokens and the already generated target tokens. \n Question: What is different in BERT-gen from standard BERT?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-aca39023638e4dbe8afdaba9222168c9",
            "input": "TABREF5 displays the output behavior of models that we damaged to resemble the damage that causes aphasia. Because the output languages in all of our domains use tokens to represent meanings in many cases, it is expected that the analog to Wernicke's area is responsible for maintaining a high precision. \n Question: Do they perform a quantitative analysis of their model displaying knowledge distortions?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-b94a5e3d08ff4a339a564bc6f544de60",
            "input": "We obtained 1260 unique utterances having commands to AMIE from our in-cabin data-set. We expanded this data-set via Amazon Mechanical Turk and ended up with 3347 utterances having intents. The annotations for intents and slots are obtained on the transcribed utterances by majority voting of 3 annotators. \n Question: What is the size of their collected dataset?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-57d182d03a194dccba9e59274ab8b816",
            "input": "Firstly, LastStateRNN is the classic RNN model, where the last state passes through an MLP and then the LR Layer estimates the corresponding probability. In contrast, in the AvgRNN model we consider the average vector of all states that come out of the cells. The AttentionRNN model is the one that it has been presented in BIBREF9. Moreover, we introduce the MultiAttentionRNN model for the harassment language detection, which instead of one attention, it includes four attentions, one for each category. \n Question: What baseline model is used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-f1ed4160aa9d462d9aaf398b0adeb62a",
            "input": "For the future work, we plan to solve the triples with multiple entities as the second entity, which is excluded from problem scope in this paper. The input of QA4IE is a document $D$ with an existing knowledge base $K$ and the output is a set of relation triples $R = \\lbrace e_i, r_{ij}, e_j\\rbrace $ in $D$ where $e_i$ and $e_j$ are two individual entities and $r_{ij}$ is their relation. \n Question: Can this approach model n-ary relations?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-eeae2390349044a4a47e7cb9df8cbe7f",
            "input": "In light of this, we suggest that care is needed when applying graph embeddings in NLP pipelines, and work needed to develop robust methods to debias such embeddings. \n Question: Do they propose any solution to debias the embeddings?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-4168bccca5314dbcafaa490365753926",
            "input": "In natural language, subjectivity refers to the aspects of communication used to express opinions, evaluations, and speculationsBIBREF0, often influenced by one's emotional state and viewpoints. In this work, we investigate the application of BERT-based models for the task of subjective language detection. Experiments ::: Dataset and Experimental Settings\nWe perform our experiments on the WNC dataset open-sourced by the authors of BIBREF2. It consists of aligned pre and post neutralized sentences made by Wikipedia editors under the neutral point of view. It contains $180k$ biased sentences, and their neutral counterparts crawled from $423,823$ Wikipedia revisions between 2004 and 2019 \n Question: Which experiments are perfomed?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-a9f70a1b5a9c4c3495d1bb534f9c3cf7",
            "input": "We present LOG-Cad, a neural network-based description generator (Figure FIGREF1 ) to directly solve this task. Given a word with its context, our generator takes advantage of the target word's embedding, pre-trained from massive text (global contexts), while also encoding the given local context, combining both to generate a natural language description. \n Question: Do they use pretrained word embeddings?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-adda08d320cd418dba49b7e09b1b9e45",
            "input": "First, in order to deal with the unseen relations, we propose to break the relation names into word sequences for question-relation matching. Second, noticing that original relation names can sometimes help to match longer question contexts, we propose to build both relation-level and word-level relation representations. Third, we use deep bidirectional LSTMs (BiLSTMs) to learn different levels of question representations in order to match the different levels of relation information. Finally, we propose a residual learning method for sequence matching, which makes the model training easier and results in more abstract (deeper) question representations, thus improves hierarchical matching. \n Question: What they use in their propsoed framework?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-117e3ebfb613417294417f048c70d497",
            "input": "The retweeting behavior of MEPs is captured by their retweet network. Each MEP active on Twitter is a node in this network. An edge in the network between two MEPs exists when one MEP retweeted the other. The weight of the edge is the number of retweets between the two MEPs We measure the cohesion of a political group INLINEFORM0 as the average retweets, i.e., the ratio of the number of retweets between the MEPs in the group INLINEFORM1 to the number of MEPs in the group INLINEFORM2  \n Question: Do they authors account for differences in usage of Twitter amongst MPs into their model?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-a555416029774d3aad782d87722fb082",
            "input": "Our semi-supervised approach is quite straightforward: first a model is trained on the training set and then this model is used to predict the labels of the silver data. This silver data is then simply added to our training set, after which the model is retrained. However, an extra step is applied to ensure that the silver data is of reasonable quality. \n Question: What semi-supervised learning is applied?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-66d082d41f06437aa8707d8b2b9dc8e3",
            "input": "The resource is composed of data from two different surveys. In both surveys subjects were asked to draw on a map (displayed under a Mercator projection) a polygon representing a given geographical descriptor, in the context of the geography of Galicia in Northwestern Spain (see Fig. FIGREF1 ). The first survey was run in order to obtain a high number of responses to be used as an evaluation testbed for modeling algorithms. It was answered by 15/16 year old students in a high school in Pontevedra (located in Western Galicia). 99 students provided answers for a list of 7 descriptors (including cardinal points, coast, inland, and a proper name). The second survey was addressed to meteorologists in the Galician Weather Agency BIBREF12 . \n Question: Which two datasets does the resource come from?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-77240966771d4ce596054d84e2ab78f4",
            "input": "Specifically, we take the CORD-19 dataset BIBREF2, which contains over 45,000 scholarly articles, including over 33,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses. We develop sentence classification methods to identify all sentences narrating radiological findings from COVID-19.  \n Question: What is the CORD-19 dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-7d3ba6b1a3524382b194c4430bd8ed63",
            "input": "The task of Word Sense Induction (WSI) can be seen as an unsupervised version of WSD. WSI aims at clustering word senses and does not require to map each cluster to a predefined sense. Instead of that, word sense inventories are induced automatically from the clusters, treating each cluster as a single sense of a word. We suggest a more advanced procedure of graph construction that uses the interpretability of vector addition and subtraction operations in word embedding space BIBREF6 while the previous algorithm only relies on the list of nearest neighbours in word embedding space. The key innovation of our algorithm is the use of vector subtraction to find pairs of most dissimilar graph nodes and construct the graph only from the nodes included in such “anti-edges”. \n Question: Is the method described in this work a clustering-based method?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-cd69fa05e7a04de0a3240e000ea73e3d",
            "input": "his reduces significantly the needed training time: SBERT can be tuned in less than 20 minutes, while yielding better results than comparable sentence embedding methods. \n Question: How much time takes its training?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-1c7fa58f12db4155966fddc7b25099c9",
            "input": "In this case, the decoding function is a linear projection, which is $= f_{\\text{de}}()=+ $ , where $\\in ^{d_\\times d_}$ is a trainable weight matrix and $\\in ^{d_\\times 1}$ is the bias term. A family of bijective transformation was designed in NICE BIBREF17 , and the simplest continuous bijective function $f:^D\\rightarrow ^D$ and its inverse $f^{-1}$ is defined as:\n\n$$h: \\hspace{14.22636pt} _1 &= _1, & _2 &= _2+m(_1) \\nonumber \\\\ h^{-1}: \\hspace{14.22636pt} _1 &= _1, & _2 &= _2-m(_1) \\nonumber $$ (Eq. 15)\n\nwhere $_1$ is a $d$ -dimensional partition of the input $\\in ^D$ , and $m:^d\\rightarrow ^{D-d}$ is an arbitrary continuous function, which could be a trainable multi-layer feedforward neural network with non-linear activation functions. It is named as an `additive coupling layer' BIBREF17 , which has unit Jacobian determinant. To allow the learning system to explore more powerful transformation, we follow the design of the `affine coupling layer' BIBREF24 :\n\n$$h: \\hspace{5.69046pt} _1 &= _1, & _2 &= _2 \\odot \\text{exp}(s(_1)) + t(_1) \\nonumber \\\\ h^{-1}: \\hspace{5.69046pt} _1 &= _1, & _2 &= (_2-t(_1)) \\odot \\text{exp}(-s(_1)) \\nonumber $$ (Eq. 16)\n\nwhere $s:^d\\rightarrow ^{D-d}$ and $t:^d\\rightarrow ^{D-d}$ are both neural networks with linear output units.\n\nThe requirement of the continuous bijective transformation is that, the dimensionality of the input $$ and the output $$ need to match exactly. \n Question: What are the two decoding functions?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-9f35346c6aca424681603fd4a37c66ec",
            "input": "Our system, including software and corpus, is available as an open source project for free research purpose and we believe that it is a good baseline for the development and comparison of future Vietnamese SRL systems.  \n Question: Are their corpus and software public?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-33bb2d3fe2c0427e9b2883275e7e7e90",
            "input": " In order to fully leverage gloss information, we propose GlossBERT to construct context-gloss pairs from all possible senses of the target word in WordNet, thus treating WSD task as a sentence-pair classification problem. \n Question: Do they incoprorate WordNet into the model?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-8bdac86f69664101919701c9eab41484",
            "input": "Table 2 lists the accuracies of the all methods on two classifier architectures. The results show that, for various datasets on different classifier architectures, our conditional BERT contextual augmentation improves the model performances most. BERT can also augments sentences to some extent, but not as much as conditional BERT does. \n Question: Do the authors report performance of conditional bert on tasks without data augmentation?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-8c14bf40c4ae4976ad8724c772bd47a7",
            "input": "With the goal of building a generalizable sentiment analysis model, we used three different training sets as provided in Table TABREF5 . One of these three datasets (Amazon reviews BIBREF23 , BIBREF24 ) is larger and has product reviews from several different categories including book reviews, electronics products reviews, and application reviews. The other two datasets are to make the model more specialized in the domain. In this paper we focus on restaurant reviews as our domain and use Yelp restaurant reviews dataset extracted from Yelp Dataset Challenge BIBREF25 and restaurant reviews dataset as part of a Kaggle competition BIBREF26 . \n Question: what dataset was used for training?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-66073653b20d4faea17964046e65aefc",
            "input": "In this paper, we propose a simple and effective method for learning affective events that only requires a very small seed lexicon and a large raw corpus. As illustrated in Figure FIGREF1, our key idea is that we can exploit discourse relations BIBREF4 to efficiently propagate polarity from seed predicates that directly report one's emotions (e.g., “to be glad” is positive). \n Question: How does their model learn using mostly raw data?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-9bf7caf055164716b4cda7ee89b0bf74",
            "input": "In this study, we focused on the LDA approaches to recommendation systems and given the importance of research, we have studied recent impressive articles on this subject and presented a taxonomy of recommendation systems based on LDA of the recent research. we evaluated ISWC and WWW conferences articles from DBLP website and used the Gibbs sampling algorithm as an evaluation parameter. We succeeded in discovering the relationship between LDA topics and paper features and also obtained the researchers' interest in research field. To perform approximate inference and learning LDA, there are many inference methods for LDA topic model such as Gibbs sampling, collapsed Variational Bayes, Expectation Maximization. Gibbs sampling is a popular technique because of its simplicity and low latency. However, for large numbers of topics, Gibbs sampling can become unwieldy. In this paper, we use Gibbs Sampling in our experiment in section 5. \n Question: How they utilize LDA and Gibbs sampling to evaluate ISWC and WWW publications?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-932d542299ac49fd8720162c4a2f3a84",
            "input": "We evaluate our model on a publicly available dataset, KARA ONE BIBREF17 , composed of multimodal data for stimulus-based, imagined and articulated speech state corresponding to 7 phonemic/syllabic ( /iy/, /piy/, /tiy/, /diy/, /uw/, /m/, /n/ ) as well as 4 words(pat, pot, knew and gnaw). The dataset consists of 14 participants, with each prompt presented 11 times to each individual.  \n Question: How many subjects does the EEG data come from?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-ca4a7dafe6fb46daa65eca1b388d949f",
            "input": "We analyze which discourse phenomena are hard to capture using monolingual data only. Among the four phenomena in the test sets we use (deixis, lexical cohesion, VP ellipsis and ellipsis which affects NP inflection) we find VP ellipsis to be the hardest phenomenon to be captured using round-trip translations. \n Question: what phenomena do they mention is hard to capture?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-bffda0251613402eb00e72cd0c259148",
            "input": "We crowd-sourced the collection of the dataset on Amazon Mechanical Turk (MTurk). \n Question: Did the authors use crowdsourcing platforms?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-e0f3e064dc7a47a0b53a3a2bb4947328",
            "input": "We train both of the systems on the WMT15 German-to-English training data, see Table TABREF18 for some statistics. For this purpose, we use manual alignments provided by RWTH German-English dataset as the hard alignments. \n Question: What datasets are used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-8545e368fc194ffd80f5a9c2a824760b",
            "input": "After averaging over all concepts, we lose information on the lexical variation that each concept presents but on the other hand one can now investigate which regions show similar geolectal variation, yielding well defined linguistic varieties. Those cells that have similar colors in either figure FIGREF16 or figure FIGREF17 are expected to be ascribed to the same dialect zone. Thus, we can distinguish two main regions or clusters in the maps. The purple background covers most of the map and represents rural regions with small, scattered population. Our analysis shows that this group of cells possesses more specific words in their lexicon. In contrast, the green and yellow cells form a second cluster that is largely concentrated on the center and along the coastline, which correspond to big cities and industrialized areas. In these cells, the use of standard Spanish language is widespread due probably to school education, media, travelers, etc. The character of its vocabulary is more uniform as compared with the purple group. While the purple cluster prefer particular utterances, the lexicon of the urban group includes most of the keywords. \n Question: What are the characteristics of the city dialect?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-c205eab7fc8f4af1815b62bf55029f6b",
            "input": "To enable this, we use a bifocal attention mechanism which computes an attention over fields at a macro level and over values at a micro level. We then fuse these attention weights such that the attention weight for a field also influences the attention over the values within it. Fused Bifocal Attention Mechanism\nIntuitively, when a human writes a description from a table she keeps track of information at two levels. At the macro level, it is important to decide which is the appropriate field to attend to next and at a micro level (i.e., within a field) it is important to know which values to attend to next. To capture this behavior, we use a bifocal attention mechanism as described below. Fused Attention: Intuitively, the attention weights assigned to a field should have an influence on all the values belonging to the particular field. To ensure this, we reweigh the micro level attention weights based on the corresponding macro level attention weights. In other words, we fuse the attention weights at the two levels as: DISPLAYFORM0\n\nwhere INLINEFORM0 is the field corresponding to the INLINEFORM1 -th value, INLINEFORM2 is the macro level context vector. \n Question: What is a bifocal attention mechanism?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-f4e4d298a205457999562ae8b3892e07",
            "input": " In particular, we use Semeval 2014 BIBREF34 Twitter Sentiment Analysis Dataset for the training Sarcasm Datasets Used in the Experiment\nThis dataset was created by BIBREF8 . The tweets were downloaded from Twitter using #sarcasm as a marker for sarcastic tweets. It is a monolingual English dataset which consists of a balanced distribution of 50,000 sarcastic tweets and 50,000 non-sarcastic tweets.\n\nSince sarcastic tweets are less frequently used BIBREF8 , we also need to investigate the robustness of the selected features and the model trained on these features on an imbalanced dataset. To this end, we used another English dataset from BIBREF8 . It consists of 25,000 sarcastic tweets and 75,000 non-sarcastic tweets.\n\nWe have obtained this dataset from The Sarcasm Detector. It contains 120,000 tweets, out of which 20,000 are sarcastic and 100,000 are non-sarcastic. We randomly sampled 10,000 sarcastic and 20,000 non-sarcastic tweets from the dataset. Visualization of both the original and subset data show similar characteristics. \n Question: Which benchmark datasets are used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-a50e17ef813145478fe80cd998329852",
            "input": "For our study, we used the number of retweets to single-out those that went viral within our sample. Tweets within that subset (viral tweets hereafter) are varied and relate to different topics. We consider that a tweet contains fake news if its text falls within any of the following categories described by Rubin et al. BIBREF7 (see next section for the details of such categories): serious fabrication, large-scale hoaxes, jokes taken at face value, slanted reporting of real facts and stories where the truth is contentious. The dataset BIBREF8 , manually labelled by an expert, has been publicly released and is available to researchers and interested parties. \n Question: How did they determine fake news tweets?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-c6dd02a46efe4f2598654796e8c10b9c",
            "input": "We address this tension by training vector space models to represent the data, in which each unique word in a large corpus is represented by a vector (embedding) in high-dimensional space. The geometry of the resulting vector space captures many semantic relations between words.  We address this tension by training vector space models to represent the data, in which each unique word in a large corpus is represented by a vector (embedding) in high-dimensional space. The geometry of the resulting vector space captures many semantic relations between words.  \n Question: Do they model semantics ",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-1c6c708ea08e478eb88589d1abe61f55",
            "input": "Even if LangID-High does not present a more accurate result, it does present a more compact one: LangID-High is 15.4 MB, while the combined wFST high resource models are 197.5 MB. \n Question: how is model compactness measured?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-1241a3b4b25742e8affd39b6299cdb40",
            "input": "In this subsection, we see the influence of each component of a model on performance by removing or replacing its components. the SNLI dataset is used for experiments, and the best performing configuration is used as a baseline for modifications. We consider the following variants: (i) models that use plain stacked LSTMs, (ii) models with different INLINEFORM0 , (iii) models without INLINEFORM1 , and (iv) models that integrate lower contexts via peephole connections. \n Question: What were the baselines?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-32f1f19b824e4c2fbb431b0d314a541a",
            "input": "We compare the results of our model to a baseline multi-task architecture inspired by yang2016multi. In our baseline model there are no explicit connections between tasks - the only shared parameters are in the hidden layer. \n Question: What is the baseline?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-7c80f27d34e94eac92e31ad4d54ad759",
            "input": "We cluster the embeddings with INLINEFORM0 -Means.  \n Question: How were the cluster extracted? ",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-348dd5e2cda642bea9d2bee480b6b0f3",
            "input": "We now seek to know if a pre-trained multi-BERT has ability to solve RC tasks in the zero-shot setting. \n Question: What model is used as a baseline?  ",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-97d15b9694954c338cc27a1f76d12565",
            "input": "Conditional Random Fields\nConditional Random Fields (CRF) BIBREF10 are a standard approach when dealing with sequential data in the context of sequence labeling. BiLSTM-CRF\nPrior to the emergence of deep neural language models, BiLSTM-CRF models BIBREF11 had achieved state-of-the-art results for the task of sequence labeling. Multi-Task Learning\nMulti-Task Learning (MTL) BIBREF15 has become popular with the progress in deep learning. BioBERT\nDeep neural language models have recently evolved to a successful method for representing text. In particular, Bidirectional Encoder Representations from Transformers (BERT) outperformed previous state-of-the-art methods by a large margin on various NLP tasks BIBREF17. \n Question: What baseline systems are proposed?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-7541532d02dd4596ace5067d50f2431d",
            "input": "More specifically, we explore three regularization terms to address the problem: (1) a regularization term associated with neutral features; (2) the maximum entropy of class distribution regularization term; and (3) the KL divergence between reference and predicted class distribution. \n Question: What are the three regularization terms?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-492d916f150c441e825c2b359c96321b",
            "input": "To test the effectiveness of this new architecture in forecasting derailment of online conversations, we develop and distribute two new datasets. The first triples in size the highly curated `Conversations Gone Awry' dataset BIBREF9, where civil-starting Wikipedia Talk Page conversations are crowd-labeled according to whether they eventually lead to personal attacks; the second relies on in-the-wild moderation of the popular subreddit ChangeMyView, where the aim is to forecast whether a discussion will later be subject to moderator action due to “rude or hostile” behavior.  \n Question: What are two datasets model is applied to?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-93f18327a1f84551a05181946a16cad6",
            "input": "We therefore propose attention models to infer the latent context, i.e., the series of posts that trigger an intervention. \n Question: What type of latent context is used to predict instructor intervention?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-94da89ebe4e74b32a2f76a4aec87a191",
            "input": "Consequently, we investigate ways to detect suspicious accounts by considering their tweets in groups (chunks). Our hypothesis is that suspicious accounts have a unique pattern in posting tweet sequences. Since their intention is to mislead, the way they transition from one set of tweets to the next has a hidden signature, biased by their intentions. Given a news Twitter account, we read its tweets from the account's timeline. Then we sort the tweets by the posting date in ascending way and we split them into $N$ chunks. Each chunk consists of a sorted sequence of tweets labeled by the label of its corresponding account. \n Question: How is a \"chunk of posts\" defined in this work?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-41bda16034dd4189a7150fae550d07c7",
            "input": "In the proposed model, in addition to the co-occurrence edges, we link two nodes (words) if the corresponding word embedding representation is similar. \n Question: Do the use word embeddings alone or they replace some previous features of the model with word embeddings?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-2ef337d0f1ce413aa0a0666d3c65eb04",
            "input": "In addition, we note that our approach can still lead to erroneous facts or even hallucinations. An interesting perspective might be to further constrain the model on the data structure in order to prevent inaccurate of even contradictory descriptions. \n Question: What future possible improvements are listed?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-45576f8d09dd4bb1b00549c341529af0",
            "input": "Since the input of our method is textual data, we follow the approach of BIBREF15 and map the text into a fixed-size vector representation. To this end, we use word embeddings that were successfully applied in other domains. We follow BIBREF5 and use pre-trained GloVe word vectors BIBREF16 to initialize the embedding layer (also known as look-up table). Section SECREF18 discusses the embedding layer in more details. \n Question: Which pretrained word vectors did they use?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-99f730370a9c4aa9ad65eb44656e51e8",
            "input": "Causal documents were chosen to contain one occurrence only of the exact unigrams: `caused', `causing', or `causes'. \n Question: How do they extract causality from text?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-a9bd70f4810d4d9abfea64f174811642",
            "input": "We compare our approaches with related approaches of pivoting, multilingual NMT (MNMT) BIBREF19, and cross-lingual transfer without pretraining BIBREF16. Although it is challenging for one model to translate all zero-shot directions between multiple distant language pairs of MultiUN, MLM+BRLM-SA still achieves better performances on Es $\\rightarrow $ Ar and Es $\\rightarrow $ Ru than strong pivoting$_{\\rm m}$, which uses MNMT to translate source to pivot then to target in two separate steps with each step receiving supervised signal of parallel corpora.  \n Question: what are the pivot-based baselines?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-0177657ede9846978b13be734dfa77cc",
            "input": "Benchmark Evaluation ::: Classifier Models\nSVM: A linear support vector machine with bag-of-words sentence representations.\n\nMLP: A multi-layer perceptron with USE embeddings BIBREF4 as input.\n\nFastText: A shallow neural network that averages embeddings of n-grams BIBREF5.\n\nCNN: A convolutional neural network with non-static word embeddings initialized with GloVe BIBREF6.\n\nBERT: A neural network that is trained to predict elided words in text and then fine-tuned on our data BIBREF1.\n\nPlatforms: Several platforms exist for the development of task-oriented agents. We consider Google's DialogFlow and Rasa NLU with spacy-sklearn. \n Question: Which classifiers are evaluated?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-94e46cf17cca4e0896990b83657cb692",
            "input": "The final collection of Facebook pages for the experiments described in this paper is as follows: FoxNews, CNN, ESPN, New York Times, Time magazine, Huffington Post Weird News, The Guardian, Cartoon Network, Cooking Light, Home Cooking Adventure, Justin Bieber, Nickelodeon, Spongebob, Disney. \n Question: Which Facebook pages did they look at?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-66933cff3fd2440f980bf706d10b5441",
            "input": "We used two datasets for the task - AMR Bank BIBREF10 and CNN-Dailymail ( BIBREF11 BIBREF12 ).  \n Question: What dataset is used in this paper?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-88ebb6ddf2124ff9a9e4621e58dc72e5",
            "input": "The second Turkish dataset is the Twitter corpus which is formed of tweets about Turkish mobile network operators. Those tweets are mostly much noisier and shorter compared to the reviews in the movie corpus. In total, there are 1,716 tweets. 973 of them are negative and 743 of them are positive. These tweets are manually annotated by two humans, where the labels are either positive or negative. \n Question: What details are given about the Twitter dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-1080678bfeed438bb411f9e25025e297",
            "input": "Unfortunately, $\\mathit {PMI}(w,c)$ goes to negative infinity when the word-context pair $(w,c)$ does not appear in the training corpus. Due to unreliable statistics, this happens very frequently in finite corpora.  \n Question: Why are statistics from finite corpora unreliable?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-6db4c28013c74e7daefc046dea9dd081",
            "input": "For the McGurk effect, we attempt an illusion for a language token (e.g. phoneme, word, sentence) $x$ by creating a video where an audio stream of $x$ is visually dubbed over by a person saying $x^{\\prime }\\ne x$ . The illusion $f(x^{\\prime },x)$ affects a listener if they perceive what is being said to be $y\\ne x$ if they watched the illusory video whereas they perceive $x$ if they had either listened to the audio stream without watching the video or had watched the original unaltered video, depending on specification.  A prototypical example is that the audio of the phoneme “baa,” accompanied by a video of someone mouthing “vaa”, can be perceived as “vaa” or “gaa” (Figure 1 ). \n Question: What is the McGurk effect?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-ce373f88339447e7ac7d80a59111a7b2",
            "input": "For dataset collection, we utilize the headlines collected in qin2018automatic, lin2019learning from Tencent News, one of the most popular Chinese news websites, as the positive samples. We follow the same data split as the original paper. As some of the links are not available any more, we get 170,754 training samples and 4,511 validation samples. For the negative training samples collection, we randomly select generated headlines from a pointer generator BIBREF0 model trained on LCSTS dataset BIBREF5 and create a balanced training corpus which includes 351,508 training samples and 9,022 validation samples. To evaluate our trained classifier, we construct a test set by randomly sampling 100 headlines from the test split of LCSTS dataset and the labels are obtained by 11 human annotators. We use LCSTS BIBREF5 as our dataset to train the summarization model. The dataset is collected from the Chinese microblogging website Sina Weibo. It contains over 2 million Chinese short texts with corresponding headlines given by the author of each text. The dataset is split into 2,400,591 samples for training, 10,666 samples for validation and 725 samples for testing. We tokenize each sentence with Jieba and a vocabulary size of 50000 is saved. \n Question: Did they used dataset from another domain for evaluation?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-f2affef914a24a8b99c8b76cbc76350b",
            "input": "Thus, we can distinguish two main regions or clusters in the maps. The purple background covers most of the map and represents rural regions with small, scattered population. Our analysis shows that this group of cells possesses more specific words in their lexicon. In contrast, the green and yellow cells form a second cluster that is largely concentrated on the center and along the coastline, which correspond to big cities and industrialized areas.  In these cells, the use of standard Spanish language is widespread due probably to school education, media, travelers, etc. \n Question: Do the authors mention any possible confounds in their study?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-e5df132c1ea248a19e894d9eb4ca4c63",
            "input": "We consider QA PGNet and Multi-decoder QA PGNet with lookup table embedding as baseline models and improve on the baselines with other variations described below. \n Question: What is the baseline?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-dabd4740a858443cb4fa50e1d5b44b79",
            "input": "In Table TABREF15 we measure BLEU BIBREF19, NIST BIBREF20, METEOR BIBREF21, ROUGE-L BIBREF22 and CIDEr BIBREF23 metrics on the 2018 E2E NLG Challenge test data using the evaluation script provided by the organizers.  We use the gold standard selection during training and validation of the text generation model, as well as the automatic evaluation. As we deploy our text generation model for manual evaluation, we use a Conditional Random Field (CRF) model to predict which events to mention. The second human evaluation aimed at judging the acceptability of the output for production use in a news agency.  In the minimum edit evaluation, carried out by the annotator who created the news corpus, only factual mistakes and grammatical errors are corrected, resulting in text which may remain awkward or unfluent. The word error rate (WER) of the generated text compared to its corrected variant as a reference is 5.6% (6.2% disregarding punctuation).  The factual errors and their types are summarized in Table TABREF23. From the total of 510 game events generated by the system, 78 of these contained a factual error, i.e. 84.7% were generated without factual errors. Most fluency issues relate to the overall flow and structure of the report. Addressing these issues would require the model to take into account multiple events in a game, and combine the information more flexibly to avoid repetition.  The second human evaluation aimed at judging the acceptability of the output for production use in a news agency. The output is evaluated in terms of its usability for a news channel labelled as being machine-generated, i.e. not aiming at the level of a human journalist equipped with substantial background information. The evaluation was carried out by two journalists from the STT agency, who split the 59 games among themselves approximately evenly. \n Question: What evaluation criteria and metrics were used to evaluate the generated text?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-47b93960ec664de5992ef7ec32162c81",
            "input": "After using Kytea, Japanese texts are applied LSW algorithm to replace OOV words by their synonyms.  \n Question: Are synonymous relation taken into account in the Japanese-Vietnamese task?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-6d6c2ed136c0465f8be248c1ad90e4e7",
            "input": "Looking at the current WinoGrande leaderboard, it appears that the previous state of the art is based on RoBERTa BIBREF2, which can be characterized as an encoder-only transformer architecture. \n Question: What is the previous state of the art?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-2d0a6815421d4ed0942af800a7fd3858",
            "input": "While manually analyzing the raw dataset, we noticed that looking at the tweet one has replied to or has quoted, provides significant contextual information. We call these, “context tweets\". As humans can better understand a tweet with the reference of its context, our assumption is that computers also benefit from taking context tweets into account in detecting abusive language.\n\nAs shown in the examples below, (2) is labeled abusive due to the use of vulgar language. However, the intention of the user can be better understood with its context tweet (1).\n\n(1) I hate when I'm sitting in front of the bus and somebody with a wheelchair get on.\n\nINLINEFORM0 (2) I hate it when I'm trying to board a bus and there's already an as**ole on it.\n\nSimilarly, context tweet (3) is important in understanding the abusive tweet (4), especially in identifying the target of the malice.\n\n(3) Survivors of #Syria Gas Attack Recount `a Cruel Scene'.\n\nINLINEFORM0 (4) Who the HELL is “LIKE\" ING this post? Sick people....\n\nHuang et al. huang2016modeling used several attributes of context tweets for sentiment analysis in order to improve the baseline LSTM model. However, their approach was limited because the meta-information they focused on—author information, conversation type, use of the same hashtags or emojis—are all highly dependent on data.\n\nIn order to avoid data dependency, text sequences of context tweets are directly used as an additional feature of neural network models. We use the same baseline model to convert context tweets to vectors, then concatenate these vectors with outputs of their corresponding labeled tweets. More specifically, we concatenate max-pooled layers of context and labeled tweets for the CNN baseline model. As for RNN, the last hidden states of context and labeled tweets are concatenated. \n Question: What additional features and context are proposed?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-ddd0e1fc11b64befa17d6dc960cec1fd",
            "input": "Baselines. We compare our models against a neural baseline models, hierarchical LSTM (hLSTM), with the attention ablated but with access to the complete context, and a strong, open-sourced feature-rich baseline BIBREF7 . We choose BIBREF7 over other prior works such as BIBREF0 since we do not have access to the dataset or the system used in their papers for replication. BIBREF7 is a logistic regression classifier with features inclusive of bag-of-words representation of the unigrams and thread length, normalised counts of agreements to previous posts, counts of non-lexical reference items such as URLs, and the Coursera forum type in which a thread appeared. We also report aggregated results from a hLSTM model with access only to the last post as context for comparison. Table TABREF17 compares the performance of these baselines against our proposed methods. \n Question: What was the previous state of the art for this task?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-c9a6ee6711e54383a06e0ef75a6fefde",
            "input": "Our data has been developed by crawling and pre-processing an OSG web forum. The forum has a great variety of different groups such as depression, anxiety, stress, relationship, cancer, sexually transmitted diseases, etc. \n Question: How did they obtain the OSG dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-27378a4cf9624f35a95a6437dae286bb",
            "input": "We collected a simplified dataset from Simple English Wikipedia that are freely available, which has been previously used for many text simplification methods BIBREF0 , BIBREF10 , BIBREF3 . \n Question: what language does this paper focus on?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-506c0b709f5c4e0f9be1e0babd12bb7d",
            "input": "Hence we have determined our targets as Galatasaray (namely Target-1) and Fenerbahçe (namely, Target-2) which are two of the most popular football clubs in Turkey. \n Question: Which sports clubs are the targets?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-973741167f95473ca71e9037f10855a0",
            "input": "Performing the Welch's t-test, both changes after weGAN training are statistically significant at a INLINEFORM0 significance level.  \n Question: Do they evaluate grammaticality of generated text?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-d3018b52a5ad40be9aa1b1ffa1863254",
            "input": "While a host of input modalities have been considered in other NLG tasks, such as text summarization BIBREF24 , image captioning BIBREF25 and table-to-text generation BIBREF26 , traditional QG mainly focused on textual inputs, especially declarative sentences, explained by the original application domains of question answering and education, which also typically featured textual inputs.\n\nRecently, with the growth of various QA applications such as Knowledge Base Question Answering (KBQA) BIBREF27 and Visual Question Answering (VQA) BIBREF28 , NQG research has also widened the spectrum of sources to include knowledge bases BIBREF29 and images BIBREF10 . \n Question: What are all the input modalities considered in prior work in question generation?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-1a92080114a8449d8c0a49575a50e0a1",
            "input": "For this reason as a baseline algorithm for English dataset we refer to results from BIBREF0, and as for Russian dataset, we used the probabilistic language model, described in BIBREF8. \n Question: Is the RNN model evaluated against any baseline?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-3cb0d2b13b5242a6a01b8d87ad5f33ae",
            "input": "We show that by determining and integrating heterogeneous set of features from different modalities - aesthetic features from posted images (colorfulness, hue variance, sharpness, brightness, blurriness, naturalness), choice of profile picture (for gender, age, and facial expression), the screen name, the language features from both textual content and profile's description (n-gram, emotion, sentiment), and finally sociability from ego-network, and user engagement - we can reliably detect likely depressed individuals in a data set of 8,770 human-annotated Twitter users. \n Question: What is the source of the visual data? ",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-f87d536519e8473183c741676c4957e6",
            "input": "Manning and Schütze argue that, even though not quite correct, language text can be modeled as stationary, ergodic random processes BIBREF29, an assumption that we follow. Moreover, given the diversity of language production, we assume this stationary ergodic random process with finite alphabet $\\mathcal {A}$ denoted $X = \\lbrace X_i, -\\infty < i < \\infty \\rbrace $ is non-null in the sense that always $P(x_{-m}^{-1}) > 0$ and\n\nThis is sometimes called the smoothing requirement. \n Question: Is the assumption that natural language is stationary and ergodic valid?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-cac0b18fe1ad4c9681a1f341adac097d",
            "input": "We measure how contextual a word representation is using three different metrics: self-similarity, intra-sentence similarity, and maximum explainable variance. \n Question: What experiments are proposed to test that upper layers produce context-specific embeddings?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-1c8a1f6afe074b97b3e1820160d0535a",
            "input": "Therefore, there are 44 tasks (22 INLINEFORM10 2) in total. In addition, there are 2 human summaries for each task. We selected three competitive systems (SumBasic, ILP, and ILP+MC) and therefore we have 3 system-system pairs (ILP+MC vs. ILP, ILP+MC vs. SumBasic, and ILP vs. SumBasic) for each task and each human summary. \n Question: Do they build one model per topic or on all topics?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-1b6c37b02be5462390172bd7324a1f9e",
            "input": "First we are comparing BiLSTMBIBREF14, BiLSTM+CNNBIBREF20, BiLSTM+CRFBIBREF1, BiLSTM+CNN+CRFBIBREF2 models with CNN modelBIBREF0 and Stanford CRF modelBIBREF21. \n Question: What is the baseline?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-1a80e1d8b3034f8784ae78fe5bb06749",
            "input": "We attribute the significant performance gain of our model over the baseline to a combination of several factors as described below: \n Question: Do they train a different training method except from scheduled sampling?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-93fb5fab42d04d969a42a6c7e30896e3",
            "input": "Machine translation finds use in cheminformatics in “translation\" from one language (e.g. reactants) to another (e.g. products). The variational Auto-encoder (VAE) is another widely adopted text generation architecture BIBREF101. Generative Adversarial Network (GAN) models generate novel molecules by using two components: the generator network generates novel molecules, and the discriminator network aims to distinguish between the generated molecules and real molecules BIBREF107. \n Question: Are this models usually semi/supervised or unsupervised?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-1bb33ebc063247969793753d3d40acca",
            "input": "In this work, we use the datasets released by BIBREF1 and HEOT dataset provided by BIBREF0 .  This was then followed by transliteration using the Xlit-Crowd conversion dictionary and translation of each word to English using Hindi to English dictionary.  \n Question: Do all the instances contain code-switching?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-d1b7660ef8614036a86b8a8cc11e5afb",
            "input": "In ConMask, we use a similar idea to select the most related words given some relationship and mask irrelevant words by assigning a relationship-dependent similarity score to words in the given entity description. ConMask selects words that are related to the given relationship to mitigate the inclusion of irrelevant and noisy words. \n Question: Can the model add new relations to the knowledge graph, or just new entities?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-49a256872e2d4a11b34054aeb04b8346",
            "input": "The CreateDebate dataset was collected from an English online debate forum discussing four topics: abortion (ABO), gay rights (GAY), Obama (OBA), and marijuana (MAR).  \n Question: What topics are included in the debate data?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-131a178c9d7f411b98f53a677c85ad40",
            "input": "The corpus is freely available at the following link. \n Question: Is this dataset publicly available?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-713dfe124bf04edf91ae46264cd06b48",
            "input": "The model consists of one layer of LSTM followed by three dense layers. The LSTM layer uses a dropout value of 0.2.  \n Question: Do they use dropout?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-41469bbeb2be4fdfb00a9f1f777781d9",
            "input": "It was developed by querying the Followerwonk Web service API with location-neutral seed words known to be used by gang members across the U.S. in their Twitter profiles. \n Question: Do the authors report on English datasets only?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-41f0fa92717c4176916806a2b669b39d",
            "input": "Each one of these tokenizers, we combine with some types of representation methods, including word to vector methods such as continuous bag of words BIBREF5, pre-trained embedding as fasttext (trained on Wiki Vietnamese language) BIBREF6 and sonvx (trained on Vietnamese newspaper) BIBREF7 The dataset in this HSD task is really imbalance. Clean class dominates with 91.5%, offensive class takes 5% and the rest belongs to hate class with 3.5%. \n Question: What dataset do they use?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-41cd4618d30c4bf1b27780a8a527154d",
            "input": "For our experiment we decided to use the Europarl dataset, using the data from the WMT11 . \n Question: Are any experiments performed to try this approach to word embeddings?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-ab4730a52ef542b3a19efa30aa12978f",
            "input": "Contexts are either ground-truth contexts from that dataset, or they are Wikipedia passages retrieved using TF-IDF BIBREF24 based on a HotpotQA question. A new non-overlapping set of contexts was again constructed from Wikipedia via HotpotQA using the same method as Round 1. In addition to contexts from Wikipedia for Round 3, we also included contexts from the following domains: News (extracted from Common Crawl), fiction (extracted from BIBREF27, and BIBREF28), formal spoken text (excerpted from court and presidential debate transcripts in the Manually Annotated Sub-Corpus (MASC) of the Open American National Corpus), and causal or procedural text, which describes sequences of events or actions, extracted from WikiHow. Finally, we also collected annotations using the longer contexts present in the GLUE RTE training data, which came from the RTE5 dataset BIBREF29. \n Question: What data sources do they use for creating their dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-106cee1626ca428f97e47c8c43d807c2",
            "input": "Since pre-trained models operate on subword level, we need to estimate subword translation probabilities. For the purpose of evaluating the contextual representations learned by our model, we do not use part-of-speech tags. Contextualized representations are directly fed into Deep-Biaffine layers to predict arc and label scores. Table TABREF34 presents the Labeled Attachment Scores (LAS) for zero-shot dependency parsing. \n Question: What metrics are used for evaluation?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-686735697f584324a1337bf0f6fb4ea2",
            "input": "Firstly, LastStateRNN is the classic RNN model, where the last state passes through an MLP and then the LR Layer estimates the corresponding probability. In contrast, in the AvgRNN model we consider the average vector of all states that come out of the cells. The AttentionRNN model is the one that it has been presented in BIBREF9. \n Question: What was the baseline?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-a55cf999c46443faab3b0d66d4441235",
            "input": " In recent years, researchers have proposed various filter based feature selection methods to raise the performance of document text classification BIBREF19.  Furthermore, in order to rank the terms based on their discriminative power among the classes, we use filter based feature selection method named as Normalized Difference Measure (NDM)BIBREF5. Considering the features contour plot, Rehman et al. BIBREF5 suggested that all those features which exist in top left, and bottom right corners of the contour are extremely significant as compared to those features which exist around diagonals. State-of-the-art filter based feature selection algorithms such as ACC2 treat all those features in the same fashion which exist around the diagonals BIBREF5. \n Question: Is the filter based feature selection (FSE) a form of regularization?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-c6584ec2fb1f45bcad48958d88cfcade",
            "input": "In this work, we propose to generate unanswerable questions by editing an answerable question and conditioning on the corresponding paragraph that contains the answer. So the generated unanswerable questions are more lexically similar and relevant to the context. Moreover, by using the answerable question as a prototype and its answer span as a plausible answer, the generated examples can provide more discriminative training signal to the question answering model. \n Question: Does their approach require a dataset of unanswerable questions mapped to similar answerable questions?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-341f082cf92c4c6fab5da522f0b7b456",
            "input": "One common point in all the approaches yet has been the use of only textual features available in the dataset. Our model not only incorporates textual features, modeled using BiLSTM and augmented with an attention mechanism, but also considers related images for the task. \n Question: What are the differences with previous applications of neural networks for this task?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-792ae9b4f0684399992992e77003206e",
            "input": "We use two datasets for experiments (Table TABREF29 ): (a) STAN INLINEFORM0 , created by BIBREF10 BansalBV15, which consists of 1,108 unique English hashtags from 1,268 randomly selected tweets in the Stanford Sentiment Analysis Dataset BIBREF36 along with their crowdsourced segmentations and our additional corrections; and (b) STAN INLINEFORM1 , our new expert curated dataset, which includes all 12,594 unique English hashtags and their associated tweets from the same Stanford dataset. \n Question: How is the dataset of hashtags sourced?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-fcbd06c983a44646833618bf1534eaf1",
            "input": "We conduct experiments on the SQuAD dataset BIBREF3. \n Question: On what datasets are experiments performed?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-f37a6d9664d74f1f85db1bddb59e6f8e",
            "input": "For each task, the training data that was made available by the organizers is used, which is a selection of tweets with for each tweet a label describing the intensity of the emotion or sentiment BIBREF1 .  Since the present study focuses on Spanish tweets, all tweets from the English datasets were translated into Spanish. This new set of “Spanish” data was then added to our original training set. Again, the machine translation platform Apertium BIBREF5 was used for the translation of the datasets. \n Question: What dataset did they use?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-56c1f4282349474fbfdc8d57aff53023",
            "input": "Five attributes, specifying certain details of clinical significance, are defined to characterize the answer types of INLINEFORM4 : (1) the time the patient has been experiencing the symptom, (2) activities that trigger the symptom (to occur or worsen), (3) the extent of seriousness, (4) the frequency occurrence of the symptom, and (5) the location of symptom. For each symptom/attribute, it can take on different linguistic expressions, defined as entities. Note that if the queried symptom or attribute is not mentioned in the dialogue, the groundtruth output is “No Answer”, as in BIBREF6 . \n Question: What labels do they create on their dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-849f437881eb40809b54a41f185fbbf5",
            "input": "We compare our approaches with related approaches of pivoting, multilingual NMT (MNMT) BIBREF19, and cross-lingual transfer without pretraining BIBREF16.  The results show that our approaches consistently outperform other approaches across languages and datasets, especially surpass pivoting, which is a strong baseline in the zero-shot scenario that multilingual NMT systems often fail to beat BIBREF19, BIBREF20, BIBREF23. \n Question: which multilingual approaches do they compare with?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-e92273cf39bd4735a784e64c4cf007df",
            "input": "Advanced neural architectures based on character input (CNNs, BPE, etc) are supposed to be able to learn how to handle spelling and morphology variations themselves, even for languages with rich morphology: `just add more layers!'. Contextualised embedding models follow this tradition: as a rule, they are trained on raw text collections, with minimal linguistic pre-processing. Below, we show that this is not entirely true. \n Question: Why is lemmatization not necessary in English?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-07d040f42001496898cf4455e6cd3f08",
            "input": "We asked medical doctors experienced in extracting knowledge related to medical entities from texts to annotate the entities described above. Initially, we asked four annotators to test our guidelines on two texts. Subsequently, identified issues were discussed and resolved. Following this pilot annotation phase, we asked two different annotators to annotate two case reports according to our guidelines. The same annotators annotated an overall collection of 53 case reports. The annotation was performed using WebAnno BIBREF7, a web-based tool for linguistic annotation. The annotators could choose between a pre-annotated version or a blank version of each text. The pre-annotated versions contained suggested entity spans based on string matches from lists of conditions and findings synonym lists. \n Question: How was annotation performed?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-aa8632b9c482406797eaa549d9df9312",
            "input": "We would expect that explanation performance should correlate with prediction performance. Since Possible-answer knowledge is primarily needed to decide if the net has enough information to answer the challenge question without guessing and relevant-variable knowledge is needed for the net to know what to query, we analyzed the network's performance on querying and answering separately. The memory network has particular difficulty learning to query relevant variables, reaching only about .5 accuracy when querying. At the same time, it learns to answer very well, reaching over .9 accuracy there. Since these two parts of the interaction are what we ask it to explain in the two modes, we find that the quality of the explanations strongly correlates with the quality of the algorithm executed by the network. \n Question: How do they measure correlation between the prediction and explanation quality?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-f5880da437eb4bf9876ecdfec841e227",
            "input": "We used recordings of nurse-initiated telephone conversations for congestive heart failure patients undergoing telemonitoring, post-discharge from the hospital. To analyze the linguistic structure of the inquiry-response pairs in the entire 41-hour dataset, we randomly sampled a seed dataset consisting of 1,200 turns and manually categorized them to different types, which are summarized in Table TABREF14 along with the corresponding occurrence frequency statistics. \n Question: Which data do they use as a starting point for the dialogue dataset?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-1a5d67192da4402d922176ade76cd91a",
            "input": "Twitter provides well-documented API, which allows to request any information about Tweets, users and their profiles, with respect to rate limits. There is special type of API, called Streaming API, that provides a real-time stream of tweets. The key difference with regular API is that connection is kept alive as long as possible, and Tweets are sent in real-time to the client. There are three endpoints of Streaming API of our interest: “sample”, “filter” and “firehose”. The first one provides a sample (random subset) of the full Tweet stream. The second one allows to receive Tweets matching some search criteria: matching to one or more search keywords, produced by subset of users, or coming from certain geo location. The last one provides the full set of Tweets, although it is not available by default. In order to get Twitter “firehose” one can contact Twitter, or buy this stream from third-parties.\n\nIn our case the simplest approach would be to use “sample” endpoint, but it provides Tweets in all possible languages from all over the World, while we are concerned only about one language (Russian). In order to use this endpoint we implemented filtering based on language. The filter is simple: if Tweet does not contain a substring of 3 or more cyrillic symbols, it is considered non-Russian. Although this approach keeps Tweets in Mongolian, Ukrainian and other slavic languages (because they use cyrillic alphabet), the total amount of false-positives in this case is negligible. To demonstrate this we conducted simple experiment: on a random sample of 200 tweets only 5 were in a language different from Russian. In order not to rely on Twitter language detection, we chose to proceed with this method of language-based filtering.\n\nHowever, the amount of Tweets received through “sample” endpoint was not satisfying. This is probably because “sample” endpoint always streams the same content to all its clients, and small portion of it comes in Russian language. In order to force mining of Tweets in Russian language, we chose \"filter\" endpoint, which requires some search query. We constructed heuristic query, containing some auxiliary words, specific to Russian language: conjunctions, pronouns, prepositions. The full list is as follows:\n\nrussian я, у, к, в, по, на, ты, мы, до, на, она, он, и, да. \n Question: Which Twitter corpus was used to train the word vectors?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-c50f97b970a144e7ba3cb1b5d35f6951",
            "input": "We then proceed to connect these mentions i) if they co-occur within the same document (we will refer to this as DOC-BASED edges), ii) if the pair of named entity mentions is identical (MATCH edges—these may connect nodes across and within documents), or iii) if they are in the same coreference chain, as predicted by the external coreference system (COREF edges). \n Question: Did they use a relation extraction method to construct the edges in the graph?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-6c2cd2b9e3644853876d5fc760ba2747",
            "input": " Last, we evaluate our approaches in 9 commonly used text classification datasets. We evaluate our methods on several commonly used datasets whose themes range from sentiment, web-page, science to medical and healthcare. \n Question: What NLP tasks do they consider?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-08776962a7334f7ebc8e1363efdf7339",
            "input": "Based on annotated corpora and token-based features, studies used machine learning approaches to build word segmentation systems with accuracy about 94%-97%. \n Question: How successful are the approaches used to solve word segmentation in Vietnamese?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-ebd958333ad841e191f409d739b998e4",
            "input": " For a fair comparison with previous work, we train the baseline MT system on the data released by BIBREF11.  \n Question: what was the baseline?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-9bf3a1857f8847a792c0f009d5edf461",
            "input": "In Table TABREF1, we summarize the quantitative results of the above previous studies. It includes the model basis, feature description, languages classified and the used dataset along with accuracy obtained. The table also lists the overall results of our proposed models (at the top). The languages used by various authors along with their acronyms are English (En), Spanish (Es), French (Fr), German (De), Russian (Ru), Italian (It), Bengali (Ben), Hindi (Hi) and Telegu (Tel). \n Question: Is the performance compared against a baseline model?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-4b56b963d6dc424eb8e1a59b4a705a20",
            "input": "In this section, we describe our data collection for these 3 components. We first describe a series of pilot studies that we conducted in order to collect commonsense inference questions (Section SECREF4 ). In Section SECREF5 , we discuss the resulting data collection of questions, texts and answers via crowdsourcing on Amazon Mechanical Turk (henceforth MTurk). Section SECREF17 gives information about some necessary postprocessing steps and the dataset validation.  \n Question: how was the data collected?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-553f7101ca6e42ea910586c883c8926a",
            "input": "Thus we consider the task of answer generation for TweetQA and we use several standard metrics for natural language generation to evaluate QA systems on our dataset, namely we consider BLEU-1 BIBREF16 , Meteor BIBREF17 and Rouge-L BIBREF18 in this paper. \n Question: What evaluation metrics do they use?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-398e00012cba47f0801ae5d4a2d7d23e",
            "input": "Outlier Detection. The Outlier Detection task BIBREF0 is to determine which word in a list INLINEFORM0 of INLINEFORM1 words is unrelated to the other INLINEFORM2 which were chosen to be related. For each INLINEFORM3 , one can compute its compactness score INLINEFORM4 , which is the compactness of INLINEFORM5 . Sentiment analysis. We also consider sentiment analysis as described by BIBREF31 . \n Question: Do they test their word embeddings on downstream tasks?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-4f35e02bf587458db2ceaae4dde0da40",
            "input": "Recently bhat-EtAl:2017:EACLshort provided a CS dataset for the evaluation of their parsing models which they trained on the Hindi and English Universal Dependency (UD) treebanks. We extend this dataset by annotating 1,448 more sentences. \n Question: How big is the provided treebank?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-6e56c50184084efabdd914df405eb87e",
            "input": "Recommendation diversity. As defined in BIBREF18 , we calculate recommendation diversity as the average dissimilarity of all pairs of tags in the list of recommended tags. \n Question: how is diversity measured?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-fbf9f4b1c07943e284aba85eb3f9d34f",
            "input": "On another note, we apply our formalization to evaluate multilingual $\\textsc {bert} $'s syntax knowledge on a set of six typologically diverse languages. Although it does encode a large amount of information about syntax (more than $81\\%$ in all languages), it only encodes at most $5\\%$ more information than some trivial baseline knowledge (a type-level representation). This indicates that the task of POS labeling (word-level POS tagging) is not an ideal task for contemplating the syntactic understanding of contextual word embeddings. We see that—in all analysed languages—type level embeddings can already capture most of the uncertainty in POS tagging. We also see that BERT only shares a small amount of extra information with the task, having small (or even negative) gains in all languages. Finally, when put into perspective, multilingual $\\textsc {bert} $'s representations do not seem to encode much more information about syntax than a trivial baseline. $\\textsc {bert} $ only improves upon fastText in three of the six analysed languages—and even in those, it encodes at most (in English) $5\\%$ additional information. \n Question: Was any variation in results observed based on language typology?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-54067e1cab5042db8788ec420538809f",
            "input": "In Figure FIGREF28, we show some examples of the annotation results in CORD-19-NER. We can see that our distantly- or weakly supervised methods achieve high quality recognizing the new entity types, requiring only several seed examples as the input. For example, we recognized “SARS-CoV-2\" as the “CORONAVIRUS\" type, “bat\" and “pangolins\" as the “WILDLIFE\" type and “Van der Waals forces\" as the “PHYSICAL_SCIENCE\" type. This NER annotation results help downstream text mining tasks in discovering the origin and the physical nature of the virus. Our NER methods are domain-independent that can be applied to corpus in different domains. In addition, we show another example of NER annotation on New York Times with our system in Figure FIGREF29.\n\nIn Figure FIGREF30, we show the comparison of our annotation results with existing NER/BioNER systems. In Figure FIGREF30, we can see that only our method can identify “SARS-CoV-2\" as a coronavirus. In Figure FIGREF30, we can see that our method can identify many more entities such as “pylogenetic\" as a evolution term and “bat\" as a wildlife. In Figure FIGREF30, we can also see that our method can identify many more entities such as “racism\" as a social behavior. In summary, our distantly- and weakly-supervised NER methods are reliable for high-quality entity recognition without requiring human effort for training data annotation. \n Question: Did they experiment with the dataset?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-16cf61e72b4345b999903eb41bd36c58",
            "input": "Both the audio and the transcript are de-identified (by removing the identifying information) with digital zeros and [de-identified] tags, respectively. \n Question: Is the data de-identified?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-3e7dd3cdc36d40c5b6bcda5229255daa",
            "input": "In this section we present two datasets used in our experiments: The NowThisNews dataset, collected for the purpose of this paper, and The BreakingNews dataset BIBREF4 , publicly available dataset of news articles.\n\ncontains 4090 posts with associated videos from NowThisNews Facebook page collected between 07/2015 and 07/2016. For each post we collected its title and the number of views of the corresponding video, which we consider our popularity metric. Due to a fairly lengthy data collection process, we decided to normalize our data by first grouping posts according to their publication month and then labeling the posts for which the popularity metric exceeds the median monthly value as popular, the remaining part as unpopular. \n Question: Where do they obtain the news videos from?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-bd0271f0b70b4b21b8fcaac1256681bf",
            "input": "To evaluate the influence of our hypersphere feature for off-the-shelf NER systems, we perform the NE recognition on two standard NER benchmark datasets, CoNLL2003 and ONTONOTES 5.0. \n Question: Do they evaluate on NER data sets?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-43d6aef24593463388160d27aae4094e",
            "input": "The first step is to pre-train the model on one MCQA dataset referred to as the source task, which usually contains abundant training data. The second step is to fine-tune the same model on the other MCQA dataset, which is referred to as the target task, that we actually care about, but that usually contains much less training data.  \n Question: How different is the dataset size of source and target?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-b995a02b5d8a4391a6c7809a7e566149",
            "input": "The only requirement is that the model accepts as input, an embedding layer (for entities and relations). If a model fulfills this requirement (which a large number of neural models on knowledge graphs do), we can just use Dolores embeddings as a drop-in replacement. We just initialize the corresponding embedding layer with Dolores embeddings. \n Question: Is fine-tuning required to incorporate these embeddings into existing models?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-59d15d17521a40ba8883ebddc71de762",
            "input": "An emerging body of work in natural language processing and computational social science has investigated how NLP systems can detect moral sentiment in online text. For example, moral rhetoric in social media and political discourse BIBREF19, BIBREF20, BIBREF21, the relation between moralization in social media and violent protests BIBREF22, and bias toward refugees in talk radio shows BIBREF23 have been some of the topics explored in this line of inquiry. In contrast to this line of research, the development of a formal framework for moral sentiment change is still under-explored, with no existing systematic and formal treatment of this topic BIBREF16. \n Question: Does the paper discuss previous models which have been applied to the same task?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-83ed6acbd66b4328b7caa54a7252bde5",
            "input": "The dataset at HASOC 2019 were given in three languages: Hindi, English, and German. Dataset in Hindi and English had three subtasks each, while German had only two subtasks. We participated in all the tasks provided by the organisers and decided to develop a single model that would be language agnostic. We used the same model architecture for all the three languages. The training dataset of Hindi dataset was more balanced than English or German dataset. Hence, the results were around 0.78. As the dataset in German language was highly imbalanced, the results drops to 0.62. \n Question: What are the languages used to test the model?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-3a0d3d5fed144aaa93b7d87d603a1715",
            "input": "It is interesting to observe that the baseline model amplifies the bias in the training data set as measured by INLINEFORM0 and INLINEFORM1 . From measurements using the described bias metrics, our method effectively mitigates bias in language modelling without a significant increase in perplexity. \n Question: how is mitigation of gender bias evaluated?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-26be285238fc4e8f9c5fbf7aba8952e8",
            "input": "We compare with the following baselines:\n\n(1) Naive: A non-domain-adaptive baseline with bag-of-words representations and SVM classifier trained on the source domain.\n\n(2) mSDA BIBREF7 : This is the state-of-the-art method based on discrete input features. Top 1000 bag-of-words features are kept as pivot features. We set the number of stacked layers to 3 and the corruption probability to 0.5.\n\n(3) NaiveNN: This is a non-domain-adaptive CNN trained on source domain, which is a variant of our model by setting INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 to zeros.\n\n(4) AuxNN BIBREF4 : This is a neural model that exploits auxiliary tasks, which has achieved state-of-the-art results on cross-domain sentiment classification. The sentence encoder used in this model is the same as ours.\n\n(5) ADAN BIBREF16 : This method exploits adversarial training to reduce representation difference between domains. The original paper uses a simple feedforward network as encoder. For fair comparison, we replace it with our CNN-based encoder. We train 5 iterations on the discriminator per iteration on the encoder and sentiment classifier as suggested in their paper.\n\n(6) MMD: MMD has been widely used for minimizing domain discrepancy on images. In those works BIBREF9 , BIBREF13 , variants of deep CNNs are used for encoding images and the MMDs of multiple layers are jointly minimized. In NLP, adding more layers of CNNs may not be very helpful and thus those models from image-related tasks can not be directly applied to our problem. To compare with MMD-based method, we train a model that jointly minimize the classification loss INLINEFORM0 on the source domain and MMD between INLINEFORM1 and INLINEFORM2 . For computing MMD, we use a Gaussian RBF which is a common choice for characteristic kernel. \n Question: What are the baseline methods?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-3b9f84bf07c34313ab61c24fbf23af32",
            "input": "To achieve this, a method is proposed that transforms the goal-labels of the used dataset (DSTC2) into labels whose behaviour can be replicated during deployment. \n Question: what corpus is used to learn behavior?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-13064c9f833745419afa2926c9665ee8",
            "input": "In the sentiment analysis experiments, we found that a tweet may contain multiple sentiments. The groundtruth labels contain 210 positive sentiments, 521 neutral sentiments, and 305 negative sentiments to the candidates. \n Question: Is datasets for sentiment analysis balanced?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-9ca6fb3261ad4d0bbe3ba9e05de3cb78",
            "input": "As previous systems collect relevant data from knowledge bases after observing questions during evaluation BIBREF24 , BIBREF25 , we also explore using this option. Namely, we build a customized text corpus based on questions in commonsense reasoning tasks.  \n Question: Do they fine-tune their model on the end task?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-ef63d8f7dd32484fbc2d3a7033c6e65a",
            "input": "n this paper we provide a novel real-time and adaptive cryptocurrency price prediction platform based on Twitter sentiments. The integrative and modular platform copes with the three aforementioned challenges in several ways. Firstly, it provides a Spark-based architecture which handles the large volume of incoming data in a persistent and fault tolerant way. Secondly, the proposed platform offers an approach that supports sentiment analysis based on VADER which can respond to large amounts of natural language processing queries in real time. Thirdly, the platform supports a predictive approach based on online learning in which a machine learning model adapts its weights to cope with new prices and sentiments. Finally, the platform is modular and integrative in the sense that it combines these different solutions to provide novel real-time tool support for bitcoin price prediction that is more scalable, data-rich, and proactive, and can help accelerate decision-making, uncover new opportunities and provide more timely insights based on the available and ever-larger financial data volume and variety. \n Question: Which elements of the platform are modular?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-fa32729dbcfd454f989fb82257258149",
            "input": "We define a metric called the Semantic Text Exchange Score (STES) that evaluates the overall ability of a model to perform STE, and an adjustable parameter masking (replacement) rate threshold (MRT/RRT) that can be used to control the amount of semantic change. \n Question: Has STES been previously used in the literature to evaluate similar tasks?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-7627f6dd149847c780b9ec56096da1c8",
            "input": "We propose extended middle context, a new context representation for CNNs for relation classification. The contexts are split into three disjoint regions based on the two relation arguments: the left context, the middle context and the right context.  Hence, we propose to use two contexts: (1) a combination of the left context, the left entity and the middle context; and (2) a combination of the middle context, the right entity and the right context. The two contexts are processed by two independent convolutional and max-pooling layers. After pooling, the results are concatenated to form the sentence representation. \n Question: How do they obtain the new context represetation?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-7c4e3eb9da724b6ba717fef4cd716cef",
            "input": "To identify legally sound answers, we recruit seven experts with legal training to construct answers to Turker questions.  \n Question: Who were the experts used for annotation?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-c86ac8e4aa9c43d785b5f693304d0aec",
            "input": "In this study, we investigate the performance of our proposed models on WSJ BIBREF5 .  \n Question: Which dataset do they use?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-2e60701c7cbd46c6ba5e1684439fab58",
            "input": "Thus, for this work, we build upon Hybrid Code Networks (HCN) BIBREF4 since HCNs achieve state-of-the-art performance in a data-efficient way for task-oriented dialogs, and propose AE-HCNs which extend HCNs with an autoencoder (Figure FIGREF8 ).  AE-HCN(-CNN) outperforms HCN on Test-OOD by a large margin about 17(20) points on average while keeping the minimum performance trade-off compared to Test.  \n Question: By how much does their method outperform state-of-the-art OOD detection?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-fcc210ac1cad44bdbcf9e8ede8aea4a4",
            "input": "For training and testing we use the train-validation-test split of WikiTableQuestions BIBREF0 , a dataset containing 22,033 pairs of questions and answers based on 2,108 Wikipedia tables. This dataset is also used by our baselines, BIBREF0 , BIBREF3 . \n Question: Does the dataset they use differ from the one used by Pasupat and Liang, 2015?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-588cd041816b4ba5b75488f690d3342d",
            "input": "The above pseudocode is agnostic with respect to the choice of fragmentation and environment functions; task-specific choices are described in more detail for each experiment below.\n\nDiscussion \n Question: Which languages do they test on?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-b105c7c72f0944f2b82b8418694ba2fc",
            "input": "Contributors record voice clips by reading from a bank of donated sentences. \n Question: What domains are covered in the corpus?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-9ff5a82107ed444791cdd26249ea418b",
            "input": "We compare our approach to two other, the first one uses almost the same tweet data as we use for training, and the second one is the CrowdFlower dataset annotated for emotions. In the first one Wang et al. BIBREF21 downloaded over 5M tweets which included one of 131 emotional hashtags based on Parrott's three-level categorization of emotions in seven categories: joy, sadness, anger, love, fear, thankfulness, surprise.  \n Question: What data is used in experiments?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-d88fd0e3236a41a0a99be268e149bfa3",
            "input": "We make copies of the monolingual model for each language and add additional crosslingual latent variables (CLVs) to couple the monolingual models, capturing crosslingual semantic role patterns. Concretely, when training on parallel sentences, whenever the head words of the arguments are aligned, we add a CLV as a parent of the two corresponding role variables. \n Question: Do they add one latent variable for each language pair in their Bayesian model?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-06125df59ef340789d3462947b07da6d",
            "input": "For the 2019 Workshop on Neural Generation of Text (WNGT) Efficiency shared task BIBREF0, the Notre Dame Natural Language Processing (NDNLP) group looked at a method of inducing sparsity in parameters called auto-sizing in order to reduce the number of parameters in the Transformer at the cost of a relatively minimal drop in performance. \n Question: What is WNGT 2019 shared task?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-6fa0bc8b485947deb6aa426dd8bbae39",
            "input": "We manually investigate the errors made by artificial neural networks for morphological inflection in a target language after pretraining on different source languages. For our qualitative analysis, we make use of the validation set. Therefore, we show validation set accuracies in Table TABREF19 for comparison. We manually annotate the outputs for the first 75 development examples for each source-target language combination. All found errors are categorized as belonging to one of the following categories. \n Question: How is the performance on the task evaluated?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-a2803c45f1bd42f5a13d642918f475aa",
            "input": "We are using the dataset of the competition, which includes text from tweets having the aforementioned categories.  \n Question: What dataset is used for this work?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-a200a5880a9d4cd0aa6582d1db3d1e54",
            "input": "Methods ::: Combining the two methods\nWe further propose to use the two methods together to combine their strengths. In fact, while the length token acts as a soft constraint to bias NMT to produce short or long translation with respect to the source, actually no length information is given to the network. On the other side, length encoding leverages information about the target length, but it is agnostic of the source length. \n Question: Do they experiment with combining both methods?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-25a90cecb1914cc7b27f35f8d8bfe186",
            "input": "Macaw also supports Wizard of Oz studies or intermediary-based information seeking studies. The architecture of Macaw for such setup is presented in FIGREF16. As shown in the figure, the seeker interacts with a real conversational interface that supports multi-modal and mixed-initiative interactions in multiple devices. The intermediary (or the wizard) receives the seeker's message and performs different information seeking actions with Macaw. All seeker-intermediary and intermediary-system interactions will be logged for further analysis. \n Question: What is a wizard of oz setup?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-5155455d6e4643aaaa52d1292bfc833b",
            "input": "This results in three additional vectors corresponding to INLINEFORM3 , INLINEFORM4 and INLINEFORM5 difference vectors.\n\nResults \n Question: How are the EAU text spans annotated?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-896604b065b741f6b2870130a7c637aa",
            "input": "In order to evaluate the models trained on generated data, we manually annotated a named entities dataset comprising 53453 tokens and 2566 sentences selected from over 250 news texts from ilur.am. \n Question: what is the source of the news sentences?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-9045ee0df8d043be8fd6159c6d330a32",
            "input": " We performed the annotation with freely available tools for the Portuguese language. \n Question: Are the annotations automatic or manually created?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-66765b7a1e4b48b1bd5b171673a6f89b",
            "input": "They can answer each question with either `yes', `rather yes', `rather no', or `no'. They can supplement each answer with a comment of at most 500 characters. \n Question: What annotations are present in dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-630b9ee6cd5b4059b25dab2bd55d3f47",
            "input": "In this experiment set, our BERT implementation is compared to several systems that participated in the MEDDOCAN challenge: a CRF classifier BIBREF18, a spaCy entity recogniser BIBREF18, and NLNDE BIBREF12, the winner of the shared task and current state of the art for sensitive information detection and classification in Spanish clinical text.  However, attending to the obtained results, BERT remains only 0.3 F1-score points behind, and would have achieved the second position among all the MEDDOCAN shared task competitors. Taking into account that only 3% of the gold labels remain incorrectly annotated, the task can be considered almost solved, and it is not clear if the differences among the systems are actually significant, or whether they stem from minor variations in initialisation or a long-tail of minor labelling inconsistencies. \n Question: Does BERT reach the best performance among all the algorithms compared?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-75348e45afe048af9fac278efca7c310",
            "input": "We approached the first and second challenges by using a Bayesian approach to learn which terms were associated with events, regardless of whether they are standard language, acronyms, or even a made-up word, so long as they match the events of interest. The third and fourth challenges are approached by using word-pairs, where we extract all the pairs of co-occurring words within each tweet. To find the words most associated with events, we search for the words that achieve the highest number of spikes matching the days of events.  \n Question: How are the keywords associated with events such as protests selected?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-21d7f81e76b84effb3cc78a01c880a5b",
            "input": "We used the following affective resources relying on different emotion models.\n\nEmolex: it contains 14,182 words associated with eight primary emotion based on the Plutchik model BIBREF10 , BIBREF11 .\n\nEmoSenticNet(EmoSN): it is an enriched version of SenticNet BIBREF12 including 13,189 words labeled by six Ekman's basic emotion BIBREF13 , BIBREF14 .\n\nDictionary of Affect in Language (DAL): includes 8,742 English words labeled by three scores representing three dimensions: Pleasantness, Activation and Imagery BIBREF15 .\n\nAffective Norms for English Words (ANEW): consists of 1,034 English words BIBREF16 rated with ratings based on the Valence-Arousal-Dominance (VAD) model BIBREF17 .\n\nLinguistic Inquiry and Word Count (LIWC): this psycholinguistic resource BIBREF18 includes 4,500 words distributed into 64 emotional categories including positive (PosEMO) and negative (NegEMO). \n Question: What affective-based features are used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-5ff990e86c7847bdb575116cde4edc43",
            "input": "Span detector. We adopt a multi-turn answer module for the span detector BIBREF1 . Formally, at time step INLINEFORM0 in the range of INLINEFORM1 , the state is defined by INLINEFORM2 . The initial state INLINEFORM3 is the summary of the INLINEFORM4 : INLINEFORM5 , where INLINEFORM6 . Here, INLINEFORM7 is computed from the previous state INLINEFORM8 and memory INLINEFORM9 : INLINEFORM10 and INLINEFORM11 . Finally, a bilinear function is used to find the begin and end point of answer spans at each reasoning step INLINEFORM12 : DISPLAYFORM0 DISPLAYFORM1\n\nThe final prediction is the average of each time step: INLINEFORM0 . We randomly apply dropout on the step level in each time step during training, as done in BIBREF1 . \n Question: What is the architecture of the span detector?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-aec936f801544f1ebe6734322baffd5e",
            "input": "The corpus consists of 53 documents, which contain an average number of 156.1 sentences per document, each with 19.55 tokens on average. The corpus comprises 8,275 sentences and 167,739 words in total. \n Question: How large is the corpus?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-e14181e26cb6441eb7e006a43e99308a",
            "input": "jiant provides support for cutting-edge sentence encoder models, including support for Huggingface's Transformers. Supported models include: BERT BIBREF17, RoBERTa BIBREF27, XLNet BIBREF28, XLM BIBREF29, GPT BIBREF30, GPT-2 BIBREF31, ALBERT BIBREF32 and ELMo BIBREF33. jiant also supports the from-scratch training of (bidirectional) LSTMs BIBREF34 and deep bag of words models BIBREF35, as well as syntax-aware models such as PRPN BIBREF36 and ON-LSTM BIBREF37. jiant also supports word embeddings such as GloVe BIBREF38. \n Question: Is jiant compatible with models in any programming language?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-f210ae7dcfad46c08c042aa6a7e216f7",
            "input": "Table TABREF34 shows the inference speed of our method when implementing the sequnece modeling layer with the LSTM-based, CNN-based, and Transformer-based architecture, respectively.  From the table, we can see that our method has a much faster inference speed than Lattice-LSTM when using the LSTM-based sequence modeling layer, and it was also much faster than LR-CNN, which used an CNN architecture to implement the sequence modeling layer. And as expected, our method with the CNN-based sequence modeling layer showed some advantage in inference speed than those with the LSTM-based and Transformer-based sequence model layer. \n Question:  What percentage of improvement in inference speed is obtained by the proposed method over the newest state-of-the-art methods?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-0b68ebd370b44a97af8fd0bd14d2928b",
            "input": "In recent years, there has been a rapid growth in the usage of social media. People post their day-to-day happenings on regular basis. BIBREF0 propose four tasks for detecting drug names, classifying medication intake, classifying adverse drug reaction and detecting vaccination behavior from tweets. We participated in the Task2 and Task4. \n Question: Was the system only evaluated over the second shared task?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-349c5bf05e7244bf9be81b92452354ef",
            "input": "Data was collected from a 10% uniform sample of Twitter posts made during 2013, specifically the Gardenhose API. Control documents were also selected. These documents did not contain any of `caused', `causing', or `causes', nor any bidirectional words, and are further matched temporally to obtain the same number of control documents as causal documents in each fifteen-minute period during 2013. Control documents were otherwise selected randomly; causal synonyms may be present. \n Question: how do they collect the comparable corpus?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-3c3a0a9c88a540c0a627c2e65b762460",
            "input": "It also needs to be noted that due to the issues in actual practice, the objective of this work is to generate candidate prescriptions to facilitate the prescribing procedure instead of substituting the human practitioners completely. \n Question: Why did they think this was a good idea?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-da8f469985ae4b4ba5c4bec903ebce15",
            "input": "A quantitative measure is required to reliably evaluate the achieved improvement. One of the methods proposed to measure the interpretability is the word intrusion test BIBREF41 . But, this method is expensive to apply since it requires evaluations from multiple human evaluators for each embedding dimension. In this study, we use a semantic category-based approach based on the method and category dataset (SEMCAT) introduced in BIBREF27 to quantify interpretability. Specifically, we apply a modified version of the approach presented in BIBREF40 in order to consider possible sub-groupings within the categories in SEMCAT.  \n Question: What experiments do they use to quantify the extent of interpretability?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-7c1a8dd195584d7fbd72c12bd377ccb6",
            "input": "For MSA, we acquired the diacritized corpus that was used to train the RDI BIBREF7 diacritizer and the Farasa diacritizer BIBREF31. The corpus contains 9.7M tokens with approximately 194K unique surface forms (excluding numbers and punctuation marks). For testing, we used the freely available WikiNews test set BIBREF31, which is composed of 70 MSA WikiNews articles (18,300 tokens) and evenly covers a variety of genres including politics, economics, health, science and technology, sports, arts and culture. For CA, we obtained a large collection of fully diacritized classical texts (2.7M tokens) from a book publisher, and we held-out a small subset of 5,000 sentences (approximately 400k words) for testing. \n Question: what datasets were used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-42b3f8e76cb44b519417760c0e59c785",
            "input": "Consequently, our architecture is based on Embeddings from Language Model or ELMo BIBREF10 Concretely, we use a pre-trained ELMo model, obtained using the 1 Billion Word Benchmark which contains about 800M tokens of news crawl data from WMT 2011 BIBREF24 . Subsequently, the contextualized embeddings are passed on to a BiLSTM with 2,048 hidden units. We aggregate the LSTM hidden states using max-pooling, which in our preliminary experiments offered us better results, and feed the resulting vector to a 2-layer feed-forward network, where each layer has 512 units. The output of this is then fed to the final layer of the model, which performs the binary classification. \n Question: What type of model are the ELMo representations used in?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-db8d279884824d77a222e44992ee8c9b",
            "input": "To study the lexical and semantic diversities of responses, we performed three analyses. First, we aggregated all worker responses to a particular question into a single list corresponding to that question. Second, we compared the diversity of individual responses between Control and AUI for each question. To measure diversity for a question, we computed the number of responses divided by the number of unique responses to that question. We call this the response density.  \n Question: How was lexical diversity measured?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-aefe600d4a524db5bf4c9b0c9f946791",
            "input": "The resulting dataset consists of 22,880 users, 41,094 blogs, and 561,003 posts. Table TABREF2 presents additional statistics of our dataset. \n Question: How many users do they look at?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-40443d80102942e99adecfbd345e8593",
            "input": "We use a siamese neural network, shown to perform state-of-the-art few-shot learning BIBREF11, as our baseline model. We modify the original to account for sequential data, with each twin composed of an embedding layer, a Long-Short Term Memory (LSTM) BIBREF12 layer, and a feed-forward layer with Rectified Linear Unit (ReLU) activations. \n Question: What were the baselines?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-494bb7a296f147f7a46e8f1e4726201f",
            "input": "The total F-1 score on the OntoNotes dataset is 88%, and the total F-1 cross-validation score on the 112 class Wiki(gold) dataset is 53%.  \n Question: What results do they achieve using their proposed approach?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-e9b6f5e57dc04b5ab6f3b8ef8dc7f9c1",
            "input": "As a document classifier we employ a word-based CNN similar to Kim consisting of the following sequence of layers: $ \\texttt {Conv} \\xrightarrow{} \\texttt {ReLU} \\xrightarrow{} \\texttt {1-Max-Pool} \\xrightarrow{} \\texttt {FC} \\\\ $ Future work would include applying LRP to other neural network architectures (e.g. character-based or recurrent models) on further NLP tasks, as well as exploring how relevance information could be taken into account to improve the classifier's training procedure or prediction performance. \n Question: Do the experiments explore how various architectures and layers contribute towards certain decisions?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-2e7105f5e2a64aa1ab990e54cde79ab0",
            "input": "In test batch 4, our system (called FACTOIDS) achieved highest recall score of ‘0.7033’ but low precision of 0.1119, leaving open the question of how could we have better balanced the two measures. \n Question: What was their highest recall score?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-603598dc016d4af894a560d08d7cdbb5",
            "input": "Baseline Results ::: Speech Synthesis\nIn our previous work on building speech systems on found data in 700 languages, BIBREF7, we addressed alignment issues (when audio is not segmented into turn/sentence sized chunks) and correctness issues (when the audio does not match the transcription). We used the same techniques here, as described above.\n\nFor the best quality speech synthesis we need a few hours of phonetically-balanced, single-speaker, read speech. Our first step was to use the start and end points for each turn in the dialogues, and select those of the most frequent speaker, nmlch. This gave us around 18250 segments. We further automatically removed excessive silence from the start, middle and end of these turns (based on occurrence of F0). This gave us 13 hours and 48 minutes of speech.\n\nWe phonetically aligned this data and built a speech clustergen statistical speech synthesizer BIBREF9 from all of this data. For speech recognition (ASR) we used Kaldi BIBREF11. We built neural end-to-end machine translation systems between Mapudungun and Spanish in both directions, using state-of-the-art Transformer architecture BIBREF14 with the toolkit of BIBREF15. \n Question: What are the models used for the baseline of the three NLP tasks?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-219057dfcd1f413c820a51c4ce676fce",
            "input": "SVM: We define 3 sets of features to characterize each question. The first is a simple bag-of-words set of features over the question (SVM-BOW), the second is bag-of-words features of the question as well as length of the question in words (SVM-BOW + LEN), and lastly we extract bag-of-words features, length of the question in words as well as part-of-speech tags for the question (SVM-BOW + LEN + POS). This results in vectors of 200, 201 and 228 dimensions respectively, which are provided to an SVM with a linear kernel. No-Answer Baseline (NA) : Most of the questions we receive are difficult to answer in a legally-sound way on the basis of information present in the privacy policy. We establish a simple baseline to quantify the effect of identifying every question as unanswerable. Word Count Baseline : To quantify the effect of using simple lexical matching to answer the questions, we retrieve the top candidate policy sentences for each question using a word count baseline BIBREF53, which counts the number of question words that also appear in a sentence. We include the top 2, 3 and 5 candidates as baselines. Human Performance: We pick each reference answer provided by an annotator, and compute the F1 with respect to the remaining references, as described in section 4.2.1. Each reference answer is treated as the prediction, and the remaining n-1 answers are treated as the gold reference. The average of the maximum F1 across all reference answers is computed as the human baseline. \n Question: Were other baselines tested to compare with the neural baseline?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-8bf4c79528e2498fb911a1b1d1fe0de9",
            "input": "We evaluate our newly proposed models and related baselines in several low-resource settings across different languages with real, distantly supervised data with non-synthetic noise. The advanced modeling of the noisy labels substantially improves the performance up to 36% over methods without noise-handling and up to 9% over all other noise-handling baselines. \n Question: Did they evaluate against baseline?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-b62a1107fc8c4cc193c7a2446f6d5202",
            "input": "Some authors use pretrained embeddings (especially when their data set is too small to train their own embeddings) or try to modify these embeddings and adjust to their set. But the biggest drawback of these approaches is that the corpus for training embeddings can be not related to the specific task where embeddings are utilized. A lot of medical concepts are not contained in well-known embeddings bases. Furthermore, the similarity of words may vary in different contexts. Then, we compute embeddings of concepts (by GloVe) for interview descriptions and for examination descriptions separately. \n Question: Do they fine-tune the used word embeddings on their medical texts?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-eb29e8c976614b1991a60b9cb159788b",
            "input": "Our approach consists of structuring input to the transformer network to use and guide the self-attention of the transformers, conditioning it on the entity. Our main mode of encoding the input, the entity-first method, is shown in Figure FIGREF4.  We also have an entity-last variant where the entity is primarily observed just before the classification token to condition the [CLS] token's self-attention accordingly.  As an additional variation, we can either run the transformer once per document with multiple [CLS] tokens (a document-level model as shown in Figure FIGREF4) or specialize the prediction to a single timestep (a sentence-level model). \n Question: In what way is the input restructured?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-b54a0b0eb2304c7f9c23707479708593",
            "input": "Level B categorizes the type of offense and two labels are used: targeted (TIN) and untargeted (INT) insults and threats.\n\nTargeted Insult (TIN): Posts which contain an insult/threat to an individual, group, or others (see next layer);\n\nUntargeted (UNT): Posts containing non-targeted profanity and swearing. Posts with general profanity are not targeted, but they contain non-acceptable language. Level C categorizes the targets of insults and threats as individual (IND), group (GRP), and other (OTH).\n\nIndividual (IND): Posts targeting an individual. It can be a a famous person, a named individual or an unnamed participant in the conversation. Insults and threats targeted at individuals are often defined as cyberbulling.\n\nGroup (GRP): The target of these offensive posts is a group of people considered as a unity due to the same ethnicity, gender or sexual orientation, political affiliation, religious belief, or other common characteristic. Many of the insults and threats targeted at a group correspond to what is commonly understood as hate speech.\n\nOther (OTH): The target of these offensive posts does not belong to any of the previous two categories (e.g. an organization, a situation, an event, or an issue). \n Question: What kinds of offensive content are explored?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-88f5702aa5a64e368131e346b0d5a771",
            "input": "In HotpotQA, on average we can find 6 candidate chains (2-hop) in a instance, and the human labeled true reasoning chain is unique. A predicted chain is correct if the chain only contains all supporting passages (exact match of passages).\n\nIn MedHop, on average we can find 30 candidate chains (3-hop). For each candidate chain our human annotators labeled whether it is correct or not, and the correct reasoning chain is not unique. A predicted chain is correct if it is one of the chains that human labeled as correct.\n\nThe accuracy is defined as the ratio: The accuracy is defined as the ratio: \n Question: What benchmarks are created?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-7eb5fe8b34054b64b54fc74a45efc6e5",
            "input": "We use the pre-trained uncased BERT$_\\mathrm {BASE}$ model for fine-tuning, because we find that BERT$_\\mathrm {LARGE}$ model performs slightly worse than BERT$_\\mathrm {BASE}$ in this task.  \n Question: Do they use large or small BERT?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-e062bb75845442ea8bd64b9fa87f7b94",
            "input": "The SG model achieved a high average similarity score of 0.650 followed by CBoW with a 0.632 average similarity score. The GloVe also achieved a considerable average score of 0.591 respectively. However, the average similarity score of SdfastText is 0.388 and the word pair Microsoft-Bill Gates is not available in the vocabulary of SdfastText. Moreover, the average semantic relatedness similarity score between countries and their capitals is shown in Table TABREF78 with English translation, where SG also yields the best average score of 0.663 followed by CBoW with 0.611 similarity score. The GloVe also yields better semantic relatedness of 0.576 and the SdfastText yield an average score of 0.391. \n Question: How does proposed word embeddings compare to Sindhi fastText word representations?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-b7409f0dc1d64dae96f2c8d1a49090c4",
            "input": "In cooperation with the ruling regime, Weibo sets strict control over the content published under its service BIBREF0. According to Zhu et al. zhu-etal:2013, Weibo uses a variety of strategies to target censorable posts, ranging from keyword list filtering to individual user monitoring. Among all posts that are eventually censored, nearly 30% of them are censored within 5-30 minutes, and nearly 90% within 24 hours BIBREF1. We hypothesize that the former are done automatically, while the latter are removed by human censors. \n Question: Is is known whether Sina Weibo posts are censored by humans or some automatic classifier?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-36c68b51c6014f40a0586bdb905ffcb0",
            "input": "Three datasets annotated with emotions are commonly used for the development and evaluation of emotion detection systems, namely the Affective Text dataset, the Fairy Tales dataset, and the ISEAR dataset. A summary is provided in Table TABREF8 , which also shows, in the bottom row, what role each dataset has in our experiments: apart from the development portion of the Affective Text, which we used to develop our models (Section SECREF4 ), all three have been used as benchmarks for our evaluation. \n Question: Which existing benchmarks did they compare to?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-866a3fae890a409ab599a073ce5d7fc7",
            "input": "We use the existing large-scale antonym and synonym pairs previously used by Nguyen:16. Originally, the data pairs were collected from WordNet BIBREF9 and Wordnik. \n Question: What dataset do they use to evaluate their method?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-f1efc42d4b77484eb87b8f668572ff5e",
            "input": "MIMIC-III is a freely available, deidentified database containing electronic health records of patients admitted to an Intensive Care Unit (ICU) at Beth Israel Deaconess Medical Center between 2001 and 2012 \n Question: what datasets were used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-7437e315350b44a2997b74da3eef7ceb",
            "input": "Intuitively, under-translated input words should contribute little to the NMT outputs, yielding much smaller word importance.  By exploiting the word importance calculated by Attribution method, we can identify the under-translation errors automatically without the involvement of human interpreters.  \n Question: How do they measure which words are under-translated by NMT models?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-222da16ae2a340b0826f63f9cc84b325",
            "input": "In this work, we analyze a set of tweets related to a specific classical music radio channel, BBC Radio 3, interested in detecting two types of musical named entities, Contributor and Musical Work. \n Question: What language is the Twitter content in?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-ecd4e960116a42f98334b9e24748b457",
            "input": "Understanding a user's intent and sentiment is of utmost importance for current intelligent chatbots to respond appropriately to human requests. However, current systems are not able to perform to their best capacity when presented with incomplete data, meaning sentences with missing or incorrect words. This scenario is likely to happen when one considers human error done in writing. In fact, it is rather naive to assume that users will always type fully grammatically correct sentences.  \n Question: How do the authors define or exemplify 'incorrect words'?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-25ecefa07abd41d8a37280b60933d132",
            "input": "As we show below, STransE performs better than the SE and TransE models and other state-of-the-art link prediction models on two standard link prediction datasets WN18 and FB15k, so it can serve as a new baseline for KB completion. \n Question: What datasets are used to evaluate the model?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-8993d12167614d6e850aca11cbbb1486",
            "input": "During each evaluation, target terms are masked, predicted, and then compared to the masked (known) value. \n Question: Are the Transformers masked?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-fb87f5a6125a4e9cb382f40bcd3662ee",
            "input": "We focus primarily on the smaller data set of 645 hand-labeled articles provided to task participants, both for training and for validation.  We pre-trained BERT-base on the 600,000 articles without labels by using the same Cloze task BIBREF5 that BERT had originally used for pre-training.  \n Question: How long is the dataset?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-f536038864f54a85b8c37cdd0e14ad06",
            "input": "We adopt a DNN-based acoustic model BIBREF0 with 11 hidden layers and the alignment used to train the model is derived from a HMM-GMM model trained with SAT criterion. \n Question: What are the deep learning architectures used in the task?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-832be5cd842b40f8a38fbe2e4bbfe440",
            "input": "Thus, to limit the number of different inputs to the classifier, we wish to reduce the number of distinct word recognition outputs that an attacker can induce, not just the number of words on which the model is “fooled”. We denote this property of a model as its sensitivity. \n Question: What does the \"sensitivity\" quantity denote?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-a50682bbf13243e783f7c2d621eff447",
            "input": "DNN based models coupled with transfer learning beat the best-known results for all three datasets. Previous best F1 scores for Wikipedia BIBREF4 and Twitter BIBREF8 datasets were 0.68 and 0.93 respectively. We achieve F1 scores of 0.94 for both these datasets using BLSTM with attention and feature level transfer learning (Table TABREF25 ). For Formspring dataset, authors have not reported F1 score. Their method has accuracy score of 78.5% BIBREF2 . We achieve F1 score of 0.95 with accuracy score of 98% for the same dataset. \n Question: What were their performance results?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-3c5832372c9741b39c1ae84b0f948630",
            "input": "We address the robustness problem on top of GE-FL BIBREF0 , a GE method which leverages labeled features as prior knowledge. \n Question: What background knowledge do they leverage?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-4c196e499ba840f781145945e6bd715d",
            "input": "A representation that is more in line with observed user behavior is a concept map BIBREF5 , a labeled graph showing concepts as nodes and relationships between them as edges (Figure FIGREF2 ). \n Question: How do the authors define a concept map?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-1496b2e5e49d433da14f7b14d8667122",
            "input": "Multiple papers have shown that 'shallow' naive Bayes classifiers BIBREF12, BIBREF8, BIBREF13, BIBREF14, SVMs BIBREF15 and similar models work very well for doing LID. The DSL 2017 paper BIBREF1, for example, gives an overview of the solutions of all of the teams that competed on the shared task and the winning approach BIBREF16 used an SVM with character n-gram, parts of speech tag features and some other engineered features. The winning approach for DSL 2015 used an ensemble naive Bayes classifier. The fasttext classifier BIBREF17 is perhaps one of the best known efficient 'shallow' text classifiers that have been used for LID . Multiple papers have proposed hierarchical stacked classifiers (including lexicons) that would for example first classify a piece of text by language group and then by exact language BIBREF18, BIBREF19, BIBREF8, BIBREF0. Researchers have investigated deeper LID models li \n Question: What is the approach of previous work?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-b7e8ab71312c4d67a1e65611318c53d7",
            "input": "To test if the learned representations can separate phonetic categories, we use a minimal pair ABX discrimination task BIBREF19 , BIBREF20 . For the within-speaker task, all the phones triplets belong to the same speaker (e.g. $x$3 ) Finally the scores for every pair of central phones are averaged and subtracted from 1 to yield the reported within-talker ABX error rate. For the across-speaker task, $x$4 and $x$5 belong to the same speaker, and $x$6 to a different one (e.g. $x$7 ). The scores for a given minimal pair are first averaged across all of the pairs of speakers for which this contrast can be made. As above, the resulting scores are averaged over all contexts over all pairs of central phones and converted to an error rate. \n Question: What is the metric that is measures in this paper?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-b5701507efec4bd5a820a0e0115c60d3",
            "input": "Our main contributions are (1) the publication of Turkish corpus for coarse-grained and fine-grained NER, and TC research, (2) six different versions of corpus according to noise reduction methodology and entity types, (3) an analysis of the corpus and (4) benchmark comparisons for NER and TC tasks against human annotators.  \n Question: Did they experiment with the dataset on some tasks?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-cc225001e6094b68a8777c2e34ecbc5e",
            "input": "We sparsify this graph to the union of a minimum spanning tree and a k-Nearest Neighbors (MST-kNN) graph BIBREF14, a geometric construction that removes less important similarities but preserves global connectivity for the graph and, hence, for the dataset. The MST-kNN graph is then analysed with Markov Stability BIBREF15, BIBREF16, BIBREF17, BIBREF18, a multi-resolution graph partitioning method that identifies relevant subgraphs (i.e., clusters of documents) at different levels of granularity. \n Question: What cluster identification method is used in this paper?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-a9c189ae6f424ccf9b7aa22fcada40b4",
            "input": "We demonstrate our approach on the task of answering open-domain fill-in-the-blank natural language questions. \n Question: What task do they evaluate on?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-c3a6e547ed134efb8720b87fbc181186",
            "input": "For sentiment classification, we systematically study the effect of character-level adversarial attacks on two architectures and four different input formats.  We also consider the task of paraphrase detection. \n Question: What end tasks do they evaluate on?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-81e7ff89d1f34f1490499c2232798b39",
            "input": "In order to decode spatial connections between the electrodes from the channel covariance matrix, we use a CNN BIBREF19 , in particular a four-layered 2D CNN stacking two convolutional and two fully connected hidden layers. The INLINEFORM0 feature map at a given CNN layer with input INLINEFORM1 , weight matrix INLINEFORM2 and bias INLINEFORM3 is obtained as: INLINEFORM4 . At this first level of hierarchy, the network is trained with the corresponding labels as target outputs, optimizing a cross-entropy cost function. In parallel, we apply a four-layered recurrent neural network on the channel covariance matrices to explore the hidden temporal features of the electrodes. Namely, we exploit an LSTM BIBREF20 consisting of two fully connected hidden layers, stacked with two LSTM layers and trained in a similar manner as CNN. \n Question: How was the spatial aspect of the EEG signal computed?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-5c145fcc50c742cfb43e98806378bb50",
            "input": "A total of 2,50,000 tweets over a period of August 31st, 2015 to August 25th,2016 on Microsoft are extracted from twitter API BIBREF15 . The news on twitter about Microsoft and tweets regarding the product releases were also included. Stock opening and closing prices of Microsoft from August 31st, 2015 to August 25th, 2016 are obtained from Yahoo! Finance BIBREF16 . \n Question: What dataset is used to train the model?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-896dd022b51e4276b8f87eb5cd46f6ea",
            "input": "The baselines are: (a) trained on S-SQuAD, (b) trained on T-SQuAD and then fine-tuned on S-SQuAD, and (c) previous best model trained on S-SQuAD BIBREF5 by using Dr.QA BIBREF20 . The proposed approach (row (f)) outperforms previous best model (row (c)) by 2% EM score and over 1.5% F1 score. \n Question: What was the previous best model?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-93f143cd0659416c97749b8f9b9f37cc",
            "input": "Many NLP tasks utilize POS as features, but human annotated POS sequences are difficult and expensive to obtain. Thus, it is important to know if we can learn sentences-level syntactic embeddings for low-sources languages without treebanks.\n\nWe performed zero-shot transfer of the syntactic embeddings for French, Portuguese and Indonesian. French and Portuguese are simulated low-resource languages, while Indonesian is a true low-resource language. \n Question: Do they evaluate on downstream tasks?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-a8cf1a183a8943eca57ecab8f3d24da2",
            "input": "These models define parametrized similarity scoring functions $: Q\\times T\\rightarrow \\mathbb {R}$ , where $Q$ is the set of natural language questions and $T$ is the set of paraphrases of logical forms. \n Question: Does a neural scoring function take both the question and the logical form as inputs?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-8566a2a2c3f742debdff592edb6cda1c",
            "input": "To identify new keywords in the selected microposts, we again leverage crowdsourcing, as humans are typically better than machines at providing specific explanations BIBREF18, BIBREF19. In the crowdsourcing task, workers are first asked to find those microposts where the model predictions are deemed correct. Then, from those microposts, workers are asked to find the keyword that best indicates the class of the microposts as predicted by the model. \n Question: How is the keyword specific expectation elicited from the crowd?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-174189118c554bd097dd2c4cad09727d",
            "input": "To collect article-level labels, we utilized a platform in the company that has been used by the market research team to collect surveys from the subscribers of different news publishers. The survey works as follows: The user is first presented with a set of selected pages (usually 4 pages and around 20 articles) from the print paper the day before. The user can select an article each time that he or she has read, and answer some questions about it. We added 3 questions to the existing survey that asked the level of partisanship, the polarity of partisanship, and which pro- or anti- entities the article presents. We also asked the political standpoint of the user. The complete survey can be found in Appendices. \n Question: Did they crowdsource the annotations?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-34026decd5384cf7a1ec6f209e8806e2",
            "input": "In particular, we perform an extensive set of experiments on standard benchmarks for bilingual dictionary induction and monolingual and cross-lingual word similarity, as well as on an extrinsic task: cross-lingual hypernym discovery. For both the monolingual and cross-lingual settings, we can notice that our models generally outperform the corresponding baselines.  As can be seen in Table 1 , our refinement method consistently improves over the baselines (i.e., VecMap and MUSE) on all language pairs and metrics. First and foremost, in terms of model-wise comparisons, we observe that our proposed alterations of both VecMap and MUSE improve their quality in a consistent manner, across most metrics and data configurations. \n Question: What are the tasks that this method has shown improvements?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-435f0831e696455aae595c0d27410a6f",
            "input": "For instance, the sentence “The girls sang a song and they danced” is translated into French as “Les filles ont chanté une chanson et ils ont dansé” by Google Translate (GT), Bing Translate, and Yandex. \n Question: Do the authors conduct experiments on the tasks mentioned?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-5bdfe31d529447b3a36fe7d9c5c96622",
            "input": "Since labels are unknown on INLINEFORM1 , EGL computes the expectation of the gradient norm over all possible labelings. BIBREF3 interprets EGL as “expected model change”. In the following section, we formalize the intuition for EGL and show that it follows naturally from reducing the variance of an estimator. Eq. ( EQREF7 ) indicates that to reduce INLINEFORM0 on test data, we need to minimize the expected variance INLINEFORM1 over the test set. A practical issue is that we do not know INLINEFORM0 in advance. We could instead substitute an estimate INLINEFORM1 from a pre-trained model, where it is reasonable to assume the INLINEFORM2 to be close to the true INLINEFORM3 . The batch selection then works by taking the samples that have largest gradient norms, DISPLAYFORM0\n\nFor RNNs, the gradients for each potential label can be obtained by back-propagation. Another practical issue is that EGL marginalizes over all possible labelings, but in speech recognition, the number of labelings scales exponentially in the number of timesteps. Therefore, we only marginalize over the INLINEFORM0 most probable labelings. They are obtained by beam search decoding, as in BIBREF7 . The EGL method in BIBREF3 is almost the same as Eq. ( EQREF8 ), except the gradient's norm is not squared in BIBREF3 . \n Question: How do they calculate variance from the model outputs?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-2f91df8c4a1943228929f77e3b711e8c",
            "input": "It includes a total 708 hours of French (Fr), German (De), Dutch (Nl), Russian (Ru), Spanish (Es), Italian (It), Turkish (Tr), Persian (Fa), Swedish (Sv), Mongolian (Mn) and Chinese (Zh) speeches, with French and German ones having the largest durations among existing public corpora. \n Question: Which languages are part of the corpus?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-60504910721d4075a969ca7bacc145e6",
            "input": "The Style baseline achieves an Accuracy of 83%, LDA features 89% and a combination of the two gets 90%. However, training on full poems reduces this to 42—52%. \n Question: Is the outcome of the LDA analysis evaluated in any way?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-2615fe452ed14ab4bfdcf38b97527d33",
            "input": "To implement the first tier of our framework and detect moral relevance, we complement our morally relevant seed words with a corresponding set of seed words approximating moral irrelevance based on the notion of valence, i.e., the degree of pleasantness or unpleasantness of a stimulus. We refer to the emotional valence ratings collected by BIBREF28 for approximately 14,000 English words, and choose the words with most neutral valence rating that do not occur in the MFD as our set of morally irrelevant seed words, for an equal total number of morally relevant and morally irrelevant words. \n Question: How do they quantify moral relevance?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-9e091ffeb9ec4ab7b4e9586e8fab254c",
            "input": "For the evaluation of summaries we use the standard ROGUE metric. For comparison with previous AMR based summarization methods, we report the Recall, Precision and INLINEFORM0 scores for ROGUE-1. \n Question: Which evaluation methods are used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-9204a8d28a9042a3a40f369a335ee748",
            "input": "We compare our HR-VAE model with three strong baselines using VAE for text modelling:\n\nVAE-LSTM-base: A variational autoencoder model which uses LSTM for both encoder and decoder. KL annealing is used to tackled the latent variable collapse issue BIBREF0;\n\nVAE-CNN: A variational autoencoder model with a LSTM encoder and a dilated CNN decoder BIBREF7;\n\nvMF-VAE: A variational autoencoder model using LSTM for both encoder and decoder where the prior distribution is the von Mises-Fisher (vMF) distribution rather than a Gaussian distribution BIBREF5. \n Question: Do they compare against state of the art text generation?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-a3f255f07dfe45d6b6e2358456d2d017",
            "input": "The semi-parametric models were trained on 32 GPUs with each replica split over 2 GPUs, one to train the translation model and the other for computing the CSTM. \n Question: Does their combination of a non-parametric retrieval and neural network get trained end-to-end?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-2da947215d804bb4a7d69166e10c42c4",
            "input": "We compared our models with the following state-of-the-art baselines:\n\nSequence to Sequence (Seq2Seq): A simple encoder-decoder model which concatenates four sentences to a long sentence with an attention mechanism BIBREF31 .\n\nHierarchical LSTM (HLSTM): The story context is represented by a hierarchical LSTM: a word-level LSTM for each sentence and a sentence-level LSTM connecting the four sentences BIBREF29 . A hierarchical attention mechanism is applied, which attends to the states of the two LSTMs respectively.\n\nHLSTM+Copy: The copy mechanism BIBREF32 is applied to hierarchical states to copy the words in the story context for generation.\n\nHLSTM+Graph Attention(GA): We applied multi-source attention HLSTM where commonsense knowledge is encoded by graph attention.\n\nHLSTM+Contextual Attention(CA): Contextual attention is applied to represent commonsense knowledge. \n Question: Did they compare to Transformer based large language models?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-cfd244ce0c2c46b4a726ff34ef1f0ff1",
            "input": "TF-IDF: We applied TF-IDF features as inputs to four classifiers: support vector machine classifier (SVM), logistic regression classifier (LR), Naive Bayes classifier (NB) and random forest (RF); CNN: We followed the same architecture as BIBREF9; LSTM: We applied an LSTM model BIBREF20 to classify the sentences with pre-trained word embeddings;LSTM-soft: We then added a soft-attention BIBREF21 layer on top of the LSTM model where we computed soft alignment scores over each of the hidden states; LSTM-self: We applied a self-attention layer BIBREF22 to LSTM model.  \n Question: To what baseline models is proposed model compared?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-b8fead7ef41549fe9bbc146f7b48d432",
            "input": "Spanish (SPA), in contrast, is morphologically rich, and disposes of much larger verbal paradigms than English. Like English, it is a suffixing language, and it additionally makes use of internal stem changes (e.g., o $\\rightarrow $ ue). We choose Zulu (ZUL), a Bantoid language. In contrast to the first two, it is strongly prefixing. Second, the system pretrained on HUN performing well suggests again that a source language with an agglutinative, as opposed to a fusional, morphology seems to be beneficial as well. Again, TUR and HUN obtain high accuracy, which is an additional indicator for our hypothesis that a source language with an agglutinative morphology facilitates learning of inflection in another language. \n Question: Are agglutinative languages used in the prediction of both prefixing and suffixing languages?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-df7e2d40ba6d4b6790ebdb1b57f45c80",
            "input": "Since none of the datasets from previous works have been published, we decide to build a new one. We randomly collect INLINEFORM0 cases from China Judgments Online, among which INLINEFORM1 cases are for training, INLINEFORM2 each for validation and testing. \n Question: what datasets are used in the experiment?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-a18389b12e304d3980504c45b207c1ac",
            "input": "We quantify the efficiency-accuracy tradeoff compared to two rule-based baselines: Unif and Stopword.  \n Question: What are the baselines used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-98ea3205295a40e5a533366a4746bdc1",
            "input": "Previously, datasets openly available in abusive language detection research on Twitter ranged from 10K to 35K in size BIBREF9 , BIBREF11 . This quantity is not sufficient to train the significant number of parameters in deep learning models. Due to this reason, these datasets have been mainly studied by traditional machine learning methods. Most recently, Founta et al. founta2018large introduced Hate and Abusive Speech on Twitter, a dataset containing 100K tweets with cross-validated labels. Although this corpus has great potential in training deep models with its significant size, there are no baseline reports to date.\n\nThis paper investigates the efficacy of different learning models in detecting abusive language. We compare accuracy using the most frequently studied machine learning classifiers as well as recent neural network models. Reliable baseline results are presented with the first comparative study on this dataset. Additionally, we demonstrate the effect of different features and variants, and describe the possibility for further improvements with the use of ensemble models. \n Question: Does the dataset feature only English language data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-93275b14151d44d7806a58e1b5325d7a",
            "input": "Compared to using external models for confidence modeling, an advantage of the proposed method is that the base model does not change: the binary classification loss just provides additional supervision. For supervised open IE systems, the confidence score of an assertion is typically computed based on its extraction likelihood given by the model BIBREF3 , BIBREF5  We follow the evaluation metrics described by Stanovsky:2016:OIE2016: area under the precision-recall curve (AUC) and F1 score. \n Question: How does this compare to traditional calibration methods like Platt Scaling?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-d666ddf435c2413b98857c069d35f3e4",
            "input": "We also measure the usage of words related to people's core values as reported by Boyd et al. boyd2015. The sets of words, or themes, were excavated using the Meaning Extraction Method (MEM) BIBREF10 . \n Question: How do they obtain psychological dimensions of people?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-a2dd09ec868f4986929c2d8b0f0b3851",
            "input": "Dataset Probes and Construction\nOur probing methodology starts by constructing challenge datasets (Figure FIGREF1, yellow box) from a target set of knowledge resources. Each of our probing datasets consists of multiple-choice questions that include a question $\\textbf {q}$ and a set of answer choices or candidates $\\lbrace a_{1},...a_{N}\\rbrace $. This section describes in detail the 5 different datasets we build, which are drawn from two sources of expert knowledge, namely WordNet BIBREF35 and the GNU Collaborative International Dictionary of English (GCIDE). We describe each resource in turn, and explain how the resulting dataset probes, which we call WordNetQA and DictionaryQA, are constructed.\n\nFor convenience, we will describe each source of expert knowledge as a directed, edge-labeled graph $G$. The nodes of this graph are $\\mathcal {V} = \\mathcal {C} \\cup \\mathcal {W} \\cup \\mathcal {S} \\cup \\mathcal {D}$, where $\\mathcal {C}$ is a set of atomic concepts, $\\mathcal {W}$ a set of words, $\\mathcal {S}$ a set of sentences, and $\\mathcal {D}$ a set of definitions (see Table TABREF4 for details for WordNet and GCIDE). Each edge of $G$ is directed from an atomic concept in $\\mathcal {C}$ to another node in $V$, and is labeled with a relation, such as hypernym or isa$^\\uparrow $, from a set of relations $\\mathcal {R}$ (see Table TABREF4).\n\nWhen defining our probe question templates, it will be useful to view $G$ as a set of (relation, source, target) triples $\\mathcal {T} \\subseteq \\mathcal {R} \\times \\mathcal {C} \\times \\mathcal {V}$. Due to their origin in an expert knowledge source, such triples preserve semantic consistency. For instance, when the relation in a triple is def, the corresponding edge maps a concept in $\\mathcal {C}$ to a definition in $\\mathcal {D}$.\n\nTo construct probe datasets, we rely on two heuristic functions, defined below for each individual probe: $\\textsc {gen}_{\\mathcal {Q}}(\\tau )$, which generates gold question-answer pairs $(\\textbf {q},\\textbf {a})$ from a set of triples $\\tau \\subseteq \\mathcal {T}$ and question templates $\\mathcal {Q}$, and $\\textsc {distr}(\\tau ^{\\prime })$, which generates distractor answers choices $\\lbrace a^{\\prime }_{1},...a^{\\prime }_{N-1} \\rbrace $ based on another set of triples $\\tau ^{\\prime }$ (where usually $\\tau \\subset \\tau ^{\\prime }$). For brevity, we will use $\\textsc {gen}(\\tau )$ to denote $\\textsc {gen}_{\\mathcal {Q}}(\\tau )$, leaving question templates $\\mathcal {Q}$ implicit.\n\nDataset Probes and Construction ::: WordNetQA\nWordNet is an English lexical database consisting of around 117k concepts, which are organized into groups of synsets that each contain a gloss (i.e., a definition of the target concept), a set of representative English words (called lemmas), and, in around 33k synsets, example sentences. In addition, many synsets have ISA links to other synsets that express complex taxonomic relations. Figure FIGREF6 shows an example and Table TABREF4 summarizes how we formulate WordNet as a set of triples $\\mathcal {T}$ of various types. These triples together represent a directed, edge-labeled graph $G$. Our main motivation for using WordNet, as opposed to a resource such as ConceptNet BIBREF36, is the availability of glosses ($\\mathcal {D}$) and example sentences ($\\mathcal {S}$), which allows us to construct natural language questions that contextualize the types of concepts we want to probe.\n\nDataset Probes and Construction ::: WordNetQA ::: Example Generation @!START@$\\textsc {gen}(\\tau )$@!END@.\nWe build 4 individual datasets based on semantic relations native to WordNet (see BIBREF37): hypernymy (i.e., generalization or ISA reasoning up a taxonomy, ISA$^\\uparrow $), hyponymy (ISA$^{\\downarrow }$), synonymy, and definitions. To generate a set of questions in each case, we employ a number of rule templates $\\mathcal {Q}$ that operate over tuples. A subset of such templates is shown in Table TABREF8. The templates were designed to mimic naturalistic questions we observed in our science benchmarks.\n\nFor example, suppose we wish to create a question $\\textbf {q}$ about the definition of a target concept $c \\in \\mathcal {C}$. We first select a question template from $\\mathcal {Q}$ that first introduces the concept $c$ and its lemma $l \\in \\mathcal {W}$ in context using the example sentence $s \\in \\mathcal {S}$, and then asks to identify the corresponding WordNet gloss $d \\in \\mathcal {D}$, which serves as the gold answer $\\textbf {a}$. The same is done for ISA reasoning; each question about a hypernym/hyponym relation between two concepts $c \\rightarrow ^{\\uparrow /\\downarrow } c^{\\prime } \\in \\mathcal {T}_{i}$ (e.g., $\\texttt {dog} \\rightarrow ^{\\uparrow /\\downarrow } \\texttt {animal/terrier}$) first introduces a context for $c$ and then asks for an answer that identifies $c^{\\prime }$ (which is also provided with a gloss so as to contain all available context).\n\nIn the latter case, the rules $(\\texttt {isa}^{r},c,c^{\\prime }) \\in \\mathcal {T}_i$ in Table TABREF8 cover only direct ISA links from $c$ in direction $r \\in \\lbrace \\uparrow ,\\downarrow \\rbrace $. In practice, for each $c$ and direction $r$, we construct tests that cover the set HOPS$(c,r)$ of all direct as well as derived ISA relations of $c$:\n\nThis allows us to evaluate the extent to which models are able to handle complex forms of reasoning that require several inferential steps or hops.\n\nDataset Probes and Construction ::: WordNetQA ::: Distractor Generation: @!START@$\\textsc {distr}(\\tau ^{\\prime })$@!END@.\nAn example of how distractors are generated is shown in Figure FIGREF6, which relies on similar principles as above. For each concept $c$, we choose 4 distractor answers that are close in the WordNet semantic space. For example, when constructing hypernymy tests for $c$ from the set hops$(c,\\uparrow )$, we build distractors by drawing from $\\textsc {hops}(c,\\downarrow )$ (and vice versa), as well as from the $\\ell $-deep sister family of $c$, defined as follows. The 1-deep sister family is simply $c$'s siblings or sisters, i.e., the other children $\\tilde{c} \\ne c$ of the parent node $c^{\\prime }$ of $c$. For $\\ell > 1$, the $\\ell $-deep sister family also includes all descendants of each $\\tilde{c}$ up to $\\ell -1$ levels deep, denoted $\\textsc {hops}_{\\ell -1}(\\tilde{c},\\downarrow )$. Formally:\n\nFor definitions and synonyms we build distractors from all of these sets (with a similar restriction on the depth of sister distractors as noted above). In doing this, we can systematically investigate model performance on a wide range of distractor sets.\n\nDataset Probes and Construction ::: WordNetQA ::: Perturbations and Semantic Clusters\nBased on how we generate data, for each concept $c$ (i.e., atomic WordNet synset) and probe type (i.e., definitions, hypernymy, etc.), we have a wide variety of questions related to $c$ that manipulate 1) the complexity of reasoning that is involved (e.g., the number of inferential hops) and; 2) the types of distractors (or distractor perturbations) that are employed. We call such sets semantic clusters. As we describe in the next section, semantic clusters allow us to devise new types of evaluation that reveal whether models have comprehensive and consistent knowledge of target concepts (e.g., evaluating whether a model can correctly answer several questions associated with a concept, as opposed to a few disjoint instances).\n\nDetails of the individual datasets are shown in Table TABREF12. From these sets, we follow BIBREF22 in allocating a maximum of 3k examples for training and reserve the rest for development and testing. Since we are interested in probing, having large held-out sets allows us to do detailed analysis and cluster-based evaluation.\n\nDataset Probes and Construction ::: DictionaryQA\nThe DictionaryQA dataset is created from the GCIDE dictionary, which is a comprehensive open-source English dictionary built largely from the Webster's Revised Unabridged Dictionary BIBREF38. Each entry consists of a word, its part-of-speech, its definition, and an optional example sentence (see Table TABREF14). Overall, 33k entries (out of a total of 155k) contain example sentences/usages. As with the WordNet probes, we focus on this subset so as to contextualize each word being probed. In contrast to WordNet, GCIDE does not have ISA relations or explicit synsets, so we take each unique entry to be a distinct sense. We then use the dictionary entries to create a probe that centers around word-sense disambiguation, as described below.\n\nDataset Probes and Construction ::: DictionaryQA ::: Example and Distractor Generation.\nTo generate gold questions and answers, we use the same generation templates for definitions exemplified in Figure TABREF8 for WordNetQA. To generate distractors, we simply take alternative definitions for the target words that represent a different word sense (e.g., the alternative definitions of gift shown in Table TABREF14), as well as randomly chosen definitions if needed to create a 5-way multiple choice question. As above, we reserve a maximum of 3k examples for training. Since we have only 9k examples in total in this dataset (see WordSense in Table TABREF12), we also reserve 3k each for development and testing.\n\nWe note that initial attempts to build this dataset through standard random splitting gave rise to certain systematic biases that were exploited by the choice-only baseline models described in the next section, and hence inflated overall model scores. After several efforts at filtering we found that, among other factors, using definitions from entries without example sentences as distractors (e.g., the first two entries in Table TABREF14) had a surprising correlation with such biases. This suggests that possible biases involving differences between dictionary entries with and without examples can taint the resulting automatically generated MCQA dataset (for more discussion on the pitfalls involved with automatic dataset construction, see Section SECREF5). \n Question: Are the automatically constructed datasets subject to quality control?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-dbcb1ab5a467488fbb06270ca0863f0e",
            "input": "he first dataset (WIKI) was the same set of 200 sentences from Wikipedia used in BIBREF7 .  The second dataset (SCI) was a set of 220 sentences from the scientific literature. We sourced the sentences from the OA-STM corpus. In total 2247 triples were extracted. In total, 11262 judgements were obtained after running the annotation process. \n Question: What is the size of the released dataset?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-7535a707f8c144a385f1334eb95c0cdc",
            "input": "While $\\beta $-VAE offers regularizing the ELBO via an additional coefficient $\\beta \\in {\\rm I\\!R}^+$, a simple extension BIBREF16 of its objective function incorporates an additional hyperparameter $C$ to explicitly control the magnitude of the KL term,\n\nwhere $C\\!\\! \\in \\!\\! {\\rm I\\!R}^+$ and $| . |$ denotes the absolute value. \n Question: How does explicit constraint on the KL divergence term that authors propose looks like?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-03af93b09b3f4e6892b15503a7a1de2d",
            "input": "The library design of Torch-Struct follows the distributions API used by both TensorFlow and PyTorch BIBREF29. \n Question: Does API provide ability to connect to models written in some other deep learning framework?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-0371e0ca97714781809751eb1cb87280",
            "input": "We explore how to utilize additional commonsense knowledge (i.e. rationales) as the input to the task. Like we mentioned in Section SECREF6, we search relevant sentences from the OMCS corpus as the additional distant rationales, and ground truth rationale sentences for dev/test data. The inputs are no longer the concept-sets themselves, but in a form of “[rationales$|$concept-set]” (i.e. concatenating the rationale sentences and original concept-set strings). \n Question: Are the models required to also generate rationales?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-6b9307148e614541b849afd33ce5c11a",
            "input": "There are three main components to the model: 1) input encoder 2) dynamic memory, and 3) output module. We will describe these three modules in details. The input encoder and output module implementations are similar to the Entity Network BIBREF17 and main novelty lies in the dynamic memory. We describe the operations executed by the network for a single example consisting of a document with $T$ sentences, where each sentence consists of a sequence of words represented with $K$ -dimensional word embeddings $\\lbrace e_1, \\ldots , e_N\\rbrace $ , a question on the document represented as another sequence of words and an answer to the question. \n Question: How is knowledge stored in the memory?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-4efa239957a948e68fc9cd7c0e4cbb5c",
            "input": "Here we describe the components of our probabilistic model of question generation.  The details of optimization are as follows. First, a large set of 150,000 questions is sampled in order to approximate the gradient at each step via importance sampling. Second, to run the procedure for a given model and training set, we ran 100,000 iterations of gradient ascent at a learning rate of 0.1. \n Question: Is it a neural model? How is it trained?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-6e3ad437a4bc49b1b031c9abab00c73f",
            "input": "Baselines. We compare our approach (FacTweet) to the following set of baselines:\n\n[leftmargin=4mm]\n\nLR + Bag-of-words: We aggregate the tweets of a feed and we use a bag-of-words representation with a logistic regression (LR) classifier.\n\nTweet2vec: We use the Bidirectional Gated recurrent neural network model proposed in BIBREF20. We keep the default parameters that were provided with the implementation. To represent the tweets, we use the decoded embedding produced by the model. With this baseline we aim at assessing if the tweets' hashtags may help detecting the non-factual accounts.\n\nLR + All Features (tweet-level): We extract all our features from each tweet and feed them into a LR classifier. Here, we do not aggregate over tweets and thus view each tweet independently.\n\nLR + All Features (chunk-level): We concatenate the features' vectors of the tweets in a chunk and feed them into a LR classifier.\n\nFacTweet (tweet-level): Similar to the FacTweet approach, but at tweet-level; the sequential flow of the tweets is not utilized. We aim at investigating the importance of the sequential flow of tweets.\n\nTop-$k$ replies, likes, or re-tweets: Some approaches in rumors detection use the number of replies, likes, and re-tweets to detect rumors BIBREF21. Thus, we extract top $k$ replied, liked or re-tweeted tweets from each account to assess the accounts factuality. We tested different $k$ values between 10 tweets to the max number of tweets from each account. Figure FIGREF24 shows the macro-F1 values for different $k$ values. It seems that $k=500$ for the top replied tweets achieves the highest result. Therefore, we consider this as a baseline. \n Question: What baselines do they compare to?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-9403282983c04cc8bb0618fcbeb1aa1d",
            "input": "Our architecture was designed to show that using a model that captures better information about the context and sequential nature of the text can outperform lexicon-based methods commonly used in the literature. \n Question: What meaningful information does the GRU model capture, which traditional ML models do not?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-a67c7bcc6a0d4705805c8e6bd9d3a1b5",
            "input": "In order to derive the discourse features, an entity grid is constructed by feeding the document through an NLP pipeline to identify salient entities. Two flavors of discourse features are created by populating the entity grid with either (i) grammatical relations (GR) or (ii) RST discourse relations (RST).  \n Question: What discourse features are used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-0c86e00de3134c3183574c711deec786",
            "input": "Our attempt at language pre-training fell short of our expectations in all but one tested dataset. Our pre-training was unsuccessful in improving accuracy, even when applied to networks larger than those reported. \n Question: Does pre-training on general text corpus improve performance?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-dea7dce6884a428fa9d476d8011ac830",
            "input": "We feed these representations into a clustering algorithm that produces exactly eleven clusters of five cities each and evaluate them against both Calvino's original labels and crowdsourced human judgments. While the overall correlation with Calvino's labels is low, both computers and humans can reliably identify some thematic groups associated with concrete objects. \n Question: How do they obtain human judgements?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-1b28f4a610264369bfd53cc86ee6798c",
            "input": "Our user study compares the correctness of three scenarios:\n\nParser correctness - our baseline is the percentage of examples where the top query returned by the semantic parser was correct.\n\nUser correctness - the percentage of examples where the user selected a correct query from the top-7 generated by the parser.\n\nHybrid correctness - correctness of queries returned by a combination of the previous two scenarios. The system returns the query marked by the user as correct; if the user marks all queries as incorrect it will return the parser's top candidate. \n Question: Do they conduct a user study where they show an NL interface with and without their explanation?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-89a14e0ede354254840d641e7a64f185",
            "input": "We compare our personalized models against two baselines. The first is a name-based Nearest-Neighbor model (NN). We initially adapted the Neural Checklist Model of BIBREF0 as a baseline; however, we ultimately use a simple Encoder-Decoder baseline with ingredient attention (Enc-Dec), which provides comparable performance and lower complexity. \n Question: What are the baseline models?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-c86211c508ef4531aeaedf1211ec38f6",
            "input": "we use a common sense definition of racist language, including all negative utterances, negative generalizations and insults concerning ethnicity, nationality, religion and culture. \n Question: how did they ask if a tweet was racist?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-025c0cefbe4f4fb8abe4cf80e2978a87",
            "input": "All cases exhibit high scores—in the vast majority of the cases substantially higher than reported in previous work. In particular, in BIBREF1 we assess the ability of LSTMs to learn subject-verb agreement patterns in English, and evaluate on naturally occurring wikipedia sentences.  BIBREF2 also consider subject-verb agreement, but in a “colorless green ideas” setting in which content words in naturally occurring sentences are replaced with random words with the same part-of-speech and inflection, thus ensuring a focus on syntax rather than on selectional-preferences based cues.  BIBREF3 consider a wider range of syntactic phenomena (subject-verb agreement, reflexive anaphora, negative polarity items) using manually constructed stimuli, allowing for greater coverage and control than in the naturally occurring setting. \n Question: Were any of these tasks evaluated in any previous work?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-422db75baf7246b6a0ea4e162a617e82",
            "input": "While the dataset was collected with simulated environments, no structure was imposed on the navigation instructions while crowd-sourcing data. Thus, many instructions in our dataset are ambiguous. Moreover, the order of the behaviors in the instructions is not always the same. For instance, a person said “turn right and advance” to describe part of a route, while another person said “go straight after turning right” in a similar situation. The high variability present in the natural language descriptions of our dataset makes the problem of decoding instructions into behaviors not trivial. See Appendix A of the supplementary material for additional details on our data collection effort. \n Question: What language is the experiment done in?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-5a5990e4a8364b0ab5ee2901a7e5169b",
            "input": "We extend the Amazon Conversational Bot Toolkit (CoBot) BIBREF6 which is a flexible event-driven framework. CoBot provides ASR results and natural language processing pipelines through the Alexa Skills Kit (ASK) BIBREF7. Gunrock corrects ASR according to the context (asr) and creates a natural language understanding (NLU) (nlu) module where multiple components analyze the user utterances. A dialog manager (DM) (dm) uses features from NLU to select topic dialog modules and defines an individual dialog flow. Each dialog module leverages several knowledge bases (knowledge). Then a natural language generation (NLG) (nlg) module generates a corresponding response. While we provide an overview of the system in the following sections, for detailed system implementation details, please see the technical report BIBREF1. \n Question: Do they specify the model they use for Gunrock?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-27389710a9274f80abf18e897ec9674c",
            "input": "We also are especially interested in seeing this model applied to different languages. \n Question: Have the authors tried this approach on other languages?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-1a075853a1d045ff8fc007efd2c50594",
            "input": "Our dataset is annotated based on Chinese pathology reports provided by the Department of Gastrointestinal Surgery, Ruijin Hospital. It contains 17,833 sentences, 826,987 characters and 2,714 question-answer pairs. All question-answer pairs are annotated and reviewed by four clinicians with three types of questions, namely tumor size, proximal resection margin and distal resection margin.  \n Question: Is all text in this dataset a question, or are there unrelated sentences in between questions?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-3b21f75e6fa6453b977611a3079793f9",
            "input": "The primary feed for the analysis collected INLINEFORM0 million tweets containing the keywords `breast' AND `cancer'.  \n Question: How were breast cancer related posts compiled from the Twitter streaming API?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-3eab09704d2d469796e7cd9134fbe8a7",
            "input": "We use MS COCO, Bing and Flickr datasets from BIBREF26 to train the model that generates questions. These datasets contain natural questions about images with the purpose of knowing more about the picture. As can be seen in the Figure FIGREF8, questions cannot be answered by only looking at the image. Each source contains 5,000 images with 5 questions per image, adding a total of 15,000 images with 75,000 questions. We use two datasets to train our chatbot model. The first one is the Persona-chat BIBREF15 which contains dialogues between two people with different profiles that are trying to know each other. It is complemented by the Cornell-movie dialogues dataset BIBREF27, which contains a collection of fictional conversations extracted from raw movie scripts. Persona-chat's sentences have a maximum of 15 words, making it easier to learn for machines and a total of 162,064 utterances over 10,907 dialogues. While Cornell-movie dataset contains 304,713 utterances over 220,579 conversational exchanges between 10,292 pairs of movie characters. \n Question: How big dataset is used for training this system?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-5235f1b74cb94e33ad73721a72091b52",
            "input": "A growing body of evidence shows that state-of-the-art models learn to exploit spurious statistical patterns in datasets BIBREF12, BIBREF13, BIBREF14, BIBREF15, BIBREF16, BIBREF17, instead of learning meaning in the flexible and generalizable way that humans do. Given this, human annotators—be they seasoned NLP researchers or non-experts—might easily be able to construct examples that expose model brittleness. \n Question: What are the weaknesses found by non-expert annotators of current state-of-the-art NLI models?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-6636d3092abd4aa8a01c67376690561a",
            "input": "Generic Sarcasm. We first examine the different patterns learned on the Gen dataset.  We observe that the not-sarcastic patterns appear to capture technical and scientific language, while the sarcastic patterns tend to capture subjective language that is not topic-specific. We observe an abundance of adjective and adverb patterns for the sarcastic class, although we do not use adjective and adverb patterns in our regex retrieval method. Rhetorical Questions. We notice that while the not-sarcastic patterns generated for RQs are similar to the topic-specific not-sarcastic patterns we find in the general dataset, there are some interesting features of the sarcastic patterns that are more unique to the RQs.\n\nMany of our sarcastic questions focus specifically on attacks on the mental abilities of the addressee. This generalization is made clear when we extract and analyze the verb, subject, and object arguments using the Stanford dependency parser BIBREF32 for the questions in the RQ dataset. Hyperbole. One common pattern for hyperbole involves adverbs and adjectives, as noted above. We did not use this pattern to retrieve hyperbole, but because each hyperbolic sarcastic utterance contains multiple cues, we learn an expanded class of patterns for hyperbole.  We learn a number of verbal patterns that we had not previously associated with hyperbole, as shown in Table TABREF34 . Interestingly, many of these instantiate the observations of CanoMora2009 on hyperbole and its related semantic fields: creating contrast by exclusion, e.g. no limit and no way, or by expanding a predicated class, e.g. everyone knows. Many of them are also contrastive.  \n Question: What are the linguistic differences between each class?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-d00042fc55ff46bb81640fba286fee62",
            "input": "We test the character and word-level variants by predicting hashtags for a held-out test set of posts. \n Question: What other tasks do they test their method on?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-7bddf643ce9c49f2975c324dba78bfe6",
            "input": "Experimental Setup \n Question: what english datasets were used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-e03f78b8eb5346a3b8ed4bcea38aa099",
            "input": "Visual Question Generation (VQG) is another emerging topic which aims to ask questions given an image. Motivated by this, BIBREF10 proposed open-ended VQG that aims to generate natural and engaging questions about an image. \n Question: Do they survey visual question generation work?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-acf296cbc486451c8c723b55b1976a20",
            "input": "While BERT-QC achieves large gains over existing methods on the ARC dataset, here we demonstrate that BERT-QC also matches state-of-the-art performance on TREC BIBREF6 , while surpassing state-of-the-art performance on the GARD corpus of consumer health questions BIBREF3 and MLBioMedLAT corpus of biomedical questions BIBREF4 . \n Question: Which datasets are used for evaluation?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-84b02ae45bef48a8987c7934dcc2c44f",
            "input": "We choose the following three models as the baselines:\n\nK-means is a well known data clustering algorithm, we implement the algorithm using sklearn toolbox, and represent documents using bag-of-words weighted by TF-IDF.\n\nLEM BIBREF13 is a Bayesian modeling approach for open-domain event extraction. It treats an event as a latent variable and models the generation of an event as a joint distribution of its individual event elements. We implement the algorithm with the default configuration.\n\nDPEMM BIBREF14 is a non-parametric mixture model for event extraction. It addresses the limitation of LEM that the number of events should be known beforehand. We implement the model with the default configuration. \n Question: What baseline approaches does this approach out-perform?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-3527f1654928446a946b6b91b426832e",
            "input": "Therefore, we decided to compile our own corpora based on English documents, which we crawled from different publicly accessible sources. \n Question: Do they report results only on English data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-0cff684a9e8c4470a26b41d12fd7979c",
            "input": "Inspired by the StackLite tag recommendation task on Kaggle, we build a new benchmark based on the public StackExchange data. We use questions with titles as source, and user-assigned tags as target keyphrases.\n\nSince oftentimes the questions on StackExchange contain less information than in scientific publications, there are fewer keyphrases per data point in StackEx. Furthermore, StackExchange uses a tag recommendation system that suggests topic-relevant tags to users while submitting questions; therefore, we are more likely to see general terminology such as Linux and Java. This characteristic challenges models with respect to their ability to distill major topics of a question rather than selecting specific snippets from the text. \n Question: How was the StackExchange dataset collected?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-8c6a1b90dde241daaeb6bf4a059d53a4",
            "input": "We assume that speech transcripts can be extracted from audio signals with high accuracy, given the advancement of ASR technologies BIBREF7 . \n Question: Do they use datasets with transcribed text or do they determine text from the audio?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-38fd8bec77944c279b5b5b65f4ad6fbb",
            "input": " In contrast to most encoder-decoder architectures, which are purely based on recurrent neural network (RNNs) framework, we construct an encoder with several convolutional layers BIBREF14 followed by NIN layers BIBREF15 as the lower part in the encoder and integrate them with deep bidirectional long short-term memory (Bi-LSTM) BIBREF16 at the higher part. On the decoder side, we use a standard deep unidirectional LSTM with global attention BIBREF13 that is calculated by a multi-layer perceptron (MLP) as described in Eq. EQREF2 . \n Question: Which architecture do they use for the encoder and decoder?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-cbed0fd21fb14aef93543d31f4ee0b41",
            "input": "We experimented with four different classifiers, namely, support vector machine BIBREF18 , random forest, extra tree and naive bayes classifier BIBREF19 . Chi square feature selection algorithm is applied to reduces the size of our feature vector. For training our system classifier, we used Scikit-learn BIBREF19 . \n Question: What type of system does the baseline classification use?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-111f3420af1841d0bdf59b2cfc8c2fb8",
            "input": "The previously mentioned datasets are all in English \n Question: For which languages most of the existing MRC datasets are created?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-85117a144e4847fc89023f8f963a75ff",
            "input": "We evaluate the QRNN architecture on a popular document-level sentiment classification benchmark, the IMDb movie review dataset BIBREF17 . The dataset consists of a balanced sample of 25,000 positive and 25,000 negative reviews, divided into equal-size train and test sets, with an average document length of 231 words BIBREF18 . We compare only to other results that do not make use of additional unlabeled data (thus excluding e.g., BIBREF19 ). \n Question: What sentiment classification dataset is used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-1557374a7a564b0b968693f8fb117115",
            "input": "We chose SVMhmm BIBREF111 implementation of Structural Support Vector Machines for sequence labeling. SVM implementation of   \n Question: Which machine learning methods are used in experiments?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-5e477e179ef54cce857307102cc04555",
            "input": "We have training and testing sets in three different languages: English, Chinese and Korean. When fine-tuning, we simply adopted the official training script of BERT, with default hyperparameters, to fine-tune each model until training loss converged. \n Question: what does the model learn in zero-shot setting?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-9957b95d75f64f0087db29fb9769dba0",
            "input": "We test our proposed approach for binary classification on either sarcasm or irony, on seven benchmark datasets retrieved from different media sources. Below we describe each dataset, please see Table TABREF1 below for a summary. Twitter: We use the Twitter dataset provided for the SemEval 2018 Task 3, Irony Detection in English Tweets BIBREF18  Finally, we use the dataset by BIBREF20 , who collected a user self-annotated corpus of tweets with the #sarcasm hashtag. Reddit: BIBREF21 collected SARC, a corpus comprising of 600.000 sarcastic comments on Reddit. We use main subset, SARC 2.0, and the political subset, SARC 2.0 pol. Online Dialogues: We utilize the Sarcasm Corpus V1 (SC-V1) and the Sarcasm Corpus V2 (SC-V2), which are subsets of the Internet Argument Corpus (IAC). Compared to other datasets in our selection, these differ mainly in text length and structure complexity BIBREF22 .  To augment each dataset with our external data, we first filter out tweets that are not in English using language guessing systems. \n Question: Do they evaluate only on English?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-7e0ac42abe294bbc88c89e1f3640b5cc",
            "input": "Last but not least, ethics and fairness are important considerations, that deserve to be studied. In that sense, detection of individual and global bias should be prioritized in order to give useful feedbacks to practitioners. Furthermore we are considering using adversarial learning as in BIBREF33 in order to ensure fairness during the training process. \n Question: Is there any ethical consideration in the research?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-973e1d5a24864e30a2c3ea0836f4f4cb",
            "input": "The test-set accuracies obtained by different learning methods, including the current state-of-the-art results, are presented in Table TABREF11 . \n Question: what models did they compare to?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-b408c166bea94d21b3625a587f32b63b",
            "input": "Davidson et al. BIBREF2 showed that there were systematic and substantial racial biases in five benchmark Twitter datasets annotated for offensive language detection. By examining more samples and with respect to recently studies BIBREF2, BIBREF24, BIBREF19, it is clear that many errors are due to biases from data collection BIBREF19 and rules of annotation BIBREF24 and not the classifier itself.  \n Question: What are the existing biases?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-758d8b616f074138bbcff0e45a02c70f",
            "input": "We compare the SBERT sentence embeddings to other sentence embeddings methods on the following seven SentEval transfer tasks:\n\nMR: Sentiment prediction for movie reviews snippets on a five start scale BIBREF25.\n\nCR: Sentiment prediction of customer product reviews BIBREF26.\n\nSUBJ: Subjectivity prediction of sentences from movie reviews and plot summaries BIBREF27.\n\nMPQA: Phrase level opinion polarity classification from newswire BIBREF28.\n\nSST: Stanford Sentiment Treebank with binary labels BIBREF29.\n\nTREC: Fine grained question-type classification from TREC BIBREF30.\n\nMRPC: Microsoft Research Paraphrase Corpus from parallel news sources BIBREF31. \n Question: What transfer learning tasks are evaluated?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-0daa848207184e19bb7dfd6c4aba5fc5",
            "input": "We demonstrated the utility of Katecheo by deploying the system for question answering in two topics, Medical Sciences and Christianity. \n Question: how many domains did they experiment with?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-a19b0c218e2c479e94ace1d7dd995b84",
            "input": "The resulting dataset is nearly balanced, with 52.3% of the data (1,857 instances) labeled stressful. \n Question: Is the dataset balanced across categories?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-d065c6be8df84d59af347bcfece5708a",
            "input": "In addition to the traditional VAE structure, we introduces an extra context-aware latent variable in CWVAE to learn the event background knowledge. In the pretrain stage, CWVAE is trained on an auxiliary dataset (consists of three narrative story corpora and contains rich event background knowledge), to learn the event background information by using the context-aware latent variable. Subsequently, in the finetune stage, CWVAE is trained on the task-specific dataset to adapt the event background information to each specific aspect of If-Then inferential target (e.g., intents, reactions, etc.). \n Question: How does the context-aware variational autoencoder learn event background information?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-7614c449ef864863b523e0536556bee0",
            "input": "Table TABREF14 shows the results of our main experiments on the 2016 and 2018 test sets for French and German. We use Meteor BIBREF31 as the main metric, as in the WMT tasks BIBREF25 . We compare our transformer baseline to transformer models enriched with image information, as well as to the deliberation models, with or without image information.\n\nWe first note that our multimodal models achieve the state of the art performance for transformer networks (constrained models) on the English-German dataset, as compared to BIBREF30 . Second, our deliberation models lead to significant improvements over this baseline across test sets (average INLINEFORM0 , INLINEFORM1 ). \n Question: What dataset does this approach achieve state of the art results on?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-b4549bf2c09a46c0afa53d125fec6c73",
            "input": "We validate our approach on the Gigaword corpus, which comprises of a training set of 3.8M article headlines (considered to be the full text) and titles (summaries), along with 200K validation pairs, and we report test performance on the same 2K set used in BIBREF7.  \n Question: What dataset they use for evaluation?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-c18fbec8f71f490ca4fba848b669cf09",
            "input": "We use UltraSuite: a repository of ultrasound and acoustic data gathered from child speech therapy sessions BIBREF15 . We used all three datasets from the repository: UXTD (recorded with typically developing children), and UXSSD and UPX (recorded with children with speech sound disorders). In total, the dataset contains 13,815 spoken utterances from 86 speakers, corresponding to 35.9 hours of recordings. The utterances have been categorised by the type of task the child was given, and are labelled as: Words (A), Non-words (B), Sentence (C), Articulatory (D), Non-speech (E), or Conversations (F).  \n Question: Do they annotate their own dataset or use an existing one?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-445d1c1a3ac241d09196e23cb2245feb",
            "input": "For the purpose of our experiments, we use data released by the APDA shared task organizers. The dataset is divided into train and test by organizers. The training set is distributed with labels for the three tasks of age, dialect, and gender. \n Question: What are the three datasets used in the paper?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-6b186d9dc4a04eeca894d125d646c2ea",
            "input": ". We could observe that compared with the base model, we have an improvement of 7.36% on accuracy and 9.69% on F1 score.  \n Question: How big are improvements of small-scale unbalanced datasets when sentence representation is enhanced with topic information?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-8e918b45f7b145ee8d81c99098b0dd37",
            "input": "We collect a novel dataset of 230K+ recipe texts and 1M+ user interactions (reviews) over 18 years (2000-2018) from Food.com. \n Question: How did they obtain the interactions?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-bafac7f7b2ef4073841b3d8181ecbeeb",
            "input": "We conducted an experiment to determine whether we could maintain or improve classifier performances by applying the following three-tiered feature elimination approach:\n\nReduction We reduced the dataset encoded for each class by eliminating features that occur less than twice in the full dataset.\n\nSelection We iteratively applied Chi-Square feature selection on the reduced dataset, selecting the top percentile of highest ranked features in increments of 5 percent to train and test the support vector model using a linear kernel and 5-fold, stratified cross-validation.\n\nRank We cumulatively plotted the average F1-score performances of each incrementally added percentile of top ranked features. We report the percentile and count of features resulting in the first occurrence of the highest average F1-score for each class. \n Question: What are the three steps to feature elimination?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-ef8ff3a9e7aa442fbc7ed65ac7ff7722",
            "input": " To remain consistent with experiments performed with LSTM's we use the morfessor for the subword tokenization in the Finnish Language. \n Question: Is the LSTM baseline a sub-word model?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-ded7f3b4b5404519ae07e332c2e2ab66",
            "input": "After Korektor, there are 4262 unique sentences (including 150 seed sentences) that form the COSTRA 1.0 dataset. \n Question: How many sentence transformations on average are available per unique sentence in dataset?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-83f5906fe0764863bd26963ccda57643",
            "input": "The first data batch consists of tweets relevant to blizzards, hurricanes, and wildfires, under the constraint that they are tweeted by “influential\" tweeters, who we define as individuals certain to have a classifiable sentiment regarding the topic at hand. For example, we assume that any tweet composed by Al Gore regarding climate change is a positive sample, whereas any tweet from conspiracy account @ClimateHiJinx is a negative sample. The assumption we make in ensuing methods (confirmed as reasonable in Section SECREF2 ) is that influential tweeters can be used to label tweets in bulk in the absence of manually-labeled tweets.  \n Question: What methodology is used to compensate for limited labelled data?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-e2c9747a1f124ce39506d133917cc8ae",
            "input": "Co-Reference Resolution: To support multi-turn interactions, it is sometimes necessary to use co-reference resolution techniques for effective retrieval. In Macaw, we identify all the co-references from the last request of user to the conversation history. The same co-reference resolution outputs can be used for different query generation components. This can be a generic or action-specific component.\n\nQuery Generation: This component generates a query based on the past user-system interactions. The query generation component may take advantage of co-reference resolution for query expansion or re-writing.\n\nRetrieval Model: This is the core ranking component that retrieves documents or passages from a large collection. Macaw can retrieve documents from an arbitrary document collection using the Indri python interface BIBREF9, BIBREF10. We also provide the support for web search using the Bing Web Search API. Macaw also allows multi-stage document re-ranking.\n\nResult Generation: The retrieved documents can be too long to be presented using some interfaces. Result generation is basically a post-processing step ran on the retrieved result list. In case of question answering, it can employ answer selection or generation techniques, such as machine reading comprehension models. For example, Macaw features the DrQA model BIBREF11 for question answering. \n Question: What functionality does Macaw provide?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-a483f78a1f8f44f6bb42856c48057daa",
            "input": "We built a phrase-based Chinese-to-English SMT system by using Moses BIBREF18 .  In the end, the final parallel text consists of around 8.8M sentence pairs, 228M Chinese tokens, and 254M English tokens (a token can be a word or punctuation symbol).  The total number of words in these two corpora is 1.81M for Chinese and 2.03M for English. \n Question: Did they only experiment with one language pair?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-66c7e263b2c44c8fa398d1b403a9ab32",
            "input": "Encoders with induced latent structures have been shown to benefit several tasks including document classification, natural language inference BIBREF12, BIBREF13, and machine translation BIBREF11.  \n Question: Is there any evidence that encoders with latent structures work well on other tasks?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-cd22974267324a7abaa4b86e51f6f597",
            "input": "We train language models on two languages: One model is estimated on the German newscrawl distributed by WMT'18 comprising 260M sentences or 6B tokens. Another model is trained on the English newscrawl data comprising 193M sentences or 5B tokens. We learn a joint Byte-Pair-Encoding (BPE; Sennrich et al., 2016) vocabulary of 37K types on the German and English newscrawl and train the language models with this vocabulary. We consider two benchmarks: Most experiments are run on the WMT'18 English-German (en-de) news translation task and we validate our findings on the WMT'18 English-Turkish (en-tr) news task. For WMT'18 English-German, the training corpus consists of all available bitext excluding the ParaCrawl corpus and we remove sentences longer than 250 tokens as well as sentence-pairs with a source/target length ratio exceeding 1.5. This results in 5.18M sentence pairs. We tokenize all data with the Moses tokenizer BIBREF8 and apply the BPE vocabulary learned on the monolingual corpora. \n Question: What dataset do they use?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-1290d003f347471e84150d7fc17c090b",
            "input": "On the other hand, Go-Explore Seq2Seq shows promising results by solving almost half of the unseen games. Figure FIGREF62 (in Appendix SECREF60) shows that most of the lost games are in the hardest set, where a very long sequence of actions is required for winning the game. These results demonstrate both the relative effectiveness of training a Seq2Seq model on Go-Explore trajectories, but they also indicate that additional effort needed for designing reinforcement learning algorithms that effectively generalize to unseen games. \n Question: How do the authors show that their learned policy generalize better than existing solutions to unseen games?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-3c76d8db70424645aa7b565721c58add",
            "input": " To be able to employ the trained model, test sets are first translated to English via machine translation and then inference takes place.   For machine translation, Google translation API is used. \n Question: how did the authors translate the reviews to other languages?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-1a083ef00805418399dad90a23c6ac58",
            "input": "The dataset consists of 5000 reviews in bahasa Indonesia. \n Question: Does the dataset contain non-English reviews?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-fb3fa54cd5124ec6a743b2547c0881c2",
            "input": "We choose the state-of-the-art Transformer BIBREF1 model and the conventional RNN-Search model BIBREF0 as our test bed. \n Question: Which model architectures do they test their word importance approach on?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-daffce8cb01744f58e8c543927d6f757",
            "input": "Using DSC loss improves the F1 score by +0.58 for MRPC and +0.73 for QQP. \n Question: What are method improvements of F1 for paraphrase identification?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-c47d2f0f81aa4e41bb68545e3e86c0b4",
            "input": "We experiment with five benchmark attacking methods for texts: FGM, FGVM, DeepFool BIBREF5, HotFlip BIBREF3) and TYC BIBREF4. \n Question: What are the benchmark attacking methods?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-3b2a89f25e28444a93293bfd6d97735e",
            "input": "We use the annotations from experts for an abstract if it exists otherwise use crowd annotations.  \n Question: How do they match annotators to instances?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-2741e59e47004528ba13663bb2ebbc62",
            "input": "We focus primarily on the word vector representations (word embeddings) created specifically using the twitter dataset. GloVe BIBREF13 is an unsupervised learning algorithm for obtaining vector representations for words. 200-dimensional GloVe embeddings trained on 2 Billion tweets are integrated. Edinburgh embeddings BIBREF14 are obtained by training skip-gram model on Edinburgh corpus BIBREF15 . Since tweets are abundant with emojis, Emoji embeddings BIBREF16 which are learned from the emoji descriptions have been used. Embeddings for each tweet are obtained by summing up individual word vectors and then dividing by the number of tokens in the tweet. \n Question: what pretrained word embeddings were used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-cebf1bbfb777414e90428d06bebe4ebc",
            "input": "For instance, a message can be regarded as harmless on its own, but when taking previous threads into account it may be seen as abusive, and vice versa. This aspect makes detecting abusive language extremely laborious even for human annotators; therefore it is difficult to build a large and reliable dataset BIBREF10 . \n Question: What examples of the difficulties presented by the context-dependent nature of online aggression do they authors give?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-f682e36721ab4d0ea91f27c0ace434f4",
            "input": "We initialized the embeddings of these words with 300 dimensional Glove embeddings BIBREF31 .  \n Question: Do they use pretrained embeddings?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-ecfda9077cf44f3b86a45f31ab37c10c",
            "input": "Across all languages, 145 human annotators were asked to score all 1,888 pairs (in their given language). We finally collect at least ten valid annotations for each word pair in each language. All annotators were required to abide by the following instructions:\n\n1. Each annotator must assign an integer score between 0 and 6 (inclusive) indicating how semantically similar the two words in a given pair are. A score of 6 indicates very high similarity (i.e., perfect synonymy), while zero indicates no similarity.\n\n2. Each annotator must score the entire set of 1,888 pairs in the dataset. The pairs must not be shared between different annotators.\n\n3. Annotators are able to break the workload over a period of approximately 2-3 weeks, and are able to use external sources (e.g. dictionaries, thesauri, WordNet) if required.\n\n4. Annotators are kept anonymous, and are not able to communicate with each other during the annotation process. \n Question: How were the datasets annotated?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-9cafbc406d444fccbec274b765144f94",
            "input": "As before, the training and test sets contain some 30,000 and 5,000 sentence pairs, respectively \n Question: How many samples did they generate for the artificial language?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-4be5caabfba34ac99dacf311a9e5231c",
            "input": "Next, we compared our VTN model with an RNN-based seq2seq VC model called ATTS2S BIBREF8. This model is based on the Tacotron model BIBREF32 with the help of context preservation loss and guided attention loss to stabilize training and maintain linguistic consistency after conversion. We followed the configurations in BIBREF8 but used mel spectrograms instead of WORLD features. \n Question: What is the baseline model?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-568a533e153142a98163b264d2abb0aa",
            "input": "Baselines. We use one strong non-DNN baseline, NBSVM (with unigrams or bigrams features) BIBREF23 and six DNN baselines. The first DNN baseline is CNN BIBREF25, which does not handle noisy labels. The other five were designed to handle noisy labels.\n\nThe comparison results are shown in Table TABREF12. From the results, we can make the following observations. (1) Our NetAb model achieves the best ACC and F1 on all datasets except for F1 of negative class on Laptop. \n Question: Is the model evaluated against a CNN baseline?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-5f3ce3b9988647c0a8a9b8ea0d3e4630",
            "input": "We collect data by using the Twitter API for saved data, available for public use. For our experiments we collect 3200 tweets filtered by keywords like “fire”, “earthquake”, “theft”, “robbery”, “drunk driving”, “drunk driving accident” etc. Later, we manually label tweets with and labels for classification as stage one. Our dataset contains 1313 tweet with positive label and 1887 tweets with a negative label . We create another dataset with the positively labeled tweets and provide them with category labels like “fire”, “accident”, “earthquake” etc. \n Question: Are the tweets specific to a region?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-5087ff68e61441e4aa1a65723fe93e48",
            "input": "We apply our adaptively sparse Transformers on four machine translation tasks.  IWSLT 2017 German $\\rightarrow $ English BIBREF27: 200K sentence pairs.\n\nKFTT Japanese $\\rightarrow $ English BIBREF28: 300K sentence pairs.\n\nWMT 2016 Romanian $\\rightarrow $ English BIBREF29: 600K sentence pairs.\n\nWMT 2014 English $\\rightarrow $ German BIBREF30: 4.5M sentence pairs. \n Question: What tasks are used for evaluation?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-1579cdd22a2d49e590d90232a03ee464",
            "input": "BIBREF6 introduce the KG-A2C, which uses a knowledge graph based state-representation to aid in the section of actions in a combinatorially-sized action-space—specifically they use the knowledge graph to constrain the kinds of entities that can be filled in the blanks in the template action-space. They test their approach on Zork1, showing the combination of the knowledge graph and template action selection resulted in improvements over existing methods. They note that their approach reaches a score of 40 which corresponds to a bottleneck in Zork1 where the player is eaten by a “grue” (resulting in negative reward) if the player has not first lit a lamp. \n Question: What are the baselines?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-019a2a55811349edb2afad79f8f59112",
            "input": "In this section, we evaluate our method and compare its performance against the competitive approaches. We use INLINEFORM0 -fold evaluation protocol with INLINEFORM1 with random dataset split. We measure the performance using standard accuracy metric which we define as a ratio between correctly classified data samples from test dataset and all test samples. \n Question: What evaluation metrics are used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-cd65836d1bd8455e81980d5e7e0ee62d",
            "input": "Thus we propose two kinds of relation-specific meta information: relation meta and gradient meta corresponding to afore mentioned two perspectives respectively. In our proposed framework MetaR, relation meta is the high-order representation of a relation connecting head and tail entities. Gradient meta is the loss gradient of relation meta which will be used to make a rapid update before transferring relation meta to incomplete triples during prediction. \n Question: What meta-information is being transferred?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-81aacfd7e186451bb29a1585c2a1a01c",
            "input": "A higher score indicates better step ordering (with a maximum score of 2). tab:coherencemetrics shows that our personalized models achieve average recipe-level coherence scores of 1.78-1.82, surpassing the baseline at 1.77. On average, human evaluators preferred personalized model outputs to baseline 63% of the time, confirming that personalized attention improves the semantic plausibility of generated recipes. \n Question: What were their results on the new dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-e9ab4a7b6bf54795bbd08429538fd6d0",
            "input": "$D_a$ contained all of the tweets collected on the attack day of the five attacks mentioned in section 4.2. And $D_b$ contained all of the tweets collected before the five attacks. There are 1180 tweets in $D_a$ and 7979 tweets in $D_b$. The tweets on the attack days ($D_a$) are manually annotated and only 50 percent of those tweets are actually about a DDoS attack. \n Question: Do twitter users tend to tweet about the DOS attack when it occurs? How much data supports this assumption?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-94ffc3f7831e422585ee41fb722d0a63",
            "input": "We therefore use reduction in softmax probability of the correct relation as our signaling strength metric for the model. We call this metric ${\\Delta }_s$ (for delta-softmax), which can be written as:\n\nwhere $rel$ is the true relation of the EDU pair, $t_i$ represents the token at index $i$ of $N$ tokens, and $X_{mask=i}$ represents the input sequence with the masked position $i$ (for $i \\in 1 \\ldots N$ ignoring separators, or $\\phi $, the empty set). \n Question: How is the delta-softmax calculated?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-55b5944c5fb141c4a497703d2e033883",
            "input": "(2) Different from LEM and DPEMM, AEM uses a generator network to capture the event-related patterns and is able to mine events from different text sources (short and long). Moreover, unlike traditional inference procedure, such as Gibbs sampling used in LEM and DPEMM, AEM could extract the events more efficiently due to the CUDA acceleration; \n Question: What alternative to Gibbs sampling is used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-01e478f7035b4e819ef8b3de34997a62",
            "input": "To tackle this issue we applying a back-translation method BIBREF13, where we translate indirect and physical harassment tweets of the train set from english to german, french and greek. After that, we translate them back to english in order to achieve data augmentation. \n Question: What language(s) is/are represented in the dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-f62c117e9e4143b397425c93bdc3403f",
            "input": "Maximum matching (MM) is one of the most popular fundamental and structural segmentation algorithms for word segmentation BIBREF19 .  However, it does not solve the problem of ambiguous words and unknown words that do not exist in the dictionary. \n Question: What are the limitations of existing Vietnamese word segmentation systems?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-1e860719609743ef920695a12a15ba6b",
            "input": "To further improve the performance, we adopt system combination on the decoding lattice level. By combining systems, we can take advantage of the strength of each model that is optimized for different domains.  The best result for vlsp2018 of 4.85% WER is obtained by the combination weights 0.6:0.4 where 0.6 is given to the general language model and 0.4 is given to the conversation one. On the vlsp2019 set, the ratio is change slightly by 0.7:0.3 to deliver the best result of 15.09%. \n Question: What is the language model combination technique used in the paper?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-9921ee8ccb1444e3a392256f77661cdc",
            "input": "I also performed several case studies. I obtained document embeddings, in the same latent space as the topic embeddings, by summing the posterior mean vectors INLINEFORM0 for each token, and visualized them in two dimensions using INLINEFORM1 -SNE BIBREF24 (all vectors were normalized to unit length). The state of the Union addresses (Figure FIGREF27 ) are embedded almost linearly by year, with a major jump around the New Deal (1930s), and are well separated by party at any given time period.  \n Question: What is an example of a computational social science NLP task?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-9b693f99415047d3bd807d5ca07ead12",
            "input": "We also evaluate all five models on downstream tasks from the VecEval suite BIBREF13 , using only the tasks for which training and evaluation data is freely available: chunking, sentiment and question classification, and natural language identification (NLI). The default settings from the suite are used, but we run only the fixed settings, where the embeddings themselves are not tunable parameters of the models, forcing the system to use only the information already in the embeddings.\n\nFinally, we use LV-N, LV-M, and FT to generate OOV word representations for the following words: 1) “hellooo”: a greeting commonly used in instant messaging which emphasizes a syllable. 2) “marvelicious”: a made-up word obtained by merging “marvelous” and “delicious”. 3) “louisana”: a misspelling of the proper name “Louisiana”. 4) “rereread”: recursive use of prefix “re”. 5) “tuzread”: made-up prefix “tuz”. \n Question: How do they evaluate their resulting word embeddings?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-9c958f92848a4a919e34f4caa4c3ea95",
            "input": "In this section, we propose new simple disentanglement models that perform better than prior methods, and re-examine prior work. \n Question: Did they experiment with the corpus?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-db73eb7d13f8473db9fc1c3034101d51",
            "input": "For comparison, we also report the performance of the error detection system by Rei2016, trained using the same FCE dataset. \n Question: What was the baseline used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-6e18e91867164439b6d4147998272566",
            "input": "Also, we compared our models with other existing works on this dataset including OpATT BIBREF6 and Neural Content Planning with conditional copy (NCP+CC) BIBREF4. \n Question: What is the state-of-the-art model for the task?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-da36d5f8c99f4afaa5ef5aa9bfeb76f2",
            "input": "We evaluate our model and baseline models on three versions of the dataset. The first one (Inc) only considers the original data, containing naturally incorrect tweets, and achieves accuracy of 80$\\%$ against BERT's 72$\\%$. The second version (Corr) considers the corrected tweets, and shows higher accuracy given that it is less noisy.  \n Question: Should their approach be applied only when dealing with incomplete data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-3f7b7798726441cfa60bc8ebe110a653",
            "input": "For the two single sentence tasks—the syntax-oriented CoLA task and the SST sentiment task—we find somewhat deteriorated performance. \n Question: Does the additional training on supervised tasks hurt performance in some tasks?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-8ec021efaf3b489b8f5f074a9da245fb",
            "input": "For all tasks, we use a TensorFlow implementation. AraNet predicts age, dialect, gender, emotion, irony, and sentiment from social media posts. It delivers state-of-the-art and competitive performance on these tasks and has the advantage of using a unified, simple framework based on the recently-developed BERT model.  \n Question: Did they experiment on all the tasks?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-87138e7e5a0148f4b2c6080f15f2b644",
            "input": "Our official scores (column Ens Test in Table TABREF19 ) have placed us second (EI-Reg, EI-Oc), fourth (V-Reg) and fifth (V-Oc) on the SemEval AIT-2018 leaderboard. \n Question: What were the scores of their system?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-d9da0cb9b8af4911a2414e8fe60be7f1",
            "input": "The dataset consists of queries and the corresponding image search results. Each token in each query is given a language tag based on the user-set home language of the user making the search on Google Images. \n Question: Do the images have multilingual annotations or monolingual ones?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-435a0635c3fd4f94944438cbe2f634d9",
            "input": "For the NER task, we consider both Chinese datasets, i.e., OntoNotes4.0 BIBREF34 and MSRA BIBREF35, and English datasets, i.e., CoNLL2003 BIBREF36 and OntoNotes5.0 BIBREF37. Table shows experimental results on NER datasets. For English datasets including CoNLL2003 and OntoNotes5.0, our proposed method outperforms BERT-MRCBIBREF38 by +0.29 and +0.96 respectively. We observe huge performance boosts on Chinese datasets, achieving F1 improvements by +0.97 and +2.36 on MSRA and OntoNotes4.0, respectively. As far as we are concerned, we are setting new SOTA performances on all of the four NER datasets. \n Question: What are method's improvements of F1 for NER task for English and Chinese datasets?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-86b4dc3515b64c56b7b14a2b6267e68a",
            "input": "We introduce a surrogate training objective that avoids these problems and as a result is fully continuous.  Specifically, we form a continuous function softLB that seeks to approximate the result of running our decoder on input INLINEFORM0 and then evaluating the result against INLINEFORM1 using INLINEFORM2 . \n Question: Do they provide a framework for building a sub-differentiable for any final loss metric?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-1245704c04fe41a6834a96eba66f9511",
            "input": "Table TABREF19 displays the performance of the 4 baselines on the ReviewQA's test set. These results are the performance achieved by our own implementation of these 4 models. \n Question: What tasks were evaluated?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-b44813fc8fa54526b007637f2a6a1d9f",
            "input": "It may be difficult to spot patterns by just looking at a collection of images. Another method is to tag all descriptions with part-of-speech information, so that it becomes possible to see e.g. which adjectives are most commonly used for particular nouns. One method readers may find particularly useful is to leverage the structure of Flickr30K Entities BIBREF8 . Following this, I applied Louvain clustering BIBREF9 to the coreference graph, resulting in clusters of expressions that refer to similar entities. \n Question: Which methods are considered to find examples of biases and unwarranted inferences??",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-0129fbfcacf74463827bbbc3241c4667",
            "input": "Results on LangID and NoLangID are compared to the system presented by deri2016grapheme, which is identified in our results as wFST. \n Question: what was the baseline?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-c81c6c00b72e43929d857897a06f02cb",
            "input": "According to the categories mentioned in BIBREF5 , irony can be classified into 3 classes: verbal irony by means of a polarity contrast, the sentences containing expression whose polarity is inverted between the intended and the literal evaluation; other types of verbal irony, the sentences that show no polarity contrast between the literal and intended meaning but are still ironic; and situational irony, the sentences that describe situations that fail to meet some expectations. As ironies in the latter two categories are obscure and hard to understand, we decide to only focus on ironies in the first category in this work.   Because of the lack of previous work and baselines on irony generation, we implement our model based on style transfer.  \n Question: What are the difficulties in modelling the ironic pattern?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-97849a0f8a2c430686c57fe054910896",
            "input": "In total around 500 different workers were involved in the annotation. about 500 \n Question: How many people participated in their evaluation study of table-to-text models?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-a50fd1ccbb764b528d561349543e45c9",
            "input": "The answer styles considered in the experiments corresponded to the two tasks. The NLG task requires a well-formed answer that is an abstractive summary of the question and ten passages, averaging 16.6 words. The Q&A task also requires an abstractive answer but prefers a more concise answer than the NLG task, averaging 13.1 words, where many of the answers do not contain the context of the question. \n Question: Is there exactly one \"answer style\" per dataset?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-7d82fa67a51045b096f7efb364fd399b",
            "input": "We work with a dataset composed of 3,206 news articles, each labeled real or fake, with a perfect 50/50 split between 1,603 real and fake articles, respectively. Fake articles were sourced from online sites that were tagged as fake news sites by the non-profit independent media fact-checking organization Verafiles and the National Union of Journalists in the Philippines (NUJP). Real articles were sourced from mainstream news websites in the Philippines, including Pilipino Star Ngayon, Abante, and Bandera. \n Question: What is the source of the dataset?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-e9ec7e521a9e4c43809b2a0a4f57ebe3",
            "input": "In the sixth international workshop on Vietnamese Language and Speech Processing (VLSP 2019), the Hate Speech Detection (HSD) task is proposed as one of the shared-tasks to handle the problem related to controlling content in SNSs. The fundamental idea of this system is how to make a system that has the diversity of viewing an input. That because of the variety of the meaning in Vietnamese language especially with the acronym, teen code type. \n Question: Is the data all in Vietnamese?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-fd0c209f27924b85b1f2d7e540e8a49d",
            "input": "Next, we analyzed all tweets labeled as offensive to better understand how Arabic speakers use offensive language. Here is a breakdown of usage:\n\nDirect name calling: The most frequent attack is to call a person an animal name, and the most used animals were كلب> (“klb” - “dog”), حمار> (“HmAr” - “donkey”), and بهيم> (“bhym” - “beast”). The second most common was insulting mental abilities using words such as غبي> (“gby” - “stupid”) and عبيط> (“EbyT” -“idiot”). Some culture-specific differences should be considered. Not all animal names are used as insults. For example, animals such as أسد> (“Asd” - “lion”), صقر> (“Sqr” - “falcon”), and غزال> (“gzAl” - “gazelle”) are typically used for praise. For other insults, people use: some bird names such as دجاجة> (“djAjp” - “chicken”), بومة> (“bwmp” - “owl”), and غراب> (“grAb” - “crow”); insects such as ذبابة> (“*bAbp” - “fly”), صرصور> (“SrSwr” - “cockroach”), and حشرة> (“H$rp” - “insect”); microorganisms such as جرثومة> (“jrvwmp” - “microbe”) and طحالب> (“THAlb” - “algae”); inanimate objects such as جزمة> (“jzmp” - “shoes”) and سطل> (“sTl” - “bucket”) among other usages.\n\nSimile and metaphor: Users use simile and metaphor were they would compare a person to: an animal as in زي الثور> (“zy Alvwr” - “like a bull”), سمعني نهيقك> (“smEny nhyqk” - “let me hear your braying”), and هز ديلك> (“hz dylk” - “wag your tail”); a person with mental or physical disability such as منغولي> (“mngwly” - “Mongolian (down-syndrome)”), معوق> (“mEwq” - “disabled”), and قزم> (“qzm” - “dwarf”); and to the opposite gender such as جيش نوال> (“jy$ nwAl” - “Nawal's army (Nawal is female name)”) and نادي زيزي> (“nAdy zyzy” - “Zizi's club (Zizi is a female pet name)”).\n\nIndirect speech: This type of offensive language includes: sarcasm such as أذكى إخواتك> (“A*kY AxwAtk” - “smartest one of your siblings”) and فيلسوف الحمير> (“fylswf AlHmyr” - “the donkeys' philosopher”); questions such as ايه كل الغباء ده> (“Ayh kl AlgbA dh” - “what is all this stupidity”); and indirect speech such as النقاش مع البهايم غير مثمر> (“AlnqA$ mE AlbhAym gyr mvmr” - “no use talking to cattle”).\n\nWishing Evil: This entails wishing death or major harm to befall someone such as ربنا ياخدك> (“rbnA yAxdk” - “May God take (kill) you”), الله يلعنك> (“Allh ylEnk” - “may Allah/God curse you”), and روح في داهية> (“rwH fy dAhyp” - equivalent to “go to hell”).\n\nName alteration: One common way to insult others is to change a letter or two in their names to produce new offensive words that rhyme with the original names. Some examples of such include changing الجزيرة> (“Aljzyrp” - “Aljazeera (channel)”) to الخنزيرة> (“Alxnzyrp” - “the pig”) and خلفان> (“xlfAn” - “Khalfan (person name)”) to خرفان> (“xrfAn” - “crazed”).\n\nSocietal stratification: Some insults are associated with: certain jobs such as بواب> (“bwAb” - “doorman”) or خادم> (“xAdm” - “servant”); and specific societal components such بدوي> (“bdwy” - “bedouin”) and فلاح> (“flAH” - “farmer”).\n\nImmoral behavior: These insults are associated with negative moral traits or behaviors such as حقير> (“Hqyr” - “vile”), خاين> (“xAyn” - “traitor”), and منافق> (“mnAfq” - “hypocrite”).\n\nSexually related: They include expressions such as خول> (“xwl” - “gay”), وسخة> (“wsxp” - “prostitute”), and عرص> (“ErS” - “pimp”). \n Question: What are the distinctive characteristics of how Arabic speakers use offensive language?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-6e63ca3a8026472994c42141b1e0c309",
            "input": "Finally, the morphosyntactic annotation was performed automatically, which may lead to errors, especially in the case of noisy web text. \n Question: Did they use a crowdsourcing platform for annotations?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-9fea37ab99b84bb2a44bc8b9fd2d99e2",
            "input": "First, we simply calculate the average unique predictions produced by both INLINEFORM0 and INLINEFORM1 in experiments shown in Section SECREF36 . \n Question: What two metrics are proposed?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-c016af8a15ce453aa422de4c2749ef8e",
            "input": "Each classifier is implemented with the following specifications:\n\nNaïve Bayes (NB): Multinomial NB with additive smoothing constant 1\n\nLogistic Regression (LR): Linear LR with L2 regularization constant 1 and limited-memory BFGS optimization\n\nSupport Vector Machine (SVM): Linear SVM with L2 regularization constant 1 and logistic loss function\n\nRandom Forests (RF): Averaging probabilistic predictions of 10 randomized decision trees\n\nGradient Boosted Trees (GBT): Tree boosting with learning rate 1 and logistic loss function Along with traditional machine learning approaches, we investigate neural network based models to evaluate their efficacy within a larger dataset. In particular, we explore Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and their variant models. \n Question: What learning models are used on the dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-cc00ce8e846d4312ada37a8e5be40e92",
            "input": "To address this drawback in ROUGE, we propose a new evaluation metric: Critical Information Completeness (CIC). Formally, CIC is a recall of semantic slot information between a candidate summary and a reference summary. CIC is defined as follows:\n\nwhere $V$ stands for a set of delexicalized values in the reference summary, $Count_{match}(v)$ is the number of values co-occurring in the candidate summary and reference summary, and $m$ is the number of values in set $V$. In our experiments, CIC is computed as the arithmetic mean over all the dialog domains to retain the overall performance.\n\nCIC is a suitable complementary metric to ROUGE because it accounts for the most important information within each dialog domain. CIC can be applied to any summarization task with predefined essential entities. \n Question: How does new evaluation metric considers critical informative entities?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-3f87389dbff94a6bacd76c91b2fc6b03",
            "input": "However, recent work has found that many NLI datasets contain biases, or annotation artifacts, i.e., features present in hypotheses that enable models to perform surprisingly well using only the hypothesis, without learning the relationship between two texts BIBREF2 , BIBREF3 , BIBREF4 . For instance, in some datasets, negation words like “not” and “nobody” are often associated with a relationship of contradiction. As a ramification of such biases, models may not generalize well to other datasets that contain different or no such biases. \n Question: Is such bias caused by bad annotation?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-e936bc2ddff74cbf90f74cdcd925c333",
            "input": "Fenerbahçe We have decided to consider tweets about popular sports clubs as our domain for stance detection.  Hence we have determined our targets as Galatasaray (namely Target-1) and Fenerbahçe (namely, Target-2) which are two of the most popular football clubs in Turkey.  Then, we have annotated the stance information in the tweets for these targets as Favor or Against. For the purposes of the current study, we have not annotated any tweets with the Neither class. \n Question: How were the tweets annotated?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-e0b3e10d7d904a6cb8dd9ff6639d1d7c",
            "input": "Datasets\nFor training English-Hindi NMT systems, we use the IITB English-Hindi parallel corpus BIBREF22 ( INLINEFORM0 sentences from the training set) and the ILCI English-Hindi parallel corpus ( INLINEFORM1 sentences). The ILCI (Indian Language Corpora Initiative) multilingual parallel corpus BIBREF23 spans multiple Indian languages from the health and tourism domains. We use the 520-sentence dev-set of the IITB parallel corpus for validation. For each child task, we use INLINEFORM2 sentences from ILCI corpus as the test set. \n Question: Which dataset(s) do they experiment with?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-d3a7ba23ee434e6ebc6e2de0f67a9ad6",
            "input": "Hypothesis 4 (H4) : The proposed template-based synthesis model outperforms a simple word replacement model. Table TABREF13 supports H4 by showing that the proposed synthesis model outperforms the WordNet baseline in training (rows 7, 8 and 19, 20) except Stat2016, and tuning (10, 11 and 22, 23) over all courses. It also shows that while adding synthetic data from the baseline is not always helpful, adding synthetic data from the template model helps to improve both the training and the tuning process. In both CS and ENGR courses, tuning with synthetic data enhances all ROUGE scores compared to tuning with only the original data. (rows 9 and 11). As for Stat2015, R-1 and R-$L$ improved, while R-2 decreased. For Stat2016, R-2 and R-$L$ improved, and R-1 decreased (rows 21 and 23). Training with both student reflection data and synthetic data compared to training with only student reflection data yields similar improvements, supporting H3 (rows 6, 8 and 18, 20). \n Question: Is the template-based model realistic?  ",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-7e014fb27b0b45f28921149fb8208a1f",
            "input": "The quality class of a Wikipedia article is assigned by Wikipedia reviewers or any registered user, who can discuss through the article's talk page to reach consensus. The arXiv dataset BIBREF2 consists of three subsets of academic articles under the arXiv repository of Computer Science (cs), from the three subject areas of: Artificial Intelligence (cs.ai), Computation and Language (cs.cl), and Machine Learning (cs.lg). In line with the original dataset formulation BIBREF2 , a paper is considered to have been accepted (i.e. is positively labeled) if it matches a paper in the DBLP database or is otherwise accepted by any of the following conferences: ACL, EMNLP, NAACL, EACL, TACL, NIPS, ICML, ICLR, or AAAI. Failing this, it is considered to be rejected (noting that some of the papers may not have been submitted to one of these conferences). \n Question: Where do they get their ground truth quality judgments?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-13242395e80f419cbed7dad9263a3ccc",
            "input": "We use a benchmark dataset created by ceccarelli2013learning from the CoNLL 2003 data.  \n Question: What is the benchmark dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-a7978a43f43443898fecafb07f21bf3f",
            "input": "In addition, we evaluate the effectiveness of pre-training by comparing it with a jointly-trained model of forward translation and back-translation. Experimental results show that the encoder-decoder-reconstructor offers significant improvement in BLEU scores and alleviates the problem of repeating and missing words in the translation on English-Japanese translation task, and the encoder-decoder-reconstructor can not be trained well without pre-training, so it proves that we have to train the forward translation model in a manner similar to the conventional attention-based NMT as pre-training. \n Question: Is pre-training effective in their evaluation?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-ed3352c0bc4d4d22bf3c7adf8182d8de",
            "input": "Maximum Entropy theory is applied to solve Vietnamese word segmentation BIBREF15 , BIBREF22 , BIBREF23 . There are several studies about the Vietnamese word segmentation task over the last decade. Dinh et al. started this task with Weighted Finite State Transducer (WFST) approach and Neural Network approach BIBREF9 . In addition, machine learning approaches are studied and widely applied to natural language processing and word segmentation as well. In fact, several studies used support vector machines (SVM) and conditional random fields (CRF) for the word segmentation task BIBREF7 , BIBREF8 . \n Question: Which approaches have been applied to solve word segmentation in Vietnamese?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-86dde69ff4dc4c9099be38c735340562",
            "input": "We use a new dataset of GD statements from 1970 to 2016, the UN General Debate Corpus (UNGDC), to examine the international development agenda in the UN BIBREF3 .  \n Question: Is the dataset multilingual?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-cb257d7fad244f69b62367e0d63bd504",
            "input": "As can be seen, the proposed DSC loss outperforms the best baseline results by a large margin, i.e., outperforming BERT-tagger by +1.86 in terms of F1 score on CTB5, +1.80 on CTB6 and +2.19 on UD1.4. \n Question: What are method's improvements of F1 w.r.t. baseline BERT tagger for Chinese POS datasets?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-3f37fd9dc12748d1b043a43a9c4a68c8",
            "input": "Considering the improvements over the majority baseline achieved by the RNN model for both non-English (on the average 22.76% relative improvement; 15.82% relative improvement on Spanish, 72.71% vs. 84.21%, 30.53% relative improvement on Turkish, 56.97% vs. 74.36%, 37.13% relative improvement on Dutch, 59.63% vs. 81.77%, and 7.55% relative improvement on Russian, 79.60% vs. 85.62%) and English test sets (27.34% relative improvement), we can draw the conclusion that our model is robust to handle multiple languages. \n Question: which non-english language was the had the worst results?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-4b055173615341aeb5a00772e8ba80b1",
            "input": "This allows Macaw to support multi-modal interactions, such as text, speech, image, click, etc. \n Question: What modalities are supported by Macaw?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-860e526d48b34d4082613be38ba1e83b",
            "input": "We focus on social media posts from the website Twitter, which are an excellent testing ground for character based models due to the noisy nature of text. Heavy use of slang and abundant misspellings means that there are many orthographically and semantically similar tokens, and special characters such as emojis are also immensely popular and carry useful semantic information. In our moderately sized training dataset of 2 million tweets, there were about 0.92 million unique word types. \n Question: Does the paper clearly establish that the challenges listed here exist in this dataset and task?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-aa6350962a974d95a9d5fc23da90f56f",
            "input": "One example illustrates that the mistakes the model makes can be associated with changes in the party policy. \n Question: Do changes in policies of the political actors account for all of the mistakes the model made?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-c442e338c5364f76a1a2958c90e2135a",
            "input": "Experiments ::: Baselines\nTo comprehensively evaluate our AGDT, we compare the AGDT with several competitive models. \n Question: Is the model evaluated against other Aspect-Based models?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-bede0b391142455db95e5ff7ed8bda33",
            "input": "In this step we extract the AMR graphs of the summary sentences using story sentence AMRs. We divide this task in two parts. First is finding the important sentences from the story and then extracting the key information from those sentences using their AMR graphs. \n Question: How are sentences selected from the summary graph?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-e98fea7009694de1b2dfef594eb5bcf8",
            "input": "We use the publicly available dataset KVRET BIBREF5 in our experiments. There are 2,425 dialogues for training, 302 for validation and 302 for testing, as shown in the upper half of Table TABREF12. \n Question: What is the size of the dataset?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-c67b22cc56c543dfa90ca350273b2419",
            "input": "The summaries generated by our ILP-based approach were compared with the corresponding manual summaries using the ROUGE BIBREF22 unigram score. \n Question: What evaluation metrics were used for the summarization task?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-9494505da32946d1a165bcd248ea372f",
            "input": "We use the same baseline as used by lang2011unsupervised which has been shown to be difficult to outperform. This baseline assigns a semantic role to a constituent based on its syntactic function, i.e. the dependency relation to its head. \n Question: What baseline model is used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-73499816fa344634b415a1ecd4742335",
            "input": "We also consider the following baselines. BOW-Tags represents locations using a bag-of-words representation, using the same tag weighting as the embedding model. BOW-KL(Tags) uses the same representation but after term selection, using the same KL-based method as the embedding model. BOW-All combines the bag-of-words representation with the structured information, encoded as proposed in BIBREF7 . GloVe uses the objective from the original GloVe model for learning location vectors, i.e. this variant differs from EGEL-Tags in that instead of INLINEFORM1 we use the number of co-occurrences of tag INLINEFORM2 near location INLINEFORM3 , measured as INLINEFORM4 . \n Question: what are the existing approaches?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-a2bbd60505a64d99b4fdd18c296f5d93",
            "input": " We opted for an automatic approach instead, that can be scaled arbitrarily and at little cost: we generate synthetic sentence pairs $(, \\tilde{})$ by randomly perturbing 1.8 million segments $$ from Wikipedia. We use three techniques: mask-filling with BERT, backtranslation, and randomly dropping out words.  \n Question: How are the synthetic examples generated?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-926090d9918b4bb3a19815c42c9d4768",
            "input": "After excluding all the words that are not used by at least three separate users in our training set, we build our AllWords model by counting the frequencies of all the remaining words and training a multinomial Naive Bayes classifier.  \n Question: What model did they use for their system?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-b612df02396049ac85717bf260ac1213",
            "input": "CIC is a suitable complementary metric to ROUGE because it accounts for the most important information within each dialog domain. CIC can be applied to any summarization task with predefined essential entities. \n Question: Is new evaluation metric extension of ROGUE?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-5742d0449f834229bd8e89426ae88336",
            "input": "To investigate the trade-off between speed and performance, we compare our technique to standard models with and without attention on a Sequence Copy Task of varying length like in BIBREF14 . For this purpose we use 4 large machine translation datasets of WMT'17 on the following language pairs: English-Czech (en-cs, 52M examples), English-German (en-de, 5.9M examples), English-Finish (en-fi, 2.6M examples), and English-Turkish (en-tr, 207,373 examples). \n Question: Which datasets are used in experiments?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-3d4eb61e177e4ad19fa94bc4dd3da99b",
            "input": "The input of our model are the words in the input text $x[1], ... , x[n]$ and query $q[1], ... , q[n]$ . We concatenate pre-trained word embeddings from GloVe BIBREF40 and character embeddings trained by CharCNN BIBREF41 to represent input words. The $2d$ -dimension embedding vectors of input text $x_1, ... , x_n$ and query $q_1, ... , q_n$ are then fed into a Highway Layer BIBREF42 to improve the capability of word embeddings and character embeddings as\n\n$$\\begin{split} g_t &= {\\rm sigmoid}(W_gx_t+b_g) \\\\ s_t &= {\\rm relu } (W_xx_t+b_x) \\\\ u_t &= g_t \\odot s_t + (1 - g_t) \\odot x_t~. \\end{split}$$ (Eq. 18) The same Highway Layer is applied to $q_t$ and produces $v_t$ . Next, $u_t$ and $v_t$ are fed into a Bi-Directional Long Short-Term Memory Network (BiLSTM) BIBREF44 respectively in order to model the temporal interactions between sequence words: Then we feed $\\mathbf {U}$ and $\\mathbf {V}$ into the attention flow layer BIBREF27 to model the interactions between the input text and query. Therefore, we introduce Self-Matching Layer BIBREF29 in our model as\n\n$$\\begin{split} o_t &= {\\rm BiLSTM}(o_{t-1}, [h_t, c_t]) \\\\ s_j^t &= w^T {\\rm tanh}(W_hh_j+\\tilde{W_h}h_t)\\\\ \\alpha _i^t &= {\\rm exp}(s_i^t)/\\Sigma _{j=1}^n{\\rm exp}(s_j^t)\\\\ c_t &= \\Sigma _{i=1}^n\\alpha _i^th_i ~. \\end{split}$$ (Eq. 20) Finally we feed the embeddings $\\mathbf {O} = [o_1, ... , o_n]$ into a Pointer Network BIBREF39 to decode the answer sequence as\n\n$$\\begin{split} p_t &= {\\rm LSTM}(p_{t-1}, c_t) \\\\ s_j^t &= w^T {\\rm tanh}(W_oo_j+W_pp_{t-1})\\\\ \\beta _i^t &= {\\rm exp}(s_i^t)/\\Sigma _{j=1}^n{\\rm exp}(s_j^t)\\\\ c_t &= \\Sigma _{i=1}^n\\beta _i^to_i~. \\end{split}$$ (Eq. 21) Therefore, the probability of generating the answer sequence $\\textbf {a}$ is as follows\n\n$${\\rm P}(\\textbf {a}|\\mathbf {O}) = \\prod _t {\\rm P}(a^t | a^1, ... , a^{t-1}, \\mathbf {O})~.$$ (Eq. 23) \n Question: What QA models were used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-2b5411279b8d45b6988408ed15a858e4",
            "input": "Suppose there are INLINEFORM0 mentions in documents on average, among these global models, NCEL not surprisingly has the lowest time complexity INLINEFORM1 since it only considers adjacent mentions, where INLINEFORM2 is the number of sub-GCN layers indicating the iterations until convergence. \n Question: Do they only use adjacent entity mentions or use more than that in some cases (next to adjacent)?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-baa929c9f2854e9e96bdeba73b099c94",
            "input": "Table TABREF14 presents the distribution of the tweets by country before and after the filtering process. A large portion of the samples is from India because the MeToo movement has peaked towards the end of 2018 in India. There are very few samples from Russia likely because of content moderation and regulations on social media usage in the country. \n Question: Do the tweets come from a specific region?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-c942f066a92148cebe663fc2dce0a875",
            "input": "What similarities and/or differences do these topics have with non-violent, non-Islamic religious material addressed specifically to women? As these questions suggest, to understand what, if anything, makes extremist appeals distinctive, we need a point of comparison in terms of the outreach efforts to women from a mainstream, non-violent religious group. For this purpose, we rely on an online Catholic women's forum. Comparison between Catholic material and the content of ISIS' online magazines allows for novel insight into the distinctiveness of extremist rhetoric when targeted towards the female population. To accomplish this task, we employ topic modeling and an unsupervised emotion detection method. \n Question: How are similarities and differences between the texts from violent and non-violent religious groups analyzed?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-385a5c5c746e481cac92b0e682333d18",
            "input": "The baseline approach is based on BIBREF20 . It divides the task of interpreting commands for behavioral navigation into two steps: path generation, and path verification. For path generation, this baseline uses a standard sequence-to-sequence model augmented with an attention mechanism, similar to BIBREF23 , BIBREF6 . For path verification, the baseline uses depth-first search to find a route in the graph that matches the sequence of predicted behaviors. If no route matches perfectly, the baseline changes up to three behaviors in the predicted sequence to try to turn it into a valid path. \n Question: What baselines did they compare their model with?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-1c9e5b20930540bdad95fd2b862565b6",
            "input": "Grammatical error correction (GEC) is a challenging task due to the variability of the type of errors and the syntactic and semantic dependencies of the errors on the surrounding context. Most of the grammatical error correction systems use classification and rule-based approaches for correcting specific error types. However, these systems use several linguistic cues as features. The standard linguistic analysis tools like part-of-speech (POS) taggers and parsers are often trained on well-formed text and perform poorly on ungrammatical text. This introduces further errors and limits the performance of rule-based and classification approaches to GEC. As a consequence, the phrase-based statistical machine translation (SMT) approach to GEC has gained popularity because of its ability to learn text transformations from erroneous text to correct text from error-corrected parallel corpora without any additional linguistic information.  We model our GEC system based on the phrase-based SMT approach. However, traditional phrase-based SMT systems treat words and phrases as discrete entities. We take advantage of continuous space representation by adding two neural network components that have been shown to improve SMT systems BIBREF3 , BIBREF4 . To train NNJM, we use the publicly available implementation, Neural Probabilistic Language Model (NPLM) BIBREF14 . The latest version of Moses can incorporate NNJM trained using NPLM as a feature while decoding. Similar to NNGLM, we use the parallel text used for training the translation model in order to train NNJM. We use a source context window size of 5 and a target context window size of 4. We select a source context vocabulary of 16,000 most frequent words from the source side. The target context vocabulary and output vocabulary is set to the 32,000 most frequent words. We use a single hidden layer to speed up training and decoding with an input embedding dimension of 192 and 512 hidden layer nodes.  \n Question: Do they use pretrained word representations in their neural network models?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-67a5d14ca1194fb496b8920f0be830b4",
            "input": "We adopt the annotation machinery of BIBREF5 implemented using Amazon's Mechanical Turk, and annotate each predicate by 2 trained workers independently, while a third consolidates their annotations into a final set of roles and arguments. In this consolidation task, the worker validates questions, merges, splits or modifies answers for the same role according to guidelines, and removes redundant roles by picking the more naturally phrased questions.  \n Question: What is different in the improved annotation protocol?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-8d8fcf6d6b814657ad136701e7bc4161",
            "input": "Through our experiments, we make subtle points related to: (a) the performance of our features, (b) how our approach compares against human ability to detect drunk-texting, (c) most discriminative stylistic features, and (d) an error analysis that points to future work. \n Question: Do the authors mention any confounds to their study?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-939c347819e940f981914ac345a8a232",
            "input": "We extract user's age by applying regular expression patterns to profile descriptions (such as \"17 years old, self-harm, anxiety, depression\") BIBREF41 . We compile \"age prefixes\" and \"age suffixes\", and use three age-extraction rules: 1. I am X years old 2. Born in X 3. X years old, where X is a \"date\" or age (e.g., 1994). We selected a subset of 1464 users INLINEFORM0 from INLINEFORM1 who disclose their gender in their profile description. \n Question: Where does the information on individual-level demographics come from?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-21282a89d62b463c83f67932a94191f9",
            "input": "In Table TABREF26 we report the results of our model on the three datasets standardly used for the evaluation of emotion classification, which we have described in Section SECREF3 . \n Question: What was their performance on emotion detection?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-2edb296d892d4572a212b88c0c0bd4b7",
            "input": "We draw on a recently released corpus of state speeches delivered during the annual UN General Debate that provides the first dataset of textual output from states that is recorded at regular time-series intervals and includes a sample of all countries that deliver speeches BIBREF11 .  \n Question: Which dataset do they use?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-2f6c221154d942bd92d3ee35d1b6df1e",
            "input": "Based on the gained insights, we have improved the state-of-the-art in XNLI for the Translate-Test and Zero-Shot approaches by a substantial margin. \n Question: Is the improvement over state-of-the-art statistically significant?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-1d814df64b6b424cbe623a6e5a9450c1",
            "input": "One way to analyze the model is to compute model gradients with respect to input features BIBREF26, BIBREF25. Figure FIGREF37 shows that in this particular example, the most important model inputs are verbs possibly associated with the entity butter, in addition to the entity's mentions themselves. It further shows that the model learns to extract shallow clues of identifying actions exerted upon only the entity being tracked, regardless of other entities, by leveraging verb semantics. \n Question: What evidence do they present that the model attends to shallow context clues?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-6a4bf5ef2db04ae98725a7dcd7d99485",
            "input": "Given the fact that the research on offensive language detection has to a large extent been focused on the English language, we set out to explore the design of models that can successfully be used for both English and Danish. \n Question: What is the challenge for other language except English",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-8c5808afb4564221918f5605417215df",
            "input": "We describe our rules for WikiSQL here. We first detect WHERE values, which exactly match to table cells. After that, if a cell appears at more than one column, we choose the column name with more overlapped words with the question, with a constraint that the number of co-occurred words is larger than 1. By default, a WHERE operator is INLINEFORM0 , except for the case that surrounding words of a value contain keywords for INLINEFORM1 and INLINEFORM2 . Then, we deal with the SELECT column, which has the largest number of co-occurred words and cannot be same with any WHERE column. By default, the SELECT AGG is NONE, except for matching to any keywords in Table TABREF8 . The coverage of our rule on training set is 78.4%, with execution accuracy of 77.9%. Our rule for KBQA is simple without using a curated mapping dictionary. The pipeline of rules in SequentialQA is similar to that of WikiSQL. Compared to the grammar of WikiSQL, the grammar of SequentialQA has additional actions including copying the previous-turn logical form, no greater than, no more than, and negation. \n Question: How many rules had to be defined?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-8b067fec11ba40a8933c0d81b085d96f",
            "input": "Three metrics in text simplification are chosen in this paper. BLEU BIBREF5 is one traditional machine translation metric to assess the degree to which translated simplifications differed from reference simplifications. FKGL measures the readability of the output BIBREF23 . A small FKGL represents simpler output. SARI is a recent text-simplification metric by comparing the output against the source and reference simplifications BIBREF20 . \n Question: what evaluation metrics did they use?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-f9e34b93cf7e453589f975d5be16c9b8",
            "input": "This framework has potential applications when comparing different gold standards, considering the design choices for a new gold standard and performing qualitative error analyses for a proposed approach. \n Question: Have they made any attempt to correct MRC gold standards according to their findings? ",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-fb898cdadbb44458839186cb8b32ca64",
            "input": "terms extracted from Yahoo! Answers tend to be more related, in terms of the number of correlated terms, to attributes related to religion or ethnicity compared to terms from Twitter. However, for two particular attributes (i.e., Price and Buddhist), the number of correlated terms from Twitter is higher than the ones from Yahoo! Answers .  \n Question: On Twitter, do the demographic attributes and answers show more correlations than on Yahoo! Answers?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-0bd6f981c8c349b2a0214ec94050859a",
            "input": "A context is upward entailing (shown by [... $\\leavevmode {\\color {red!80!black}\\uparrow }$ ]) that allows an inference from ( \"Introduction\" ) to ( \"Introduction\" ), where French dinner is replaced by a more general concept dinner.  On the other hand, a downward entailing context (shown by [... $\\leavevmode {\\color {blue!80!black}\\downarrow }$ ]) allows an inference from ( \"Introduction\" ) to ( \"Introduction\" ), where workers is replaced by a more specific concept new workers. All [ workers $\\leavevmode {\\color {blue!80!black}\\downarrow }$ ] [joined for a French dinner $\\leavevmode {\\color {red!80!black}\\uparrow }$ ] All workers joined for a dinner All new workers joined for a French dinner Not all [new workers $\\leavevmode {\\color {red!80!black}\\uparrow }$ ] joined for a dinner Not all workers joined for a dinner \n Question: How do they define upward and downward reasoning?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-bc012dc21a894069ac3d8ba9c868c3a1",
            "input": "Stanford - Twitter Sentiment Corpus (STS Corpus): STS Corpus contains 1,600K training tweets collected by a crawler from BIBREF0 . BIBREF0 constructed a test set manually with 177 negative and 182 positive tweets. The Stanford test set is small. However, it has been widely used in different evaluation tasks BIBREF0 BIBREF5 BIBREF13 .\n\nSanders - Twitter Sentiment Corpus: This dataset consists of hand-classified tweets collected by using search terms: INLINEFORM0 , #google, #microsoft and #twitter. We construct the dataset as BIBREF14 for binary classification.\n\nHealth Care Reform (HCR): This dataset was constructed by crawling tweets containing the hashtag #hcr BIBREF15 . Task is to predict positive/negative tweets BIBREF14 . Table IV shows the result of our model for sentiment classification against other models. We compare our model performance with the approaches of BIBREF0 BIBREF5 on STS Corpus.  For Sanders and HCR datasets, we compare results with the model of BIBREF14 that used a ensemble of multiple base classifiers (ENS) such as NB, Random Forest (RF), SVM and Logistic Regression (LR).  \n Question: Are results reported only on English datasets?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-2675d1ae0d2948b5b6754e6ac27d4491",
            "input": "We thus also conducted human studies on Amazon MTurk to evaluate the generated responses with pairwise comparison for dialogue quality. We compare our models with an advanced decoding algorithm MMI BIBREF2 and two models, namely LSTM BIBREF0 and VHRED BIBREF7, both with additive attention. To our best knowledge, LSTM and VHRED were the primary models with which F1's were reported on the Ubuntu dataset. Following BIBREF5 (BIBREF5), we employ two criteria: Plausibility and Content Richness. The first criterion measures whether the response is plausible given the context, while the second gauges whether the response is diverse and informative.  \n Question: How is human evaluation performed, what was the criteria?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-a7ca207021b647039145fc1e861f6464",
            "input": "Especially for Davidson-dataset, some tweets with specific language (written within the African American Vernacular English) and geographic restriction (United States of America) are oversampled such as tweets containing disparage words “nigga\", “faggot\", “coon\", or “queer\", result in high rates of misclassification. However, these misclassifications do not confirm the low performance of our classifier because annotators tended to annotate many samples containing disrespectful words as hate or offensive without any presumption about the social context of tweeters such as the speaker’s identity or dialect, whereas they were just offensive or even neither tweets.  \n Question: What biases does their model capture?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-b39e301e2dee4989a978153124477258",
            "input": "TextCat is the most well-known Perl implementation of the out-of-place method, it lists models for 76 languages in its off-the-shelf configuration; the program is not actively maintained. TextCat is the most well-known Perl implementation of the out-of-place method, it lists models for 76 languages in its off-the-shelf configuration; the program is not actively maintained. TextCat is not the only example of an off-the-shelf implementation of the out-of-place method: other implementations include libtextcat with 76 language models, JTCL with 15 languages, and mguesser with 104 models for different language-encoding pairs. The main issue addressed by later implementations is classification speed: TextCat is implemented in Perl and is not optimized for speed, whereas implementations such as libtextcat and mguesser have been specifically written to be fast and efficient. whatlang-rs uses an algorithm based on character trigrams and refers the user to the BIBREF7 article. It comes pre-trained with 83 languages.\n\nis the language identifier embedded in the Google Chrome web browser. It uses a NB classifier, and script-specific classification strategies. assumes that all the input is in UTF-8, and assigns the responsibility of encoding detection and transcoding to the user. uses Unicode information to determine the script of the input. also implements a number of pre-processing heuristics to help boost performance on its target domain (web pages), such as stripping character sequences like .jpg. The standard implementation supports 83 languages, and an extended model is also available, that supports 160 languages.\n\nis a Java library that implements a language identifier based on a NB classifier trained over character . The software comes with pre-trained models for 53 languages, using data from Wikipedia. makes use of a range of normalization heuristics to improve the performance on particular languages, including: (1) clustering of Chinese/Japanese/Korean characters to reduce sparseness; (2) removal of “language-independent” characters, and other text normalization; and (3) normalization of Arabic characters.\n\nis a Python implementation of the method described by BIBREF150 , which exploits training data for the same language across multiple different sources of text to identify sequences of characters that are strongly predictive of a given language, regardless of the source of the text. This feature set is combined with a NB classifier, and is distributed with a pre-trained model for 97 languages prepared using data from 5 different text sources. BIBREF151 provide an empirical comparison of to , and and find that it compares favorably both in terms of accuracy and classification speed. There are also implementations of the classifier component (but not the training portion) of in Java, C, and JavaScript.\n\nBIBREF153 uses a vector-space model with per-feature weighting on character sequences. One particular feature of is that it uses discriminative training in selecting features, i.e. it specifically makes use of features that are strong evidence against a particular language, which is generally not captured by NB models. Another feature of is that it uses inter-string smoothing to exploit sentence-level locality in making language predictions, under the assumption that adjacent sentences are likely to be in the same language. BIBREF153 reports that this substantially improves the accuracy of the identifier. Another distinguishing feature of is that it comes pre-trained with data for 1400 languages, which is the highest number by a large margin of any off-the-shelf system.\n\nwhatthelang is a recent language identifier written in Python, which utilizes the FastText NN-based text classification algorithm. It supports 176 languages.\n\nimplements an off-the-shelf classifier trained using Wikipedia data, covering 122 languages. Although not described as such, the actual classification algorithm used is a linear model, and is thus closely related to both NB and a cosine-based vector space model.\n\nIn addition to the above-mentioned general-purpose language identifiers, there have also been efforts to produce pre-trained language identifiers targeted specifically at Twitter messages. is a Twitter-specific tool with built-in models for 19 languages. It uses a document representation based on tries BIBREF401 . The algorithm is a LR classifier using all possible substrings of the data, which is important to maximize the available information from the relatively short Twitter messages.\n\nBIBREF152 provides a comparison of 8 off-the-shelf language identifiers applied without re-training to Twitter messages. One issue they report is that comparing the accuracy of off-the-shelf systems is difficult because of the different subset of languages supported by each system, which may also not fully cover the languages present in the target data. The authors choose to compare accuracy over the full set of languages, arguing that this best reflects the likely use-case of applying an off-the-shelf system to new data. They find that the best individual systems are , and , but that slightly higher accuracy can be attained by a simple voting-based ensemble classifier involving these three systems.\n\nIn addition to this, commercial or other closed-source language identifiers and language identifier services exist, of which we name a few. The Polyglot 3000 and Lextek Language Identifier are standalone language identifiers for Windows. Open Xerox Language Identifier is a web service with available REST and SOAP APIs. \n Question: what are the off-the-shelf systems discussed in the paper?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-6fc6a9924eaa438dbe483b9c859bea6d",
            "input": "We focus here on the Europarl domain, for which we have ample data in several languages, and use as in-domain training data the Europarl corpus BIBREF5 for two translation directions: English INLINEFORM0 German and English INLINEFORM1 French. \n Question: what language pairs are explored?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-f0aa90c85c6d4cb488664ce59ab79ed3",
            "input": "Our idea is the BQ generation for MQ and, at the same time, we only want the minimum number of BQ to represent the MQ, so modeling our problem as $LASSO$ optimization problem is an appropriate way \n Question: What they formulate the question generation as?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-9db4b1cf479f4e1f82570ac673417da3",
            "input": "Considering the improvements over the majority baseline achieved by the RNN model for both non-English (on the average 22.76% relative improvement; 15.82% relative improvement on Spanish, 72.71% vs. 84.21%, 30.53% relative improvement on Turkish, 56.97% vs. 74.36%, 37.13% relative improvement on Dutch, 59.63% vs. 81.77%, and 7.55% relative improvement on Russian, 79.60% vs. 85.62%) and English test sets (27.34% relative improvement), we can draw the conclusion that our model is robust to handle multiple languages. \n Question: which non-english language had the best performance?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-bb553812cfb64400ad284ce1fc652c5f",
            "input": "The PolyResponse restaurant search is currently available in 8 languages and for 8 cities around the world: English (Edinburgh), German (Berlin), Spanish (Madrid), Mandarin (Taipei), Polish (Warsaw), Russian (Moscow), Korean (Seoul), and Serbian (Belgrade). \n Question: In what 8 languages is PolyResponse engine used for restourant search and booking system?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-51f01c252dfa4f29ba7a85a1ea1b2499",
            "input": "In this work, we introduce a methodology that provides VQA algorithms with the ability to generate human interpretable attention maps which effectively ground the answer to the relevant image regions. We accomplish this by leveraging region descriptions and object annotations available in the Visual Genome dataset, and using these to automatically construct attention maps that can be used for attention supervision, instead of requiring human annotators to manually provide grounding labels. \n Question: How do they obtain region descriptions and object annotations?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-9e301397e8e244fc8d228da7c6350411",
            "input": "The experiment dataset comes from Microsoft Research (MSR) . It contains three domains: movie, taxi, and restaurant. The total count of dialogues per domain and train/valid/test split is reported in Table TABREF11. At every turn both user and agent acts are annotated, we use only the agent side as targets in our experiment. The acts are ordered in the dataset (each output sentence aligns with one act). \n Question: What datasets are used for training/testing models? ",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-938b3087461b45b5819df49800786518",
            "input": "In this paper, we use three encoders (NBOW, LSTM and attentive LSTM) to model the text descriptions. \n Question: What neural models are used to encode the text?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-0b4fbd595ac14219896c119704dbacca",
            "input": "We compare our best F1 score with other state-of-the-art approaches in table TABREF39 , which shows our model has competitive advantage in dealing with drug-drug interaction extraction. \n Question: What is the performance of their model?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-8e8c8964a66143b4a59f0bfd5e94f376",
            "input": "To further study the information encoded in the discourse embeddings, we perform t-SNE clustering BIBREF20 on them, using the best performing model CNN2-DE (global). \n Question: How are discourse embeddings analyzed?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-e0d4ab6a1b4948bfa2081dd43533178f",
            "input": "Table TABREF14 show the results obtained for proposed s2sL approach in comparison to that of MLP for the tasks of Speech/Music and Neutral/Sad classification, by considering different proportions of training data. \n Question: Which models/frameworks do they compare to?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-8310540f3e7b455a9ad81eb1285943db",
            "input": "In this paper, we present an extraction-then-synthesis framework for machine reading comprehension shown in Figure 1 , in which the answer is synthesized from the extraction results. We build an evidence extraction model to predict the most important sub-spans from the passages as evidence, and then develop an answer synthesis model which takes the evidence as additional features along with the question and passage to further elaborate the final answers. \n Question: What two components are included in their proposed framework?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-faa053a2645141e6b8dc1d0e9dfe3a0e",
            "input": "Task: given a caption and a paired image (if used), the goal is to label every token in a caption in BIO scheme (B: beginning, I: inside, O: outside) BIBREF27 .  \n Question: Can named entities in SnapCaptions be discontigious?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-5da2c2bff434417d8def87737aad12d7",
            "input": "Noticeably, despite being both based on and integrated into a BERT$_\\text{base}$ model, our architecture even outperforms a standalone BERT$_\\text{large}$ model by a large margin. \n Question: What models other than standalone BERT is new model compared to?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-f51a2ce9205b4a3190b1f0e36db1956c",
            "input": "Figure FIGREF10 illustrates the UTCNN model. As more than one user may interact with a given post, we first add a maximum pooling layer after the user matrix embedding layer and user vector embedding layer to form a moderator matrix embedding INLINEFORM0 and a moderator vector embedding INLINEFORM1 for moderator INLINEFORM2 respectively, where INLINEFORM3 is used for the semantic transformation in the document composition process, as mentioned in the previous section. The term moderator here is to denote the pseudo user who provides the overall semantic/sentiment of all the engaged users for one document. The embedding INLINEFORM4 models the moderator stance preference, that is, the pattern of the revealed user stance: whether a user is willing to show his preference, whether a user likes to show impartiality with neutral statements and reasonable arguments, or just wants to show strong support for one stance. Ideally, the latent user stance is modeled by INLINEFORM5 for each user. Likewise, for topic information, a maximum pooling layer is added after the topic matrix embedding layer and topic vector embedding layer to form a joint topic matrix embedding INLINEFORM6 and a joint topic vector embedding INLINEFORM7 for topic INLINEFORM8 respectively, where INLINEFORM9 models the semantic transformation of topic INLINEFORM10 as in users and INLINEFORM11 models the topic stance tendency. The latent topic stance is also modeled by INLINEFORM12 for each topic.\n\nAs for comments, we view them as short documents with authors only but without likers nor their own comments. Therefore we apply document composition on comments although here users are commenters (users who comment). It is noticed that the word embeddings INLINEFORM0 for the same word in the posts and comments are the same, but after being transformed to INLINEFORM1 in the document composition process shown in Figure FIGREF4 , they might become different because of their different engaged users. The output comment representation together with the commenter vector embedding INLINEFORM2 and topic vector embedding INLINEFORM3 are concatenated and a maximum pooling layer is added to select the most important feature for comments. Instead of requiring that the comment stance agree with the post, UTCNN simply extracts the most important features of the comment contents; they could be helpful, whether they show obvious agreement or disagreement. Therefore when combining comment information here, the maximum pooling layer is more appropriate than other pooling or merging layers. Indeed, we believe this is one reason for UTCNN's performance gains.\n\nFinally, the pooled comment representation, together with user vector embedding INLINEFORM0 , topic vector embedding INLINEFORM1 , and document representation are fed to a fully connected network, and softmax is applied to yield the final stance label prediction for the post. \n Question: How many layers does the UTCNN model have?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-959b3402e444456783c474639deb94ef",
            "input": "Finally, we combine our CNN and RNN models using a voting process. For each sentence in the test set, we apply several CNN and RNN models presented in Tables TABREF12 and TABREF14 and predict the class with the most votes. In case of a tie, we pick one of the most frequent classes randomly. \n Question: How does their simple voting scheme work?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-08e32399bef1479ea11843d1596759f9",
            "input": "We use the BLEU BIBREF30 metric on the validation set for the VQG model training. BLEU is a measure of similitude between generated and target sequences of words, widely used in natural language processing. It assumes that valid generated responses have significant word overlap with the ground truth responses. We use it because in this case we have five different references for each of the generated questions. We obtain a BLEU score of 2.07.\n\nOur chatbot model instead, only have one reference ground truth in training when generating a sequence of words. We considered that it was not a good metric to apply as in some occasions responses have the same meaning, but do not share any words in common. Thus, we save several models with different hyperparameters and at different number of training iterations and compare them using human evaluation, to chose the model that performs better in a conversation. \n Question: How is performance of this system measured?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-7dcd7086cea94ac689b0e7a1eab7337e",
            "input": "Since our methodology does not use a seed list of offensive words, it is not biased by topic, target, or dialect. Using our methodology, we tagged 10,000 Arabic tweet dataset for offensiveness, where offensive tweets account for roughly 19% of the tweets. Further, we labeled tweets as vulgar or hate speech. To date, this is the largest available dataset, which we plan to make publicly available along with annotation guidelines. We use this dataset to characterize Arabic offensive language to ascertain the topics, dialects, and users' gender that are most associated with the use of offensive language.  \n Question: In what way is the offensive dataset not biased by topic, dialect or target?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-e4ce4c1d87504aa1ad62e39822d33ee5",
            "input": "Metrics. We use tolerance accuracy BIBREF16, which measures how far away the predicted span is from the gold standard span, as a metric. The rationale behind the metric is that, in practice, it suffices to recommend a rough span which contains the answer - a difference of a few seconds would not matter much to the user. Metrics. We used accuracy and MRR (Mean Reciprocal Ranking) as metrics.  Metrics. To evaluate our pipeline approach we use overall accuracy after filtering and accuracy given that the segment is in the top 10 videos.  \n Question: What evaluation metrics were used in the experiment?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-5cbded8d7b624ec2ad7601744a7433dd",
            "input": "The accuracy of our model is 7.8% higher than the best result achieved by LSVM. The results show that this model can perform better than state-of-the-art baselines including hybrid CNN BIBREF15 and LSTM with attention BIBREF16 by 3.1% on the validation set and 1% on the test set. \n Question: What are state of the art methods authors compare their work with? ",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-cad6231a940c420388e03ec9f6a3f2b5",
            "input": "We compare the performance of translation approaches based on four metrics:\n\n[align=left,leftmargin=0em,labelsep=0.4em,font=]\n\nAs in BIBREF20 , EM is 1 if a predicted plan matches exactly the ground truth; otherwise it is 0.\n\nThe harmonic average of the precision and recall over all the test set BIBREF26 .\n\nThe minimum number of insertions, deletions or swap operations required to transform a predicted sequence of behaviors into the ground truth sequence BIBREF27 .\n\nGM is 1 if a predicted plan reaches the ground truth destination (even if the full sequence of behaviors does not match exactly the ground truth). Otherwise, GM is 0. \n Question: What evaluation metrics are used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-94dfe6b500cd41519f6afb53d75b94e1",
            "input": "the basic model yields good performance for recognizing explicit discourse relations as well, which is comparable with previous best result (92.05% macro F1-score and 93.09% accuracy as reported in BIBREF11 ). After untying parameters in the softmax prediction layer, implicit discourse relation classification performance was improved across all four relations, meanwhile, the explicit discourse relation classification performance was also improved. Then we also created ensemble models by applying majority voting to combine results of ten runs. From table 5 , each ensemble model obtains performance improvements compared with single model. The full model achieves performance boosting of (51.84 - 48.82 = 3.02) and (94.17 - 93.21 = 0.96) in macro F1-scores for predicting implicit and explicit discourse relations respectively.  \n Question: What discourse relations does it work best/worst for?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-7ca977ae31f949f28211a67c03ee2285",
            "input": " On high level, we indirectly evaluate the content relevance between the candidate summary and the human summary using information retrieval. To accomplish this, we use the summaries as search queries and compare the overlaps of the retrieved results.  \n Question: In the proposed metric, how is content relevance measured?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-ef17ced6b52d4fc6907063ef3b7ddd20",
            "input": "The first challenge is to acquire very large Vietnamese corpus and to use them in building a classifier, which could further improve accuracy.  The second challenge is design and development of big data warehouse and analytic framework for Vietnamese documents, which corresponds to the rapid and continuous growth of gigantic volume of articles and/or documents from Web 2.0 applications, such as, Facebook, Twitter, and so on. The final challenge relates to building a system, which is able to incrementally learn new corpora and interactively process feedback. \n Question: Why challenges does word segmentation in Vietnamese pose?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-acae45c7db5f4b93aecbb4cfcead90c5",
            "input": "The seq2seq model with global attention gives the best results with an average target BLEU score of 29.65 on the style transfer dataset, compared with an average target BLEU score of 26.97 using the seq2seq model with pointer networks. \n Question: What is best BLEU score of language style transfer authors got?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-6372b9be25f8407eb56f70e7c580fb3a",
            "input": "In this section, we introduce an invertibility condition for our neural projector to tackle the optimization challenge. Specifically, we constrain our neural projector with two requirements: (1) INLINEFORM0 and (2) INLINEFORM1 exists.  \n Question: What is the invertibility condition?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-5a278c666bd84505842b56b79d50e5a2",
            "input": "Therefore, we just implement a simple one-layer RNN with LSTM cell to classify pre-processed sentences into ironic sentences and non-ironic sentences because LSTM networks are widely used in irony detection. \n Question: How did the authors find ironic data on twitter?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-c010ee54902742b78f423488ab89b606",
            "input": "We use three creative English datasets with distinct linguistic characteristics: (1) A corpus of 740 classical and contemporary English poems, (2) a corpus of 14950 metaphor sentences retrieved from a metaphor database website and (3) a corpus of 1500 song lyrics ranging across genres.  We first pre-train our generator on the Gutenberg dataset BIBREF24 for 20 epochs and then fine-tune BIBREF19 them to our target datasets with a language modeling objective. \n Question: Which datasets are used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-05e97c8f67e04f24a7c4e2ac27767157",
            "input": "Cloze Track User Query Track \n Question: What two types the Chinese reading comprehension dataset consists of?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-d1b661bc3fef474b920ab5074864e996",
            "input": "The baseline system for the SLC task is a very simple logistic regression classifier with default parameters, where we represent the input instances with a single feature: the length of the sentence.  The baseline for the FLC task generates spans and selects one of the 18 techniques randomly.  \n Question: What was the baseline for this task?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-801aabeb53ac465bae28f7c0fa856e99",
            "input": "The task presented here was easy and simple to analyze, however, future work may be done on natural language tasks. If these properties hold it might indicate that a new evaluation paradigm for NLP should be pushed; one that emphasizes performance on uncharacteristic (but structurally sound) inputs in addition to the data typically seen in training. \n Question: Can the findings of this paper be generalized to a general-purpose task?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-c8fa21114a38405d81794239228af93f",
            "input": "We create three additional training datasets by adding sentences involving object RCs to the original Wikipedia corpus (Section lm). To this end, we randomly pick up 30 million sentences from Wikipedia (not overlapped to any sentences in the original corpus), parse by the same parser, and filter sentences containing an object RC, amounting to 680,000 sentences.  \n Question: How do they perform data augmentation?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-3451406570a64a0387e4572c9db1f293",
            "input": "Input of the model is the concatenation of word embedding and another embedding indicating whether this word is predicate: $ \\mathbf {x}_t = [\\mathbf {W}_{\\text{emb}}(w_t), \\mathbf {W}_{\\text{mask}}(w_t = v)]. $ \n Question: What's the input representation of OpenIE tuples into the model?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-5fd6fa9b5ae447e4b50449015450c9a6",
            "input": "n the OLID dataset, we use a hierarchical annotation model split into three levels to distinguish between whether language is offensive or not (A), and type (B) and target (C) of the offensive language. Each level is described in more detail in the following subsections and examples are shown in Table TABREF10 . Level A: Offensive language Detection\nLevel A discriminates between offensive (OFF) and non-offensive (NOT) tweets. Level B: Categorization of Offensive Language\nLevel B categorizes the type of offense and two labels are used: targeted (TIN) and untargeted (INT) insults and threats. Level C: Offensive Language Target Identification\nLevel C categorizes the targets of insults and threats as individual (IND), group (GRP), and other (OTH). \n Question: What are the three layers of the annotation scheme?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-03f7380dd1624c23b73d299341368a0d",
            "input": "Car-speak is abstract language that pertains to a car's physical attribute(s). In this instance the physical attributes that the term “fast” pertains to could be the horsepower, or it could be the car's form factor (how the car looks). However, we do not know exactly which attributes the term “fast” refers to. \n Question: How does car speak pertains to a car's physical attributes?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-5875ebe6351340979ad5f1bbce211553",
            "input": "We will compare to the more recent cross-lingual language model XLM BIBREF12, as well as the state-of-the-art CoNLL 2018 shared task results with predicted tokenisation and segmentation in an updated version of the paper. In French, no extensive work has been done due to the limited availability of NER corpora. We compare our model with the strong baselines settled by BIBREF49, who trained both CRF and BiLSTM-CRF architectures on the FTB and enhanced them using heuristics and pretrained word embeddings. In the TRANSLATE-TRAIN setting, we report the best scores from previous literature along with ours. BiLSTM-max is the best model in the original XNLI paper, mBERT which has been reported in French in BIBREF52 and XLM (MLM+TLM) is the best-presented model from BIBREF50. \n Question: What is the state of the art?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-75296877879b439784ea5ca264352881",
            "input": "We defined the reward as being 1 for successfully completing the task, and 0 otherwise. A discount of $0.95$ was used to incentivize the system to complete dialogs faster rather than slower, yielding return 0 for failed dialogs, and $G = 0.95^{T-1}$ for successful dialogs, where $T$ is the number of system turns in the dialog.   0.95^{T-1} reward  0.95^{T-1}  \n Question: What is the reward model for the reinforcement learning appraoch?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-1646d9b313e34f1894eb4bd62715c886",
            "input": "Our model introduces a multi-turns inference mechanism to process multi-perspective matching features.  \n Question: Which matching features do they employ?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-12c23819b491481793d2713c12ea1492",
            "input": "We compare FSDM with four baseline methods and two ablations.\n\nNDM BIBREF7 proposes a modular end-to-end trainable network. It applies de-lexicalization on user utterances and responses.\n\nLIDM BIBREF9 improves over NDM by employing a discrete latent variable to learn underlying dialogue acts. This allows the system to be refined by reinforcement learning.\n\nKVRN BIBREF13 adopts a copy-augmented Seq2Seq model for agent response generation and uses an attention mechanism on the KB. It does not perform belief state tracking.\n\nTSCP/RL BIBREF10 is a two-stage CopyNet which consists of one encoder and two copy-mechanism-augmented decoders for belief state and response generation. \n Question: What baselines have been used in this work?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-cefc022d20a54e8b89341d2cff11a64b",
            "input": "As illustrated in Figure FIGREF1, our key idea is that we can exploit discourse relations BIBREF4 to efficiently propagate polarity from seed predicates that directly report one's emotions (e.g., “to be glad” is positive). Suppose that events $x_1$ are $x_2$ are in the discourse relation of Cause (i.e., $x_1$ causes $x_2$). If the seed lexicon suggests $x_2$ is positive, $x_1$ is also likely to be positive because it triggers the positive emotion. The fact that $x_2$ is known to be negative indicates the negative polarity of $x_1$. Similarly, if $x_1$ and $x_2$ are in the discourse relation of Concession (i.e., $x_2$ in spite of $x_1$), the reverse of $x_2$'s polarity can be propagated to $x_1$. Even if $x_2$'s polarity is not known in advance, we can exploit the tendency of $x_1$ and $x_2$ to be of the same polarity (for Cause) or of the reverse polarity (for Concession) although the heuristic is not exempt from counterexamples. We transform this idea into objective functions and train neural network models that predict the polarity of a given event. \n Question: How are relations used to propagate polarity?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-fe9352df76424a0293424fb13bc78029",
            "input": "Questions: We make use of the 7,787 science exam questions of the Aristo Reasoning Challenge (ARC) corpus BIBREF31 , which contains standardized 3rd to 9th grade science questions from 12 US states from the past decade.  \n Question: How was the dataset collected?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-6b312c750fb84e7088df95615e7b0d81",
            "input": "Also, note that the performance for this task is not expected to achieve a perfect accuracy, as there may be situations where more than one action is reasonable, and also because writers tell a story playing with elements such as surprise or uncertainty. \n Question: Why do they think this task is hard?  What is the baseline performance?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-0a08ddfc42834d8683aae5b29902215c",
            "input": "Crowdsourced annotators assigned similarity to word pairs during the word similarity task.  \n Question: did they use a crowdsourcing platform for annotations?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-d7426cf3a5344d81ab0414fe603aa919",
            "input": "Results reveal that, contrary to the common belief, correlations between Rouge and the Pyramid scores are weak, which challenges its effectiveness for scientific summarization. Furthermore, we show a large variance of correlations between different Rouge variants and the manual evaluations which further makes the reliability of Rouge for evaluating scientific summaries less clear. \n Question: What is the common belief that this paper refutes? (c.f. 'contrary to the common belief, ROUGE is not much [sic] reliable'",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-16aba0c7cb9f4c7f8a268be5d3cf10fd",
            "input": "DrQA is a CRC baseline coming with the CoQA dataset. Note that this implementation of DrQA is different from DrQA for SQuAD BIBREF8 in that it is modified to support answering no answer questions by having a special token unknown at the end of the document.  DrQA+CoQA is the above baseline pre-tuned on CoQA dataset and then fine-tuned on $(\\text{RC})_2$ .  BERT is the vanilla BERT model directly fine-tuned on $(\\text{RC})_2$ . We use this baseline for ablation study on the effectiveness of pre-tuning. BERT+review first tunes BERT on domain reviews using the same objectives as BERT pre-training and then fine-tunes on $(\\text{RC})_2$ . We use this baseline to show that a simple domain-adaptation of BERT is not good. BERT+CoQA first fine-tunes BERT on the supervised CoQA data and then fine-tunes on $(\\text{RC})_2$ . We use this baseline to show that pre-tuning is very competitive even compared with models trained from large-scale supervised data. \n Question: What is the baseline model used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-681216a44cf84c7cb7918aab606ffae6",
            "input": "The corpus of supervisor assessment has 26972 sentences.  \n Question: What is the size of the real-life dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-d0fdba1011bf4a19971ed5d044efea6f",
            "input": "(proposed) Bi-LSTM/CRF + Bi-CharLSTM with modality attention (W+C): uses the modality attention to merge word and character embeddings.\n\n(proposed) Bi-LSTM/CRF + Bi-CharLSTM + Inception (W+C+V): takes as input visual contexts extracted from InceptionNet as well, concatenated with word and char vectors.\n\n(proposed) Bi-LSTM/CRF + Bi-CharLSTM + Inception with modality attention (W+C+V): uses the modality attention to merge word, character, and visual embeddings as input to entity LSTM. \n Question: Does their NER model learn NER from both text and images?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-f1f8218f22f04e25be2edb2a26e53205",
            "input": "Traditional text-to-speech (TTS) systems are composed of complex pipelines BIBREF0 , these often include accoustic frontends, duration model, acoustic prediction model and vocoder models. The architecture of our model utilizes RNN-based Seq2Seq model for generating mel spectrogram from text. The architecture is similar to that of Tacotron 2 BIBREF4  Direct comparison of model parameters between ours and the open-source tacotron 2, our model contains 4.5 million parameters, whereas the Tacotron 2 contains around 13 million parameters with default setting. By helping our model learn attention alignment faster, we can afford to use a smaller overall model to achieve similar quality speech quality. \n Question: Do they reduce the number of parameters in their architecture compared to other direct text-to-speech models?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-c060a5eb5d8b478884512f74fd5bfe4c",
            "input": "We build the iSQuAD and iNewsQA datasets based on SQuAD v1.1 BIBREF0 and NewsQA BIBREF1. iMRC: Making MRC Interactive ::: Evaluation Metric\nSince iMRC involves both MRC and RL, we adopt evaluation metrics from both settings. First, as a question answering task, we use $\\text{F}_1$ score to compare predicted answers against ground-truth, as in previous works. When there exist multiple ground-truth answers, we report the max $\\text{F}_1$ score. Second, mastering multiple games remains quite challenging for RL agents. Therefore, we evaluate an agent's performance during both its training and testing phases. During training, we report training curves averaged over 3 random seeds. During test, we follow common practice in supervised learning tasks where we report the agent's test performance corresponding to its best validation performance . \n Question: What are the models evaluated on?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-1fdbe1ebc94d4b74ab9265a0f266d01e",
            "input": "The annotation projection method, which we follow in this work, is one way to address this problem. It was introduced for POS tagging, base noun phrase bracketing, NER tagging, and inflectional morphological analysis BIBREF29 but it has also been used for dependency parsing BIBREF30 , role labeling BIBREF31 , BIBREF32 and semantic parsing BIBREF26 . \n Question: Do the authors test their annotation projection techniques on tasks other than AMR?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-dcb0f6b0dbe0452ca45cdadd992f4a87",
            "input": "The encoder is a bidirectional Long-Short Term Memory(LSTM) cell BIBREF14 and the decoder a single LSTM cell with attention mechanism.  \n Question: How many attention layers are there in their model?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-91cf02e10bbe4b8a9506e40799d8ea2e",
            "input": "The data for this project are two parts, the first part is the historical S&P 500 component stocks, which are downloaded from the Yahoo Finance. We use the data over the period of from 12/07/2017 to 06/01/2018. The second part is the news article from financial domain are collected with the same time period as stock data. Hence, only news article from financial domain are collected. The data is mainly taken from Webhose archived data, which consists of 306242 news articles present in JSON format, dating from December 2017 up to end of June 2018. \n Question: What is the dataset used in the paper?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-4f984c07ef2749e3aa24e475f675d2ec",
            "input": "Our study focused primarily on English tweets, since this was the language of our diagnostic training sample. \n Question: Do the authors report results only on English datasets?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-dcc34aeecd2d42b2a617e9ad0a8bd8d7",
            "input": "We evaluate with all commonly-used metrics in question generation BIBREF13: BLEU-1 (B1), BLEU-2 (B2), BLEU-3 (B3), BLEU-4 (B4) BIBREF17, METEOR (MET) BIBREF18 and ROUGE-L (R-L) BIBREF19. We use the evaluation script released by Chen2015MicrosoftCC. \n Question: What metrics do they use?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-6e5494e023294c09b775a28daa695970",
            "input": "For example, the original sentence `We went shop on Saturday' and the corrected version `We went shopping on Saturday' would produce the following pattern:\n\n(VVD shop_VV0 II, VVD shopping_VVG II)\n\nAfter collecting statistics from the background corpus, errors can be inserted into error-free text. The learned patterns are now reversed, looking for the correct side of the tuple in the input sentence. We only use patterns with frequency INLINEFORM0 , which yields a total of 35,625 patterns from our training data.  \n Question: What textual patterns are extracted?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-2aed0842a5914f2a9057d65a6b5f68de",
            "input": "For each candidate mention span, we retrieve the top 10 entities according to the Freebase API. We then create a lattice in which the nodes correspond to mention-entity pairs, scored by their Freebase API scores, and the edges encode the fact that no joint assignment of entities to mentions can contain overlapping spans. We take the top 10 paths through the lattice as possible entity disambiguations. For each possibility, we generate $n$ -best paraphrases that contains the entity mention spans. In the end, this process creates a total of $10n$ paraphrases.  \n Question: How many paraphrases are generated per question?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-2affc26573fe43928a76c08b9dc86a14",
            "input": "CoVoST is currently built on that release and covers the following 11 languages: French, German, Dutch, Russian, Spanish, Italian, Turkish, Persian, Swedish, Mongolian and Chinese. \n Question: Is Arabic one of the 11 languages in CoVost?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-6339b57df75f4241a3bd28500c4da71f",
            "input": "We first introduce a method that detects bottlenecks in text-games using the overall reward gained and the knowledge graph state. This method freezes the policy used to reach the bottleneck and restarts the training from there on out, additionally conducting a backtracking search to ensure that a sub-optimal policy has not been frozen. The second contribution explore how to leverage knowledge graphs to improve existing exploration algorithms for dealing with combinatorial action-spaces such as Go-Explore BIBREF9.  \n Question: What are the two new strategies?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-bd72bfc9264946389eebefd00bd38872",
            "input": "In Table TABREF1, we summarize the quantitative results of the above previous studies. In Table TABREF1, we summarize the quantitative results of the above previous studies. \n Question: What is the accuracy reported by state-of-the-art methods?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-e7b64d4cb24a4538928d0b98e3a0a436",
            "input": "We evaluated the phrase compositionality models on the adjective-noun and noun-noun phrase similarity tasks compiled by Mitchell2010, using the same evaluation scheme as in the original work. Spearman's INLINEFORM0 between phrasal similarities derived from our compositional functions and the human annotators (computed individually per annotator and then averaged across all annotators) was the evaluation measure. \n Question: How do they score phrasal compositionality?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-872d7f01057c4bd09eda60775347ed8c",
            "input": "We use Waseem and Hovy BIBREF5, Davidson et al. BIBREF9, and Waseem et al. BIBREF10 as baselines and compare the results with our different fine-tuning strategies using pre-trained BERTbase model.  \n Question: What existing approaches do they compare to?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-e9d9031ad2c94b28872002310ba5172b",
            "input": "We then proceed to connect these mentions i) if they co-occur within the same document (we will refer to this as DOC-BASED edges), ii) if the pair of named entity mentions is identical (MATCH edges—these may connect nodes across and within documents), or iii) if they are in the same coreference chain, as predicted by the external coreference system (COREF edges).  \n Question: How did they get relations between mentions?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-2c1ff4cf0fe94eb48d3557368d4c92fe",
            "input": "Three incremental levels of document preprocessing are experimented with: raw text, text cleaning through document logical structure detection, and removal of keyphrase sparse sections of the document. In doing so, we present the first consistent comparison of different keyphrase extraction models and study their robustness over noisy text. \n Question: what levels of document preprocessing are looked at?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-8a95a51338bf4551b203594ed69a64a5",
            "input": "Our model contains five independent decoders, one for each image in the sequence. This allows each decoder to learn a specific language model for each position of the sequence. \n Question: Do the decoder LSTMs all have the same weights?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-d399e513904d43faa2d052547bbdd264",
            "input": "We employ a huge dataset of 45,000 self-reported depressed users introduced in BIBREF46 where a lexicon of depression symptoms consisting of 1500 depression-indicative terms was created with the help of psychologist clinician and employed for collecting self-declared depressed individual's profiles. A subset of 8,770 users (24 million time-stamped tweets) containing 3981 depressed and 4789 control users (that do not show any depressive behavior) were verified by two human judges BIBREF46 . \n Question: How is the data annotated?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-cb9c1246c645494fa2e9d07516560e00",
            "input": "We performed experiments on synthetically generated dataset since it gives us a better control over the distribution of the data. Specifically we compared the gains obtained using our approach versus the variance of the distribution. We created dataset from the following generative process. [H] Generative Process [1] Generate data\n\nPick k points INLINEFORM0 as domain -1 means and a corresponding set of k points INLINEFORM1 as domain-2 means, and covariance matrices INLINEFORM2\n\niter INLINEFORM0 upto num INLINEFORM1 samples Sample class INLINEFORM2 Sample INLINEFORM3 Sample INLINEFORM4 Add q and a so sampled to the list of q,a pairs We generated the dataset from the above sampling process with means selected on a 2 dimensional grid of size INLINEFORM5 with variance set as INLINEFORM6 in each dimension.10000 sample points were generated. The parameter INLINEFORM7 of the above algorithm was set to 0.5 and k was set to 9 (since the points could be generated from one of the 9 gaussians with centroids on a INLINEFORM8 grid). \n Question: How do they generate the synthetic dataset?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-fd82ba723788457991b32fdb4f2b146f",
            "input": "The input for both tasks consists of news articles in free-text format, collected from 36 propagandist and 12 non-propagandist news outlets and then annotated by professional annotators.  \n Question: What dataset was used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-b30bdaaf2f2446beac46018a502f5a95",
            "input": "Later, BIBREF8 introduced an RNN with an external stack memory to learn simple context-free languages, such as $a^n b^m$ , $a^nb^ncb^ma^m$ , and $a^{n+m} b^n c^m$ . Similar studies BIBREF15 , BIBREF16 , BIBREF17 , BIBREF10 , BIBREF11 have explored the existence of stable counting mechanisms in simple RNNs, which would enable them to learn various context-free and context-sensitive languages BIBREF9 , on the other hand, proposed a variant of Long Short-Term Memory (LSTM) networks to learn two context-free languages, $a^n b^n$ , $a^n b^m B^m A^n$ , and one strictly context-sensitive language, $a^n b^n c^n$ . \n Question: How do they get the formal languages?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-0f5e87eb779e4704b0b9e27d64fd2726",
            "input": "To investigate whether our BERT-based model can transfer knowledge beyond language, we consider image features as simple visual tokens that can be presented to the model analogously to textual token embeddings. In order to make the $o_j$ vectors (of dimension $2048+4=2052$) comparable to BERT embeddings (of dimension 768), we use a simple linear cross-modal projection layer $W$ of dimensions $2052\\hspace{-1.00006pt}\\times \\hspace{-1.00006pt}768$. The $N$ object regions detected in an image, are thus represented as $X_{img} = (W.o_1,...,W.o_N)$. Once mapped into the BERT embedding space with $W$, the image is seen by the rest of the model as a sequence of units with no explicit indication if it is a text or an image embedding. \n Question: How are multimodal representations combined?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-049e2df6a9154726af9bacb4fe9d5dc7",
            "input": "Our ASR and ST models follow the architecture in berard2018end, but have 3 decoder layers like that in pino2019harnessing. \n Question: What is the architecture of their model?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-f91141a6c9a2402791bbb2c9f590074c",
            "input": "Validated transcripts were sent to professional translators. In order to control the quality of the professional translations, we applied various sanity checks to the translations BIBREF11. We also sanity check the overlaps of train, development and test sets in terms of transcripts and voice clips (via MD5 file hashing), and confirm they are totally disjoint. \n Question: How is the quality of the data empirically evaluated? ",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-d9b9dfc7bc814869b4fd83b6d4c91204",
            "input": "The final annotation categories for the dataset are: Joy, Sadness, Anger, Fear, Anticipation, Surprise, Love, Disgust, Neutral \n Question: How many emotions do they look at?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-0f7862ea9b3b49f990de356a2d0a89be",
            "input": "Moreover, QA and QG have probabilistic correlation as both tasks relate to the joint probability between $q$ and $a$ . Given a question-answer pair $\\langle q, a \\rangle $ , the joint probability $P(q, a)$ can be computed in two equivalent ways.\r\n\r\n$$P(q, a) = P(a) P(q|a) = P(q)P(a|q)$$ (Eq. 1)\r\n\r\nThe conditional distribution $P(q|a)$ is exactly the QG model, and the conditional distribution $P(a|q)$ is closely related to the QA model. Existing studies typically learn the QA model and the QG model separately by minimizing their own loss functions, while ignoring the probabilistic correlation between them.\r\n\r\nBased on these considerations, we introduce a training framework that exploits the duality of QA and QG to improve both tasks. There might be different ways of exploiting the duality of QA and QG. In this work, we leverage the probabilistic correlation between QA and QG as the regularization term to influence the training process of both tasks. Specifically, the training objective of our framework is to jointly learn the QA model parameterized by $\\theta _{qa}$ and the QG model parameterized by $\\theta _{qg}$ by minimizing their loss functions subject to the following constraint.\r\n\r\n$$P_a(a) P(q|a;\\theta _{qg}) = P_q(q)P(a|q;\\theta _{qa})$$ (Eq. 3)\r\n\r\n$P_a(a)$ and $P_q(q)$ are the language models for answer sentences and question sentences, respectively. Overall, the framework includes three components, namely a QA model, a QG model and a regularization term that reflects the duality of QA and QG. The QA specific objective aims to minimize the loss function $l_{qa}(f_{qa}(a,q;\\theta _{qa}), label)$ , where $label$ is 0 or 1 that indicates whether $a$ is the correct answer of $q$ or not. For each correct question-answer pair, the QG specific objective is to minimize the following loss function,\r\n\r\n$$l_{qg}(q, a) = -log P_{qg}(q|a;\\theta _{qg})$$ (Eq. 6)\r\n\r\nwhere $a$ is the correct answer of $q$ . The third objective is the regularization term which satisfies the probabilistic duality constrains as given in Equation 3 . Specifically, given a correct $\\langle q, a \\rangle $ pair, we would like to minimize the following loss function,\r\n\r\n$$ \\nonumber l_{dual}(a,q;\\theta _{qa}, \\theta _{qg}) &= [logP_a(a) + log P(q|a;\\theta _{qg}) \\\\ & - logP_q(q) - logP(a|q;\\theta _{qa})]^2$$ (Eq. 9)\r\n\r\nwhere $P_a(a)$ and $P_q(q)$ are marginal distributions, which could be easily obtained through language model. \n Question: What does \"explicitly leverages their probabilistic correlation to guide the training process of both models\" mean?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-e4995c81865a440aaac771a541b596f8",
            "input": "Second, we compute three metrics on the extracted information:\n\n$\\bullet $ Relation Generation (RG) estimates how well the system is able to generate text containing factual (i.e., correct) records. We measure the precision and absolute number (denoted respectively RG-P% and RG-#) of unique relations $r$ extracted from $\\hat{y}_{1:T}$ that also appear in $s$.\n\n$\\bullet $ Content Selection (CS) measures how well the generated document matches the gold document in terms of mentioned records. We measure the precision and recall (denoted respectively CS-P% and CS-R%) of unique relations $r$ extracted from $\\hat{y}_{1:T}$ that are also extracted from $y_{1:T}$.\n\n$\\bullet $ Content Ordering (CO) analyzes how well the system orders the records discussed in the description. We measure the normalized Damerau-Levenshtein distance BIBREF36 between the sequences of records extracted from $\\hat{y}_{1:T}$ that are also extracted from $y_{1:T}$. \n Question: Which qualitative metric are used for evaluation?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-2fcb162dd7c84a8088b2d4186aeba156",
            "input": "A single speaker narrated the 2000 sentences, which took several days.  \n Question: How many annotators participated?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-c26657aa0828436ba5d12851889b5ade",
            "input": " To evaluate real-world performance of our selected classifier (i.e., performance in the absence of model and parameter bias), we perform classification of the holdout set. On this set, our classifier had an accuracy and F1-score of 89.6% and 89.2%, respectively.  \n Question: What was their system's performance?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-acc91f7b1888430b92161214a7de1db1",
            "input": "The study of the mathematical structure of grammar has indicated that the fundamental things making up sentences are not the words, but some atomic grammatical types, such as the noun-type and the sentence-type BIBREF23 , BIBREF24 , BIBREF25 . The transitive verb-type is not an atomic grammatical type, but a composite made up of two noun-types and one sentence-type. Hence, particularly interesting here is that atomic doesn't really mean smallest... \n Question: Do they argue that all words can be derived from other (elementary) words?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-8318bc74e1824286bfe34d26fdf6259b",
            "input": "Following the formula, we can calculate the contribution of every input word makes to every output word, forming a contribution matrix of size $M \\times N$, where $N$ is the output sentence length. Given the contribution matrix, we can obtain the word importance of each input word to the entire output sentence. To this end, for each input word, we first aggregate its contribution values to all output words by the sum operation, and then normalize all sums through the Softmax function. \n Question: How do their models decide how much improtance to give to the output words?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-4d8661d8f89d464fa93132b728653861",
            "input": "To better demonstrate the effectiveness of the proposed model, we compare with baselines and show the results in Table TABREF12 . \n Question: What was the score of the proposed model?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-4e6db47a7e5e4caf8a7076e75e141969",
            "input": "The problem with morpheme segmentation is that the vocabulary of stem units is still very large, which leads to many rare and unknown words at the training time.  Therefore, we learn a BPE model on the stem units in the training corpus rather than the words, and then apply it on the stem unit of each word after morpheme segmentation. \n Question: How is morphology knowledge implemented in the method?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-4961931b76b74389877d02baa6431f9a",
            "input": "For those willing to have a more tightly-controlled installation of Seshat on their system, we also fully specify the manual installation steps in our online documentation). \n Question: Is this software available to the public?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-7c0532fac9934b739b462e27981fa3a4",
            "input": "KryptoOracle has been built in the Apache ecosystem and uses Apache Spark. Data structures in Spark are based on resilient distributed datasets (RDD), a read only multi-set of data which can be distributed over a cluster of machines and is fault tolerant.  Spark RDD has the innate capability to recover itself because it stores all execution steps in a lineage graph. In case of any faults in the system, Spark redoes all the previous executions from the built DAG and recovers itself to the previous steady state from any fault such as memory overload. Spark RDDs lie in the core of KryptoOracle and therefore make it easier for it to recover from faults. Moreover, faults like memory overload or system crashes may require for the whole system to hard reboot. However, due to the duplicate copies of the RDDs in Apache Hive and the stored previous state of the machine learning model, KryptoOracle can easily recover to the previous steady state.\n\n \n Question: How is the architecture fault-tolerant?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-69cb9e495b6e4b1f9e3bbb92a643510e",
            "input": "The answer styles considered in the experiments corresponded to the two tasks. The NLG task requires a well-formed answer that is an abstractive summary of the question and ten passages, averaging 16.6 words. The Q&A task also requires an abstractive answer but prefers a more concise answer than the NLG task, averaging 13.1 words, where many of the answers do not contain the context of the question. \n Question: What do they mean by answer styles?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-adddc9037e3a490fb70d7caafbf70a83",
            "input": "We build and test our MMT models on the Multi30K dataset BIBREF21 . Each image in Multi30K contains one English (EN) description taken from Flickr30K BIBREF22 and human translations into German (DE), French (FR) and Czech BIBREF23 , BIBREF24 , BIBREF25 . The dataset contains 29,000 instances for training, 1,014 for development, and 1,000 for test. We only experiment with German and French, which are languages for which we have in-house expertise for the type of analysis we present. In addition to the official Multi30K test set (test 2016), we also use the test set from the latest WMT evaluation competition, test 2018 BIBREF25 . \n Question: Do they report results only on English dataset?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-3473d8aeb9c94e919bf787f1009ff55c",
            "input": "One of the most important operations on a word is to obtain the set of words whose meaning is similar to the word, or whose usage in text is similar to the word. We call this set the neighbor of the word. We have observed that the neighbor of a polysemic word consists of words that resemble the primary sense of the polysemic word.  Even though a word may be a polysemic, it usually corresponds to a single vector in distributed representation. This vector is primarily determined by the major sense, which is most frequently used. The information about a word's minor sense is subtle, and the effect of a minor sense is difficult to distinguish from statistical fluctuation. To measure the effect of a minor sense, this paper proposes to use the concept of surrounding uniformity. The surrounding uniformity roughly corresponds to statistical fluctuation in the vectors that correspond to the words in the neighbor Surrounding Uniformity (SU) can be expressed as follows: $SU(\\vec{w}) = \\frac{|\\vec{s}(\\vec{w})|}{|\\vec{w}| + \\sum _{i}^{N}|\\vec{a_i}(\\vec{w})|}$\n\nwhere $\\vec{s}(\\vec{w}) = \\vec{w} + \\sum _{i}^{N} \\vec{a_i}(\\vec{w}).$ \n Question: How is the fluctuation in the sense of the word and its neighbors measured?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-59b011262a5f45359fd136b5d504b840",
            "input": "However, no prior work has explored the target of the offensive language, which is important in many scenarios, e.g., when studying hate speech with respect to a specific target. \n Question: What are the differences between this dataset and pre-existing ones?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-465a2ad6fc004c4a9f09c763cff491a2",
            "input": "The reports are published by Météo France and the Met Office, its British counterpart. They are publicly available on the respective websites of the organizations. Both corpora span on the same period as the corresponding time series and given their daily nature, it yields a total of 4,261 and 4,748 documents respectively. \n Question: How big is dataset used for training/testing?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-5f2d5257d02c48ee94dacc1e80179cf2",
            "input": "A group of 50 native people who were well-versed in both English and Tamil languages acted as annotators for the evaluation. A collection of samples of about 100 sentences were taken from the test set results for comparison. This set included a randomized selection of the translation results to ensure the objectivity of evaluation. Fluency and adequacy results for the RNNMorph results are tabulated. Adequacy rating was calculated on a 5-point scale of how much of the meaning is conveyed by the translation (All, Most, Much, Little, None). The fluency rating was calculated based on grammatical correctness on a 5-point scale of (Flawless, Good, Non-native, Disfluent, Incomprehensive). For the comparison process, the RNNMorph and the RNNSearch + Word2Vec models’ sentence level translations were individually ranked between each other, permitting the two translations to have ties in the ranking. \n Question: How were the human judgements assembled?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-e6ede0fd70be4413a2992631978b295d",
            "input": "For the input, we build a parallel monolingual corpus by translating the mixed language sequence using Google NMT to English ( INLINEFORM0 ) and Mandarin ( INLINEFORM1 ) sequences. \n Question: What parallel corpus did they use?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-5d9ef4bcd4d045e19a08fca66fddae6f",
            "input": ". This work can be extended to predict future events with one day in advance, where we will use the same method for feature selection in addition to to time series analysis of the historical patterns of the word-pairs. \n Question: Do the authors suggest any future extensions to this work?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-f609a35795cb43e4815018cc63aedb3c",
            "input": "The MP framework is based on the core idea of recursive neighborhood aggregation. That is, at every iteration, the representation of each vertex is updated based on messages received from its neighbors. All spectral GNNs can be described in terms of the MP framework.\n\nGNNs have been applied with great success to bioinformatics and social network data, for node classification, link prediction, and graph classification. However, a few studies only have focused on the application of the MP framework to representation learning on text. This paper proposes one such application. The concept of message passing over graphs has been around for many years BIBREF0, BIBREF1, as well as that of graph neural networks \n Question: What is the message passing framework?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-d10907702a4046dfb977db893e67b418",
            "input": "Building Extractive CNN/Daily Mail \n Question: What is the problem with existing metrics that they are trying to address?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-8442c6afeba24bb7aab80b4df881bbc4",
            "input": "Since those sets are expected to reflect social norms, they are referred as Dos and Don'ts hereafter. Analogously, some of the negative words just describe inappropriate behaviour, like slur or misdeal, whereas others are real crimes as murder. \n Question: Do they report results only on English data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-a3da490f024c4014a9db6fdfd80a27e5",
            "input": "image feature pre-selection part which models the tendency where people focus to ask questions We propose to perform saliency-like pre-selection operation to alleviate the problems and model the RoI patterns. The image is first divided into $g\\times g$ grids as illustrated in Figure. 2 . Taking $m\\times m$ grids as a region, with $s$ grids as the stride, we obtain $n\\times n$ regions, where $n=\\left\\lfloor \\frac{g-m}{s}\\right\\rfloor +1$ . We then feed the regions to a pre-trained ResNet BIBREF24 deep convolutional neural network to produce $n\\times n\\times d_I$ -dimensional region features, where $d_I$ is the dimension of feature from the layer before the last fully-connected layer. \n Question: Does the new system utilize pre-extracted bounding boxes and/or features?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-6aa733fa81c247c0b47d6be65118e1aa",
            "input": "To do so, we constructed a new dataset with 23,700 queries that are short and unstructured, in the same style made by real users of task-oriented systems.  \n Question: What is the size of this dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-b9d477b3163b42998729c2c2366a30a6",
            "input": ". While automated evaluation metrics like ROUGE measure lexical similarity between machine and human summaries, humans can better measure how coherent and readable a summary is. Our evaluation study investigates whether tuning the PG-net model increases summary coherence, by asking evaluators to select which of three summaries for the same document they like most: the PG-net model trained on CNN/DM; the model trained on student reflections; and finally the model trained on CNN/DM and tuned on student reflections. 20 evaluators were recruited from our institution and asked to each perform 20 annotations. Summaries are presented to evaluators in random order. Evaluators are then asked to select the summary they feel to be most readable and coherent. \n Question: Who were the human evaluators used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-f87a0aae520a44a3ba64c23e862d8745",
            "input": "The SnapCaptions dataset is composed of 10K user-generated image (snap) and textual caption pairs where named entities in captions are manually labeled by expert human annotators (entity types: PER, LOC, ORG, MISC). \n Question: How large is their MNER SnapCaptions dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-b6391d7a54464e9fa7c6987fddc24505",
            "input": "We recognise features that add ambiguity to the supporting facts, for example when information is only expressed implicitly by using an Ellipsis. As opposed to redundant words, we annotate Restrictivity and Factivity modifiers, words and phrases whose presence does change the meaning of a sentence with regard to the expected answer, and occurrences of intra- or inter-sentence Coreference in supporting facts (that is relevant to the question). Lastly, we mark ambiguous syntactic features, when their resolution is required in order to obtain the answer. Concretely, we mark argument collection with con- and disjunctions (Listing) and ambiguous Prepositions, Coordination Scope and Relative clauses/Adverbial phrases/Appositions. We recognise features that add ambiguity to the supporting facts, for example when information is only expressed implicitly by using an Ellipsis. As opposed to redundant words, we annotate Restrictivity and Factivity modifiers, words and phrases whose presence does change the meaning of a sentence with regard to the expected answer, and occurrences of intra- or inter-sentence Coreference in supporting facts (that is relevant to the question). Lastly, we mark ambiguous syntactic features, when their resolution is required in order to obtain the answer. Concretely, we mark argument collection with con- and disjunctions (Listing) and ambiguous Prepositions, Coordination Scope and Relative clauses/Adverbial phrases/Appositions. \n Question: What features are absent from MRC gold standards that can result in potential lexical ambiguity?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-b1dd19289797403fb396faebbbf515c4",
            "input": "We use word alignments, similarly to other annotation projection work, to project the AMR alignments to the target languages. Our approach depends on an underlying assumption that we make: if a source word is word-aligned to a target word and it is AMR aligned with an AMR node, then the target word is also aligned to that AMR node. Word alignments were generated using fast_align BIBREF10 , while AMR alignments were generated with JAMR BIBREF11 . \n Question: How is annotation projection done when languages have different word order?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-40d8bf629410412b9c5d92a7cab634f4",
            "input": "In contrast, in order to exploit section information, in this paper we propose to capture a distributed representation of both the global (the whole document) and the local context (e.g., the section/topic) when deciding if a sentence should be included in the summary \n Question: What do they mean by global and local context?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-358907b64f23448485a55e6aea92f85f",
            "input": "As a result, transfer training with only 1000 hours data can match equivalent performance for full training with 7300 hours data. \n Question: how small of a dataset did they train on?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-36f8b48ff1d24bb681a565285c7e7727",
            "input": "We have collected total 3500 questions from the Internet and other sources such as books of general knowledge questions, history etc. The corpus contains the questions and the classes each question belongs to. \n Question: what datasets did they use?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-aac509a8e08347aebee2550cd6de0d4c",
            "input": "All corpora provide datasets/splits for answer selection, whereas only (WikiQA, SQuAD) and (WikiQA, SelQA) provide datasets for answer extraction and answer triggering, respectively. SQuAD is much larger in size although questions in this corpus are often paraphrased multiple times. On the contrary, SQuAD's average candidates per question ( INLINEFORM0 ) is the smallest because SQuAD extracts answer candidates from paragraphs whereas the others extract them from sections or infoboxes that consist of bigger contexts. Although InfoboxQA is larger than WikiQA or SelQA, the number of token types ( INLINEFORM1 ) in InfoboxQA is smaller than those two, due to the repetitive nature of infoboxes.\n\nAll corpora show similar average answer candidate lengths ( INLINEFORM0 ), except for InfoboxQA where each line in the infobox is considered a candidate. SelQA and SQuAD show similar average question lengths ( INLINEFORM1 ) because of the similarity between their annotation schemes. It is not surprising that WikiQA's average question length is the smallest, considering their questions are taken from search queries. InfoboxQA's average question length is relatively small, due to the restricted information that can be asked from the infoboxes. InfoboxQA and WikiQA show the least question-answer word overlaps over questions and answers ( INLINEFORM2 and INLINEFORM3 in Table TABREF2 ), respectively. In terms of the F1-score for overlapping words ( INLINEFORM4 ), SQuAD gives the least portion of overlaps between question-answer pairs although WikiQA comes very close. \n Question: How do they analyze contextual similaries across datasets?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-48f2322f7ed346d086420dd5f99346c9",
            "input": "N-grams, POS, Hierarchical features: A baseline bag-of-words model incorporating both tagged and untagged unigrams and bigams.  CNN: Kim BIBREF28 demonstrated near state-of-the-art performance on a number of sentence classification tasks (including TREC question classification) by using pre-trained word embeddings BIBREF40 as feature extractors in a CNN model. \n Question: What previous methods is their model compared to?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-143fa2dae7484b9f9c9da681c00de33d",
            "input": "The FBFans dataset contains data from anti-nuclear-power Chinese Facebook fan groups from September 2013 to August 2014, including posts and their author and liker IDs. \n Question: What topic is covered in the Chinese Facebook data? ",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-d3d7a0133db74a4fa18c611583effcec",
            "input": "In order to represent individual sentences, we insert external [cls] tokens at the start of each sentence, and each [cls] symbol collects features for the sentence preceding it. We also use interval segment embeddings to distinguish multiple sentences within a document. This way, document representations are learned hierarchically where lower Transformer layers represent adjacent sentences, while higher layers, in combination with self-attention, represent multi-sentence discourse. Position embeddings in the original Bert model have a maximum length of 512; we overcome this limitation by adding more position embeddings that are initialized randomly and fine-tuned with other parameters in the encoder. \n Question: What is novel about their document-level encoder?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-aca8fa8a2eff4bb6b0302da682b69be6",
            "input": "We use the VisDial v1.0 BIBREF0 dataset to train our models, where one example has an image with its caption, 9 question-answer pairs, and follow-up questions and candidate answers for each round. At round $r$, the caption and the previous question-answer pairs become conversational context. The whole dataset is split into 123,287/2,000/8,000 images for train/validation/test, respectively. Unlike the images in the train and validation sets, the images in the test set have only one follow-up question and candidate answers and their corresponding conversational context. \n Question: How big is dataset for this challenge?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-ea351435746e4c4494f6be833f89405a",
            "input": "The baseline classifier uses a linear Support Vector Machine BIBREF7 , which is suited for a high number of features.  \n Question: What baseline is used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-d01b8476f1844eccb027ed313bc7954c",
            "input": "Comparing the natural and artificial sources of our parallel data wrt. several linguistic and distributional properties, we observe that (see Fig. FIGREF21 - FIGREF22 ):\n\nartificial sources are on average shorter than natural ones: when using BT, cases where the source is shorter than the target are rarer; cases when they have the same length are more frequent.\n\nautomatic word alignments between artificial sources tend to be more monotonic than when using natural sources, as measured by the average Kendall INLINEFORM0 of source-target alignments BIBREF22 : for French-English the respective numbers are 0.048 (natural) and 0.018 (artificial); for German-English 0.068 and 0.053.  The intuition is that properties (i) and (ii) should help translation as compared to natural source, while property (iv) should be detrimental. \n Question: what is their explanation for the effectiveness of back-translation?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-bfd1a3ceb0d44b9e9090844319bc90ed",
            "input": "We use two datasets for experiments (Table TABREF29 ): (a) STAN INLINEFORM0 , created by BIBREF10 BansalBV15, which consists of 1,108 unique English hashtags from 1,268 randomly selected tweets in the Stanford Sentiment Analysis Dataset BIBREF36 along with their crowdsourced segmentations and our additional corrections; and (b) STAN INLINEFORM1 , our new expert curated dataset, which includes all 12,594 unique English hashtags and their associated tweets from the same Stanford dataset. \n Question: Do the hashtag and SemEval datasets contain only English data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-08d9b4e80e24482294aa48fb7bb6337a",
            "input": "Baseline Experiments\nIn this section, we briefly describe a baseline and evaluation scripts that we release, with a detailed documentation, along with the corpus. \n Question: What type of evaluation is proposed for this task?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-64cb29ed071b45eb9259cf38c5ed54b2",
            "input": "Despite the focus on sharing datasets and source codes on popular software development platforms such as GitHub (github.com) or Zenodo (zenodo.org), it is still a challenge to use data or code from other groups. \n Question: Are datasets publicly available?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-1661c357879d4133b4b5f9b33cddd238",
            "input": "To identify the most suitable classifier for classifying the scalars associated with each text, we perform evaluations using the stochastic gradient descent, naive bayes, decision tree, and random forest classifiers. \n Question: What was the baseline?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-4006dab0c888434bb3b1f134ad0c31d2",
            "input": "CodeInternational: A tool which can translate code between human languages, powered by Google Translate.\n\n \n Question: Is this auto translation tool based on neural networks?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-5e1ef7025a204955940cd67e0ce23bd3",
            "input": "Tweets IDs 6, 8, and 10 are some samples containing offensive words and slurs which arenot hate or offensive in all cases and writers of them used this type of language in their daily communications. Given these pieces of evidence, by considering the content of tweets, we can see in tweets IDs 3, 4, and 9 that our BERT-based classifier can discriminate tweets in which neither and implicit hatred content exist. \n Question: What evidence do the authors present that the model can capture some biases in data annotation and collection?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-c4072db42b7843d4b1b796c92bc1008d",
            "input": "In Sec. \"Materials and Methods\" we discuss our materials and methods, including the dataset we studied, how we preprocessed that data and extracted a `causal' corpus and a corresponding `control' corpus, and the details of the statistical and language analysis tools we studied these corpora with. \n Question: Do they use expert annotations, crowdsourcing, or only automatic methods to analyze the corpora?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-6cd55b40e01b4863b6fb2ed6d0d30e8b",
            "input": "Recently, a number of researchers have endeavored to explore methods for simultaneous translation in the context of NMT BIBREF6, BIBREF7, BIBREF8, BIBREF9. Some of them propose sophisticated training frameworks explicitly designed for simultaneous translation BIBREF5, BIBREF10.  \n Question: Has there been previous work on SNMT?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-30e32abe96e24b5c8639f84cabd06170",
            "input": "CoinCollector BIBREF8 is a class of text-based games where the objective is to find and collect a coin from a specific location in a given set of connected rooms . The agent wins the game after it collects the coin, at which point (for the first and only time) a reward of +1 is received by the agent. The environment parses only five admissible commands (go north, go east, go south, go west, and take coin) made by two worlds; CookingWorld BIBREF14 in this challenge, there are 4,440 games with 222 different levels of difficulty, with 20 games per level of difficulty, each with different entities and maps. The goal of each game is to cook and eat food from a given recipe, which includes the task of collecting ingredients (e.g. tomato, potato, etc.), objects (e.g. knife), and processing them according to the recipe (e.g. cook potato, slice tomato, etc.). The parser of each game accepts 18 verbs and 51 entities with a predefined grammar, but the overall size of the vocabulary of the observations is 20,000. In Appendix SECREF36 we provide more details about the levels and the games' grammar. \n Question: On what Text-Based Games are experiments performed?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-7b480307c687493e80fbab9dfb3ddb23",
            "input": "Past research took a reductionist approach, separately considering these two problems of “what” and “how” via content selection and question construction.  In contrast, neural models motivate an end-to-end architectures. Deep learned frameworks contrast with the reductionist approach, admitting approaches that jointly optimize for both the “what” and “how” in an unified framework.  \n Question: What learning paradigms do they cover in this survey?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-a7186399c19e423689e1677139b96b09",
            "input": "BIBREF4 contains a variety of news-related information such as images, captions, geo-location information and comments which could be used as a proxy for article popularity. The articles in this dataset were collected between January and December 2014. Although we tried to retrieve the entire dataset, we were able to download only 38,182 articles due to the dead links published in the dataset. The retrieved articles were published in main news channels, such as Yahoo News, The Guardian or The Washington Post. Similarly, to The NowThisNews dataset we normalize the data by grouping articles per publisher, and classifying them as popular, when the number of comments exceeds the median value for given publisher. \n Question: What is the source of the news articles?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-193de252f23841ad9ba0ff836f612a46",
            "input": "Word2vec representation is far better, advanced and a recent technique which functions by mapping words to a 300 dimensional vector representations. \n Question: What is the dimension of the embeddings?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-4d85970f3f234ed39301922fb98b7e29",
            "input": "Bi-LSTM BIBREF11 is a baseline for neural models. Bi-LSTM$_{+ att. + LEX + POS}$ BIBREF10 is a multi-task learning framework for WSD, POS tagging, and LEX with self-attention mechanism, which converts WSD to a sequence learning task. GAS$_{ext}$ BIBREF12 is a variant of GAS which is a gloss-augmented variant of the memory network by extending gloss knowledge. CAN$^s$ and HCAN BIBREF13 are sentence-level and hierarchical co-attention neural network models which leverage gloss knowledge. \n Question: How does the neural network architecture accomodate an unknown amount of senses per word?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-75952c7d992c4e41a96de2acd2c6bf88",
            "input": "This work also contributes a new dataset of INLINEFORM0 pairs of free-form natural language instructions and high-level navigation plans. This dataset was collected through Mechanical Turk using 100 simulated environments with a corresponding topological map and, to the best of our knowledge, it is the first of its kind for behavioral navigation.  \n Question: Did the authors use a crowdsourcing platform?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-f505daa98ce44ee69a48b9f0c856aab1",
            "input": "For the training data, we use Daily Mail news articles released by BIBREF9 .  \n Question: what dataset was used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-318730f2b33145fa988f7553104143bd",
            "input": "Another insight gained from these charts is that a random summarizer resulted in scores more than 50% in all measures, and without using document-aware features, the model achieves a small improvement over a random summarizer. \n Question: Is new approach tested against state of the art?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-c15caa525efc4d0295ecef6c553bb2b3",
            "input": "The decoder is the second LSTM network that uses the information obtained from the encoder to generate the sequence's story. The first input $x_0$ to the decoder is the image for which the text is being generated. The last hidden state from the encoder $h_e^{(t)}$ is used to initialize the first hidden state of the decoder $h_d^{(0)}$ . With this strategy, we provide the decoder with the context of the whole sequence and the content of the current image (i.e. global and local information) to generate the corresponding text that will contribute to the overall story. \n Question: How is the sequential nature of the story captured?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-c3fbb7e866a5403cb66d4f04fdd50884",
            "input": "Most prior work focuses on a different aspect of offensive language such as abusive language BIBREF0 , BIBREF1 , (cyber-)aggression BIBREF2 , (cyber-)bullying BIBREF3 , BIBREF4 , toxic comments INLINEFORM0 , hate speech BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , and offensive language BIBREF11 . Prior work has focused on these aspects of offensive language in Twitter BIBREF3 , BIBREF7 , BIBREF8 , BIBREF11 , Wikipedia comments, and Facebook posts BIBREF2 . \n Question: What is the definition of offensive language?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-7dff285404ce4f0fbf005257250386ca",
            "input": "We extract 5 Surface and Lexical features, namely sequence length in number of tokens, average word length, type-token ratio, and lexical to tokens ratio (ratio of adjectives, verbs, nouns, and adverbs to tokens). We extracted two features that use a learned representation: Firstly, we get a sentence embedding feature that is built by averaging the word embeddings of an input sentence. Secondly, we extract a fastText representation using the fastText library with the same parameters as reported in Joulin et al. joulin2016bag. \n Question: Do they differentiate insights where they are dealing with learned or engineered representations?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-4aa9c980c1654d78963060c6ea358d76",
            "input": "In our experiments, the Memory Neural Networks (MemNNs) proposed in babidataset serve as the baselines. \n Question: What word level and character level model baselines are used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-8f3d0d3f72814581870f33f93de78f61",
            "input": "The corpus of supervisor assessment has 26972 sentences. The summary statistics about the number of words in a sentence is: min:4 max:217 average:15.5 STDEV:9.2 Q1:9 Q2:14 Q3:19. \n Question: What is the average length of the sentences?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-22a3d0afa72244a6a677d8192500e8dd",
            "input": "This study focuses on Switchboard-300, a standard 300-hour English conversational speech recognition task. As a contrast to our best results on Switchboard-300, we also train a seq2seq model on the 2000-hour Switchboard+Fisher data.  \n Question: How much bigger is Switchboard-2000 than Switchboard-300 database?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-92c0faca7c2c4a0fa8e04ea639bee384",
            "input": "We created our own causal explanation dataset by collecting 3,268 random Facebook status update messages. \n Question: What types of social media did they consider?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-b6089ddcad39497fb1e725768abbf10e",
            "input": "Overall, this confirms our initial intuition that combining the best performing popularity-based approach with the best similarity-based approach should result in the highest accuracy (i.e., INLINEFORM7 for INLINEFORM8 ). \n Question: which algorithm was the highest performer?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-74420a0281f24991914295b062ca06a6",
            "input": "Very recently, the success of deep neural networks in many natural language processing tasks ( BIBREF20 ) has inspired new work in abstractive summarization . BIBREF2 propose a neural attention model with a convolutional encoder to solve this task.  More recently, BIBREF4 extended BIBREF2 's work with an RNN decoder, and BIBREF8 proposed an RNN encoder-decoder architecture for summarization. Both techniques are currently the state-of-the-art on the DUC competition. \n Question: What is the state-of-the art?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-689acbec8c3c42a2b0c618bd92186721",
            "input": "The focus of this publication is on representing information that is valuable for these tasks but that hitherto has largely been ignored in machine learning approaches centering around simplified language, specifically, text structure (e.g., paragraphs, lines), typography (e.g., font type, font style), and image (content, position, and dimensions) information. Information on the font type and font style (e.g., italics, bold print) of a token and its position on the physical page (for PDFs only) was specified as attributes to the token elements of the tokens layer (cf. Figure FIGREF34 for an example) \n Question: Which information about typography is included in the corpus?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-aec77a6689ed4fc787bc7b84bb26f074",
            "input": "In the experiments, we compare model performance between GluonCV/NLP and other open source implementations with Caffe, Caffe2, Theano, and TensorFlow, including ResNet BIBREF8 and MobileNet BIBREF9 for image classification (ImageNet), Faster R-CNN BIBREF10 for object detection (COCO), Mask R-CNN BIBREF11 for instance segmentation, Simple Pose BIBREF12 for pose estimation (COCO), textCNN BIBREF13 for sentiment analysis (TREC), and BERT BIBREF14 for question answering (SQuAD 1.1), sentiment analysis (SST-2), natural langauge inference (MNLI-m), and paraphrasing (MRPC). \n Question: Do they experiment with the toolkits?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-a306d0d0c1df4267a7f531349efc5fbc",
            "input": "We use the LSTM network as follows. The $x$ vector is fed through the LSTM network which outputs a vector $\\overrightarrow{h_i}$ for each time step $i$ from 0 to $n-1$. This is the forward LSTM. As we have access to the complete vector $x$, we can process a backward LSTM as well. This is done by computing a vector $\\overleftarrow{h_i}$ for each time step $i$ from $n-1$ to 0. Finally, we concatenate the backward LSTM with the forward LSTM:\n\nBoth $\\overrightarrow{h_i}$ and $\\overleftarrow{h_i}$ have a dimension of $l$, which is an optimized hyperparameter. The BiLSTM output $h$ thus has dimension $2l\\times n$. \n Question: Is the LSTM bidirectional?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-ae325565dca24320bab5dccf73910a90",
            "input": "We introduce a new crowdsourced dataset of 23,700 queries, including 22,500 in-scope queries covering 150 intents, which can be grouped into 10 general domains.  \n Question: Where does the data come from?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-7901e7d2eb204419862768ce6f071e90",
            "input": "Our model BiLSTM+CNN(grapheme-level) and BiLSTM+CNN(G)+POS outperforms all other model experimented in OurNepali and ILPRL dataset respectively. \n Question: What is the best model?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-2c9349b99f624a50a706e1d9d9e69482",
            "input": "Stacked LSTMs Cell-aware Stacked LSTMs\nNow we extend the stacked LSTM formulation defined above to address the problem noted in the previous subsection. Sentence Encoders\nThe sentence encoder network we use in our experiments takes INLINEFORM0 words (assumed to be one-hot vectors) as input. Top-layer Classifiers\nFor the natural language inference experiments, we use the following heuristic function proposed by BIBREF36 in feature extraction: DISPLAYFORM0\n\nwhere INLINEFORM0 means vector concatenation, and INLINEFORM1 and INLINEFORM2 are applied element-wise. \n Question: Which models did they experiment with?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-7217260bc0f04e99b2226b9b3a5c25ba",
            "input": "The task is set up to mimic (albeit, in an oversimplified manner) the input-output symbol alignments and local syntactic properties that models must learn in many natural language tasks, such as translation, tagging and summarization. \n Question: Why does the proposed task a good proxy for the general-purpose sequence to sequence tasks?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-5bb6fac2816e47f3984269732eddf986",
            "input": "Our current feature set for training a machine learning classifier is relatively small, consisting of paragraph domain scores, bag-of-words, length of stay, and number of previous admissions, but we intend to factor in many additional features that extend beyond the scope of the present study. These include a deeper analysis of clinical narratives in EHRs: our next task will be to extend our EHR data pipeline by distinguishing between clinically positive and negative phenomena within each risk factor domain. This will involve a series of annotation tasks that will allow us to generate lexicon-based and corpus-based sentiment analysis tools. We can then use these clinical sentiment scores to generate a gradient of patient improvement or deterioration over time. \n Question: What additional features are proposed for future work?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-b00731531d3f49da9b7076814c9e41d5",
            "input": "To deal with negative values, we propose clipped $\\mathit {PMI}$,\n\nwhich is equivalent to $\\mathit {PPMI}$ when $z = 0$. In practice we find this works poorly if done symmetrically, so we introduce a variant called $\\mathit {NNEGPMI}$ which only normalizes $\\mathit {\\texttt {-}PMI}$: \n Question: What novel PMI variants are introduced?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-402f24780bed484eb6dabb7bde3391bf",
            "input": "For every input data point and possible target class, LRP delivers one scalar relevance value per input variable, hereby indicating whether the corresponding part of the input is contributing for or against a specific classifier decision, or if this input variable is rather uninvolved and irrelevant to the classification task at all. \n Question: Does the LRP method work in settings that contextualize the words with respect to one another?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-06fcbb55fc064f6b91da4f0f2deb300e",
            "input": "With an intention to exploit the contribution of hashtag use to stance detection, we have also used the existence of hashtags in tweets as an additional feature to unigrams. \n Question: What are hashtag features?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-c85699fa2f6441bbae7f37cdc7a06830",
            "input": "We deployed the task on Amazon Mechanical Turk (AMT). To see how reasoning varies across workers, we hire 3 crowdworkers per one instance. We hire reliable crowdworkers with $\\ge 5,000$ HITs experiences and an approval rate of $\\ge $ 99.0%, and pay ¢20 as a reward per instance. \n Question: Did they use any crowdsourcing platform?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-8b593cf2e3674814875ed1472fc5aa74",
            "input": "In this research, we briefly discuss the steps of QA system and compare the performance of seven machine learning based classifiers (Multi-Layer Perceptron (MLP), Naive Bayes Classifier (NBC), Support Vector Machine (SVM), Gradient Boosting Classifier (GBC), Stochastic Gradient Descent (SGD), K Nearest Neighbour (K-NN) and Random Forest (RF)) in classifying Bengali questions to classes based on their anticipated answers. \n Question: what ml based approaches were compared?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-54d0f16b9a4740ed8883ea71d01b6628",
            "input": "We evaluate our method on two widely-studied datasets provided by Waseem and Hovey BIBREF5 and Davidson et al. BIBREF9. \n Question: Which publicly available datasets are used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-8bd08f745e08477fa5e34c3043d1190c",
            "input": "At the same time, some information about inflected word forms in the context can be useful, but it is lost during lemmatization, and this leads to the decreased score. Arguably, this means that lemmatization brings along both advantages and disadvantages for WSD with ELMo.  \n Question: Do the authors mention any downside of lemmatizing input before training ELMo?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-1dee9955e00d4f1e895f013c374ee56c",
            "input": "We ran UTD over all 104 telephone calls, which pair 11 hours of audio with Spanish transcripts and their crowdsourced English translations. The transcripts contain 168,195 Spanish word tokens (10,674 types), and the translations contain 159,777 English word tokens (6,723 types). \n Question: what is the size of the speech corpus?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-d5fc144007e34e679249ab8fac46258a",
            "input": "For the cross-domain polarity classification experiments, we use the second version of Multi-Domain Sentiment Dataset BIBREF0 . The data set contains Amazon product reviews of four different domains: Books (B), DVDs (D), Electronics (E) and Kitchen appliances (K). Reviews contain star ratings (from 1 to 5) which are converted into binary labels as follows: reviews rated with more than 3 stars are labeled as positive, and those with less than 3 stars as negative. In each domain, there are 1000 positive and 1000 negative reviews. \n Question: How long is the dataset?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-a12962212269491b90cd7f12b4824623",
            "input": "For all experiments, the dimensions of word embeddings and recurrent hidden states are both set to 512. \n Question: Do they use the same architecture as LSTM-s and GRUs with just replacing with the LAU unit?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-90dee997da0149c58ebebca98d874459",
            "input": "Differently to textual data, our goal in this paper is to explore the large amount of categorical data that is often collected in travel surveys. This includes trip purpose, education level, or family type. We also consider other variables that are not necessarily of categorical nature, but typically end up as dummy encoding, due to segmentation, such as age, income, or even origin/destination pair. \n Question: How do they model travel behavior?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-7f856d1baac64d449d0f84747dcd932c",
            "input": "We see that parent quality is a simple yet effective feature and SVM model with this feature can achieve significantly higher ($p<0.001$) F1 score ($46.61\\%$) than distance from the thesis and linguistic features. Although the BiLSTM model with attention and FastText baselines performs better than the SVM with distance from the thesis and linguistic features, it has similar performance to the parent quality baseline. We find that the flat representation of the context achieves the highest F1 score. It may be more difficult for the models with a larger number of parameters to perform better than the flat representation since the dataset is small. We also observe that modeling 3 claims on the argument path before the target claim achieves the best F1 score ($55.98\\%$). \n Question: How better are results compared to baseline models?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-42d37c21083b4461bbbcf43a5d815a3f",
            "input": "A prototypical implementation, in which the words are assumed to be in the fundamental representation of the special orthogonal group, INLINEFORM0 , and are conditioned on losses sensitive to the relative actions of words, is the subject of another manuscript presently in preparation. \n Question: Is there a formal proof that the RNNs form a representation of the group?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-b437c55e1abb482dab4bf1879c9ec950",
            "input": "In this experiment, we compare our Enitity-GCN against recent prior work on the same task. We present test and development results (when present) for both versions of the dataset in Table 2 . From BIBREF0 , we list an oracle based on human performance as well as two standard reading comprehension models, namely BiDAF BIBREF3 and FastQA BIBREF6 . We also compare against Coref-GRU BIBREF12 , MHPGM BIBREF11 , and Weaver BIBREF10 . Additionally, we include results of MHQA-GRN BIBREF23 , from a recent arXiv preprint describing concurrent work. They jointly train graph neural networks and recurrent encoders. We report single runs of our two best single models and an ensemble one on the unmasked test set (recall that the test set is not publicly available and the task organizers only report unmasked results) as well as both versions of the validation set. \n Question: What baseline did they compare Entity-GCN to?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-8e91072b3890493eba7b94e52eacf1f5",
            "input": "In this paper we present our work, which is a part of the SociaL Media And Harassment Competition of the ECML PKDD 2019 Conference.   We are using the dataset of the competition, which includes text from tweets having the aforementioned categories. \n Question: What were the datasets used in this paper?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-9c4c70a36f714cc0af341a34860820c3",
            "input": "To collect a diverse training dataset, we have randomly sampled 1000 posts each from the subreddits politics, business, science, and AskReddit, and 1000 additional posts from the Reddit frontpage. \n Question: what are the topics pulled from Reddit?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-90a9697a5d634358b1b2a8b2db87e75c",
            "input": "We used the Reuters-8 dataset without stop words from BIBREF27 aiming at single-label classification, which is a preprocessed format of the Reuters-21578. \n Question: Which dataset has been used in this work?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-ec24b94564d14a0eb904082545b814ff",
            "input": "We use one public dataset Social Honeypot dataset and one self-collected dataset Weibo dataset to validate the effectiveness of our proposed features. Before directly performing the experiments on the employed datasets, we first delete some accounts with few posts in the two employed since the number of tweets is highly indicative of spammers. For the English Honeypot dataset, we remove stopwords, punctuations, non-ASCII words and apply stemming. For the Chinese Weibo dataset, we perform segmentation with \"Jieba\", a Chinese text segmentation tool. After preprocessing steps, the Weibo dataset contains 2197 legitimate users and 802 spammers, and the honeypot dataset contains 2218 legitimate users and 2947 spammers. \n Question: What is the benchmark dataset and is its quality high?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-633153c6079a472787699e21df50efe5",
            "input": "For training data, 3 million webpages from this corpus were processed with a CCG parser to produce logical forms BIBREF10 . We also used the test set created by Krishnamurthy and Mitchell, which contains 220 queries generated in the same fashion as the training data from a separate section of ClueWeb. However, as they did not release a development set with their data, we used this set as a development set. This final test set contains 307 queries. \n Question: How big is their dataset?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-83f58486b85f4bac998d8abcbf72d85c",
            "input": "Each tweet was tokenized using NLTK TweetTokenizer and classified as one of 10 potential accounts from which it may have originated. The accounts were chosen based on the distinct topics each is known to typically tweet about. \n Question: What text classification task is considered?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-df82378c1d3a4f77bf2570a3752540e3",
            "input": "In this section, we evaluate the proposed method intrinsically in terms of whether the co-occurrence matrix after the low-rank approximation is able to capture similar concepts on student response data sets, and also extrinsically in terms of the end task of summarization on all corpora. In the following experiments, summary length is set to be the average number of words in human summaries or less. An alternative way to evaluate the hypothesis is to let humans judge whether two bigrams are similar or not, which we leave for future work. \n Question: Do they quantitavely or qualitatively evalute the output of their low-rank approximation to verify the grouping of lexical items?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-11da73f72bbf44a4bda8fd168466c531",
            "input": "As we can see that, all variants of our CRU model could give substantial improvements over the traditional GRU model, where a maximum gain of 2.7%, 1.0%, and 1.9% can be observed in three datasets, respectively. \n Question: Do experiment results show consistent significant improvement of new approach over traditional CNN and RNN models?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-870daf98eae4475393bd1d90042d9262",
            "input": "Tourist: I can't go straight any further.\n\nGuide: ok. turn so that the theater is on your right.\n\nGuide: then go straight\n\nTourist: That would be going back the way I came\n\nGuide: yeah. I was looking at the wrong bank\n\nTourist: I'll notify when I am back at the brooks brothers, and the bank.\n\nTourist: ACTION:TURNRIGHT\n\nGuide: make a right when the bank is on your left\n\nTourist: ACTION:FORWARD ACTION:FORWARD ACTION:TURNRIGHT\n\nTourist: Making the right at the bank.\n\nTourist: ACTION:FORWARD ACTION:FORWARD\n\nTourist: I can't go that way.\n\nTourist: ACTION:TURNLEFT\n\nTourist: Bank is ahead of me on the right\n\nTourist: ACTION:FORWARD ACTION:FORWARD ACTION:TURNLEFT\n\nGuide: turn around on that intersection\n\nTourist: I can only go to the left or back the way I just came.\n\nTourist: ACTION:TURNLEFT\n\nGuide: you're in the right place. do you see shops on the corners?\n\nGuide: If you're on the corner with the bank, cross the street\n\nTourist: I'm back where I started by the shop and the bank.\n\nTourist: ACTION:TURNRIGHT \n Question: What language do the agents talk in?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-7eefe190eb2f426ab5b26504e9a7a85a",
            "input": "However, a significant bottleneck that has been overlooked by previous efforts is the input vocabulary size and its corresponding word embedding matrix, often accounting for a significant proportion of all model parameters. For instance, the embedding table of the BERTBASE model, comprising over 30K WordPiece tokens BIBREF14, accounts for over $21\\%$ of the model size. While there has been existing work on reducing NLP model vocabulary sizes BIBREF15, distillation techniques cannot utilize these, since they require the student and teacher models to share the same vocabulary and output space. This profoundly limits their potential to further reduce model sizes. \n Question: Why are prior knowledge distillation techniques models are ineffective in producing student models with vocabularies different from the original teacher models?  ",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-7b193bbaf9d14fce8db0c39e77fbccd8",
            "input": "In this approach the similarity between two words is not strictly based on their co-occurrence frequencies, but rather on the frequencies of the other words which occur with both of them (i.e., second order co-occurrences). This approach has been shown to be successful in quantifying semantic relatedness BIBREF12 , BIBREF13 . \n Question: What is a second order co-ocurrence matrix?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-9fdcf697781144cc9c9ce25acae5c011",
            "input": "We seek a function controlled by gates that can mix states across timesteps, but which acts independently on each channel of the state vector. The simplest option, which BIBREF12 term “dynamic average pooling”, uses only a forget gate: DISPLAYFORM0 \n Question: What pooling function is used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-ae09c3504a704fae8d10f83bc0355637",
            "input": "Contextualized word embeddings, sentence embeddings, such as deep contextualized word representations BIBREF20 , BERT BIBREF22 , encode the complex characteristics and meanings of words in various context by jointly training a bidirectional language model.  The second method is to use sentence embeddings, BERT. It is used to a generate single 786-dimensional sentence embedding from 10k-dimensional one-hot vector or distribution over previous words and then merge into a single context vector with two different merging methods. Using this approach, we can obtain a more dense, informative, fixed-length vectors to encode conversational-context information, $e^k_{context}$ to be used in next $k$ -th utterance prediction. We use contextual gating mechanism in our decoder network to combine the conversational-context embeddings with speech and word embeddings effectively. Our gating is contextual in the sense that multiple embeddings compute a gate value that is dependent on the context of multiple utterances that occur in a conversation.  Let $e_w = e_w(y_{u-1})$ be our previous word embedding for a word $y_{u-1}$ , and let $e_s = e_s(x^k_{1:T})$ be a speech embedding for the acoustic features of current $k$ -th utterance $x^k_{1:T}$ and $e_c = e_c(s_{k-1-n:k-1})$ be our conversational-context embedding for $n$ -number of preceding utterances ${s_{k-1-n:k-1}}$ . Then using a gating mechanism:\n\n$$g = \\sigma (e_c, e_w, e_s)$$ (Eq. 15)\n\nwhere $\\sigma $ is a 1 hidden layer DNN with $\\texttt {sigmoid}$ activation, the gated embedding $e$ is calcuated as\n\n$$e = g \\odot (e_c, e_w, e_s) \\\\ h = \\text{LSTM}(e)$$ (Eq. 16)\n\nand fed into the LSTM decoder hidden layer.  The output of the decoder $h$ is then combined with conversational-context embedding $e_c$ again with a gating mechanism,\n\n$$g = \\sigma (e_C, h) \\\\ \\hat{h} = g \\odot (e_c, h)$$ (Eq. 17)\n\nThen the next hidden layer takes these gated activations, $\\hat{h}$ , and so on. \n Question: How are sentence embeddings incorporated into the speech recognition system?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-e9cda47caecd42d190c1a2e60942408d",
            "input": "After removing 135 questions during the validation, the final dataset comprises 13,939 questions, 3,827 of which require commonsense knowledge (i.e. 27.4%).  \n Question: what is the size of their dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-421eaed10cc842d5afeedd542d93f8fb",
            "input": "We evaluate our model on two benchmark datasets BIBREF9 . The homographic dataset contains 2,250 contexts, 1,607 of which contain a pun. The heterographic dataset consists of 1,780 contexts with 1,271 containing a pun. We notice there is no standard splitting information provided for both datasets. Thus we apply 10-fold cross validation. To make direct comparisons with prior studies, following BIBREF4 , we accumulated the predictions for all ten folds and calculate the scores in the end. \n Question: What datasets are used in evaluation?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-38e267f66e854125b445b83cfce546bc",
            "input": "We compare our approach to two other, the first one uses almost the same tweet data as we use for training, and the second one is the CrowdFlower dataset annotated for emotions.\n\nIn the first one Wang et al. BIBREF21 downloaded over 5M tweets which included one of 131 emotional hashtags based on Parrott's three-level categorization of emotions in seven categories: joy, sadness, anger, love, fear, thankfulness, surprise.  In the second one, the reported results are from a paper by BIBREF33 in which they used maximum entropy classifier with bag of words model to classify various emotional datasets. Here we only report part of their result for CrowdFlower dataset that can be mapped to one of our seven labels. \n Question: What baseline is used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-0973641d55124166b27cc45a0353f776",
            "input": "Methodology ::: US dataset\nWe collected tweets associated to a dozen US mainstream news websites, i.e. most trusted sources described in BIBREF18, with the Streaming API, and we referred to Hoaxy API BIBREF16 for what concerns tweets containing links to 100+ US disinformation outlets.  Methodology ::: Italian dataset\nFor what concerns the Italian scenario we first collected tweets with the Streaming API in a 3-week period (April 19th, 2019-May 5th, 2019), filtering those containing URLs pointing to Italian official newspapers websites as described in BIBREF22; these correspond to the list provided by the association for the verification of newspaper circulation in Italy (Accertamenti Diffusione Stampa). We instead referred to the dataset provided by BIBREF23 to obtain a set of tweets, collected continuously since January 2019 using the same Twitter endpoint, which contain URLs to 60+ Italian disinformation websites. \n Question: What are the two large-scale datasets used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-77e4ac4510e74c2b9a19376a62bd762b",
            "input": "Table TABREF18 shows the Spearman correlation values of GM$\\_$KL model evaluated on the benchmark word similarity datasets: SL BIBREF20, WS, WS-R, WS-S BIBREF21, MEN BIBREF22, MC BIBREF23, RG BIBREF24, YP BIBREF25, MTurk-287 and MTurk-771 BIBREF26, BIBREF27, and RW BIBREF28.  Table TABREF19 shows the evaluation results of GM$\\_$KL model on the entailment datasets such as entailment pairs dataset BIBREF29 created from WordNet with both positive and negative labels, a crowdsourced dataset BIBREF30 of 79 semantic relations labelled as entailed or not and annotated distributionally similar nouns dataset BIBREF31 \n Question: What are the qualitative experiments performed on benchmark datasets?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-4b1e8f6e3fa746e69c5ffd3632eeb8a9",
            "input": "Work on VAE in BIBREF17 to learn acoustic embeddings conducted experiments using the TIMIT data set. The baseline performance for VAE based phone classification experiments in BIBREF17 report an accuracy of 72.2%. The re-implementation forming the basis for our work gave an accuracy of 72.0%, a result that was considered to provide a credible basis for further work. \n Question: What classification baselines are used for comparison?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-dd56f183a169452789711b81a1e6b459",
            "input": "To overcome the size issue of the student reflection dataset, we first explore the effect of incorporating domain transfer into a recent abstractive summarization model: pointer networks with coverage mechanism (PG-net)BIBREF0.  \n Question: What is the recent abstractive summarization method in this paper?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-9c459763718d471e837ca5357f0a9ee5",
            "input": "Based on the resulting 1.8 million lists of about 169,000 distinct user ids, we compute a topic model with INLINEFORM0 topics using Latent Dirichlet Allocation BIBREF3 . For each of the user ids, we extract the most probable topic from the inferred user id-topic distribution as cluster id. This results in a thematic cluster id for most of the user ids in our background corpus grouping together accounts such as American or German political actors, musicians, media websites or sports clubs (see Table TABREF17 ).  \n Question: What topic clusters are identified by LDA?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-3300e5bb5174456dbd9ae863dbe82d38",
            "input": "We use two datasets, NELL-One and Wiki-One which are constructed by BIBREF11 . \n Question: What datasets are used to evaluate the approach?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-3ba88a3e9e0a43eb8475e4e078df86e8",
            "input": " The annotation scheme was developed to capture three aspects of classroom talk that are theorized in the literature as important to discussion quality and learning opportunities: argumentation (the process of systematically reasoning in support of an idea), specificity (the quality of belonging or relating uniquely to a particular subject), and knowledge domain (area of expertise represented in the content of the talk). \n Question: how do they measure discussion quality?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-d4a28225dcba48db8a8e93d8d81e4aa8",
            "input": "They used 20-million-words data randomly sampled from the raw text released by the CoNLL 2017 Shared Task - Automatically Annotated Raw Texts and Word Embeddings BIBREF8, which is a combination of Wikipedia dump and common crawl.  For example, we compared the Latvian model by ELMoForManyLangs with a model we trained on a complete (wikidump + common crawl) Latvian corpus, which has about 280 million tokens. \n Question: How larger are the training sets of these versions of ELMo compared to the previous ones?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-0d473df3e4d74178bbb20d4c47664d11",
            "input": "The key idea behind our proposed interactive MRC (iMRC) is to restrict the document context that a model observes at one time. Concretely, we split a supporting document into its component sentences and withhold these sentences from the model. Given a question, the model must issue commands to observe sentences in the withheld set; we equip models with actions such as Ctrl+F (search for token) and stop for searching through partially observed documents. A model searches iteratively, conditioning each command on the input question and the sentences it has observed previously. Thus, our task requires models to `feed themselves' rather than spoon-feeding them with information. This casts MRC as a sequential decision-making problem amenable to reinforcement learning (RL). \n Question: Do they provide decision sequences as supervision while training models?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-0c807e7d4dd64fa6a35a95bcd853cd83",
            "input": "A causal attribution dataset is a collection of text pairs that reflect cause-effect relationships proposed by humans (for example, “virus causes sickness”). These written statements identify the nodes of the network (see also our graph fusion algorithm for dealing with semantically equivalent statements) while cause-effect relationships form the directed edges (“virus” $\\rightarrow $ “sickness”) of the causal attribution network. \n Question: What are causal attribution networks?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-f276fe26eba04903a30eb29e17868047",
            "input": "We evaluate our newly proposed models and related baselines in several low-resource settings across different languages with real, distantly supervised data with non-synthetic noise.  \n Question: How they evaluate their approach?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-10ffb776be074394a88cc865de5d9b8e",
            "input": "Because our conversion rules are interpretable, we identify shortcomings in both resources, using each as validation for the other. We were able to find specific instances of incorrectly applied UniMorph annotation, as well as specific instances of cross-lingual inconsistency in both resources. \n Question: Do they look for inconsistencies between different languages' annotations in UniMorph?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-4585b33cdc2a40798b9ff8ee7f4d9494",
            "input": "For the sake of simplicity, we focus our analysis on Airbnb listings from Manhattan, NY, during the time period of January 1, 2016, to January 1, 2017. The data provided to us contained information for roughly 40,000 Manhattan listings that were posted on Airbnb during this defined time period. For each listing, we were given information of the amenities of the listing (number of bathrooms, number of bedrooms …), the listing’s zip code, the host’s description of the listing, the price of the listing, and the occupancy rate of the listing.  \n Question: What is the size of the Airbnb?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-1a3af02a9da44872b4bdb47b564c5155",
            "input": "The gated mechanism learn the domain agnostic representations. They together control the information that has to flow through further fully connected output layer after max pooling. The effectiveness of gated architectures rely on the idea of training a gate with sole purpose of identifying a weightage. In the task of sentiment analysis this weightage corresponds to what weights will lead to a decrement in final loss or in other words, most accurate prediction of sentiment. In doing so, the gate architecture learns which words or n-grams contribute to the sentiment the most, these words or n-grams often co-relate with domain independent words. On the other hand the gate gives less weightage to n-grams which are largely either specific to domain or function word chunks which contribute negligible to the overall sentiment. This is what makes gated architectures effective at Domain Adaptation. We see that gated architectures almost always outperform recurrent, attention and linear models BoW, TFIDF, PV. This is largely because while training and testing on same domains, these models especially recurrent and attention based may perform better. However, for Domain Adaptation, as they lack gated structure which is trained in parallel to learn importance, their performance on target domain is poor as compared to gated architectures. As gated architectures are based on convolutions, they exploit parallelization to give significant boost in time complexity as compared to other models.  \n Question: Are there conceptual benefits to using GCNs over more complex architectures like attention?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-d41707b4893a41d493fd835b84f0ee87",
            "input": "This algorithm finds the optimal number by minimising a cost function based on the eigenvector structure of the word similarity matrix. We refer the reader to the relevant literature for further details.\n\nResults and Discussion \n Question: What are the six target languages?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-c79b13f4ea6749788457afbb1006604f",
            "input": "In this work, we build a novel regression model, based on linguistic, content, behavioral and topic features to detect Arabic Twitter bots to understand the impact of bots in spreading religious hatred in Arabic Twitter space.  \n Question: Do they propose a new model to better detect Arabic bots specifically?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-6cdcc63723b649b1836824991348e643",
            "input": "We defined the intents with guidance from queries collected using a scoping crowdsourcing task, which prompted crowd workers to provide questions and commands related to topic domains in the manner they would interact with an artificially intelligent assistant. We manually grouped data generated by scoping tasks into intents.  \n Question: How was the dataset annotated?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-3e342444b4dd4cd89650deed949ebc0c",
            "input": "Different from scaled dot-product attention, Gaussian-masked directional attention expects to pay attention to the adjacent characters of each positions and cast the localness relationship between characters as a fix Gaussian weight for attention. We assume that the Gaussian weight only relys on the distance between characters. \n Question: How does Gaussian-masked directional multi-head attention works?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-65a2ad044e3f4bb5b013714be4d394d4",
            "input": "Last but not least, ethics and fairness are important considerations, that deserve to be studied. In that sense, detection of individual and global bias should be prioritized in order to give useful feedbacks to practitioners. Furthermore we are considering using adversarial learning as in BIBREF33 in order to ensure fairness during the training process. \n Question: Do they analyze if their system has any bias?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-8be2a560052f4a508d088b1c82830a32",
            "input": "We call these techniques Recurrence over BERT (RoBERT) and Transformer over BERT (ToBERT). We observed that ToBERT outperforms RoBERT on pre-trained BERT features and fine-tuned BERT features for all our tasks.  \n Question: On top of BERT does the RNN layer work better or the transformer layer?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-1ed4859ce72e43e09907711a3575713a",
            "input": "We map each relation type $R(x,y)$ to at least one parametrized natural-language question $q_x$ whose answer is $y$ . \n Question: How is the input triple translated to a slot-filling task?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-337588b882af49ffa2bd39b0459aa345",
            "input": "In this section we describe how to evaluate and compare the outcomes of algorithms which assign relevance scores to words (such as LRP or SA) through intrinsic validation. Furthermore, we propose a measure of model explanatory power based on an extrinsic validation procedure. \n Question: Are the document vectors that the authors introduce evaluated in any way other than the new way the authors propose?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-1bf47a54b32b490cbb28a8c1eb020f00",
            "input": "We validate the performance of the proposed s2sL by providing the preliminary results obtained on two different tasks namely, Speech/Music discrimination and emotion classification. We considered the GTZAN Music-Speech dataset [17], consisting of 120 audio files (60 speech and 60 music), for task of classifying speech and music. Each audio file (of 2 seconds duration) is represented using a 13-dimensional mel-frequency cepstral coefficient (MFCC) vector, where each MFCC vector is the average of all the frame level (frame size of 30 msec and an overlap of 10 msec) MFCC vectors. It is to be noted that our main intention for this task is not better feature selection, but to demonstrate the effectiveness of our approach, in particular for low data scenarios. The standard Berlin speech emotion database (EMO-DB) [18] consisting of 535 utterances corresponding to 7 different emotions is considered for the task of emotion classification.  \n Question: Up to how many samples do they experiment with?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-2bf198c41d1041deb0d7c73f2bf599d0",
            "input": "We used a set of global network indicators which allow us to encode each network layer by a tuple of features. Then we simply concatenated tuples as to represent each multi-layer network with a single feature vector. We used the following global network properties:\n\nNumber of Strongly Connected Components (SCC): a Strongly Connected Component of a directed graph is a maximal (sub)graph where for each pair of vertices $u,v$ there is a path in each direction ($u\\rightarrow v$, $v\\rightarrow u$).\n\nSize of the Largest Strongly Connected Component (LSCC): the number of nodes in the largest strongly connected component of a given graph.\n\nNumber of Weakly Connected Components (WCC): a Weakly Connected Component of a directed graph is a maximal (sub)graph where for each pair of vertices $(u, v)$ there is a path $u \\leftrightarrow v$ ignoring edge directions.\n\nSize of the Largest Weakly Connected Component (LWCC): the number of nodes in the largest weakly connected component of a given graph.\n\nDiameter of the Largest Weakly Connected Component (DWCC): the largest distance (length of the shortest path) between two nodes in the (undirected version of) largest weakly connected component of a graph.\n\nAverage Clustering Coefficient (CC): the average of the local clustering coefficients of all nodes in a graph; the local clustering coefficient of a node quantifies how close its neighbours are to being a complete graph (or a clique). It is computed according to BIBREF28.\n\nMain K-core Number (KC): a K-core BIBREF13 of a graph is a maximal sub-graph that contains nodes of internal degree $k$ or more; the main K-core number is the highest value of $k$ (in directed graphs the total degree is considered).\n\nDensity (d): the density for directed graphs is $d=\\frac{|E|}{|V||V-1|}$, where $|E|$ is the number of edges and $|N|$ is the number of vertices in the graph; the density equals 0 for a graph without edges and 1 for a complete graph.\n\nStructural virality of the largest weakly connected component (SV): this measure is defined in BIBREF14 as the average distance between all pairs of nodes in a cascade tree or, equivalently, as the average depth of nodes, averaged over all nodes in turn acting as a root; for $|V| > 1$ vertices, $SV=\\frac{1}{|V||V-1|}\\sum _i\\sum _j d_{ij}$ where $d_{ij}$ denotes the length of the shortest path between nodes $i$ and $j$. This is equivalent to compute the Wiener's index BIBREF29 of the graph and multiply it by a factor $\\frac{1}{|V||V-1|}$. In our case we computed it for the undirected equivalent graph of the largest weakly connected component, setting it to 0 whenever $V=1$. \n Question: What are the global network features which quantify different aspects of the sharing process?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-5a2daf09c1204f79a9d9db7d60888646",
            "input": "So, we can impose the constraint that our model's representation of the input's syntax cannot contain this context-invariant information. This regularization is strictly preferable to allowing all aspects of word meaning to propagate into the input's syntax representation. Without such a constraint, all inputs could, in principle, be given their own syntactic categories. \n Question: Does having constrained neural units imply word meanings are fixed across different context?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-e87eafd804e5469995fd796d8bc42483",
            "input": "When compared with baselines that randomly choose one of the neighbors, or assume that the fact with the lowest score is incorrect, we see that outperforms both of these with a considerable gap, obtaining an accuracy of $42\\%$ and $55\\%$ in detecting errors. \n Question: Can this adversarial approach be used to directly improve model accuracy?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-4b1a3e2aa6a64980a28db5e3803de0b0",
            "input": "The utterance is concatenated with a special symbol marking the end of the sequence. We initialize our word embeddings using 300-dimensional GloVe BIBREF30 and then fine-tune them during training. \n Question: Do they use pretrained word vectors for dialogue context embedding?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-62c1174f757f437b915761e97510443b",
            "input": "Specifically, we design three tasks for seq2seq model pre-training, namely Sentence Reordering (SR), Next Sentence Generation (NSG), and Masked Document Generation (MDG).  \n Question: What is masked document generation?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-1db25bade9644427b79f112afe45f466",
            "input": "BERT: We implement two BERT-based baselines BIBREF51 for evidence identification. First, we train BERT on each query-policy sentence pair as a binary classification task to identify if the sentence is evidence for the question or not (Bert). We also experiment with a two-stage classifier, where we separately train the model on questions only to predict answerability. At inference time, if the answerable classifier predicts the question is answerable, the evidence identification classifier produces a set of candidate sentences (Bert + Unanswerable).\n\n \n Question: What type of neural model was used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-99f01dc4af2d47d18b23ac099c1fd257",
            "input": " Since we are interested in the zero-shot capabilities of our representation, we trained our sentiment analysis model only on the english IMDB Large Movie Review dataset and tested it on the chinese ChnSentiCorp dataset and german SB-10K BIBREF24 , BIBREF25 . A natural language inference task consists of two sentences; a premise and a hypothesis which are either contradictions, entailments or neutral. Learning a NLI task takes a certain nuanced understanding of language. Therefore it is of interest whether or not UG-WGAN captures the necessary linguistic features.  \n Question: Did they experiment with tasks other than word problems in math?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-1184a0528d9e48b98ec578dda44de15b",
            "input": "We embedded COSTRA sentences with LASER BIBREF15, the method that performed very well in revealing linear relations in BaBo2019. \n Question: Are some baseline models trained on this dataset?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-52834863a60e40f7b612c39a60a43510",
            "input": "We conduct all our experiments on 635hrs of audio data for 7 Indian languages collected from $\\textbf {All India Radio}$ news channel.  We collected and curated around 635Hrs of audio data for 7 Indian languages, namely Kannada, Hindi, Telugu, Malayalam, Bengali, and English. We collected the data from the All India Radio news channel where an actor will be reading news for about 5-10 mins. To cover many speakers for the dataset, we crawled data from 2010 to 2019. Since the audio is very long to train any deep neural network directly, we segment the audio clips into smaller chunks using Voice activity detector. Since the audio clips will have music embedded during the news, we use Inhouse music detection model to remove the music segments from the dataset to make the dataset clean and our dataset contains 635Hrs of clean audio which is divided into 520Hrs of training data containing 165K utterances and 115Hrs of testing data containing 35K utterances.  \n Question: How was the audio data gathered?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-e47b6a10fefd4f18b25fc3d0f667dffd",
            "input": "We show that by determining and integrating heterogeneous set of features from different modalities - aesthetic features from posted images (colorfulness, hue variance, sharpness, brightness, blurriness, naturalness), choice of profile picture (for gender, age, and facial expression), the screen name, the language features from both textual content and profile's description (n-gram, emotion, sentiment), and finally sociability from ego-network, and user engagement - we can reliably detect likely depressed individuals in a data set of 8,770 human-annotated Twitter users. \n Question: What is the source of the user interaction data? ",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-caee8653263d4f7abac5245202a89fdc",
            "input": "Bengali, Gujarati, Marathi, Malayalam and Tamil are the primary source languages, and translation from these to Hindi constitute the child tasks.  \n Question: On how many language pairs do they show that preordering assisting language sentences helps translation quality?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-f9d7a755e1bc43adb17edc47170f6048",
            "input": "The SemEval-2010 benchmark dataset BIBREF0 is composed of 244 scientific articles collected from the ACM Digital Library (conference and workshop papers). \n Question: how many articles are in the dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-34d4904cbfa14978b5cb80acafd53c14",
            "input": "This dataset was collected through Mechanical Turk using 100 simulated environments with a corresponding topological map and, to the best of our knowledge, it is the first of its kind for behavioral navigation.  We created a new dataset for the problem of following navigation instructions under the behavioral navigation framework of BIBREF5 . This dataset was created using Amazon Mechanical Turk and 100 maps of simulated indoor environments, each with 6 to 65 rooms. To the best of our knowledge, this is the first benchmark for comparing translation models in the context of behavioral robot navigation.\n\nAs shown in Table TABREF16 , the dataset consists of 8066 pairs of free-form natural language instructions and navigation plans for training. This training data was collected from 88 unique simulated environments, totaling 6064 distinct navigation plans (2002 plans have two different navigation instructions each; the rest has one). The dataset contains two test set variants:\n\nWhile the dataset was collected with simulated environments, no structure was imposed on the navigation instructions while crowd-sourcing data. Thus, many instructions in our dataset are ambiguous. Moreover, the order of the behaviors in the instructions is not always the same. For instance, a person said “turn right and advance” to describe part of a route, while another person said “go straight after turning right” in a similar situation. The high variability present in the natural language descriptions of our dataset makes the problem of decoding instructions into behaviors not trivial. See Appendix A of the supplementary material for additional details on our data collection effort.\n\n \n Question: How were the navigation instructions collected?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-3b3be431008c4557beda6ad458cd8ca7",
            "input": "Car-speak is abstract language that pertains to a car's physical attribute(s). In this instance the physical attributes that the term “fast” pertains to could be the horsepower, or it could be the car's form factor (how the car looks). However, we do not know exactly which attributes the term “fast” refers to. We train a series of classifiers in order to classify car-speak. We train three classifiers on the review vectors that we prepared in Section SECREF8. The classifiers we use are K Nearest Neighbors (KNN), Random Forest (RF), Support Vector Machine (SVM), and Multi-layer Perceptron (MLP) BIBREF13. \n Question: Is car-speak language collection of abstract features that classifier is later trained on?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-50d3339e31c142e9b4fe31ea82be2c87",
            "input": "We optimized our single-task baseline to get a strong baseline in order to exclude better results in multi-task learning in comparison to single-task learning only because of these two following points: network parameters suit the multi-task learning approach better and a better randomness while training in the multi-task learning. To exclude the first point, we tested different hyperparameters for the single-task baseline. We tested all the combinations of the following hyperparameter values: 256, 512, or 1024 as the sizes for the hidden states of the LSTMs, 256, 512, or 1024 as word embedding sizes, and a dropout of 30 %, 40 %, or 50 %. We used subword units generated by byte-pair encoding (BPE) BIBREF16 as inputs for our model. To avoid bad subword generation for the synthetic datasets, in addition to the training dataset, we considered the validation and test dataset for the generating of the BPE merge operations list. We trained the configurations for 14 epochs and trained every configuration three times. We chose the training with the best quality with regard to the validation F1-score to exclude disadvantages of a bad randomness. We got the best quality with regard to the F1-score with 256 as the size of the hidden states of the LSTMs, 1024 as word embedding size, and a dropout of 30 %. For the batch size, we used 64. \n Question: What are the strong baselines you have?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-07d25ce863234c90a0f2f4f213e0a79f",
            "input": "The DeepMine database consists of three parts. The first one contains fixed common phrases to perform text-dependent speaker verification. The second part consists of random sequences of words useful for text-prompted speaker verification, and the last part includes phrases with word- and phoneme-level transcription, useful for text-independent speaker verification using a random phrase (similar to Part4 of RedDots). DeepMine Database Parts ::: Part1 - Text-dependent (TD)\nThis part contains a set of fixed phrases which are used to verify speakers in text-dependent mode. Each speaker utters 5 Persian phrases, and if the speaker can read English, 5 phrases selected from Part1 of the RedDots database are also recorded.\n\nWe have created three experimental setups with different numbers of speakers in the evaluation set. Similar to the text-dependent case, three experimental setups with different number of speaker in the evaluation set are defined (corresponding to the rows in Table TABREF16). However, different strategy is used for defining trials: Depending on the enrollment condition (1- to 3-sess), trials are enrolled on utterances of all words from 1 to 3 different sessions (i.e. 3 to 9 utterances). Further, we consider two conditions for test utterances: seq test utterance with only 3 or 4 words and full test utterances with all words (i.e. same words as in enrollment but in different order). Based on the recording sessions, we created two experimental setups for speaker verification. In the first one, respondents with at least 17 recording sessions are included to the evaluation set, respondents with 16 sessions to the development and the rest of respondents to the background set (can be used as training data). In the second setup, respondents with at least 8 sessions are included to the evaluation set, respondents with 6 or 7 sessions to the development and the rest of respondents to the background set.  two experimental setups for speaker verification \n Question: what evaluation protocols are provided?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-404da2303bd042118a60d790948d6982",
            "input": "Analogous to word embeddings, sentence embeddings, e.g. the Universal Sentence Encoder BIBREF8 and Sentence-BERT BIBREF6, allow one to calculate the cosine similarity of various different sentences, as for instance the similarity of a question and the corresponding answer. The more appropriate a specific answer is to a given question, the stronger is its cosine similarity expected to be. When considering two opposite answers, it is therefore possible to determine a bias value:\n\nwhere $\\vec{q}$ is the vector representation of the question and $\\vec{a}$ and $\\vec{b}$ the representations of the two answers/choices. \n Question: How is moral bias measured?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-ab5626d8dfdf4e66b3a6e7ae0f24008a",
            "input": "Among all three pre-training tasks, SR works slightly better than the other two tasks (i.e., NSG and MDG). \n Question: Which of the three pretraining tasks is the most helpful?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-bd6a4221123046be825d3acbe31198ea",
            "input": "In the multiple-choice setting, which is the variety of question-answering (QA) that we focus on in this paper, there is also pragmatic reasoning involved in selecting optimal answer choices (e.g., while greenhouse effect might in some other context be a reasonable answer to the second question in Figure FIGREF1, global warming is a preferable candidate). \n Question: Do they focus on Reading Comprehension or multiple choice question answering?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-7392c950a9964bb89edd38b1864ca5d9",
            "input": "In a cooperation with researchers from the German Institute for International Educational Research we identified the following current controversial topics in education in English-speaking countries: (1) homeschooling, (2) public versus private schools, (3) redshirting — intentionally delaying the entry of an age-eligible child into kindergarten, allowing their child more time to mature emotionally and physically BIBREF51 , (4) prayer in schools — whether prayer in schools should be allowed and taken as a part of education or banned completely, (5) single-sex education — single-sex classes (males and females separate) versus mixed-sex classes (“co-ed”), and (6) mainstreaming — including children with special needs into regular classes. \n Question: Do they report results only on English data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-c1840d07f3b74ed4a132bd553ceba875",
            "input": "In the future, we aim to use the corpus for annotation projects such as parts-of-speech tagging, named entity recognition. Furthermore, the generated word embeddings will be utilized for the automatic construction of Sindhi WordNet. \n Question: Are trained word embeddings used for any other NLP task?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-5292089326a647f0bf07f698c06adad3",
            "input": "Of the 144 schemas in the collection at WinogradSchemas there are 33 that can plausibly be translated this way. \n Question: Did they collect their own datasets?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-7503bed6fb4f4682ba456a2a6ebef091",
            "input": "The real-time tweets scores were calculated in the same way as the historical data and summed up for a minute and sent to the machine learning model with the Bitcoin price in the previous minute and the rolling average price. It predicted the next minute's Bitcoin price from the given data. After the actual price arrived, the RMS value was calculated and the machine learning model updated itself to predict with better understanding the next value. \n Question: What experimental evaluation is used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-42875494f7c6480eb0479da796737d44",
            "input": "The following data sources were used to train the RNN-T and associated RNN-LMs in this study.\n\nSource-domain baseline RNN-T: approximately 120M segmented utterances (190,000 hours of audio) from YouTube videos, with associated transcripts obtained from semi-supervised caption filtering BIBREF28.\n\nSource-domain normalizing RNN-LM: transcripts from the same 120M utterance YouTube training set. This corresponds to about 3B tokens of the sub-word units used (see below, Section SECREF30).\n\nTarget-domain RNN-LM: 21M text-only utterance-level transcripts from anonymized, manually transcribed audio data, representative of data from a Voice Search service. This corresponds to about 275M sub-word tokens.\n\nTarget-domain RNN-T fine-tuning data: 10K, 100K, 1M and 21M utterance-level {audio, transcript} pairs taken from anonymized, transcribed Voice Search data. These fine-tuning sets roughly correspond to 10 hours, 100 hours, 1000 hours and 21,000 hours of audio, respectively. \n Question: How much training data is used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-f90f3809eea646d484614a93d1f72902",
            "input": "Furthermore, the basic seq2seq assumes a strict order between generated tokens, but in reality, we should not severely punish the model when it predicts the correct tokens in the wrong order. \n Question: Do they impose any grammatical constraints over the generated output?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-0ece99348a7f49b19f0574532f1991dd",
            "input": "As of January 2020, the latest CoVo 2019-06-12 release includes 29 languages. CoVoST is currently built on that release and covers the following 11 languages: French, German, Dutch, Russian, Spanish, Italian, Turkish, Persian, Swedish, Mongolian and Chinese. Common Voice BIBREF10 is a crowdsourcing speech recognition corpus with an open CC0 license. Contributors record voice clips by reading from a bank of donated sentences \n Question: How was the dataset collected?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-e788a07a1e864d5cae33e86cb971718e",
            "input": "We compare the performance of our Chinese word embedding vectors in the task of synonym discovery against another set of embedding vectors that was constructed with a co-occurrence model BIBREF1 . Our embeddings also proved to perform better than our benchmark dataset. \n Question: Does this approach perform better than context-based word embeddings?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-67d454e10c8544d4bd9bed92fa2f1a22",
            "input": "On Twitter we can see results that are consistent with the RCV results for the left-to-center political spectrum. The exception, which clearly stands out, is the right-wing groups ENL and EFDD that seem to be the most cohesive ones. This is the direct opposite of what was observed in the RCV data. We speculate that this phenomenon can be attributed to the fact that European right-wing groups, on a European but also on a national level, rely to a large degree on social media to spread their narratives critical of European integration. \n Question: Do the authors mention any possible confounds in their study?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-84d1146aea6e4d319793de0355a7186f",
            "input": "The experiment settings from this paper and evaluation protocol for the Mboshi corpus (Boundary F-scores using the ZRC speech reference) are the same from BIBREF8. Table presents the results for bilingual UWS and multilingual leveraging. For the former, we reach our best result by using as aligned information the French, the original aligned language for this dataset. Languages closely related to French (Spanish and Portuguese) ranked better, while our worst result used German. English also performs notably well in our experiments. We believe this is due to the statistics features of the resulting text. We observe in Table that the English portion of the dataset contains the smallest vocabulary among all languages. Since we train our systems in very low-resource settings, vocabulary-related features can impact greatly the system's capacity to language-model, and consequently the final quality of the produced alignments. Even in high-resource settings, it was already attested that some languages are more difficult to model than others BIBREF9. \n Question: Is the model evaluated against any baseline?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-55464a235060477e9e59fdaafa9ba797",
            "input": "Deep convolutional neural networks (CNNs) with 2D convolutions and small kernels BIBREF1, have achieved state-of-the-art results for several speech recognition tasks BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6. r Models: Our baseline CNN model BIBREF21 consists of 15 convolutional and one fully-connected layer. \n Question: Is model compared against state of the art models on these datasets?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-7b23f61176f040e5820226cd20dc07e5",
            "input": "This work analyzes the understanding of pretrained language models of factual and commonsense knowledge stored in negated statements. To this end, we introduce the negated LAMA dataset. We construct it by simply inserting negation elements (e.g., “not”) in LAMA cloze statement (e.g., “The theory of relativity was not developed by [MASK].”). \n Question: How did they extend LAMA evaluation framework to focus on negation?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-89ab976fbeed43229d0228770ee98374",
            "input": "We started from an initial corpus that did not have revisions, and kept adding new documents and revising existing documents. The revision operations we used were deletion, addition and replacement of words, sentences, paragraphs, section names and document titles. The addition of words, ..., section names, and new documents were pulled from the Wikipedia abstracts. We generated six data sets using different random seeds, and each data set contained six corpora (Corpus 0 - 5). \n Question: What are simulated datasets collected?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-3ef2f9d890814cd1b14c71eacd9126f0",
            "input": "Ablation Study\nTo further investigate the efficacy of the key components in our framework, namely, THA and STN, we perform ablation study as shown in the second block of Table TABREF39 . The results show that each of THA and STN is helpful for improving the performance, and the contribution of STN is slightly larger than THA. “OURS w/o THA & STN” only keeps the basic bi-linear attention. Although it performs not bad, it is still less competitive compared with the strongest baseline (i.e., CMLA), suggesting that only using attention mechanism to distill opinion summary is not enough. After inserting the STN component before the bi-linear attention, i.e. “OURS w/o THA”, we get about 1% absolute gains on each dataset, and then the performance is comparable to CMLA. By adding THA, i.e. “OURS”, the performance is further improved, and all state-of-the-art methods are surpassed. \n Question: Do they explore how useful is the detection history and opinion summary?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-89978d398809429b91a757b83041455e",
            "input": "We evaluate the proposed approach on the Chinese social media text summarization task, based on the sequence-to-sequence model. Large-Scale Chinese Short Text Summarization Dataset (LCSTS) is constructed by BIBREF1 . The dataset consists of more than 2.4 million text-summary pairs in total, constructed from a famous Chinese social media microblogging service Weibo.  \n Question: Are results reported only for English data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-788ca5623e9349e7a1f5f7f7842606a2",
            "input": "The Flickr30K dataset BIBREF0 is a collection of over 30,000 images with 5 crowdsourced descriptions each. It is commonly used to train and evaluate neural network models that generate image descriptions (e.g. BIBREF2 ). This paper aims to give an overview of linguistic bias and unwarranted inferences resulting from stereotypes and prejudices. I will build on earlier work on linguistic bias in general BIBREF3 , providing examples from the Flickr30K data, and present a taxonomy of unwarranted inferences. Finally, I will discuss several methods to analyze the data in order to detect biases. \n Question: What is the size of the dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-d8300ca9c9fe4e47930d0a9705611100",
            "input": "Using our methodology, we tagged 10,000 Arabic tweet dataset for offensiveness, where offensive tweets account for roughly 19% of the tweets. Further, we labeled tweets as vulgar or hate speech.  \n Question: How many tweets are in the dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-bdcda166b4944884b30abb729d1987cf",
            "input": "There are various possible extensions for this work. For example, using all frames assigned to a phone, rather than using only the middle frame. \n Question: Do they propose any further additions that could be made to improve generalisation to unseen speakers?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-4f895e4b40db45568eccd87c4a035e12",
            "input": "We collect data by using the Twitter API for saved data, available for public use. For our experiments we collect 3200 tweets filtered by keywords like “fire”, “earthquake”, “theft”, “robbery”, “drunk driving”, “drunk driving accident” etc.  \n Question: Do the tweets come from any individual?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-740572e1594047a5b5e0454110f55113",
            "input": "We collect a novel dataset of 230K+ recipe texts and 1M+ user interactions (reviews) over 18 years (2000-2018) from Food.com. \n Question: Where do they get the recipes from?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-6defba2409394f12852a481368392f82",
            "input": "Each dataset consists of a collection of records with one QA problem per record. For each record, we include some question text, a context document relevant to the question, a set of candidate solutions, and the correct solution. The context document for each record consists of a list of ranked and scored pseudodocuments relevant to the question. Several baselines rely on the retrieved context to extract the answer to a question. For these, we refer to the fraction of instances for which the correct answer is present in the context as Search Accuracy. Naturally, the search accuracy increases as the context size increases, however at the same time reading performance decreases since the task of extracting the answer becomes harder for longer documents. \n Question: Which retrieval system was used for baselines?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-04ec241d6c664a2fa26e74d80cf1a45a",
            "input": "Following previous work BIBREF13, BIBREF12, BIBREF10, BIBREF17, BIBREF9, BIBREF7, we choose SemCor3.0 as training corpus, which is the largest corpus manually annotated with WordNet sense for WSD. \n Question: Is SemCor3.0 reflective of English language data in general?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-8a991a419f604b05bda5f1681143b5b9",
            "input": "In all our experiments, we used the out-of-the-box BERT models without any task-specific fine-tuning. \n Question: How does their model differ from BERT?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-a6a9f618255541d994b7f34d7ed5294f",
            "input": "LiLi should have the following capabilities: \n Question: What are the components of the general knowledge learning engine?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-3fadd5c5931b46b395614ae7bd2faa5e",
            "input": "We label each review vector with the car it reviews.  \n Question: What are labels in car speak language dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-75dedd6a75364d9da2cabcc17f3967b4",
            "input": "What is the impact of pre-trained representations with less transcribed data? In order to get a better understanding of this, we train acoustic models with different amounts of labeled training data and measure accuracy with and without pre-trained representations (log-mel filterbanks). The pre-trained representations are trained on the full Librispeech corpus and we measure accuracy in terms of WER when decoding with a 4-gram language model. Figure shows that pre-training reduces WER by 32% on nov93dev when only about eight hours of transcribed data is available. Pre-training only on the audio data of WSJ ( WSJ) performs worse compared to the much larger Librispeech ( Libri). This further confirms that pre-training on more data is crucial to good performance. \n Question: Do they explore how much traning data is needed for which magnitude of improvement for WER? ",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-ece72de3370742f3b5064325ad20cdee",
            "input": "With BERT, two fully unsupervised tasks are performed. The Masked Language Model and the Next Sentence Prediction (NSP).\n\nFor this study, the NSP is used as a proxy for the relevance of response. \n Question: was bert used?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-3bdbaf6610b34d66841296f8266a9f89",
            "input": "Named Entity Recognition (NER) in the Biomedical domain usually includes recognition of entities such as proteins, genes, diseases, treatments, drugs, etc.  \n Question: What is NER?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-cfb1f5aa25334b88a74526f3e2cbaffe",
            "input": "Our model differs by learning the subword vectors and resulting representation jointly as weighted factorization of a word-context co-occurrence matrix is performed. \n Question: Which matrix factorization methods do they use?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-94901a88a04140c3bb41864a103a9424",
            "input": "The initial analyses of the embedding matrices for both the UK and France revealed that in general, words were grouped by context or influence on the electricity consumption. For instance, we observed that winter words were together and far away from summer ones.  \n Question: Is there any example where geometric property is visible for context similarity between words?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-d455953f356b45ca9acb8a9ca5342c6c",
            "input": "We learn that a number of factors can influence the performance of adversarial attacks, including architecture of the classifier, sentence length and input domain. \n Question: What other factors affect the performance?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-7adef170c5974c37b88aec3ffd9c7140",
            "input": "Table TABREF44 shows average results of our automatic and human evaluations. \n Question: How big is the difference in performance between proposed model and baselines?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-647c0b28d4a740998c15da8ef6db538d",
            "input": "In this work, we use MLP (modified to handle our data representation) as the base classifier. \n Question: Which classification algorithm do they use for s2sL?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-0978dccacf4c47548d31a0929bfaa1ff",
            "input": "The Zurich Cognitive Language Processing Corpus (ZuCo) 2.0, including raw and preprocessed eye-tracking and electroencephalography (EEG) data of 18 subjects, as well as the recording and preprocessing scripts, is publicly available. It contains physiological data of each subject reading 739 English sentences from Wikipedia (see example in Figure FIGREF1). During the recording session, the participants read 739 sentences that were selected from the Wikipedia corpus provided by culotta2006integrating.  \n Question: What kind of sentences were read?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-ea7bd39d97a74704855426edb2ce8052",
            "input": "Data was collected from a 10% uniform sample of Twitter posts made during 2013, specifically the Gardenhose API. Control documents were also selected. These documents did not contain any of `caused', `causing', or `causes', nor any bidirectional words, and are further matched temporally to obtain the same number of control documents as causal documents in each fifteen-minute period during 2013. Control documents were otherwise selected randomly; causal synonyms may be present. \n Question: What is the source of the \"control\" corpus?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-fa746913dd46495fbb10d2a07c0543ba",
            "input": "We propose a simple and practical human evaluation for evaluating text summarization, where the summary is evaluated against the source content instead of the reference. It handles both of the problems of paraphrasing and lack of high-quality reference.  To avoid the deficiencies, we propose a simple human evaluation method to assess the semantic consistency. Each summary candidate is evaluated against the text rather than the reference. If the candidate is irrelevant or incorrect to the text, or the candidate is not understandable, the candidate is labeled bad. Otherwise, the candidate is labeled good. Then, we can get an accuracy of the good summaries. The proposed evaluation is very simple and straight-forward. It focuses on the relevance between the summary and the text. The semantic consistency should be the major consideration when putting the text summarization methods into practice, but the current automatic methods cannot judge properly. For detailed guidelines in human evaluation, please refer to Appendix SECREF6 .  \n Question: What human evaluation method is proposed?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-48d25fbc9c224f379a4b9939d1279423",
            "input": "Traditional text-to-speech (TTS) systems are composed of complex pipelines BIBREF0 , these often include accoustic frontends, duration model, acoustic prediction model and vocoder models. Neural text-to-speech systems have garnered large research interest in the past 2 years. The first to fully explore this avenue of research was Google's tacotron BIBREF1 system. The architecture of our model utilizes RNN-based Seq2Seq model for generating mel spectrogram from text. The architecture is similar to that of Tacotron 2 BIBREF4 In the original Tacotron 2, the attention mechanism used was location sensitive attention BIBREF12 combined the original additive Seq2Seq BIBREF7 Bahdanau attention.\n\nWe propose to replace this attention with the simpler query-key attention from transformer model Following the logic above, we utilize a similar method from BIBREF6 that adds an additional guided attention loss to the overall loss objective, which acts to help the attention mechanism become monotoic as early as possible. As seen from FIGREF24 , an attention loss mask, INLINEFORM0 , is created applies a loss to force the attention alignment, INLINEFORM1 , to be nearly diagonal. \n Question: Which modifications do they make to well-established Seq2seq architectures?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-0878d58795a74fcabe89cb49bf7d7416",
            "input": "Our baseline is a GRU network for each of the three tasks. \n Question: Are the models compared to some baseline models?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-06fab34354df429cb23f39472de4c0fa",
            "input": "INSIGHT 1: Political handles are more likely to engage in profile changing behavior as compared to their followers.  We analyze the trends in Figure FIGREF12 and find that the political handles do not change their usernames at all. This is in contrast to the trend in Figure FIGREF15 where we see that there are a lot of handles that change their usernames multiple times.  INSIGHT 3: Political handles tend to make new changes related to previous attribute values. However, the followers make comparatively less related changes to previous attribute values. \n Question: How do profile changes vary for influential leads and their followers over the social movement?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-afe0d8ee10f84acda5e69239dfa94626",
            "input": "The output of a system with the target words in the predicted order is compared to the gold ranking of the DURel data set. As the metric to assess how well the model's output fits the gold ranking Spearman's $\\rho $ was used. The higher Spearman's rank-order correlation the better the system's performance. \n Question: How is evaluation performed?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-b8fad4acf48e45eb8141d211ef48b7eb",
            "input": "We collect human judgments on Amazon Mechanical Turk via ParlAI BIBREF18. \n Question: Do they use crowdsourcing to collect human judgements?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-62b7f545af3b40a7b8b7169e497c6747",
            "input": "We evaluate the proposed transfer learning techniques in two non-English language pairs of WMT 2019 news translation tasks: French$\\rightarrow $German and German$\\rightarrow $Czech. \n Question: Are experiments performed with any other pair of languages, how did proposed method perform compared to other models?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-729505545a09421097bc1897c0c7e99c",
            "input": "The corpus used for sentiment analysis is the IMDb dataset of movie reviews by BIBREF11 while that for NER is Groningen Meaning Bank (GMB) by BIBREF12, containing 47,959 sentence samples. \n Question: What sentiment analysis dataset is used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-632f9a06f85f4da0906d4e01bcc433ee",
            "input": "Finally, a comparison was made between the Skip-gram model W10N20 obtained at the 50th epoch and the other two W2V in Italian present in the literature (BIBREF9 and BIBREF10). The first test (Table TABREF15) was performed considering all the analogies present, and therefore evaluating as an error any analogy that was not executable (as it related to one or more words absent from the vocabulary).\n\nAs it can be seen, regardless of the metric used, our model has significantly better results than the other two models, both overall and within the two macro-areas. \n Question: Are the word embeddings evaluated?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-37d1c604955c4ab3887dd2d25c818e8b",
            "input": "The respondents in our MTurk survey had most difficulties recognizing reviews of category $(b=0.3, \\lambda=-5)$, where true positive rate was $40.4\\%$, while the true negative rate of the real class was $62.7\\%$. The precision were $16\\%$ and $86\\%$, respectively. The class-averaged F-score is $47.6\\%$, which is close to random. Detailed classification reports are shown in Table~\\ref{table:MTurk_sub} in Appendix. Our MTurk-study shows that \\emph{our NMT-Fake reviews pose a significant threat to review systems}, since \\emph{ordinary native English-speakers have very big difficulties in separating real reviews from fake reviews}. We use the review category $(b=0.3, \\lambda=-5)$ for future user tests in this paper, since MTurk participants had most difficulties detecting these reviews. We refer to this category as NMT-Fake* in this paper. The classifier is very effective in detecting reviews that humans have difficulties detecting. For example, the fake reviews MTurk users had most difficulty detecting ($b=0.3, \\lambda=-5$) are detected with an excellent 97\\% F-score. \n Question: Does their detection tool work better than human detection?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-3afae0a6c7024c46a3909935513713d5",
            "input": "The minimal change operation substantially changed the meaning of the sentence, and yet the embedding of the transformation lies very closely to the original sentence (average similarity of 0.930). \n Question: Do they do any analysis of of how the modifications changed the starting set of sentences?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-58aa608846e34c7f873c7ab62475871a",
            "input": "Irony Classifier: We implement a CNN classifier trained with our irony dataset. Sentiment Classifier for Irony: We first implement a one-layer LSTM network to classify ironic sentences in our dataset into positive and negative ironies. Sentiment Classifier for Non-irony: Similar to the training process of the sentiment classifier for irony, we first implement a one-layer LSTM network trained with the dataset for the sentiment analysis of common twitters to classify the non-ironies into positive and negative non-ironies. In this section, we describe some additional experiments on the transformation from ironic sentences to non-ironic sentences. \n Question: What experiments are conducted?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-07689157245745feae103391727d2669",
            "input": "We focused evaluation over a small but diversified dataset composed by 10 YouTube videos in the English language in the news context. The selected videos cover different topics like technology, human rights, terrorism and politics with a length variation between 2 and 10 minutes.  \n Question: What kind of Youtube video transcripts did they use?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-6da9690afd8745198b3ae1530da2db30",
            "input": "We plan to apply semantic slot scaffold to news summarization. Specifically, we can annotate the critical entities such as person names or location names to ensure that they are captured correctly in the generated summary. We also plan to collect a human-human dialog dataset with more diverse human-written summaries. \n Question: Is it expected to have speaker role, semantic slot and dialog domain annotations in real world datasets?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-6e64bde43cca47d68cd2e7a7724ae72d",
            "input": "In sub-task C the goal is to classify the target of the offensive language. Only posts labeled as targeted insults (TIN) in sub-task B are considered in this task BIBREF17 . Samples are annotated with one of the following:\n\nIndividual (IND): Posts targeting a named or unnamed person that is part of the conversation. In English this could be a post such as @USER Is a FRAUD Female @USER group paid for and organized by @USER. In Danish this could be a post such as USER du er sku da syg i hoved. These examples further demonstrate that this category captures the characteristics of cyberbullying, as it is defined in section \"Background\" .\n\nGroup (GRP): Posts targeting a group of people based on ethnicity, gender or sexual orientation, political affiliation, religious belief, or other characteristics. In English this could be a post such as #Antifa are mentally unstable cowards, pretending to be relevant. In Danish this could be e.g. Åh nej! Svensk lorteret!\n\nOther (OTH): The target of the offensive language does not fit the criteria of either of the previous two categories. BIBREF17 . In English this could be a post such as And these entertainment agencies just gonna have to be an ass about it.. In Danish this could be a post such as Netto er jo et tempel over lort. \n Question: How many categories of offensive language were there?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-fd6ab57c59884f02b21019727d5a98fb",
            "input": "We use Waseem and Hovy BIBREF5, Davidson et al. BIBREF9, and Waseem et al. BIBREF10 as baselines and compare the results with our different fine-tuning strategies using pre-trained BERTbase model.  \n Question: What baseline is used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-7d219c0999884f5785c226a81997f4eb",
            "input": "The question generation model provides each candidate answer with a score by measuring semantic relevance between the question and the generated question based on the semantics of the candidate answer.  \n Question: Where is a question generation model used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-f53f4124b6a6438493695f20e7fa6468",
            "input": "One of the solutions that has been proposed for mitigating gender bias on the word embedding level is Counterfactual Data Augmentation (CDA) BIBREF25. We apply this method by augmenting our dataset with a copy of every dialogue with gendered words swapped using the gendered word pair list provided by BIBREF21. \n Question: How does counterfactual data augmentation aim to tackle bias?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-fe7c3febfd4e434ba694436e012a5d1a",
            "input": "To capture this interesting property, we propose a new tagging scheme consisting of three tags, namely { INLINEFORM0 }.\n\nINLINEFORM0 tag indicates that the current word appears before the pun in the given context.\n\nINLINEFORM0 tag highlights the current word is a pun.\n\nINLINEFORM0 tag indicates that the current word appears after the pun. \n Question: What is the tagging scheme employed?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-1df0171020af4f6ab5ba448b07962dfe",
            "input": "Furthermore, we do not restrict ourselves to a test set of sequences of fixed lengths during testing. Rather, we exhaustively enumerate all the sequences in a language by their lengths and then go through the sequences in the test set one by one until our network errs $k$ times, thereby providing a more fine-grained evaluation criterion of its generalization capabilities. To study the effect of various length distributions on the learning capability and speed of LSTM models, we experimented with four discrete probability distributions supported on bounded intervals (Figure 2 ) to sample the lengths of sequences for the languages.  \n Question: Are the unobserved samples from the same distribution as the training data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-07f868d7ad4343a2abfafe3bf79d62ae",
            "input": "Evaluation Metrics. To evaluate the strategy formulation ability, we introduce a measure called Coverage( INLINEFORM0 ), defined as the fraction of total query data instances, for which LiLi has successfully formulated strategies that lead to winning. If LiLi wins on all episodes for a given dataset, INLINEFORM1 is 1.0. To evaluate the predictive performance, we use Avg. MCC and avg. +ve F1 score. \n Question: What metrics are used to establish that this makes chatbots more knowledgeable and better at learning and conversation? ",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-e5aad250a2964750bd24bd34f551d4e8",
            "input": "The final section of the paper, then, discusses how insights gained from technologically observing opinion dynamics can inform conceptual modelling efforts and approaches to on-line opinion facilitation. As such, the paper brings into view and critically evaluates the fundamental conceptual leap from machine-guided observation to debate facilitation and intervention. \n Question: Does the paper report the results of previous models applied to the same tasks?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-70f522d7376243008d00d71210c74705",
            "input": "For experiments that use parallel data to initialize foreign specific parameters, we use the same datasets in the work of BIBREF6. Specifically, we use United Nations Parallel Corpus BIBREF18 for en-ru, en-ar, en-zh, and en-fr. We collect en-hi parallel data from IIT Bombay corpus BIBREF19 and en-vi data from OpenSubtitles 2018. \n Question: What datasets are used for evaluation?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-a6c9311a85fd4c1f8039d58707c8600b",
            "input": "For generating a poem from images we use an existing actor-critic architecture BIBREF1. For Shakespearizing modern English texts, we experimented with various types of sequence to sequence models. We use a sequence-to-sequence model which consists of a single layer unidrectional LSTM encoder and a single layer LSTM decoder and pre-trained retrofitted word embeddings shared between source and target sentences. Since a pair of corresponding Shakespeare and modern English sentences have significant vocabulary overlap we extend the sequence-to-sequence model mentioned above using pointer networks BIBREF11 that provide location based attention and have been used to enable copying of tokens directly from the input. \n Question: What models are used for painting embedding and what for language style transfer?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-f9170fc6ec084072bb53351302074f4f",
            "input": "We compare eight different models in our experiments. Four of them have a Projected Layer (see Fig. FIGREF2), while the others do not have, and this is the only difference between these two groups of our models. So, we actually include four models in our experiments (having a projected layer or not). Firstly, LastStateRNN is the classic RNN model, where the last state passes through an MLP and then the LR Layer estimates the corresponding probability. In contrast, in the AvgRNN model we consider the average vector of all states that come out of the cells. The AttentionRNN model is the one that it has been presented in BIBREF9. Moreover, we introduce the MultiAttentionRNN model for the harassment language detection, which instead of one attention, it includes four attentions, one for each category. \n Question: What are the different variations of the attention-based approach which are examined?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-62e2a2d47faf41c6ba3f9f5b357b0404",
            "input": "The dataset from Twitter that we are using in our work, consists of a train set, a validation set and a test set. It was published for the \"First workshop on categorizing different types of online harassment languages in social media\". The whole dataset is divided into two categories, which are harassment and non-harassment tweets. Moreover, considering the type of the harassment, the tweets are divided into three sub-categories which are indirect harassment, sexual and physical harassment. \n Question: What types of online harassment are studied?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-92f44e98d8274ecd8972349b6ffd713f",
            "input": "Each tweet is annotated as no evidence of depression (e.g., “Citizens fear an economic depression\") or evidence of depression (e.g., “depressed over disappointment\"). If a tweet is annotated evidence of depression, then it is further annotated with one or more depressive symptoms, for example, depressed mood (e.g., “feeling down in the dumps\"), disturbed sleep (e.g., “another restless night\"), or fatigue or loss of energy (e.g., “the fatigue is unbearable\") BIBREF10 . \n Question: How is the dataset annotated?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-de459e4db2634f16804ec27e62811f7a",
            "input": "For the human evaluation, we follow the standard approach in evaluating machine translation systems BIBREF23 , as used for question generation by BIBREF9 . We asked three workers to rate 300 generated questions between 1 (poor) and 5 (good) on two separate criteria: the fluency of the language used, and the relevance of the question to the context document and answer. \n Question: What human evaluation metrics were used in the paper?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-0cdf786d06634c8bb30984650d8a8d84",
            "input": "First, it provides an annotation interface that allows users to define content elements, upload documents, and annotate documents Our platform is able to ingest documents in a variety of formats, including PDFs and Microsoft Word, and converts these formats into plain text before presenting them to the annotators. \n Question: What type of documents are supported by the annotation platform?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-febd7fe8a7f4449ebbe9a9ea9813f7d1",
            "input": "We use a default train-validation-test split of 70-15-15 for each dataset, and use all 4 datasets (BF, BA, SFU and Sherlock). \n Question: Which multiple datasets did they train on during joint training?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-e1dbaac7aa634fc3ba24b659fd4f1596",
            "input": "Table TABREF23 shows the Pearson, Spearman and Kendall correlation of Rouge and Sera, with pyramid scores. Interestingly, we observe that many variants of Rouge scores do not have high correlations with human pyramid scores. The lowest F-score correlations are for Rouge-1 and Rouge-L (with INLINEFORM0 =0.454). Weak correlation of Rouge-1 shows that matching unigrams between the candidate summary and gold summaries is not accurate in quantifying the quality of the summary. \n Question: What different correlations result when using different variants of ROUGE scores?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-cb2b2cbbf9f1405d87b4b7ffbe793d5c",
            "input": "This decoding approach closely follows Algorithm SECREF7 , but along with soft back pointers, we also compute hard back pointers at each time step. After computing all the relevant quantities like model score, loss etc., we follow the hard backpointers to obtain the best sequence INLINEFORM0 . \n Question: Do they compare partially complete sequences (created during steps of beam search) to gold/target sequences?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-f0ffcef8238e45b99b720ad0f34efc9c",
            "input": "We recruited 102 pairs of participants from Amazon Mechanical Turk and randomly assigned speaker and listener roles \n Question: Was this experiment done in a lab?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-310ed850e8ab4789bf5bbe21ac378925",
            "input": "There are several caveats in our work: first, tweet sentiment is rarely binary (this work could be extended to a multinomial or continuous model). Second, our results are constrained to Twitter users, who are known to be more negative than the general U.S. population BIBREF9 . Third, we do not take into account the aggregate effects of continued natural disasters over time. Going forward, there is clear demand in discovering whether social networks can indicate environmental metrics in a “nowcasting\" fashion. As climate change becomes more extreme, it remains to be seen what degree of predictive power exists in our current model regarding climate change sentiments with regards to natural disasters. \n Question: Do the authors mention any confounds to their study?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-76d2867d3298491dac3057582fa05b71",
            "input": "Eventually, we have a balanced testing dataset, where each term-sense pair has around 15 samples for testing (on average, each pair has 14.56 samples and the median sample number is 15), and a comparison with training dataset is shown in Figure FIGREF11. Due to the difficulty for collecting the testing dataset, we decided to only collect for a random selection of 30 terms.  \n Question: How big is dataset for testing?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-4d830ee9db004706b14000d318e16f8f",
            "input": "Clustering was performed separately for each specialty of doctors. We also examined the distribution of doctors' IDs in the obtained clusters. It turned out that some clusters covered almost exactly descriptions written by one doctor. This situation took place in the specialties where clusters are separated with large margins (e.g. psychiatry, pediatrics, cardiology). \n Question: Do they explore similarity of texts across different doctors?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-bb060384c16040dda841f69983ef7d21",
            "input": "The task of humor identification in social media texts is analyzed as a classification problem and several machine learning classification models are used. \n Question: What experiments were carried out on the corpus?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-c9a4b0a2d69c4c7e9567e30a90e7e942",
            "input": "In this study, we concentrate our effort on re-assessing keyphrase extraction performance on three increasingly sophisticated levels of document preprocessing described below. \n Question: what keyphrase extraction models were reassessed?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-e5d11ff8297b47cf92a57e1b5e45c087",
            "input": " For the embeddings, we relied on $AraVec$ BIBREF30 for Arabic, FastText BIBREF31 for French, and Word2vec Google News BIBREF32 for English .  \n Question: What monolingual word representations are used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-f9f0019384154397a7691dbfb9105c28",
            "input": "An extrinsic evaluation was carried out on the task of Open IE BIBREF7. \n Question: Is the semantic hierarchy representation used for any task?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-11e992157ae4437a8cefe8bc27c0f7d0",
            "input": "To analyse the results we chose to use the test provided by BIBREF10, which consists of $19\\,791$ analogies divided into 19 different categories: 6 related to the “semantic\" macro-area (8915 analogies) and 13 to the “syntactic\" one (10876 analogies). All the analogies are composed by two pairs of words that share a relation, schematized with the equation: $a:a^{*}=b:b^{*}$ (e.g. “man : woman = king : queen\"); where $b^{*}$ is the word to be guessed (“queen\"), $b$ is the word coupled to it (“king\"), $a$ is the word for the components to be eliminated (“man\"), and $a^{*}$ is the word for the components to be added (“woman\"). \n Question: Are the word embeddings tested on a NLP task?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-90b2a71af28342e3a699bf91eb15d534",
            "input": "For answer retrieval, a dataset is created by INLINEFORM4 , which gives INLINEFORM5 accuracy and INLINEFORM6 coverage, respectively. \n Question: Do they employ their indexing-based method to create a sample of a QA Wikipedia dataset?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-805ab17e61824ae4b816f472f0740073",
            "input": "We quantify the use of these strategies by comparing the airtime debaters devote to talking points . For a side INLINEFORM0 , let the self-coverage INLINEFORM1 be the fraction of content words uttered by INLINEFORM2 in round INLINEFORM3 that are among their own talking points INLINEFORM4 ; and the opponent-coverage INLINEFORM5 be the fraction of its content words covering opposing talking points INLINEFORM6 .  We use all conversational features discussed above. For each side INLINEFORM0 we include INLINEFORM1 , INLINEFORM2 , and their sum. We also use the drop in self-coverage given by subtracting corresponding values for INLINEFORM3 , and the number of discussion points adopted by each side. \n Question: what aspects of conversation flow do they look at?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-a330d811e7484bd1b845381c6b29a97f",
            "input": "Because of this, we do not claim that this dataset can be considered a ground truth. \n Question: How is the ground truth for fake news established?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-b79b3952bd6d42b8a9548f4ac56e29c0",
            "input": "In this section, we consider two kinds of relational classification tasks: (1) relation prediction and (2) relation extraction.  We hope to design a simple and clear experiment setup to conduct error analysis for relational prediction. Therefore, we consider a typical method TransE BIBREF3 as the subject as well as FB15K BIBREF3 as the dataset. For relation extraction, we consider the supervised relation extraction setting and TACRED dataset BIBREF10 . As for the subject model, we use the best model on TACRED dataset — position-aware neural sequence model.  \n Question: Which competitive relational classification models do they test?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-660312e98a9447e8b0eb30a6202d46c8",
            "input": "CNN can also be employed on the sarcasm datasets in order to identify sarcastic and non-sarcastic tweets. We term the features extracted from this network baseline features, the method as baseline method and the CNN architecture used in this baseline method as baseline CNN. Since the fully-connected layer has 100 neurons, we have 100 baseline features in our experiment.  \n Question: What are the network's baseline features?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-eba7cc5f86854943993b3d617d4b88a8",
            "input": "Experimental results show that our joint model outperforms the visual-only model in all cases, and the text-only model on Wikipedia and two subsets of arXiv. \n Question: Do the methods that work best on academic papers also work best on Wikipedia?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-662f3d44a9924955a6edd9e3448da09f",
            "input": "Human Judgments\nFollowing BIBREF11 , BIBREF12 and the vast amount of previous work on semantic similarity, we ask nine undergraduate subjects to assess the similarity of 360 pairs of relations from a subset of Wikidata BIBREF8 that are chosen to cover from high to low levels of similarity. In our experiment, subjects were asked to rate an integer similarity score from 0 (no similarity) to 4 (perfectly the same) for each pair.  \n Question: How do they gather human judgements for similarity between relations?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-210ade5e7bca4a64a735cd6949930401",
            "input": "Baselines. We compare our approach (FacTweet) to the following set of baselines: LR + Bag-of-words: We aggregate the tweets of a feed and we use a bag-of-words representation with a logistic regression (LR) classifier.\n\nTweet2vec: We use the Bidirectional Gated recurrent neural network model proposed in BIBREF20. We keep the default parameters that were provided with the implementation. To represent the tweets, we use the decoded embedding produced by the model. With this baseline we aim at assessing if the tweets' hashtags may help detecting the non-factual accounts.\n\nLR + All Features (tweet-level): We extract all our features from each tweet and feed them into a LR classifier. Here, we do not aggregate over tweets and thus view each tweet independently.\n\nLR + All Features (chunk-level): We concatenate the features' vectors of the tweets in a chunk and feed them into a LR classifier.\n\nFacTweet (tweet-level): Similar to the FacTweet approach, but at tweet-level; the sequential flow of the tweets is not utilized. We aim at investigating the importance of the sequential flow of tweets.\n\nTop-$k$ replies, likes, or re-tweets: Some approaches in rumors detection use the number of replies, likes, and re-tweets to detect rumors BIBREF21. Thus, we extract top $k$ replied, liked or re-tweeted tweets from each account to assess the accounts factuality. We tested different $k$ values between 10 tweets to the max number of tweets from each account. Figure FIGREF24 shows the macro-F1 values for different $k$ values. It seems that $k=500$ for the top replied tweets achieves the highest result. Therefore, we consider this as a baseline. \n Question: What baselines were used in this work?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-92adca0abe204fdabd3c0bf91a0c0acc",
            "input": " In terms of this, rather than starting from Twitter API Search, we look into the archived snapshots of two major news websites (CNN, NBC), and then extract the tweet blocks that are embedded in the news articles. \n Question: How do they determine if tweets have been used by journalists?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-3135328725fd4e80a427ec20f175ed95",
            "input": "Figure FIGREF5 shows that E-BERT performs comparable to BERT and ERNIE on unfiltered LAMA. \n Question: Which of the two ensembles yields the best performance?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-6f01ae8869ad4a9b881cdbe3ec7d3527",
            "input": "The resulting sequence of vectors is encoded using an LSTM encoder.  \n Question: What architecture does the encoder have?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-5572ef51f3584d0fb16f6b23f087cbf6",
            "input": "We began with evaluating standard MT paradigms, i.e., PBSMT BIBREF3 and NMT BIBREF1 . As for PBSMT, we also examined two advanced methods: pivot-based translation relying on a helping language BIBREF10 and induction of phrase tables from monolingual data BIBREF14 .\n\nAs for NMT, we compared two types of encoder-decoder architectures: attentional RNN-based model (RNMT) BIBREF2 and the Transformer model BIBREF18 . In addition to standard uni-directional modeling, to cope with the low-resource problem, we examined two multi-directional models: bi-directional model BIBREF11 and multi-to-multi (M2M) model BIBREF8 .\n\nAfter identifying the best model, we also examined the usefulness of a data augmentation method based on back-translation BIBREF17 . \n Question: what was the baseline?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-9fe14c057b8a4944a9ee335da084faa8",
            "input": "Following BIBREF11 , we experiment on the non-anonymized version of . When generating summaries, we follow standard practice of tuning the maximum output length and disallow repeating the same trigram BIBREF27 , BIBREF14 . For this task we train language model representations on the combination of newscrawl and the training data. Table TABREF16 shows that pre-trained embeddings can significantly improve on top of a strong baseline transformer. We also compare to BIBREF26 who use a task-specific architecture compared to our generic sequence to sequence baseline. Pre-trained representations are complementary to their method. \n Question: What other models do they compare to?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-f14e49b778334eae86f8812dc35bcaf5",
            "input": "When we include the LM embeddings in our system overall performance increases from 90.87% to 91.93% INLINEFORM0 for the CoNLL 2003 NER task, a more then 1% absolute F1 increase, and a substantial improvement over the previous state of the art. We also establish a new state of the art result (96.37% INLINEFORM1 ) for the CoNLL 2000 Chunking task. \n Question: what results do they achieve?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-906fafa8b0f342489c0f76f255238675",
            "input": "Finally, the decoder task, which predicts the target sequence probability at time INLINEFORM3 based on previous output and context information INLINEFORM4 , can be formulated as: DISPLAYFORM0 \n Question: How does their decoder generate text?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-22be38f469cc4cac9ae387d3e7185dbc",
            "input": "Additionally, we also test their performance only on the items from the Zurich and Visp dialect, because most of the samples are from this two dialects. In Table TABREF15 we show the result of the comparison of the two models. \n Question: Is the model evaluated on the graphemes-to-phonemes task?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-5c6177fc3631447593212600c64a0572",
            "input": "In contrast, our models are trained only on the 76K questions in the training set. \n Question: Do the authors also try the model on other datasets?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-646c1dc2b3b54dfd937099a6297d94cd",
            "input": "The final code-mixed tweets were forwarded to a group of three annotators who were university students and fluent in both English and Hindi. \n Question: How many annotators tagged each text?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-ebff3b6ae5714482adf6f7ef8790c913",
            "input": "Features. We argue that different kinds of features like the sentiment of the text, morality, and other text-based features are critical to detect the nonfactual Twitter accounts by utilizing their occurrence during reporting the news in an account's timeline. We employ a rich set of features borrowed from previous works in fake news, bias, and rumors detection BIBREF0, BIBREF1, BIBREF8, BIBREF9.\n\n[leftmargin=4mm]\n\nEmotion: We build an emotions vector using word occurrences of 15 emotion types from two available emotional lexicons. We use the NRC lexicon BIBREF10, which contains $\\sim $14K words labeled using the eight Plutchik's emotions BIBREF11. The other lexicon is SentiSense BIBREF12 which is a concept-based affective lexicon that attaches emotional meanings to concepts from the WordNet lexical database. It has $\\sim $5.5 words labeled with emotions from a set of 14 emotional categories We use the categories that do not exist in the NRC lexicon.\n\nSentiment: We extract the sentiment of the tweets by employing EffectWordNet BIBREF13, SenticNet BIBREF14, NRC BIBREF10, and subj_lexicon BIBREF15, where each has the two sentiment classes, positive and negative.\n\nMorality: Features based on morality foundation theory BIBREF16 where words are labeled in one of the following 10 categories (care, harm, fairness, cheating, loyalty, betrayal, authority, subversion, sanctity, and degradation).\n\nStyle: We use canonical stylistic features, such as the count of question marks, exclamation marks, consecutive characters and letters, links, hashtags, users' mentions. In addition, we extract the uppercase ratio and the tweet length.\n\nWords embeddings: We extract words embeddings of the words of the tweet using $Glove\\-840B-300d$ BIBREF17 pretrained model. The tweet final representation is obtained by averaging its words embeddings. \n Question: What features are extracted?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-d399e91d766747868663e13faf9e9990",
            "input": "All documents are segmented into paragraphs and processed at the paragraph level (both training and inference); this is acceptable because we observe that most paragraphs are less than 200 characters. The input sequences are segmented by the BERT tokenizer, with the special [CLS] token inserted at the beginning and the special [SEP] token added at the end. \n Question: Was the structure of regulatory filings exploited when training the model? ",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-8f90c23145ae4c31b2e516094381de1c",
            "input": "Knowledge Base Question Answering (KBQA) systems answer questions by obtaining information from KB tuples BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 \n Question: What does KBQA abbreviate for",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-11d921e3b03d497abe19b29ae204e20b",
            "input": "We explore an architecture based on a stack of dilated convolution layers, effectively operating on a broader scale than with standard convolutions while limiting model size. Standard convolutional networks cannot capture long temporal patterns with reasonably small models due to the increase in computational cost yielded by larger receptive fields. Dilated convolutions skip some input values so that the convolution kernel is applied over a larger area than its own. The network therefore operates on a larger scale, without the downside of increasing the number of parameters. \n Question: What are dilated convolutions?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-97d7941a5996453aa40e85ffb908b45f",
            "input": "PERPLEXITY \n Question: Does the paper consider the use of perplexity in order to identify text anomalies?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-3380a2f2fea64673a699508280efaf20",
            "input": "For every gendered character in the dataset, we ask annotators to create a new character with a persona of the opposite gender that is otherwise identical except for referring nouns or pronouns. \n Question: In the targeted data collection approach, what type of data is targetted?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-b4896049e9cf4ec79961f0bc0dc348f1",
            "input": " Finally, when investigating the relatedness between European vs. non European languages (cf. (En/Fr)$\\rightarrow $Ar), we obtain similar results than those obtained in the monolingual experiment (macro F-score 62.4 vs. 68.0) and best results are achieved by Ar $\\rightarrow $(En/Fr). This shows that there are pragmatic devices in common between both sides and, in a similar way, similar text-based patterns in the narrative way of the ironic tweets. \n Question: Do the authors identify any cultural differences in irony use?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-1b6bbe7673ba45ef8c91e9d52e048f49",
            "input": "Although our approach strongly outperforms random baselines, the relatively low F1 scores indicate that predicting which word is echoed in explanations is a very challenging task. It follows that we are only able to derive a limited understanding of how people choose to echo words in explanations. The extent to which explanation construction is fundamentally random BIBREF47, or whether there exist other unidentified patterns, is of course an open question. We hope that our study and the resources that we release encourage further work in understanding the pragmatics of explanations. \n Question: Do authors provide any explanation for intriguing patterns of word being echoed?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-23fc129d9e114bcca7424e32f1e2e644",
            "input": "An important feature when suggesting an article INLINEFORM0 to an entity INLINEFORM1 is the novelty of INLINEFORM2 w.r.t the already existing entity profile INLINEFORM3 Given an entity INLINEFORM0 and the already added news references INLINEFORM1 up to year INLINEFORM2 , the novelty of INLINEFORM3 at year INLINEFORM4 is measured by the KL divergence between the language model of INLINEFORM5 and articles in INLINEFORM6 . We combine this measure with the entity overlap of INLINEFORM7 and INLINEFORM8 . The novelty value of INLINEFORM9 is given by the minimal divergence value. Low scores indicate low novelty for the entity profile INLINEFORM10 .\n\n \n Question: What features are used to represent the novelty of news articles to entity pages?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-8722cc16d1ba450290a47a256f0c21a1",
            "input": "Motivated by this, we introduce resolution mode variables $\\Pi = \\lbrace \\pi _1, \\ldots , \\pi _n\\rbrace $ , where for each mention $j$ the variable $\\pi _j \\in \\lbrace str, prec, attr\\rbrace $ indicates in which mode the mention should be resolved. In our model, we define three resolution modes — string-matching (str), precise-construct (prec), and attribute-matching (attr) — and $\\Pi $ is deterministic when $D$ is given (i.e. $P(\\Pi |D)$ is a point distribution). We determine $\\pi _j$ for each mention $m_j$ in the following way:\n\n$\\pi _j = str$ , if there exists a mention $m_i, i < j$ such that the two mentions satisfy the String Match sieve, the Relaxed String Match sieve, or the Strict Head Match A sieve in the Stanford multi-sieve system BIBREF1 .\n\n$\\pi _j = prec$ , if there exists a mention $m_i, i < j$ such that the two mentions satisfy the Speaker Identification sieve, or the Precise Constructs sieve.\n\n$\\pi _j = attr$ , if there is no mention $m_i, i < j$ satisfies the above two conditions. \n Question: Are resolution mode variables hand crafted?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-5061c389a5384156a826ceb3fcbd9029",
            "input": "We crowdsourced multiple-choice questions in two parts, encouraging workers to be imaginative and varied in their use of language. First, workers were given a seed qualitative relation q+/-( INLINEFORM0 ) in the domain, expressed in English (e.g., “If a surface has more friction, then an object will travel slower”), and asked to enter two objects, people, or situations to compare. \n Question: Do all questions in the dataset allow the answers to pick from 2 options?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-c6ef301ca5624db49d4d81633de7eb5c",
            "input": "We have implemented the following interfaces for Macaw:\n\n[leftmargin=*]\n\nFile IO: This interface is designed for experimental purposes, such as evaluating the performance of a conversational search technique on a dataset with multiple queries. This is not an interactive interface.\n\nStandard IO: This interactive command line interface is designed for development purposes to interact with the system, see the logs, and debug or improve the system.\n\nTelegram: This interactive interface is designed for interaction with real users (see FIGREF4). Telegram is a popular instant messaging service whose client-side code is open-source. We have implemented a Telegram bot that can be used with different devices (personal computers, tablets, and mobile phones) and different operating systems (Android, iOS, Linux, Mac OS, and Windows). This interface allows multi-modal interactions (text, speech, click, image). It can be also used for speech-only interactions. For speech recognition and generation, Macaw relies on online APIs, e.g., the services provided by Google Cloud and Microsoft Azure. In addition, there exist multiple popular groups and channels in Telegram, which allows further integration of social networks with conversational systems. For example, see the Naseri and Zamani's study on news popularity in Telegram BIBREF12. \n Question: What interface does Macaw currently have?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-285b035962e34bfbb0f6581ef1f8b848",
            "input": "However, probably the most surprising result in the field is Zipf's law, which states that if one ranks words by their frequency in a large text, the resulting rank frequency distribution is approximately a power law, for all languages BIBREF0, BIBREF11. Another interesting characterization of texts is the Heaps-Herdan law, which describes how the vocabulary -that is, the set of different words- grows with the size of a text, the number of which, empirically, has been found to grow as a power of the text size BIBREF18, BIBREF19. It is worth noting that it has been argued that this law is a consequence Zipf's law. BIBREF20, BIBREF21 \n Question: How do Zipf and Herdan-Heap's laws differ?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-cb5302b1c0a149a09c7bf2b7b2d63c74",
            "input": "This step consists of generating a query out of the claim and querying a search engine (here, we experiment with Google and Bing) in order to retrieve supporting documents. Rather than querying the search engine with the full claim (as on average, a claim is two sentences long), we generate a shorter query following the lessons highlighted in BIBREF0 . We rank the words by means of tf-idf. We compute the idf values on a 2015 Wikipedia dump and the English Gigaword. BIBREF0 suggested that a good way to perform high-quality search is to only consider the verbs, the nouns and the adjectives in the claim; thus, we exclude all words in the claim that belong to other parts of speech. Moreover, claims often contain named entities (e.g., names of persons, locations, and organizations); hence, we augment the initial query with all the named entities from the claim's text. We use IBM's AlchemyAPI to identify named entities. Ultimately, we generate queries of 5-10 tokens, which we execute against a search engine. We then collect the snippets and the URLs in the results, skipping any result that points to a domain that is considered unreliable. \n Question: How are the potentially relevant text fragments identified?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-65cc8721657f439ab68fb2adde9e99ea",
            "input": "In this paper, we use three data sets from the literature to train and evaluate our own classifier. Data collected by BIBREF3 , which we term the Sexist/Racist (SR) data set, was collected using an initial Twitter search followed by analysis and filtering by the authors and their team who identified 17 common phrases, hashtags, and users that were indicative of abusive speech. BIBREF4 collected the HATE dataset by searching for tweets using a lexicon provided by Hatebase.org. The final data set we used, which we call HAR, was collected by BIBREF9 ; we removed all retweets reducing the dataset to 20,000 tweets. Tweets were labeled as “Harrassing” or “Non-Harrassing”; hate speech was not explicitly labeled, but treated as an unlabeled subset of the broader “Harrassing” category BIBREF9 . Many of the false negatives we see are specific references to characters in the TV show “My Kitchen Rules”, rather than something about women in general.  While this may be a limitation of considering only the content of the tweet, it could also be a mislabel.\n\nDebra are now my most hated team on #mkr after least night's ep. Snakes in the grass those two.\n\nAlong these lines, we also see correct predictions of innocuous speech, but find data mislabeled as hate speech:\n\n@LoveAndLonging ...how is that example \"sexism\"?\n\n@amberhasalamb ...in what way? \n Question: Do they report results only on English data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-e499c64daa96414f9c153e8451020009",
            "input": "For the MT task, we use the WMT 2014 En $\\leftrightarrow $ Fr parallel corpus. The dataset contains 36 million En $\\rightarrow $ Fr sentence pairs. We swapped the source and target sentences to obtain parallel data for the Fr $\\rightarrow $ En translation task. We use these two datasets (72 million sentence pairs) to train a single multilingual NMT model to learn both these translation directions simultaneously.  \n Question: What data were they used to train the multilingual encoder?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-b0ba8e63e71b45118012f654507fd908",
            "input": "For experiments, we train and evaluate our models in the civil law system of mainland China. We collect and construct a large-scale real-world data set of INLINEFORM0 case documents that the Supreme People's Court of People's Republic of China has made publicly available \n Question: what is the size of the real-world civil case dataset?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-36b6daadabcb4fa999bf8afb80b6b395",
            "input": "Despite the greater similarity between our task and the 2013 ShARe/CLEF Task 1, we use the clinical notes from the CE task in 2010 i2b2/VA on account of 1) the data from 2010 i2b2/VA being easier to access and parse, 2) 2013 ShARe/CLEF containing disjoint entities and hence requiring more complicated tagging schemes. \n Question: where did they obtain the annotated clinical notes from?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-ea87515e6486441d943d659dd3f58f95",
            "input": "The attention function BIBREF11 is used to compute the similarity score between passages and questions as: INLINEFORM2 \n Question: Do they use attention?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-d3fcd7cdfd72487892e3048919253a27",
            "input": "Further, we employ BLEU score to evaluate the accuracy of generations BIBREF21, and the number of distinct n-gram to evaluate the diversity of generations BIBREF6.  \n Question: How do they measure the diversity of inferences?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-d1fb778f08dd45bbbf4fa2c6ef451447",
            "input": "From the chart, it is clear that there is a higher chance of fake news coming from unverified accounts. Notice that accounts spreading viral tweets with fake news have, on average, a larger ratio of friends/followers.  Figure FIGREF24 shows that, in contrast to other kinds of viral tweets, those containing fake news were created more recently. \n Question: What are the characteristics of the accounts that spread fake news?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-0efc2d9c724740eebe940563dd9009d2",
            "input": "To evaluate the performance of our approach, we used a subset of the SNIPS BIBREF12 dataset, which is readily available in RASA nlu format.  We use accuracy of intent and entity recognition as our task and metric.  \n Question: How are their changes evaluated?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-35a263e6c97b4a73bb8e471082542f04",
            "input": "The answer styles considered in the experiments corresponded to the two tasks. The NLG task requires a well-formed answer that is an abstractive summary of the question and ten passages, averaging 16.6 words. The Q&A task also requires an abstractive answer but prefers a more concise answer than the NLG task, averaging 13.1 words, where many of the answers do not contain the context of the question. \n Question: What is an \"answer style\"?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-94883ff5b6aa4e8ab60a8b76e5342c62",
            "input": "However, we found that explicitly initializing these embeddings with vectors pre-trained over a large collection of unlabeled data significantly improved performance (see Section \"Effects of initialized embeddings and corrupt-sampling schemes\" ). \n Question: What is the new initialization method proposed in this paper?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-13f6c8393a0c477180157f04d1aa5a6b",
            "input": "Active learning sharply increases the performance of iteratively trained machine learning models by selectively determining which unlabeled samples should be annotated.  \n Question: What is active learning?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-5564bc3cd6824e438e8d1bcd0fafdc3c",
            "input": "Comparing Eq.DISPLAY_FORM14 with Eq.DISPLAY_FORM22, we can see that Eq.DISPLAY_FORM14 is actually a soft form of $F1$, using a continuous $p$ rather than the binary $\\mathbb {I}( p_{i1}>0.5)$. This gap isn't a big issue for balanced datasets, but is extremely detrimental if a big proportion of training examples are easy-negative ones: easy-negative examples can easily dominate training since their probabilities can be pushed to 0 fairly easily. Meanwhile, the model can hardly distinguish between hard-negative examples and positive ones, which has a huge negative effect on the final F1 performance.\n\nTo address this issue, we propose to multiply the soft probability $p$ with a decaying factor $(1-p)$, changing Eq.DISPLAY_FORM22 to the following form:\n\nOne can think $(1-p_{i1})$ as a weight associated with each example, which changes as training proceeds. The intuition of changing $p_{i1}$ to $(1-p_{i1}) p_{i1}$ is to push down the weight of easy examples. For easy examples whose probability are approaching 0 or 1, $(1-p_{i1}) p_{i1}$ makes the model attach significantly less focus to them. Figure FIGREF23 gives gives an explanation from the perspective in derivative: the derivative of $\\frac{(1-p)p}{1+(1-p)p}$ with respect to $p$ approaches 0 immediately after $p$ approaches 0, which means the model attends less to examples once they are correctly classified.\n\nA close look at Eq.DISPLAY_FORM14 reveals that it actually mimics the idea of focal loss (FL for short) BIBREF16 for object detection in vision. Focal loss was proposed for one-stage object detector to handle foreground-background tradeoff encountered during training. It down-weights the loss assigned to well-classified examples by adding a $(1-p)^{\\beta }$ factor, leading the final loss to be $(1-p)^{\\beta }\\log p$. \n Question: How are weights dynamically adjusted?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-f0ebcb7f7f2c4a6388ddde145c7b0b67",
            "input": "French and Russian, and Arabic can be regarded as high resource languages whereas Hindi has far less data and can be considered as low resource. \n Question: Is the system tested on low-resource languages?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-b08e4a57bc9b4ba29c90eda7511c0f27",
            "input": "In addition to it, we extract three hand-crafted polarity scores, which are minimum, mean, and maximum polarity scores, from each review. These polarity scores of words are computed as in (DISPLAY_FORM4). For example, if a review consists of five words, it would have five polarity scores and we utilise only three of these sentiment scores as mentioned. Lastly, we concatenate these three scores to the averaged word vector per review.\n\nThat is, each review is represented by the average word vector of its constituent word embeddings and three supervised scores. We then feed these inputs into the SVM approach. The flowchart of our framework is given in Figure FIGREF11. When combining the unsupervised features, which are word vectors created on a word basis, with supervised three scores extracted on a review basis, we have better state-of-the-art results. \n Question: Which hand-crafted features are combined with word2vec?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-7766aff8649842a5bdfb301337a05812",
            "input": "It is worth mentioning that the collected texts contain a large quantity of errors of several types: orthographic, syntactic, code-switched words (i.e. words not in the required language), jokes, etc. Hence, the original written sentences have been processed in order to produce “cleaner” versions, in order to make the data usable for some research purposes (e.g. to train language models, to extract features for proficiency assessment, ...). \n Question: Are any of the utterances ungrammatical?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-41e152c8c1214404bebbcc8f82db972b",
            "input": "These are four conjunctively written Nguni languages (zul, xho, nbl, ssw), Afrikaans (afr) and English (eng), three disjunctively written Sotho languages (nso, sot, tsn), as well as tshiVenda (ven) and Xitsonga (tso). The Nguni languages are similar to each other and harder to distinguish. The same is true of the Sotho languages. Similar languages are to each other are:\n- Nguni languages: zul, xho, nbl, ssw\n- Sotho languages: nso, sot, tsn \n Question: Which languages are similar to each other?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-c67e831f35484b4c9d1b75b1ae0cf2de",
            "input": "We collected the dataset with 1900 dialogs and 8533 turns.  \n Question: What is the average length of dialog?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-da13f14504f142fe8ace571dc16c74df",
            "input": "The final result submitted in public leaderboard is 0.73019 and in private leaderboard is 0.58455. It is quite different in bad way. That maybe is the result of the model too overfit on train set tuning on public test set. \n Question: What is private dashboard?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-cdffc3aad2b24f3c9792b7c73e78c4c1",
            "input": "In Section SECREF5 , we discuss the resulting data collection of questions, texts and answers via crowdsourcing on Amazon Mechanical Turk (henceforth MTurk). \n Question: what crowdsourcing platform was used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-68f09a21664a48a7a50d757391c532d2",
            "input": "Figure FIGREF29 shows the $\\text{MPAs}$ of the proposed DP-LSTM and vanilla LSTM for comparison.  \n Question: Is the model compared against a linear regression baseline?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-38888fbc766249ae8de0b63cb93b57f7",
            "input": "The current state-of-the-art approach BIBREF14 , BIBREF15 uses maximum entropy and CRF models with a combination of language model and hand-crafted features to predict if each character in the hashtag is the beginning of a new word. \n Question: What current state of the art method was used for comparison?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-0c4e6c2b763b4e8c97093be1e84912f1",
            "input": "Using the LDA model, each person in the dataset is with a topic probability vector $X_i$ . Assume $x_{ik}\\in X_{i}$ denotes the likelihood that the $\\emph {i}^{th}$ tweet account favors $\\emph {k}^{th}$ topic in the dataset. Our topic based features can be calculated as below. Global Outlier Standard Score measures the degree that a user's tweet content is related to a certain topic compared to the other users. Local Outlier Standard Score measures the degree of interest someone shows to a certain topic by considering his own homepage content only. Three baseline classification methods: Support Vector Machines (SVM), Adaboost, and Random Forests are adopted to evaluate our extracted features. \n Question: How do they detect spammers?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-3b7f5c88887e4b629cd0551902fa2962",
            "input": "The corpus consists of 53 documents, which contain an average number of 156.1 sentences per document, each with 19.55 tokens on average. \n Question: How many documents are in the new corpus?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-f246c89c2b6042bc911b120719e50fea",
            "input": "The dataset needed to train the W2V was obtained using the information extracted from a dump of the Italian Wikipedia (dated 2019.04.01), from the main categories of Italian Google News (WORLD, NATION, BUSINESS, TECHNOLOGY, ENTERTAINMENT, SPORTS, SCIENCE, HEALTH) and from some anonymized chats between users and a customer care chatbot (Laila). The dataset (composed of 2.6 GB of raw text) includes $421\\,829\\,960$ words divided into $17\\,305\\,401$ sentences. All the words were then transformed to lowercase (to avoid a double presence) finally producing a vocabulary of $618\\,224$ words. \n Question: What is the dataset used as input to the Word2Vec algorithm?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-b77cdecb272340888a546f7bfe4c9949",
            "input": "In this paper, we introduce a novel policy model to output multiple actions per turn (called multi-act), generating a sequence of tuples and expanding agents' expressive power. Each tuple is defined as $(\\textit {continue}, \\textit {act}, \\textit {slots})$, where continue indicates whether to continue or stop producing new acts, act is an act type (e.g., inform or request), and slots is a set of slots (names) associated with the current act type. Correspondingly, a novel decoder (Figure FIGREF5) is proposed to produce such sequences. Each tuple is generated by a cell called gated Continue Act Slots (gCAS, as in Figure FIGREF7), which is composed of three sequentially connected gated units handling the three components of the tuple. This decoder can generate multi-acts in a double recurrent manner BIBREF18.  \n Question: What is specific to gCAS cell?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-1a7a0a8c24a84b7eb688e944bf4314b1",
            "input": "This paper explores the potential of predicting a user's industry -the aggregate of enterprises in a particular field- by identifying industry indicative text in social media.  \n Question: What do they mean by a person's industry?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-c9b1a69997e349418bd9906a11d51bb3",
            "input": "We compare the above model to a similar model, where rather than explicitly represent $K$ features as input, we have $K$ features in the form of a genre embedding, i.e. we learn a genre specific embedding for each of the gothic, scifi, and philosophy genres, as studied in BIBREF8 and BIBREF7. \n Question: Is this style generator compared to some baseline?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-904b7f580ebe493795443a6710b57fd4",
            "input": "It is true that the above-mentioned associated caption sentences for each concept-set are human-written and do describe scenes that cover all given concepts. However, they are created under specific contexts (i.e. an image or a video) and thus might be less representative for common sense. To better measure the quality and interpretability of generative reasoners, we need to evaluate them with scenes and rationales created by using concept-sets only as the signals for annotators.\n\nWe collect more human-written scenes for each concept-set in dev and test set through crowd-sourcing via the Amazon Mechanical Turk platform. Each input concept-set is annotated by at least three different humans. The annotators are also required to give sentences as the rationales, which further encourage them to use common sense in creating their scenes. \n Question: Are the sentences in the dataset written by humans who were shown the concept-sets?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-32e9d79cace249118669158d2a330c55",
            "input": "In this study we use transcripts and results of Oxford-style debates from the public debate series “Intelligence Squared Debates” (IQ2 for short). \n Question: what debates dataset was used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-ccbc6e1da13246eab41662cb2a18e9ae",
            "input": "Since it is infeasible to manually annotate all of the comments, we process this dataset with the goal of extracting threads that involve suspected trolling attempts and the direct responses to them.  We had two human annotators who were trained on snippets (i.e., (suspected trolling attempt, responses) pairs) taken from 200 conversations and were allowed to discuss their findings. After this training stage, we asked them to independently label the four aspects for each snippet.  \n Question: Do they use a crowdsourcing platform for annotation?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-37333c04a9ba443eba6d8711a40106cc",
            "input": "To test this hypothesis we first trained UG-WGAN in English, Chinese and German following the procedure described in Section \"UG-WGAN\" . For this experiment we trained UG-WGAN on the English and Russian language following the procedure described in Section \"UG-WGAN\" .  To do so we trained 6 UG-WGAN models on the following languages: English, Russian, Arabic, Chinese, German, Spanish, French. \n Question: What are the languages they consider in this paper?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-b015c763473b41c08b23ed3d2f6493ff",
            "input": "The Wikipedia dataset consists of articles from English Wikipedia, with quality class labels assigned by the Wikipedia community. \n Question: Which languages do they use?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-7cb8be0316d440c8917764ae8dceafb3",
            "input": "We evaluated SDNet on CoQA dataset \n Question: Is the model evaluated on other datasets?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-b5898289b3da4e7494da9caf6dd3afd2",
            "input": "Hashtag prediction for social media has been addressed earlier, for example in BIBREF15 , BIBREF16 . BIBREF15 also use a neural architecture, but compose text embeddings from a lookup table of words. \n Question: Is this hashtag prediction task an established task, or something new?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-151c3d06417b4954b6f76d3206cb97f2",
            "input": "Task 1: Quora Duplicate Question Pair Detection Task 2: Ranking questions in Bing's People Also Ask \n Question: On which tasks do they test their conflict method?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-a73d206a30f54f069d9eb4465e7098ec",
            "input": "If we take the best psr ensemble trained on CBT as a baseline, improving the model architecture as in BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , continuing to use the original CBT training data, lead to improvements of INLINEFORM0 and INLINEFORM1 absolute on named entities and common nouns respectively. By contrast, inflating the training dataset provided a boost of INLINEFORM2 while using the same model. \n Question: How large are the improvements of the Attention-Sum Reader model when using the BookTest dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-cce415fa4a984d64aabd679d831cd7c2",
            "input": "Our system significantly out-performed the baseline on the joint task, as the baseline scored significantly lower for the gender task than for the variety task. \n Question: On which task does do model do best?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-ea07e429fa1c4851a0c619c05ef173d3",
            "input": "We would like to thank the organizers of the WASSA-2017 Shared Task on Emotion Intensity, for providing the data, the guidelines and timely support. \n Question: what dataset was used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-d7b15bef3a7141c5bf7db11296251488",
            "input": "We used a dataset provided by Shortir, an Indonesian news aggregator and summarizer company. The dataset contains roughly 20K news articles. Each article has the title, category, source (e.g., CNN Indonesia, Kumparan), URL to the original article, and an abstractive summary which was created manually by a total of 2 native speakers of Indonesian \n Question: Did they use a crowdsourcing platform for the summaries?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-58b8f8cffef340079b14a4bfa3e53bbb",
            "input": "Baseline Experiments\nIn this section, we briefly describe a baseline and evaluation scripts that we release, with a detailed documentation, along with the corpus. \n Question: What baseline system is proposed?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-506443df67a548a99c052c609085f785",
            "input": "In the second round, we collected 293 annotations from 12 annotators. After Korektor, there are 4262 unique sentences (including 150 seed sentences) that form the COSTRA 1.0 dataset. \n Question: How are possible sentence transformations represented in dataset, as new sentences?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-25e038dd1a984fd397e4e4c922c88c46",
            "input": "In terms of speed, our system was able to lemmatize 7.4 million words on a personal laptop in almost 2 minutes compared to 2.5 hours for MADAMIRA, i.e. 75 times faster.  \n Question: How was speed measured?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-15f0cf5cbf4a428ba4ae2f95480a3813",
            "input": "Since the present study focuses on Spanish tweets, all tweets from the English datasets were translated into Spanish.  \n Question: What other languages did they translate the data from?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-51595f8a20c643d1a1257f4e392250ff",
            "input": "Amazon Reviews Dataset BIBREF24 is a large dataset with millions of reviews from different product categories. For our experiments, we consider a subset of 20000 reviews from the domains Cell Phones and Accessories(C), Clothing and Shoes(S), Home and Kitchen(H) and Tools and Home Improvement(T). Out of 20000 reviews, 10000 are positive and 10000 are negative. We use 12800 reviews for training, 3200 reviews for validation and 4000 reviews for testing from each domain. \n Question: For the purposes of this paper, how is something determined to be domain specific knowledge?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-309a33be8c5349f4920b8c7027aaae1d",
            "input": "Specifically, Roget's Thesaurus BIBREF38 , BIBREF39 is used to derive the concepts and concept word-groups to be used as the external lexical resource for our proposed method. By using the concept word-groups, we have trained the GloVe algorithm with the proposed modification given in Section SECREF4 on a snapshot of English Wikipedia measuring 8GB in size, with the stop-words filtered out. \n Question: Do they report results only on English data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-489ec21ef45f47d98c1c61cca0415c74",
            "input": "Our model encodes the information from audio and text sequences using dual RNNs and then combines the information from these sources using a feed-forward neural model to predict the emotion class. \n Question: How do they combine audio and text sequences in their RNN?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-fc013d95893c47be95181a272c4a8e78",
            "input": "Within ICU notes (e.g., text example in top-left box in Figure 2), we first identify all abbreviations using regular expressions and then try to find all possible expansions of these abbreviations from domain-specific knowledge base as candidates. \n Question: Do they use any knowledge base to expand abbreviations?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-526b17363d184e67a9a36745bd289da9",
            "input": "After averaging over all concepts, we lose information on the lexical variation that each concept presents but on the other hand one can now investigate which regions show similar geolectal variation, yielding well defined linguistic varieties. Those cells that have similar colors in either figure FIGREF16 or figure FIGREF17 are expected to be ascribed to the same dialect zone. Thus, we can distinguish two main regions or clusters in the maps. The purple background covers most of the map and represents rural regions with small, scattered population. Our analysis shows that this group of cells possesses more specific words in their lexicon. In contrast, the green and yellow cells form a second cluster that is largely concentrated on the center and along the coastline, which correspond to big cities and industrialized areas. In these cells, the use of standard Spanish language is widespread due probably to school education, media, travelers, etc. The character of its vocabulary is more uniform as compared with the purple group. While the purple cluster prefer particular utterances, the lexicon of the urban group includes most of the keywords.  \n Question: What are the characteristics of the rural dialect?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-24f018587e58448f9e8e782d4b9efecb",
            "input": "For DQA four participants answered each question, therefore we took the average INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 values over the four evaluators as the result per question. The detailed answers by the participants and available online. To assess the correctness of the answers given both by participants in the DQA experiments, and by the QALD system, we use the classic information retrieval metrics of precision (P), recall (R), and F1. INLINEFORM0 measures the fraction of relevant (correct) answer (items) given versus all answers (answer items) given. \n Question: Do they test performance of their approaches using human judgements?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-104525eebb9f452dabaf095143e71acd",
            "input": "BIBREF5 adopt a machine learning approach for NER. Their NER system extracts medical problems, tests and treatments from discharge summaries and progress notes. The dataset used is the i2b2 2010 challenge dataset.  \n Question: Does the paper explore extraction from electronic health records?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-d20d8c503c624e8a9b07ffebeac71dd3",
            "input": "We make copies of the monolingual model for each language and add additional crosslingual latent variables (CLVs) to couple the monolingual models, capturing crosslingual semantic role patterns. Concretely, when training on parallel sentences, whenever the head words of the arguments are aligned, we add a CLV as a parent of the two corresponding role variables.  \n Question: Which additional latent variables are used in the model?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-e1b8c08507c94af49953c96b6bf9c5a4",
            "input": "We also conduct two human evaluations in order to assess (a) which type of summary participants prefer (we compare extractive and abstractive systems) and (b) how much key information from the document is preserved in the summary (we ask participants to answer questions pertaining to the content in the document by reading system summaries). \n Question: Do they use other evaluation metrics besides ROUGE?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-fbb633df0fde4e328cc8467a2e38999e",
            "input": "Then, it selects a subset of a 1,700-hour ( INLINEFORM2 1.1M instances) unlabeled dataset. \n Question: How much data samples do they start with before obtaining the initial model labels?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-ed3a6b622df64213b34116ace29a2f11",
            "input": "Our video question answering task is novel and to our knowledge, no model has been designed specifically for this task. As a first step towards solving this problem, we evaluated the performance of state-of-the-art models developed for other QA tasks, including a sentence-level prediction task and two segment retrieval tasks. Baselines ::: Baseline1: Sentence-level prediction\nGiven a transcript (a sequence of sentences) and a question, Baseline1 predicts (starting sentence index, ending sentence index). The model is based on RaSor BIBREF13, which has been developed for the SQuAD QA task BIBREF6. RaSor concatenates the embedding vectors of the starting and the ending words to represent a span. Following this idea, Baseline1 represents a span of sentences by concatenating the vectors of the starting and ending sentences. The left diagram in Fig. FIGREF15 illustrates the Baseline1 model.\n\nModel. The model takes two inputs, a transcript, $\\lbrace s_1, s_2, ... s_n\\rbrace $ where $s_i$ are individual sentences and a question, $q$. The output is the span scores, $y$, the scores over all possible spans. GLoVe BIBREF14 is used for the word representations in the transcript and the questions. We use two bi-LSTMs BIBREF15 to encode the transcript.\n\nwhere n is the number of sentences . The output of Passage-level Encoding, $p$, is a sequence of vector, $p_i$, which represents the latent meaning of each sentence. Then, the model combines each pair of sentence embeddings ($p_i$, $p_j$) to generate a span embedding.\n\nwhere [$\\cdot $,$\\cdot $] indicates the concatenation. Finally, we use a one-layer feed forward network to compute a score between each span and a question.\n\nIn training, we use cross-entropy as an objective function. In testing, the span with the highest score is picked as an answer. Baselines ::: Baseline2: Segment retrieval\nWe also considered a simpler task by casting our problem as a retrieval task. Specifically, in addition to a plain transcript, we also provided the model with the segmentation information which was created during the data collection phrase (See Section. SECREF3). Note that each segments corresponds to a candidate answer. Then, the task is to pick the best segment for given a query. This task is easier than Baseline1's task in that the segmentation information is provided to the model. Unlike Baseline1, however, it is unable to return an answer span at various granularities. Baseline2 is based on the attentive LSTM BIBREF17, which has been developed for the InsuranceQA task. The right diagram in Fig. FIGREF15 illustrates the Baseline2 model.\n\nModel. The two inputs, $s$ and $q$ represent the segment text and a question. The model first encodes the two inputs.\n\n$h^s$ is then re-weighted using attention weights.\n\nwhere $\\odot $ denotes the element-wise multiplication operation. The final score is computed using a one-layer feed-forward network.\n\nDuring training, the model requires negative samples. For each positive example, (question, ground-truth segment), all the other segments in the same transcript are used as negative samples. Cross entropy is used as an objective function.\n\n Baselines ::: Baseline3: Pipeline Segment retrieval\nWe construct a pipelined approach through another segment retrieval task, calculating the cosine similarities between the segment and question embeddings. In this task however, we want to test the accuracy of retrieving the segments given that we first retrieve the correct video from our 76 videos. First, we generate the TF-IDF embeddings for the whole video transcripts and questions. The next step involves retrieving the videos which have the lowest cosine distance between the video transcripts and question. We then filter and store the top ten videos, reducing the number of computations required in the next step. Finally, we calculate the cosine distances between the question and the segments which belong to the filtered top 10 videos, marking it as correct if found in these videos. While the task is less computationally expensive than the previous baseline, we do not learn the segment representations, as this task is a simple retrieval task based on TF-IDF embeddings.\n\nModel. The first two inputs are are the question, q, and video transcript, v, encoded by their TF-IDF vectors: BIBREF18:\n\nWe then filter the top 10 video transcripts(out of 76) with the minimum cosine distance, and further compute the TF-IDF vectors for their segments, Stop10n, where n = 10. We repeat the process for the corresponding segments:\n\nselecting the segment with the minimal cosine distance distance to the query. \n Question: What baseline algorithms were presented?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-9e8c793385aa402580b0cb6030616bd6",
            "input": " A dialog turn from one speaker may not only be a direct response to the other speaker's query, but also likely to be a continuation of his own previous statement. Thus, when modeling turn $k$ in a dialog, we propose to connect the last RNN state of turn $k-2$ directly to the starting RNN state of turn $k$ , instead of letting it to propagate through the RNN for turn $k-1$ . \n Question: How long of dialog history is captured?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-c15627f66a51468ba3013d3197f2d9f8",
            "input": "As pointed out in the section “sec:coalitionpolicy”, there is a strong separation in two blocks between supporters and opponents of European integration, which is even more clearly observed in Fig FIGREF42 B. In the area of State and Evolution of the Union we again observe a strong division in two blocks (see Fig FIGREF42 E). This is different to the Economic and monetary system, however, where we observe a far-left and far-right cooperation, where the division is along the traditional left-right axis.\n\nThe patterns of coalitions forming on Twitter closely resemble those in the European Parliament. \n Question: Does the analysis find that coalitions are formed in the same way for different policy areas?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-47f1fdda4b164b168ec020f0f774270e",
            "input": "Querying posts on Twitter with extracted lexicons led to a collection of $19,300$ tweets. In order to have lexical diversity, we added 2500 randomly sampled tweets to our dataset. \n Question: How many tweets were collected?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-4eb8a95082104623a994eec5b3f73eab",
            "input": "Our baseline is TransE since that the score function of our models is based on TransE. \n Question: What baselines are used for comparison?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-a3689a3990664516a82e959f7c75a544",
            "input": "Python package twitterscraper is used to scrap tweets from twitter. 10,478 tweets from the past two years from domains like `sports', `politics', `entertainment' were extracted. \n Question: Where did the texts in the corpus come from?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-133c5bf90a324936a3bb9963400ce5bb",
            "input": "Suppose there are $M$ total layers, and define a subset of these layers at which to apply the loss function: $K = \\lbrace k_1, k_2, ..., k_L\\rbrace \\subseteq \\lbrace 1,..,M-1\\rbrace $. The total objective function is then defined as\n\nwhere $Z_{k_l}$ is the $k_l$-th Transformer layer activations, $Y$ is the ground-truth transcription for CTC and context dependent states for hybrid ASR, and $Loss(P, Y)$ can be defined as CTC objective BIBREF11 or cross entropy for hybrid ASR. T \n Question: Do they just sum up all the loses the calculate to end up with one single loss?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-7037f5d903e946e2a9ebc29ad611730f",
            "input": "In order to derive the discourse features, an entity grid is constructed by feeding the document through an NLP pipeline to identify salient entities. Two flavors of discourse features are created by populating the entity grid with either (i) grammatical relations (GR) or (ii) RST discourse relations (RST).  Denoting a feature as such with INLINEFORM4 , we construct the pooling vector INLINEFORM5 for the char-bigrams, and concatenate INLINEFORM6 to INLINEFORM7 before feeding the resulting vector to the softmax layer. \n Question: How are discourse features incorporated into the model?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-8d966af9c45d41e297302f31ec4173c2",
            "input": "Lastly, the attention model, MDREA, also outperforms the best existing research results (WAP 0.690 to 0.688) BIBREF19 . \n Question: By how much does their model outperform the state of the art results?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-22b33a5178ef426aadb21d6bd6bfddea",
            "input": "The final result submitted in public leaderboard is 0.73019 and in private leaderboard is 0.58455. It is quite different in bad way. That maybe is the result of the model too overfit on train set tuning on public test set. \n Question: What is public dashboard?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-25880759ce7b4127aaeecbe14026bbeb",
            "input": "The embeddings for words and POS tags were pre-trained on a large unannotated corpus consisting of the first 1 billion characters from Wikipedia. \n Question: Do they use pretrained models as part of their parser?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-ab17bf59d51f4aa2906d87b9415623d2",
            "input": "We call this lifelong interactive learning and inference (LiLi). Lifelong learning is reflected by the facts that the newly acquired facts are retained in the KB and used in inference for future queries, and that the accumulated knowledge in addition to the updated KB including past inference performances are leveraged to guide future interaction and learning. \n Question: In what way does LiLi imitate how humans acquire knowledge and perform inference during an interactive conversation?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-0394de88f18d491aa7d2e1ae1256bf4b",
            "input": "We train Word2Vec with vector size $d_\\mathrm {W2V} = d_\\mathrm {LM} = 768$ on PubMed+PMC (see Appendix for details). Then, we follow the procedure described in Section SECREF3 to update the wordpiece embedding layer and tokenizer of general-domain BERT. \n Question: What in-domain text did they use?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-af29eb5caf834affa6e647a03f22fd20",
            "input": "We perform a direct comparison of expert and crowd contributors, for 1000 term groups based on the number of total annotations(200 term groups with 2 total annotations, 200 term groups with 3 total annotations, and so on up to term groups with 6 total annotations).  Evaluators were given a summary of the annotations received for the term group in the form of:The term group \"inequality inequity\" received annotations as 50.0% sadness, 33.33% disgust, 16.67% anger. Then, they were asked to evaluate on a scale from 1 to 5, how valid these annotations were considered. \n Question: How do they compare lexicons?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-f9e0083608d04a198221abed85709a22",
            "input": "Our first baseline is ROUGE-L BIBREF1 , since it is the most commonly used metric for compression tasks. We compare to the best n-gram-overlap metrics from toutanova2016dataset; We further compare to the negative LM cross-entropy, i.e., the log-probability which is only normalized by sentence length. Our next baseline is perplexity, which corresponds to the exponentiated cross-entropy: Due to its popularity, we also performed initial experiments with BLEU BIBREF17 .  \n Question: Is ROUGE their only baseline?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-7a9513865331423bb5c50ad9a1a4726b",
            "input": "The BD-4SK-ASR Dataset ::: The Language Model\nWe created the language from the transcriptions. The model was created using CMUSphinx in which (fixed) discount mass is 0.5, and backoffs are computed using the ratio method. The model includes 283 unigrams, 5337 bigrams, and 6935 trigrams. \n Question: What are the results of the experiment?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-4a354af5406544179d0bd80240363bdc",
            "input": "But we also showed that in some cases the saliency maps seem to not capture the important input features.  The second observation we can make is that the saliency map doesn't seem to highlight the right things in the input for the summary it generates \n Question: Is the explanation from saliency map correct?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-0c043e18fb15414182da88560bcc51e6",
            "input": "The improvement over all benchmark datasets also shows that conditional BERT is a general augmentation method for multi-labels sentence classification tasks. \n Question: Does the new objective perform better than the original objective bert is trained on?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-a38d864be5944becbcf33228236e7e15",
            "input": "The results show that NCEL consistently outperforms various baselines with a favorable generalization ability. \n Question: How effective is their NCEL approach overall?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-85d07a0823294c1299a48cf1c6ad37c5",
            "input": "Motivated by this, we introduce resolution mode variables $\\Pi = \\lbrace \\pi _1, \\ldots , \\pi _n\\rbrace $ , where for each mention $j$ the variable $\\pi _j \\in \\lbrace str, prec, attr\\rbrace $ indicates in which mode the mention should be resolved. In our model, we define three resolution modes — string-matching (str), precise-construct (prec), and attribute-matching (attr) — and $\\Pi $ is deterministic when $D$ is given (i.e. $P(\\Pi |D)$ is a point distribution). We determine $\\pi _j$ for each mention $m_j$ in the following way:\n\n$\\pi _j = str$ , if there exists a mention $m_i, i < j$ such that the two mentions satisfy the String Match sieve, the Relaxed String Match sieve, or the Strict Head Match A sieve in the Stanford multi-sieve system BIBREF1 .\n\n$\\pi _j = prec$ , if there exists a mention $m_i, i < j$ such that the two mentions satisfy the Speaker Identification sieve, or the Precise Constructs sieve.\n\n$\\pi _j = attr$ , if there is no mention $m_i, i < j$ satisfies the above two conditions. \n Question: What are resolution model variables?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-35d8c95f4d454acaac46680546bfd587",
            "input": "Only the information concerned with the dictionary definitions are used there, discarding the polarity scores. However, when we utilise the supervised score (+1 or -1), words of opposite polarities (e.g. “happy\" and “unhappy\") get far away from each other as they are translated across coordinate regions. \n Question: How are the supervised scores of the words calculated?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-07ded9074be841cab373d50374bdfd07",
            "input": "Given a news Twitter account, we read its tweets from the account's timeline. Then we sort the tweets by the posting date in ascending way and we split them into $N$ chunks. Each chunk consists of a sorted sequence of tweets labeled by the label of its corresponding account. Consequently, we investigate ways to detect suspicious accounts by considering their tweets in groups (chunks). Our hypothesis is that suspicious accounts have a unique pattern in posting tweet sequences. Since their intention is to mislead, the way they transition from one set of tweets to the next has a hidden signature, biased by their intentions. Therefore, reading these tweets in chunks has the potential to improve the detection of the fake news accounts. \n Question: How are chunks defined?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-2e14a76c8b86495caca25ca879cfbf17",
            "input": "We show that by determining and integrating heterogeneous set of features from different modalities - aesthetic features from posted images (colorfulness, hue variance, sharpness, brightness, blurriness, naturalness), choice of profile picture (for gender, age, and facial expression), the screen name, the language features from both textual content and profile's description (n-gram, emotion, sentiment), and finally sociability from ego-network, and user engagement - we can reliably detect likely depressed individuals in a data set of 8,770 human-annotated Twitter users. \n Question: What is the source of the textual data? ",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-b6f67844269846c081613708a07b95cd",
            "input": "To demonstrate the value of building a dedicated version of BERT for French, we first compare CamemBERT to the multilingual cased version of BERT (designated as mBERT). \n Question: Was CamemBERT compared against multilingual BERT on these tasks?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-2d71c1f6e44b4618bf2fa23b60671f15",
            "input": "The behavior model was implemented using an RNN with LSTM units and trained with the Couples Therapy Corpus. Out of the 33 behavioral codes included in the corpus we applied the behaviors Acceptance, Blame, Negativity, Positivity, and Sadness to train our models. This is motivated from previous works which showed good separability in these behaviors as well as being easy to interpret. The behavior model is pre-trained to identify the presence of each behavior from a sequence of words using a multi-label classification scheme. The pre-trained portion of the behavior model was implemented using a single layer RNN with LSTM units with dimension size 50. \n Question: How is module that analyzes behavioral state trained?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-9eab8bd9537c4d8685a27bb1932922a7",
            "input": "This corpus has multiple versions, and we choose the following two versions as their test dataset has significantly larger number of instances of multiple relation tuples with overlapping entities. (i) The first version is used by BIBREF6 (BIBREF6) (mentioned as NYT in their paper) and has 24 relations. We name this version as NYT24. (ii) The second version is used by BIBREF11 (BIBREF11) (mentioned as NYT10 in their paper) and has 29 relations. \n Question: Are there datasets with relation tuples annotated, how big are datasets available?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-ebda03c45ba14aefb499965a145b2525",
            "input": "The experiment settings from this paper and evaluation protocol for the Mboshi corpus (Boundary F-scores using the ZRC speech reference) are the same from BIBREF8. Table presents the results for bilingual UWS and multilingual leveraging.  \n Question: Does the paper report the accuracy of the model?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-27813cd1bc2848078ab70b446b0043ec",
            "input": " Here a small portion of the large parallel corpus for English-German is used as a simulation for the scenario where we do not have much parallel data: Translating texts in English to German.  \n Question: Which languages do they test on for the under-resourced scenario?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-0fcb0142d09149cbadfa21cfa1fbffb2",
            "input": "For the competition, we provided 1,000 labeled dialogues from each dataset for training, and 240 unlabeled dialogues from each dataset for evaluation.  \n Question: What is the size of the second dataset?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-40638794616e4148bd3a75a0c65815b5",
            "input": "Results and Analysis \n Question: what quantitative analysis is done?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-ba01228ad9dc4614a0088507111563d9",
            "input": "The most intuitive way to evaluate the text answer is to directly compute the Exact Match (EM) and Macro-averaged F1 scores (F1) between the predicted text answer and the ground-truth text answer. \n Question: What evaluation metrics were used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-e7b099d77ac1450684e63e0ea9395506",
            "input": "For instance, question and aux are correlated because main-clause questions in English require subject-aux inversion and in many cases the insertion of auxiliary do (Do lions meow?).  Expletives, or “dummy” arguments, are semantically inert arguments. The most common expletives in English are it and there, although not all occurrences of these items are expletives. \n Question: Do they report results only on English data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-f519f9f6029d40f7995409db0da8e259",
            "input": "We propose a novel problem of relationship recommendation (RSR). Different from the reciprocal recommendation problem on DSNs, our RSR task operates on regular social networks (RSN), estimating long-term and serious relationship compatibility based on social posts such as tweets. \n Question: Is this a task other people have worked on?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-6e2d00a82c724b2e90706148797dd2ad",
            "input": "First we are comparing BiLSTMBIBREF14, BiLSTM+CNNBIBREF20, BiLSTM+CRFBIBREF1, BiLSTM+CNN+CRFBIBREF2 models with CNN modelBIBREF0 and Stanford CRF modelBIBREF21. \n Question: Which models are used to solve NER for Nepali?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-b988fa51f98647ada633f4f37a26e815",
            "input": "This study assumes that a robot does not have any vocabularies in advance but can recognize syllables or phonemes. \n Question: Does their model start with any prior knowledge of words?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-8545c7f80d0e49a380561c7ccb800e30",
            "input": "We have also performed preliminary experiments on the Arabic portion of the SemEval-2016 cQA task.  \n Question: Did they experimnet in other languages?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-35cc1756d327438a8a589d52535261c4",
            "input": "Both beat the non-attention baseline by a significant margin. For the attention baseline, we use the standard parametrized attention BIBREF2 . \n Question: Which baseline methods are used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-dea0e55b89784e5a87e13ded5c0f66de",
            "input": "Our conversational agent uses two architectures to simulate a specialized reminiscence therapist. The block in charge of generating questions is based on the work Show, Attend and Tell BIBREF13. This work generates descriptions from pictures, also known as image captioning. \n Question: Is machine learning system underneath similar to image caption ML systems?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-fac9e58cfea9470da191c7c6aff7a332",
            "input": "We create two models both of which constitutes of three main parts: encoder, interaction and classifier and take two sequences as input. The encoder is shared among the sequences simply uses two stacked GRU layers. The interaction part consists of only attention for one model while for the another one it consists of attention and conflict combined as shown in (eqn.11) . The classifier part is simply stacked fully-connected layers.  \n Question: Which neural architecture do they use as a base for their attention conflict mechanisms?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-df42e9abf1934f4d87ded9641c483539",
            "input": "To report the result for answer selection, MAP and MRR are used; however, the answer triggering task is evaluated by F1-score. The result for MULT Method is reported in Table. 1. \n Question: Is accuracy the only metric they used to compare systems?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-12746a3d41e748258685d405bb3a4e62",
            "input": "We use two popular knowledge bases: WordNet BIBREF0 and Freebase BIBREF1 in this paper. Specifically, we use WN18 (a subset of WordNet) BIBREF24 and FB15K (a subset of Freebase) BIBREF2 since their text descriptions are easily publicly available. Table 1 lists statistics of the two datasets. \n Question: What datasets are used to evaluate this paper?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-b71004ccedd743308b7cd2e83e700ef1",
            "input": "When predicting anger, joy, or valence, the number of systems consistently giving higher scores to sentences with female noun phrases (21-25) is markedly higher than the number of systems giving higher scores to sentences with male noun phrases (8-13). (Recall that higher valence means more positive sentiment.) In contrast, on the fear task, most submissions tended to assign higher scores to sentences with male noun phrases (23) as compared to the number of systems giving higher scores to sentences with female noun phrases (12). When predicting sadness, the number of submissions that mostly assigned higher scores to sentences with female noun phrases (18) is close to the number of submissions that mostly assigned higher scores to sentences with male noun phrases (16). These results are in line with some common stereotypes, such as females are more emotional, and situations involving male agents are more fearful BIBREF27  The majority of the systems assigned higher scores to sentences with African American names on the tasks of anger, fear, and sadness intensity prediction. On the joy and valence tasks, most submissions tended to assign higher scores to sentences with European American names. These tendencies reflect some common stereotypes that associate African Americans with more negative emotions BIBREF28 . \n Question: Which race and gender are given higher sentiment intensity predictions?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-38c909e7f9f2478cab9d3ace72cb0baf",
            "input": "GE-FL reduces the heavy load of instance annotation and performs well when we provide prior knowledge with no bias. In our experiments, we observe that comparable numbers of labeled features for each class have to be supplied. We randomly select $t \\in [1, 20]$ features from the feature pool for one class, and only one feature for the other. Our methods are also evaluated on datasets with different unbalanced class distributions. We manually construct several movie datasets with class distributions of 1:2, 1:3, 1:4 by randomly removing 50%, 67%, 75% positive documents. Incorporating KL divergence is robust enough to control unbalance both in the dataset and in labeled features while the other three methods are not so competitive. \n Question: How do they define robustness of a model?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-97e026fc0ab74f9580d6fc6804d15cfd",
            "input": "Following previous studies BIBREF1, we collect event-related microposts from Twitter using 11 and 8 seed events (see Section SECREF2) for CyberAttack and PoliticianDeath, respectively. \n Question: Which real-world datasets are used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-c57e449b87974383837edaf4fd79c9a5",
            "input": "The corpus used for sentiment analysis is the IMDb dataset of movie reviews by BIBREF11 while that for NER is Groningen Meaning Bank (GMB) by BIBREF12, containing 47,959 sentence samples.  \n Question: What Named Entity Recognition dataset is used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-dca8add60a894f76bc6cdb0055b8e3e7",
            "input": "We use the additional INLINEFORM0 training articles labeled by publisher as an unsupervised data set to further train the BERT model. We first investigate the impact of pre-training on BERT-BASE's performance.  On the same computer, fine tuning the model on the small training set took only about 35 minutes for sequence length 100.  \n Question: How are the two different models trained?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-78e4e3cb94074055a36d4be533246293",
            "input": "The ratio of correct `translations' (matches) was used as an evaluation measure. \n Question: What evaluation metric do they use?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-a5bd185f7e174803a1d971e1131ce093",
            "input": "Our visual model is based on fine-tuning an Inception V3 model BIBREF1 over visual renderings of documents, while our textual model is based on a hierarchical biLSTM. We further combine the two into a joint model.  We applied neural network models to capture visual features given visual renderings of documents. Experimental results show that we achieve a 2.9% higher accuracy than state-of-the-art approaches based on textual features over Wikipedia, and performance competitive with or surpassing state-of-the-art approaches over arXiv. We further proposed a joint model, combining textual and visual representations, to predict the quality of a document.  \n Question: What kind of model do they use?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-0fe0641e1dff4b2e8ca66d2d16048de8",
            "input": "Our experiments showed that using the objects color or shape to uniquely identify an object allows the robot successfully complete the binning task in 97.6% and 96.0% of the cases. However, using the shape alone as a unique identifier, the task could only be completed in 79.0% of the cases. \n Question: What is task success rate achieved? ",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-b1fe31df74904323ab39e730a8559926",
            "input": "We compute our measures over the comments written by users in a community in time windows of months, for each sufficiently active month, and manually remove communities where the bulk of the contributions are in a foreign language.  \n Question: Do they report results only on English data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-e193c2b2e51b482095439666e69bf988",
            "input": " The flexibility of the DNN model allowed us to include many more surface level features such as affixes, leading and trailing characters in words and stems, and the presence of words in large gazetteers of named entities. As we show later, these additional features significantly lowered CEER. \n Question: what surface-level features are used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-725eb398221d4ead854e7fb1116e266d",
            "input": "Although the candidates agreed to the use of their interviews, the dataset will not be released to public outside of the scope of this study due to the videos being personal data subject to high privacy constraints. \n Question: Have the candidates given their consent to have their videos used for the research?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-9d15aa007ed145e7bb0c6b990c4e7595",
            "input": "Experiment 1 directly tested the hypothesis that speakers increase their specificity in contexts with asymmetry in visual access. We found that speakers are not only context-sensitive in choosing referring expressions that distinguish target from distractors in a shared context, but are occlusion-sensitive, adaptively compensating for uncertainty. These results strongly suggest that the speaker's informativity influences listener accuracy. In support of this hypothesis, we found a strong negative correlation between informativity and error rates across items and conditions: listeners make fewer errors when utterances are a better fit for the target relative to the distractor ( $\\rho = -0.81$ , bootstrapped 95% CI $= [-0.9, -0.7]$ ; Fig. 6 B). This result suggests that listener behavior is driven by an expectation of speaker informativity: listeners interpret utterances proportionally to how well they fit objects in context. Our Rational Speech Act (RSA) formalization of cooperative reasoning in this context predicts that speakers (directors) naturally increase the informativity of their referring expressions to hedge against the increased risk of misunderstanding; Exp. 1 presents direct evidence in support of this hypothesis. Exp. 2 is consistent with this hypothesis; when directors used underinformative scripted instructions (taken from prior work), listeners made significantly more errors than when speakers were allowed to provide referring expressions at their natural level of informativity, and speaker informativeness strongly modulated listener error rates. \n Question: Did participants behave unexpectedly?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-18a3ea406ebe4e188c78c993fb51c3f2",
            "input": " We use three creative English datasets with distinct linguistic characteristics: (1) A corpus of 740 classical and contemporary English poems, (2) a corpus of 14950 metaphor sentences retrieved from a metaphor database website and (3) a corpus of 1500 song lyrics ranging across genres.  \n Question: Do they report results only on English data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-9dfa7192a39c483fb9bcb0c529b9cb26",
            "input": "The evaluation results are quite favorable for both targets and particularly higher for Target-1, considering the fact that they are the initial experiments on the data set. \n Question: Which SVM approach resulted in the best performance?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-f191820d288147d0a291a64420d91a25",
            "input": "We compare our method to the following baselines: (1) Single-task CNN: training a CNN model for each task individually; (2) Single-task FastText: training one FastText model BIBREF23 with fixed embeddings for each individual task; (3) Fine-tuned the holistic MTL-CNN: a standard transfer-learning approach, which trains one MTL-CNN model on all the training tasks offline, then fine-tunes the classifier layer (i.e. $\\mathrm {M}^{(cls)}$ Figure 1 (a)) on each target task; (4) Matching Network: a metric-learning based few-shot learning model trained on all training tasks; (5) Prototypical Network: a variation of matching network with different prediction function as Eq. 9 ; (6) Convex combining all single-task models: training one CNN classifier on each meta-training task individually and taking the encoder, then for each target task training a linear combination of all the above single-task encoders with Eq. ( 24 ).  \n Question: Do they compare with the MAML algorithm?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-29488cc2a9ba413d95b01b85649fc044",
            "input": "For both POS tagging and dependency parsing, we run experiments on the Wall Street Journal (WSJ) portion of the Penn Treebank. \n Question: Do they evaluate only on English datasets?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-1ceda640b0a447828c8d86880e6acb3c",
            "input": "Twitter data: We used the Twitter API to scrap tweets with hashtags. All non-English tweets were filtered out by the API. \n Question: Do they evaluate only on English datasets?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-442bf5dd48c84e8a888ac20e0c0f550d",
            "input": "Baselines. As none of the existing KBC methods can solve the OKBC problem, we choose various versions of LiLi as baselines.\n\nSingle: Version of LiLi where we train a single prediction model INLINEFORM0 for all test relations.\n\nSep: We do not transfer (past learned) weights for initializing INLINEFORM0 , i.e., we disable LL.\n\nF-th): Here, we use a fixed prediction threshold 0.5 instead of relation-specific threshold INLINEFORM0 .\n\nBG: The missing or connecting links (when the user does not respond) are filled with “@-RelatedTo-@\" blindly, no guessing mechanism.\n\nw/o PTS: LiLi does not ask for additional clues via past task selection for skillset improvement. \n Question: What baseline is used in the experiments?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-53730ae122964c13816ec7325891edd2",
            "input": "To tackle this issue, we present a new evaluation dataset that covers a wide range of monotonicity reasoning that was created by crowdsourcing and collected from linguistics publications (Section \"Dataset\" ). \n Question: Do they release MED?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-6a6a685e50d743f3b73247fdc962bef5",
            "input": "Word embeddings have risen in popularity for NLP applications due to the success of models designed specifically for the big data setting.  Note that “big” datasets are not always available, particularly in computational social science NLP applications, where the data of interest are often not obtained from large scale sources such as the internet and social media, but from sources such as press releases BIBREF11 , academic journals BIBREF10 , books BIBREF12 , and transcripts of recorded speech BIBREF13 , BIBREF14 , BIBREF15  have proposed a model-based method for training interpretable corpus-specific word embeddings for computational social science, using mixed membership representations, Metropolis-Hastings-Walker sampling, and NCE. Experimental results for prediction, supervised learning, and case studies on state of the Union addresses and NIPS articles, indicate that high-quality embeddings and topics can be obtained using the method. The results highlight the fact that big data is not always best, as domain-specific data can be very valuable, even when it is small. \n Question: Why is big data not appropriate for this task?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-ab888989a0b6469bb2f8099529982948",
            "input": "The dataset includes about 6,000 triples, comprised of videos, questions, and answer spans manually collected from screencast tutorial videos with spoken narratives for a photo-editing software.  \n Question: What kind of instructional videos are in the dataset?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-57d2260341cf4180911f7bce9f04bd2e",
            "input": "As a baseline for the RNN models, we apply a uni-directional RNN which predicts the relation after processing the whole sentence. \n Question: Which variant of the recurrent neural network do they use?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-6ea3d2c6515e4076958d8c8f2b9025ea",
            "input": "Conditional Random Fields (CRF) BIBREF15 have been extensively used for tasks of sequential nature. In this paper, we propose as one of the competitive baselines a CRF classifier trained with sklearn-crfsuite for Python 3.5 and the following configuration: algorithm = lbfgs; maximum iterations = 100; c1 = c2 = 0.1; all transitions = true; optimise = false. spaCy is a widely used NLP library that implements state-of-the-art text processing pipelines, including a sequence-labelling pipeline similar to the one described by strubell2017fast. spaCy offers several pre-trained models in Spanish, which perform basic NLP tasks such as Named Entity Recognition (NER). In this paper, we have trained a new NER model to detect NUBes-PHI labels. As the simplest baseline, a sensitive data recogniser and classifier has been developed that consists of regular-expressions and dictionary look-ups. For each category to detect a specific method has been implemented. For instance, the Date, Age, Time and Doctor detectors are based on regular-expressions; Hospital, Sex, Kinship, Location, Patient and Job are looked up in dictionaries. The dictionaries are hand-crafted from the training data available, except for the Patient's case, for which the possible candidates considered are the 100 most common female and male names in Spain according to the Instituto Nacional de Estadística (INE; Spanish Statistical Office). \n Question: What are the other algorithms tested?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-aa654ec555a043709605e360f4fd7c36",
            "input": "We aim to find such content in the social media focusing on the tweets. \n Question: Does the dataset contain content from various social media platforms?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-12b08c85424a430d98dcd5d2945d45dd",
            "input": "To the best of our knowledge, the only scientific summarization benchmark is from TAC 2014 summarization track. For evaluating the effectiveness of Rouge variants and our metric (Sera), we use this benchmark, which consists of 20 topics each with a biomedical journal article and 4 gold human written summaries. Consider the following example:\n\nEndogeneous small RNAs (miRNA) were genetically screened and studied to find the miRNAs which are related to tumorigenesis. \n Question: Do the authors report results only on English data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-2a8b0f16a42c42e0854771b4208462e0",
            "input": "Our novelties include:\n\nUsing self-play learning for the neural response ranker (described in detail below).\n\nOptimizing neural models for specific metrics (e.g. diversity, coherence) in our ensemble setup.\n\nTraining a separate dialog model for each user, personalizing our socialbot and making it more consistent.\n\nUsing a response classification predictor and a response classifier to predict and control aspects of responses such as sentiment, topic, offensiveness, diversity etc.\n\nUsing a model predictor to predict the best responding model, before the response candidates are generated, reducing computational expenses.\n\nUsing our entropy-based filtering technique to filter all dialog datasets, obtaining higher quality training data BIBREF3.\n\nBuilding big, pre-trained, hierarchical BERT and GPT dialog models BIBREF6, BIBREF7, BIBREF8.\n\nConstantly monitoring the user input through our automatic metrics, ensuring that the user stays engaged. \n Question: What is novel in author's approach?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-f3678ff7dbf645fc8b71ab2ef1db5de4",
            "input": "This version highly surpasses the previous state of the art while still having fewer parameters than previous work. \n Question: Do they compare results against state-of-the-art language models?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-4c7b5df5994c48558c2bf977ba54341d",
            "input": "Methodology ::: Corpus-based Approach\nContextual information is informative in the sense that, in general, similar words tend to appear in the same contexts.  In the corpus-based approach, we capture both of these characteristics and generate word embeddings specific to a domain. Methodology ::: Dictionary-based Approach\nIn Turkish, there do not exist well-established sentiment lexicons as in English. In this approach, we made use of the TDK (Türk Dil Kurumu - “Turkish Language Institution”) dictionary to obtain word polarities. \n Question: What word-based and dictionary-based feature are used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-30f4d50b2c1e43c7bddf47cb83b1006d",
            "input": "We compare the predictions of logistic regression models based on unigram bag-of-words features (BOW), sentiment signals (SENT), the linguistic features from our earlier analyses (LING), and combinations of these features. \n Question: What predictive model do they build?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-d73f87c56a05420fb8457ee9500d3b5d",
            "input": "is the standard word counting method whereby the feature vector represents the term frequency of the words in a sentence. is similar to BoW, except that it is derived by the counting of the words in the sentence weighted by individual word's term-frequency and inverse-document-frequency BIBREF31 . is derived from clustering of words-embeddings with k-means into 5000 clusters, and follow by BoW representation of the words in 5000 clusters. \n Question: What other non-neural baselines do the authors compare to? ",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-9264c9ec73524997a57137ba42809394",
            "input": "Specifically, we firstly extract multiple relations with an off-the-shelf Open Information Extraction (OpenIE) toolbox BIBREF7, then we select the relation that is most relevant to the answer with carefully designed heuristic rules. \n Question: How they extract \"structured answer-relevant relation\"?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-113bc27639eb45f69267a22e7eac0b4f",
            "input": "We suspect the absence of decoding in maximum likelihood estimation as a cause behind this inconsistency, and suggest investigating sequence-level learning as an alternative in the future. Inconsistency may arise from the lack of decoding in solving this optimization problem. Maximum likelihood learning fits the model $p_{\\theta }$ using the data distribution, whereas a decoded sequence from the trained model follows the distribution $q_{\\mathcal {F}}$ induced by a decoding algorithm. Based on this discrepancy, we make a strong conjecture: we cannot be guaranteed to obtain a good consistent sequence generator using maximum likelihood learning and greedy decoding. \n Question: Is infinite-length sequence generation a result of training with maximum likelihood?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-6d277e7783dd4e1990844cb8da071131",
            "input": " In this work, we have restricted ourselves to the same datasets as BIBREF7 . These include nine (real-valued) numerical features, which are latitude, longitude, elevation, population, and five climate related features (avg. temperature, avg. precipitation, avg. solar radiation, avg. wind speed, and avg. water vapor pressure). In addition, 180 categorical features were used, which are CORINE land cover classes at level 1 (5 classes), level 2 (15 classes) and level 3 (44 classes) and 116 soil types (SoilGrids). Note that each location should belong to exactly 4 categories: one CORINE class at each of the three levels and a soil type. \n Question: what dataset is used in this paper?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-5cc7683cee8347ca853e5aade8975590",
            "input": "We segment a hashtag into meaningful English phrases. In order to achieve this, we use a dictionary of English words. The sentiment of url: Since almost all the articles are written in well-formatted english, we analyze the sentiment of the first paragraph of the article using Standford Sentiment Analysis tool BIBREF4 .  \n Question: Do the authors report only on English language data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-2f680ea5dfa54cb6b215c16f530907cc",
            "input": "Concretely, we selected gold standards that fit our problem definition and were published in the years 2016 to 2019, have at least $(2019 - publication\\ year) \\times 20$ citations, and bucket them according to the answer selection styles as described in Section SECREF4 \n Question: What modern MRC gold standards are analyzed?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-b3168ce4d7a64c32917f8273c44e3102",
            "input": "We obtained a corpus of approximately 300,000 sentences, from which roughly 1.5 million single-quiz question training examples were derived.  \n Question: What is the size of the dataset?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-77b171e915f04db09613f650f77e69c4",
            "input": "In this work, we use the datasets released by BIBREF1 and HEOT dataset provided by BIBREF0 .  Both the HEOT and BIBREF1 datasets contain tweets which are annotated in three categories: offensive, abusive and none (or benign) \n Question: Do they perform some annotation?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-13572a6a60cd452988c3c5a204efb684",
            "input": "The National Gang Threat Assessment Report confirms that at least tens of thousands of gang members are using social networking websites such as Twitter and video sharing websites such as YouTube in their daily life BIBREF0 . They are very active online; the 2007 National Assessment Center's survey of gang members found that 25% of individuals in gangs use the Internet for at least 4 hours a week BIBREF4 . \n Question: Do the authors provide evidence that 'most' street gang members use Twitter to intimidate others?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-d79f8de2673047e9bedd346e94240e7a",
            "input": "In Spanish, serigos2017using extracted anglicisms from a corpus of Argentinian newspapers by combining dictionary lookup (aided by TreeTagger and the NLTK lemmatizer) with automatic filtering of capitalized words and manual inspection. In serigos2017applying, a character n-gram module was added to estimate the probabilities of a word being English or Spanish. moreno2018configuracion used different pattern-matching filters and lexicon lookup to extract anglicism cadidates from a corpus of tweets in US Spanish. \n Question: Does the paper mention other works proposing methods to detect anglicisms in Spanish?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-d7d49dc65913411ca6384b06ca65c06a",
            "input": "Among all single models, LFT performs the best, followed by MinAvgOut. RL is also comparable with previous state-of-the-art models VHRED (attn) and Reranking-RL. We think that this is because LFT exerts no force in pulling the model predictions away from the ground-truth tokens, but rather just makes itself aware of how dull each response is. Consequently, its responses appear more relevant than the other two approaches. Moreover, the hybrid model (last row) outperforms all other models by a large margin.  \n Question: Which one of the four proposed models performed best?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-eb39de635b774323b586db69c7196019",
            "input": "We proposed to use visual renderings of documents to capture implicit document quality indicators, such as font choices, images, and visual layout, which are not captured in textual content. We applied neural network models to capture visual features given visual renderings of documents. Experimental results show that we achieve a 2.9% higher accuracy than state-of-the-art approaches based on textual features over Wikipedia, and performance competitive with or surpassing state-of-the-art approaches over arXiv. We further proposed a joint model, combining textual and visual representations, to predict the quality of a document. Experimental results show that our joint model outperforms the visual-only model in all cases, and the text-only model on Wikipedia and two subsets of arXiv. These results underline the feasibility of assessing document quality via visual features, and the complementarity of visual and textual document representations for quality assessment. \n Question: Which is more useful, visual or textual features?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-3308d1d401d14b0f806963fcb0e98a14",
            "input": "Our work is further connected to the topic of interpretability and transparency of machine learning models BIBREF11, BIBREF35, BIBREF12, for which humans are increasingly involved, for instance for post-hoc evaluations of the model's interpretability.  \n Question: How are the interpretability merits of the approach demonstrated?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-db913104253341419339d29756f0e90a",
            "input": "Notably, this increase is observed after the conclusion of the US presidential primaries and during the period of the Democratic and Republican National Conventions and does not reduce even after the conclusion of the US presidential elections held on November 8. \n Question: What other political events are included in the database?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-0865df4a60c24de2b101d011f42117e2",
            "input": "On the other hand, in the case of MHD, instead of the integration at attention level, we assign multiple decoders for each head and then integrate their outputs to get a final output. \n Question: Does each attention head in the decoder calculate the same output?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-abb1ced5718f4d4395d19c8abd80f6da",
            "input": "Using this annotation model, we create a new large publicly available dataset of English tweets. \n Question: In what language are the tweets?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-0ccc5494dedf4f2384e7302ab2599616",
            "input": "Since our objective is to compare character-based and word-based approaches, we have also implemented a simple word-level encoder for tweets. The input tweet is first split into tokens along white-spaces. A more sophisticated tokenizer may be used, but for a fair comparison we wanted to keep language specific preprocessing to a minimum. The encoder is essentially the same as tweet2vec, with the input as words instead of characters. A lookup table stores word vectors for the $V$ (20K here) most common words, and the rest are grouped together under the `UNK' token. \n Question: What is the word-level baseline?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-6b35a30060e844bc93ce58a39e8cf9cb",
            "input": "Besides, we report the sentiment accuracy (Senti ACC) which measures whether the output sentence has the same sentiment polarity as the input sentence based on our standardized sentiment classifiers. The BLEU score BIBREF25 between the input sentences and the output sentences is calculated to evaluate the content preservation performance. In order to evaluate the overall performance of different models, we also report the geometric mean (G2) and harmonic mean (H2) of the sentiment accuracy and the BLEU score. As for the irony accuracy, we only report it in human evaluation results because it is more accurate for the human to evaluate the quality of irony as it is very complicated. \n Question: Who judged the irony accuracy, sentiment preservation and content preservation?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-7b6c9d751a744608bfe68a1b7d084e63",
            "input": "We evaluate our approach for six target languages: French (fr), Russian (ru), Arabic (ar), Chinese (zh), Hindi (hi), and Vietnamese (vi).  \n Question: What languages are the model transferred to?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-b0ac7c0ed0f64caca9ce6531e78d7f15",
            "input": "(1) AutoJudge consistently and significantly outperforms all the baselines, including RC models and other neural text classification models, which shows the effectiveness and robustness of our model. (2) RC models achieve better performance than most text classification models (excluding GRU+Attention), which indicates that reading mechanism is a better way to integrate information from heterogeneous yet complementary inputs. (3) Comparing with conventional RC models, AutoJudge achieves significant improvement with the consideration of additional law articles. \n Question: what are their results on the constructed dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-ae5f3c0898d94e8887191f5251f3d101",
            "input": "To control the quality, we ensured that a single annotator annotates maximum 120 headlines (this protects the annotators from reading too many news headlines and from dominating the annotations). Secondly, we let only annotators who geographically reside in the U.S. contribute to the task.\n\nWe test the annotators on a set of $1,100$ test questions for the first phase (about 10% of the data) and 500 for the second phase. Annotators were required to pass 95%. \n Question: How is quality of annotation measured?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-08e77e221a2f49c5b3517b990bc61957",
            "input": "Therefore, we used the crowdsourcing platform CrowdFlower (CF) for our data collection. \n Question: How was this data collected?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-4c325256587a456da01c0a3e6998bef9",
            "input": "It was collected using crowdsourcing BIBREF4. The data collection was done using an Android application. \n Question: what is the source of the data?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-342c777edb9540dbab52de79c45d9181",
            "input": "The ensembles were formed by simply averaging the predictions from the constituent single models. \n Question: How does their ensemble method work?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-d6220a608fb74c77abce5745824257b2",
            "input": "The reviews we used have been extracted from TripAdvisor and originally proposed in BIBREF10 , BIBREF11 .  \n Question: Where are the hotel reviews from?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-48c39f9ef59c422c97ea32095df9b690",
            "input": "We experiment with two image captioning models: the Show&Tell model BIBREF0 and the LRCN1u model BIBREF1. Both models follow the basic encoder-decoder architecture design that uses a CNN encoder to condense the visual information into an image embedding, which in turn conditions an LSTM decoder to generate a natural language caption. The main difference between the two models is the way they condition the decoder. The Show&Tell model feeds the image embedding as the “predecessor word embedding” to the first produced word, while the LRCN1u model concatenates the image features with the embedded previous word as the input to the sequence model at each time step. \n Question: Which existing models are evaluated?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-c0b8bda0b61d4bd986364f853e21f159",
            "input": "Our second approach incorporates shallow syntactic information in downstream tasks via token-level chunk label embeddings. Task training (and test) data is automatically chunked, and chunk boundary information is passed into the task model via BIOUL encoding of the labels. We add randomly initialized chunk label embeddings to task-specific input encoders, which are then fine-tuned for task-specific objectives. \n Question: Which syntactic features are obtained automatically on downstream task data?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-9c09201394134d01b09395de6d937429",
            "input": "The encoder is a Convolutional Neural Network (CNN) and the decoder is a Long Short-Term Memory (LSTM) network, as presented in Figure 2 . The image is passed through the encoder generating the image representation that is used by the decoder to know the content of the image and generate the description word by word. \n Question: What model is used to encode the images?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-5faff7ddbe58431b8995d1088af090ef",
            "input": "We compare the performance of our model (Table 2 ) with traditional Bag of Words (BoW), TF-IDF, and n-grams features based classifiers. We also compare against averaged Skip-Gram BIBREF29 , Doc2Vec BIBREF30 , CNN BIBREF23 , Hierarchical Attention (HN-ATT) BIBREF24 and hierarchical network (HN) models. HN it is similar to our model HN-SA but without any self attention. We further investigated the performance on SWBD2 by examining the confusion matrix of the model. Figure 2 shows the heatmap of the normalized confusion matrix of the model on SWBD2. \n Question: Do the authors do manual evaluation?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-3ac34092e20e46b591d6a3ab57767f77",
            "input": "Logistic regression: To produce the representation of the input, we concatenate the Bag-Of-Words representation of the document with the Bag-Of-Words representation of the question. LSTM: We start with a concatenation of the sequence of indexes of the document with the sequence of indexes of the question. Them we feed an LSTM network with this vector and use the final state as the representation of the input. Finally, we apply a logistic regression over this representation to produce the final decision. End-to-end memory networks: This architecture is based on two different memory cells (input and output) that contain a representation of the document. Deep projective reader: This is a model of our own design, largely inspired by the efficient R-net reader BIBREF12 . \n Question: What baselines are presented?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-bc5aabe0b6464b9c86efb660bcefc75c",
            "input": "SMT techniques require a parallel corpus in thr source and thr target language. A text-code parallel corpus similar to Fig. FIGREF12 is used in training. This parallel corpus has 18805 aligned data in it . In source data, the expression of each line code is written in the English language. In target data, the code is written in Python programming language. \n Question: What dataset do they use?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-500c49a3a07b47ea9b75c7d114a97bef",
            "input": "Drawing on the concept of variance in mathematics, local variance loss is defined as the reciprocal of its variance expecting the attention model to be able to focus on more salient parts. The standard variance calculation is based on the mean of the distribution. However, as previous work BIBREF15, BIBREF16 mentioned that the median value is more robust to outliers than the mean value, we use the median value to calculate the variance of the attention distribution. Thus, local variance loss can be calculated as:\n\nwhere $\\hat{\\cdot }$ is a median operator and $\\epsilon $ is utilized to avoid zero in the denominator. \n Question: How do they define local variance?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-497742f46b3141028f8fa9e8545f6d9a",
            "input": "The character sets of these 7 languages have little overlap except that (i) they all include common basic Latin alphabet, and (ii) both Hindi and Marathi use Devanagari script. We took the union of 7 character sets therein as the multilingual grapheme set (Section SECREF2), which contained 432 characters. \n Question: How much of the ASR grapheme set is shared between languages?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-e7e729e8c36348b3adfec7378140a2a5",
            "input": "Comparing these topics with those that appeared on a Catholic women forum, it seems that both ISIS and non-violent groups use topics about motherhood, spousal relationship, and marriage/divorce when they address women. Moreover, we used Depechemood methods to analyze the emotions that these materials are likely to elicit in readers. The result of our emotion analysis suggests that both corpuses used words that aim to inspire readers while avoiding fear. However, the actual words that lead to these effects are very different in the two contexts. Overall, our findings indicate that, using proper methods, automated analysis of large bodies of textual data can provide novel insight insight into extremist propaganda that can assist the counterterrorism community. \n Question: What conclusions do the authors draw from their finding that the emotional appeal of ISIS and Catholic materials are similar?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-8c3c880206de4f52876e595548459ce2",
            "input": "We assessed the proposed models on four different NLG domains: finding a restaurant, finding a hotel, buying a laptop, and buying a television. The Restaurant and Hotel were collected in BIBREF4 , while the Laptop and TV datasets have been released by BIBREF22 with a much larger input space but only one training example for each DA so that the system must learn partial realization of concepts and be able to recombine and apply them to unseen DAs.  \n Question: Does the model evaluated on NLG datasets or dialog datasets?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-280d21df12394195b01b7128ab773510",
            "input": " Table 1 shows the results of all models on WikiLarge dataset. We can see that our method (NMT+synthetic) can obtain higher BLEU, lower FKGL and high SARI compared with other models, except Dress on FKGL and SBMT-SARI on SARI. It verified that including synthetic data during training is very effective, and yields an improvement over our baseline NMF by 2.11 BLEU, 1.7 FKGL and 1.07 SARI.  Results on WikiSmall dataset are shown in Table 2. We see substantial improvements (6.37 BLEU) than NMT from adding simplified training data with synthetic ordinary sentences.  \n Question: by how much did their model improve?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-55297efe1497491d951b819c891c361b",
            "input": "However, adversarial misspellings constitute a longstanding real-world problem. Spammers continually bombard email servers, subtly misspelling words in efforts to evade spam detection while preserving the emails' intended meaning BIBREF1 , BIBREF2 . \n Question: Why is the adversarial setting appropriate for misspelling recognition?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-b544cd9b0fc54dc1b83e05e06c485dde",
            "input": "In our setup, our starting point is a base model, trained on NLI data. Rather than employing automated adversarial methods, here the model's “adversary” is a human annotator. Given a context (also often called a “premise” in NLI), and a desired target label, we ask the human writer to provide a hypothesis that fools the model into misclassifying the label. One can think of the writer as a “white hat” hacker, trying to identify vulnerabilities in the system. For each human-generated example that is misclassified, we also ask the writer to provide a reason why they believe it was misclassified. \n Question: Do they use active learning to create their dataset?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-6e6566f4d5214af59d27d1a3a41a4f89",
            "input": "To test whether horizontal or vertical consistency is more helpful, we train and evaluate M-Bert on a dataset variant where all questions are in their English version. \n Question: Did they pefrorm any cross-lingual vs single language evaluation?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-681c62dde10e476693369fbe86c52026",
            "input": "To better exploit such existing data sources, we propose an end-to-end (E2E) model based on pointer networks with attention, which can be trained end-to-end on the input/output pairs of human IE tasks, without requiring token-level annotations. Since our model does not need token-level labels, we create an E2E version of each data set without token-level labels by chunking the BIO-labeled words and using the labels as fields to extract. \n Question: Do they assume sentence-level supervision?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-449bb91a9ff14c6eb13fc8413dcf574b",
            "input": "With this challenge in mind, we introduce Torch-Struct with three specific contributions:\n\nModularity: models are represented as distributions with a standard flexible API integrated into a deep learning framework.\n\nCompleteness: a broad array of classical algorithms are implemented and new models can easily be added in Python.\n\nEfficiency: implementations target computational/memory efficiency for GPUs and the backend includes extensions for optimization. \n Question: Is this library implemented into Torch or is framework agnostic?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-7f567095aac24af7a8d38c55a8b17621",
            "input": "To validate the effectiveness of AEM for extracting events from social media (e.g. Twitter) and news media sites (e.g. Google news), three datasets (FSD BIBREF12 , Twitter, and Google datasets) are employed. \n Question: What datasets are used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-4ffb26e8134b47e6a7f4ee6d6d4903cb",
            "input": " Moreover, due to their noisy nature, they are also processed using some Twitter-specific techniques such as substitution/removal of URLs, of user mentions, of hashtags, and of emoticons, spelling correction, elongation normalization, abbreviation lookup, punctuation removal, detection of amplifiers and diminishers, negation scope detection, etc. That language proved to be quite challenging with its use of creative spelling and punctuation, misspellings, slang, new words, URLs, and genre-specific terminology and abbreviations, e.g., RT for re-tweet and #hashtags. In addition to the genre difference, there is also a difference in length: social media messages are generally short, often length-limited by design as in Twitter, i.e., a sentence or a headline rather than a full document \n Question: What difficulties does sentiment analysis on Twitter have, compared to sentiment analysis in other domains?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-2ffc643631db49d1a74d09457714ea4c",
            "input": "We collected tweets related to five different DDoS attacks on three different American banks. For each attack, all the tweets containing the bank's name posted from one week before the attack until the attack day were collected. There are in total 35214 tweets in the dataset. Only the tweets from the Bank of America attack on 09/19/2012 were used in this experiment. In this subsection we evaluate how good the model generalizes. To achieve that, the dataset is divided into two groups, one is about the attacks on Bank of America and the other group is about PNC and Wells Fargo. The only difference between this experiment and the experiment in section 4.4 is the dataset. In this experiment setting $D_a$ contains only the tweets collected on the days of attack on PNC and Wells Fargo. $D_b$ only contains the tweets collected before the Bank of America attack. \n Question: What is the training and test data used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-1f234a36d2ab4a119effb6510dd389f2",
            "input": "Methods ::: Length Encoding Method\nInspired by BIBREF11, we use length encoding to provide the network with information about the remaining sentence length during decoding. We propose two types of length encoding: absolute and relative. Let pos and len be, respectively, a token position and the end of the sequence, both expressed in terms of number characters. Then, the absolute approach encodes the remaining length:\n\nwhere $i=1,\\ldots ,d/2$.\n\nSimilarly, the relative difference encodes the relative position to the end. This representation is made consistent with the absolute encoding by quantizing the space of the relative positions into a finite set of $N$ integers:\n\nwhere $q_N: [0, 1] \\rightarrow \\lbrace 0, 1, .., N\\rbrace $ is simply defined as $q_N(x) = \\lfloor {x \\times N}\\rfloor $. As we are interested in the character length of the target sequence, len and pos are given in terms of characters, but we represent the sequence as a sequence of BPE-segmented subwords BIBREF17. To solve the ambiguity, len is the character length of the sequence, while pos is the character count of all the preceding tokens. \n Question: How do they enrich the positional embedding with length information",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-59254892a7024470ac9678c0784cb7a9",
            "input": "The performance on discriminating between offensive (OFF) and non-offensive (NOT) posts is reported in Table TABREF18 . We can see that all systems perform significantly better than chance, with the neural models being substantially better than the SVM. The CNN outperforms the RNN model, achieving a macro-F1 score of 0.80. The CNN system achieved higher performance in this experiment compared to the BiLSTM, with a macro-F1 score of 0.69. All systems performed better at identifying target and threats (TIN) than untargeted offenses (UNT). \n Question: What is the best performing model?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-ba7360a74834472e8e6e45bb49691800",
            "input": "All data were downloaded from Twitter in two separate batches using the “twint\" scraping tool BIBREF5 to sample historical tweets for several different search terms; queries always included either “climate change\" or “global warming\", and further included disaster-specific search terms (e.g., “bomb cyclone,\" “blizzard,\" “snowstorm,\" etc.).  \n Question: Do they report results only on English data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-8467170af23644d7ae51ac25386c637b",
            "input": "We evaluated our detection models on three benchmarks: the FCE test data (41K tokens) and the two alternative annotations of the CoNLL 2014 Shared Task dataset (30K tokens) BIBREF3 .  \n Question: Which annotated corpus did they use?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-0bcdfa3bcb9c4367a69ad416daf50a9e",
            "input": "For the emotion recognition from text, we manually transcribe all utterances of our AMMER study. To exploit existing and available data sets which are larger than the AMMER data set, we develop a transfer learning approach. We use a neural network with an embedding layer (frozen weights, pre-trained on Common Crawl and Wikipedia BIBREF36), a bidirectional LSTM BIBREF37, and two dense layers followed by a soft max output layer. This setup is inspired by BIBREF38. We use a dropout rate of 0.3 in all layers and optimize with Adam BIBREF39 with a learning rate of $10^{-5}$ (These parameters are the same for all further experiments). We build on top of the Keras library with the TensorFlow backend. We consider this setup our baseline model. \n Question: What is the baseline method for the task?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-e2e2b18fd5c640fca2dd36e9fbd72fbf",
            "input": "We evaluate the adversarially trained models, as shown in Table TABREF18.\n\nAfter adversarial training, the performance of all the target models raises significantly, while that on the original examples remain comparable. \n Question: How much in experiments is performance improved for models trained with generated adversarial examples?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-d67bbde8f9b34d2a8f4c3fc138cbd8dd",
            "input": "We apply the same data split in BIBREF31, BIBREF30, BIBREF6, which uses news (the union of bn and nw) as the training set, a half of bc as the development set and the remaining data as the test set.\n\nWe learn the model parameters using Adam BIBREF32. We apply dropout BIBREF33 to the hidden layers to reduce overfitting. The development set is used for tuning the model hyperparameters and for early stopping. We train 5 Bi-LSTM English RE models initiated with different random seeds, apply the 5 models on the target languages, and combine the outputs by selecting the relation type labels with the highest probabilities among the 5 models. \n Question: Do they train their own RE model?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-8b60c7555e3d400987d301553f2deda8",
            "input": "With the above hyperparameter setting, the hybrid NER model achieved a F1 score of $0.995$ on synthesized queries and $0.948$ on clinical notes while the i2b2 NER model achieved a F1 score of $0.441$ on synthesized queries and $0.927$ on clinical notes (See Table TABREF23). \n Question: what were their performance results?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-48d36d0b966b4681a424d5cb75111f69",
            "input": "We evaluate the quality of the document embeddings learned by the different variants of CAHAN and the HAN baseline on three of the large-scale document classification datasets introduced by BIBREF14 and used in the original HAN paper BIBREF5. \n Question: Do they compare to other models appart from HAN?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-c2aa128699354267ac912572b953f4ec",
            "input": "An overview of the provided dataset is given in Table TABREF16 . Notice that the available amount of data differs per language. \n Question: is the dataset balanced across the four languages?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-2161a28c6eff4ba3afa9eedd27457b33",
            "input": "We use the following three evaluation metrics:\n\nPhoneme Error Rate (PER) is the Levenshtein distance between the predicted phoneme sequences and the gold standard phoneme sequences, divided by the length of the gold standard phoneme sequences.\n\nWord Error Rate (WER) is the percentage of words in which the predicted phoneme sequence does not exactly match the gold standard phoneme sequence.\n\nWord Error Rate 100 (WER 100) is the percentage of words in the test set for which the correct guess is not in the first 100 guesses of the system. \n Question: what evaluation metrics were used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-68a0c0d90a0349968446bff2df3fbcab",
            "input": "Our model consists of two neural network modules, i.e. an extractor and abstractor. The extractor encodes a source document and chooses sentences from the document, and then the abstractor paraphrases the summary candidates.  The extractor is based on the encoder-decoder framework. We adapt BERT for the encoder to exploit contextualized representations from pre-trained transformers. We use LSTM Pointer Network BIBREF22 as the decoder to select the extracted sentences based on the above sentence representations.  Our abstractor is practically identical to the one proposed in BIBREF8. \n Question: What's the method used here?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-fb5209988daf482f9dbe743bea11f2d2",
            "input": "The task, as framed above, requires to detect the semantic change between two corpora. The two corpora used in the shared task correspond to the diachronic corpus pair from BIBREF0: DTA18 and DTA19. They consist of subparts of DTA corpus BIBREF11 which is a freely available lemmatized, POS-tagged and spelling-normalized diachronic corpus of German containing texts from the 16th to the 20th century. DTA18 contains 26 million sentences published between 1750-1799 and DTA19 40 million between 1850-1899. \n Question: What is the corpus used for the task?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-7ccb7be1051840d8bda39b516b212852",
            "input": "Our data set contains $3,209$ reviews about 553 different cars from 49 different car manufacturers. \n Question: How big is dataset of car-speak language?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-844edc601b0a4a298c28bcc2f6a2d55b",
            "input": "The study of the mathematical structure of grammar has indicated that the fundamental things making up sentences are not the words, but some atomic grammatical types, such as the noun-type and the sentence-type BIBREF23 , BIBREF24 , BIBREF25 . The transitive verb-type is not an atomic grammatical type, but a composite made up of two noun-types and one sentence-type. Hence, particularly interesting here is that atomic doesn't really mean smallest...\n\nOn the other hand, just like in particle physics where we have particles and anti-particles, the atomic types include types as well as anti-types. But unlike in particle physics, there are two kinds of anti-types, namely left ones and right ones. This makes language even more non-commutative than quantum theory! \n Question: Do they break down word meanings into elementary particles as in the standard model of quantum theory?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-fa4c6db85d334defab0b61b2029b483b",
            "input": "As detailed before, we adapted the cluster-linking dataset from rupnik2016news to evaluate our online crosslingual clustering approach. \n Question: What are the sources of the datasets?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-22da29f57a354635881385cae05f5336",
            "input": "We used two parallel corpora: Asian Scientific Paper Excerpt Corpus (ASPEC) BIBREF0 and NTCIR PatentMT Parallel Corpus BIBREF1 . \n Question: What parallel corpus did they use?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-48fa707fe54441a1bf819a2c49b9dee3",
            "input": "In phase 1 (also referred to as the “exploration” phase) the algorithm explores the state space through keeping track of previously visited states by maintaining an archive. During this phase, instead of resuming the exploration from scratch, the algorithm starts exploring from promising states in the archive to find high performing trajectories. \n Question: How is trajectory with how rewards extracted?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-ccb87e0dbeb34f609e583fe74dc4fb0e",
            "input": "In this work, we introduce a new logical inference engine called MonaLog, which is based on natural logic and work on monotonicity stemming from vanBenthemEssays86. Since our logic operates over surface forms, it is straightforward to hybridize our models. We investigate using MonaLog in combination with the language model BERT BIBREF20, including for compositional data augmentation, i.e, re-generating entailed versions of examples in our training sets.  We perform two experiments to test MonaLog. We first use MonaLog to solve the problems in a commonly used natural language inference dataset, SICK BIBREF1, comparing our results with previous systems. Second, we test the quality of the data generated by MonaLog. To do this, we generate more training data (sentence pairs) from the SICK training data using our system, and performe fine-tuning on BERT BIBREF20, a language model based on the transformer architecture BIBREF23, with the expanded dataset.  \n Question: How do they combine MonaLog with BERT?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-fa094b6395a448178da8dfbcd916e267",
            "input": "BioASQ organizers provide the training and testing data. The training data consists of questions, gold standard documents, snippets, concepts, and ideal answers (which we did not use in this paper, but we used last year BIBREF2). The test data is split between phases A and B. The Phase A dataset consists of the questions, unique ids, question types. The Phase B dataset consists of the questions, golden standard documents, snippets, unique ids and question types. \n Question: What dataset did they use?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-a42476af4415448285218c3c10df496f",
            "input": "1. BERT based fine-tuning: In the first approach, which is shown in Figure FIGREF8, very few changes are applied to the BERTbase. In this architecture, only the [CLS] token output provided by BERT is used. The [CLS] output, which is equivalent to the [CLS] token output of the 12th transformer encoder, a vector of size 768, is given as input to a fully connected network without hidden layer. The softmax activation function is applied to the hidden layer to classify.\n\n2. Insert nonlinear layers: Here, the first architecture is upgraded and an architecture with a more robust classifier is provided in which instead of using a fully connected network without hidden layer, a fully connected network with two hidden layers in size 768 is used. The first two layers use the Leaky Relu activation function with negative slope = 0.01, but the final layer, as the first architecture, uses softmax activation function as shown in Figure FIGREF8.\n\n3. Insert Bi-LSTM layer: Unlike previous architectures that only use [CLS] as the input for the classifier, in this architecture all outputs of the latest transformer encoder are used in such a way that they are given as inputs to a bidirectional recurrent neural network (Bi-LSTM) as shown in Figure FIGREF8. After processing the input, the network sends the final hidden state to a fully connected network that performs classification using the softmax activation function.\n\n4. Insert CNN layer: In this architecture shown in Figure FIGREF8, the outputs of all transformer encoders are used instead of using the output of the latest transformer encoder. So that the output vectors of each transformer encoder are concatenated, and a matrix is produced. The convolutional operation is performed with a window of size (3, hidden size of BERT which is 768 in BERTbase model) and the maximum value is generated for each transformer encoder by applying max pooling on the convolution output. By concatenating these values, a vector is generated which is given as input to a fully connected network. By applying softmax on the input, the classification operation is performed. \n Question: What new fine-tuning methods are presented?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-6d658f12e8f74e59827dd031e434e4b6",
            "input": "jiant System Overview ::: jiant Components\nTasks: Tasks have references to task data, methods for processing data, references to classifier heads, and methods for calculating performance metrics, and making predictions. \n Question: Does jiant involve datasets for the 50 NLU tasks?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-84b384bbfd5145158325970376b297c4",
            "input": "For the image-aided model (W+C+V; upper row in Figure FIGREF19 ), we confirm that the modality attention successfully attenuates irrelevant signals (selfies, etc.) and amplifies relevant modality-based contexts in prediction of a given token. \n Question: Do they inspect their model to see if their model learned to associate image parts with words related to entities?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-d93cde107a8e4ae2b36d970eb8226a48",
            "input": "Following lebret2016neural, we used BLEU-4, NIST-4 and ROUGE-4 as the evaluation metrics. \n Question: What metrics are used for evaluation?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-c07555851b7d45e2b7d9a8b7c9727be7",
            "input": "We focus here on the Europarl domain, for which we have ample data in several languages, and use as in-domain training data the Europarl corpus BIBREF5 for two translation directions: English INLINEFORM0 German and English INLINEFORM1 French.  \n Question: what language is the data in?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-b3222fc6baf14a57809fc6fffda95223",
            "input": "The Machine Learning techniques used varied from Maximum Entropy Classifiers (BIBREF4) to Support Vector Machines (BIBREF5,BIBREF6,BIBREF7,BIBREF8), while the deep learning approaches included Recursive Neural Networks (BIBREF9,BIBREF10), Convolutional Neural Networks (BIBREF11) and most recently transfer learning-based architectures like Bidirectional Encoder Representation from Transformers (BERT) (BIBREF12). Figures FIGREF1 and FIGREF1 contain a summary of the papers addressing speculation detection and scope resolution (BIBREF13, BIBREF5, BIBREF9, BIBREF3, BIBREF14, BIBREF15, BIBREF16, BIBREF17, BIBREF6, BIBREF11, BIBREF18, BIBREF10, BIBREF19, BIBREF7, BIBREF4, BIBREF8). \n Question: What were the baselines?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-8ada0d37e3f84486b1e1927bfe70f49d",
            "input": "Many practical fake news detection algorithms use a kind of semantic side information, such as whether the generated text is factually correct, in addition to its statistical properties. Although statistical side information would be straightforward to incorporate in the hypothesis testing framework, it remains to understand how to cast such semantic knowledge in a statistical decision theory framework. \n Question: What semantic features help in detecting whether a piece of text is genuine or generated? of ",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-398d6bf256a242fd83a9b50a21ba6f1c",
            "input": "String kernels represent a way of using information at the character level by measuring the similarity of strings through character n-grams. \n Question: What is a string kernel?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-4974089ecc3346a1bfb03bdd797cf03e",
            "input": "The results indicate that most performance gains come from words embeddings, style, and morality features. Other features (emotion and sentiment) show lower importance: nevertheless, they still improve the overall system performance (on average 0.35% Macro-F$_1$ improvement) \n Question: Based on this paper, what is the more predictive set of features to detect fake news?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-995afbefad474894b65479ed6d420e2d",
            "input": "In this section we describe a number of experiments targeted to compare the performance of popular named entity recognition algorithms on our data. We trained and evaluated Stanford NER, spaCy 2.0, and a recurrent model similar to BIBREF13 , BIBREF14 that uses bidirectional LSTM cells for character-based feature extraction and CRF, described in Guillaume Genthial's Sequence Tagging with Tensorflow blog post BIBREF15 . Stanford NER is conditional random fields (CRF) classifier based on lexical and contextual features such as the current word, character-level n-grams of up to length 6 at its beginning and the end, previous and next words, word shape and sequence features BIBREF16 . spaCy 2.0 uses a CNN-based transition system for named entity recognition. The main model that we focused on was the recurrent model with a CRF top layer, and the above-mentioned methods served mostly as baselines.  \n Question: what ner models were evaluated?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-b58e7749ca6f4e2b9b868b6f0032cdd1",
            "input": "The average content score across the paintings is 3.7 which demonstrates that the prose generated is relevant to the painting. The average creativity score is 3.9 which demonstrates that the model captures more than basic objects in the painting successfully using poetic clues in the scene. The average style score is 3.9 which demonstrates that the prose generated is perceived to be in the style of Shakespeare. \n Question: How does final model rate on Likert scale?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-6eedbdd9f5b742539858210b0d687c56",
            "input": "A recently published dataset called SCAN BIBREF2 (Simplified version of the CommAI Navigation tasks), tests compositional generalization in a sequence-to-sequence (seq2seq) setting by systematically holding out of the training set all inputs containing a basic primitive verb (\"jump\"), and testing on sequences containing that verb. \n Question: How does the SCAN dataset evaluate compositional generalization?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-e83166514cde442d823671a62e2b667a",
            "input": "For both POS tagging and dependency parsing, we run experiments on the Wall Street Journal (WSJ) portion of the Penn Treebank. \n Question: What datasets do they evaluate on?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-d3cb3aef45c8463a9a788af7a04b7390",
            "input": "For example, the performance of even the best QA models degrades substantially on our hyponym probes (by 8-15%) when going from 1-hop links to 2-hops. Further, the accuracy of even our best models on the WordNetQA probe drops by 14-44% under our cluster-based analysis, which assesses whether a model knows several facts about each individual concept, rather than just being good at answering isolated questions.  \n Question: After how many hops does accuracy decrease?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-40f155e7eba345e48537c078229c9b00",
            "input": "Here we introduce the evaluation setup and analyze the results for the article-entity (AEP) placement task. We only report the evaluation metrics for the `relevant' news-entity pairs.  Baselines. We consider the following baselines for this task.\n\nB1. The first baseline uses only the salience-based features by Dunietz and Gillick BIBREF11 .\n\nB2. The second baseline assigns the value relevant to a pair INLINEFORM0 , if and only if INLINEFORM1 appears in the title of INLINEFORM2 . Here we show the evaluation setup for ASP task and discuss the results with a focus on three main aspects, (i) the overall performance across the years, (ii) the entity class specific performance, and (iii) the impact on entity profile expansion by suggesting missing sections to entities based on the pre-computed templates. Baselines. To the best of our knowledge, we are not aware of any comparable approach for this task. Therefore, the baselines we consider are the following:\n\nS1: Pick the section from template INLINEFORM0 with the highest lexical similarity to INLINEFORM1 : S1 INLINEFORM2\n\nS2: Place the news into the most frequent section in INLINEFORM0 \n Question: What baseline model is used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-33e309aade9f43eead307fc10de8f0d1",
            "input": "Experimental Protocol ::: Datasets ::: Training Dataset\n(i) TTS System dataset: We trained our TTS system with a mixture of neutral and newscaster style speech. For a total of 24 hours of training data, split in 20 hours of neutral (22000 utterances) and 4 hours of newscaster styled speech (3000 utterances).\n\n(ii) Embedding selection dataset: As the evaluation was carried out only on the newscaster speaking style, we restrict our linguistic search space to the utterances associated to the newscaster style: 3000 sentences.\n\nExperimental Protocol ::: Datasets ::: Evaluation Dataset\nThe systems were evaluated on two datasets:\n\n(i) Common Prosody Errors (CPE): The dataset on which the baseline Prostron model fails to generate appropriate prosody. This dataset consists of complex utterances like compound nouns (22%), “or\" questions (9%), “wh\" questions (18%). This set is further enhanced by sourcing complex utterances (51%) from BIBREF24.\n\n(ii) LFR: As demonstrated in BIBREF25, evaluating sentences in isolation does not suffice if we want to evaluate the quality of long-form speech. Thus, for evaluations on LFR we curated a dataset of news samples. The news style sentences were concatenated into full news stories, to capture the overall experience of our intended use case. \n Question: What dataset is used for train/test of this method?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-3e9f4f465aa14aad9c6d87c46ae28454",
            "input": "We leveraged an existing, annotated Twitter dataset that was constructed based on a hierarchical model of depression-related symptoms BIBREF12 , BIBREF13 . The dataset contains 9,473 annotations for 9,300 tweets. \n Question: What dataset is used for this study?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-dd11e3e99e9246ff8a8d6f824bb7144e",
            "input": "Our first task is the recently-introduced Visual Question Answering challenge (VQA) BIBREF22 . The next set of experiments we consider focuses on GeoQA, a geographical question-answering task first introduced by Krish2013Grounded. \n Question: What benchmark datasets they use?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-2250ab1776354b1882b5a5013cb00665",
            "input": "For the intitial data collection using the CRWIZ platform, 145 unique dialogues were collected (each dialogue consists of a conversation between two participants).  The average time per assignment was 10 minutes 47 seconds, very close to our initial estimate of 10 minutes, and the task was available for 5 days in AMT. Out of the 145 dialogues, 14 (9.66%) obtained the bonus of $0.2 for resolving the emergency. We predicted that only a small portion of the participants would be able to resolve the emergency in less than 6 minutes, thus it was framed as a bonus challenge rather than a requirement to get paid. The fastest time recorded to resolve the emergency was 4 minutes 13 seconds with a mean of 5 minutes 8 seconds. Table TABREF28 shows several interaction statistics for the data collected compared to the single lab-based WoZ study BIBREF4.\n\nData Analysis ::: Subjective Data\nTable TABREF33 gives the results from the post-task survey. We observe, that subjective and objective task success are similar in that the dialogues that resolved the emergency were rated consistently higher than the rest.\n\nMann-Whitney-U one-tailed tests show that the scores of the Emergency Resolved Dialogues for Q1 and Q2 were significantly higher than the scores of the Emergency Not Resolved Dialogues at the 95% confidence level (Q1: $U = 1654.5$, $p < 0.0001$; Q2: $U = 2195$, $p = 0.009$, both $p < 0.05$). This indicates that effective collaboration and information ease are key to task completion in this setting.\n\nRegarding the qualitative data, one of the objectives of the Wizard-of-Oz technique was to make the participant believe that they are interacting with an automated agent and the qualitative feedback seemed to reflect this: “The AI in the game was not helpful at all [...]” or “I was talking to Fred a bot assistant, I had no other partner in the game“.\n\nData Analysis ::: Single vs Multiple Wizards\nIn Table TABREF28, we compare various metrics from the dialogues collected with crowdsourcing with the dialogues previously collected in a lab environment for a similar task. Most figures are comparable, except the number of emergency assistant turns (and consequently the total number of turns). To further understand these differences, we have first grouped the dialogue acts in four different broader types: Updates, Actions, Interactions and Requests, and computed the relative frequency of each of these types in both data collections. In addition, Figures FIGREF29 and FIGREF30 show the distribution of the most frequent dialogue acts in the different settings. It is visible that in the lab setting where the interaction was face-to-face with a robot, the Wizard used more Interaction dialogue acts (Table TABREF32). These were often used in context where the Wizard needed to hold the turn while looking for the appropriate prompt or waiting for the robot to arrive at the specified goal in the environment. On the other hand, in the crowdsourced data collection utterances, the situation updates were a more common choice while the assistant was waiting for the robot to travel to the specified goal in the environment.\n\nPerhaps not surprisingly, the data shows a medium strong positive correlation between task success and the number of Action type dialogue acts the Wizard performs, triggering events in the world leading to success ($R=0.475$). There is also a positive correlation between task success and the number of Request dialogue acts requesting confirmation before actions ($R=0.421$), e.g., “Which robot do you want to send?”. As Table 3 shows, these are relatively rare but perhaps reflect a level of collaboration needed to further the task to completion. Table TABREF40 shows one of the dialogues collected where the Emergency Assistant continuously engaged with the Operator through these types of dialogue acts.\n\nThe task success rate was also very different between the two set-ups. In experiments reported in BIBREF4, 96% of the dialogues led to the extinction of the fire whereas in the crowdsourcing setting only 9.66% achieved the same goal. In the crowdsourced setting, the robots were slower moving at realistic speeds unlike the lab setting. A higher bonus and more time for the task might lead to a higher task success rate.\n\nData Analysis ::: Limitations\nIt is important to consider the number of available participants ready and willing to perform the task at any one time. This type of crowdsourcing requires two participants to connect within a few minutes of each other to be partnered together. As mentioned above, there were some issues with participants not collaborating and these dialogues had to be discarded as they were not of use. \n Question: Is CRWIZ already used for data collection, what are the results?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-d5597d8f14c3432883f05bec625d19bb",
            "input": "he first, OpenIE 4 BIBREF5 , descends from two popular OIE systems OLLIE BIBREF10 and Reverb BIBREF10 . The second was MinIE BIBREF7 , which is reported as performing better than OLLIE, ClauseIE BIBREF9 and Stanford OIE BIBREF9 . \n Question: Which OpenIE systems were used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-274fa539613f4c9d95f50d731bffc749",
            "input": "We decided to explore the remaining space for improvement on the CBT by testing humans on a random subset of 50 named entity and 50 common noun validation questions that the psr ensemble could not answer correctly. \n Question: How do they show there is space for further improvement?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-ca6f7c951a4f4a649131b91587ad57a5",
            "input": "In our experiments, the RNN takes in spectrograms of utterances, passing them through two 2D-convolutional layers, followed by seven bi-directional recurrent layers and a fully-connected layer with softmax activation. All recurrent layers are batch normalized. At each timestep, the softmax activations give a probability distribution over the characters. CTC loss BIBREF8 is then computed from the timestep-wise probabilities. \n Question: Which model do they use for end-to-end speech recognition?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-14b00fc23b7d4141b5c9014d6484104d",
            "input": "In addition to the majority baseline, we also compare our results with a lexicon-based approach. For experimental results, we report majority baseline for each language where the majority baseline corresponds to a model's accuracy if it always predicts the majority class in the dataset. \n Question: what are the baselines?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-2ec0d928ece94e1e898e23a84c39c2a9",
            "input": "As the first step, we build three baseline LID systems, one based on the i-vector model, and the other two based on LSTM-RNN, using the speech data of two languages from Babel: Assamese and Georgian (AG). The two RNN LID baselines are: a standard RNN LID system (AG-RNN-LID) that discriminates between the two languages in its output, and a multi-task system (AG-RNN-MLT) that was trained to discriminate between the two languages as well as the phones. \n Question: Which is the baseline model?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-0b536d6510e34ec893c59907cd710ec4",
            "input": "The 10-fold cross validation with this setting gave a token-level accuracy of roughly 71%.  \n Question: Does the paper report translation accuracy for an automatic translation model for Tunisian to Arabish words?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-b648f9417bb34462bbfec76285e56240",
            "input": "Together, these users gave Gunrock an average rating of 3.65 (median: 4.0), which was elicited at the end of the conversation (“On a scale from 1 to 5 stars, how do you feel about talking to this socialbot again?\"). \n Question: Do they gather explicit user satisfaction data on Gunrock?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-176625b4dbbf4d6bb789b4e81f5f30a7",
            "input": "We showed in BIBREF22 that Pearson correlation is badly suited for STS. Instead, we compute the Spearman's rank correlation between the cosine-similarity of the sentence embeddings and the gold labels. \n Question: What metrics are used for the STS tasks?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-d64b1d4e3afb43f9a818e4acad2dcf5d",
            "input": "In this paper we propose two neural embedding models in order to learn continuous concept vectors based on the skip-gram model BIBREF11 . Our first model is the Concept Raw Context model (CRC) which utilizes concept mentions in a large scale KB to jointly learn embeddings of both words and concepts. Our second model is the Concept-Concept Context model (3C) which learns the embeddings of concepts from their conceptual contexts (i.e., contexts containing surrounding concepts only).  \n Question: What are the two neural embedding models?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-1bd0ac7d4e674862a135835a2583bb79",
            "input": "A key benefit is that the RNN infers a latent representation of state, obviating the need for state labels. \n Question: Does the latent dialogue state heklp their model?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-86561b06b94c43c8b29fbd9d66c48d8b",
            "input": "Yet, it is possible that the precise interpretability scores that are measured here are biased by the dataset used. However, it should be noted that the change in the interpretability scores for different word coverages might be effected by non-ideal subsampling of category words. Although our word sampling method, based on words' distances to category centers, is expected to generate categories that are represented better compared to random sampling of category words, category representations might be suboptimal compared to human designed categories. \n Question: What are the weaknesses of their proposed interpretability quantification method?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-a96a3d99440c4460ab4fb7d81701a153",
            "input": " In almost all genres, DenseNMT models are significantly better than the baselines. \n Question: did they outperform previous methods?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-f32a16d58bf448e5b5a82d68c785dc47",
            "input": "In this section, we present the details of the analysis performed on the data obtained pertaining to Twitter messages from January 2020 upto now, that is the time since the news of the Coronavirus outbreak in China was spread across nations. Also, the data prominently captures the tweets in English, Spanish, and French languages. \n Question: Do they collect only English data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-806a0d457e73455eb6c963162917cfd1",
            "input": "Similar trends are seen in the performance of other two state-of-the-art approaches BIBREF9 , BIBREF8 . \n Question: What are the state of the art models?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-68f410f9594c4de88bc59488338fa262",
            "input": "This section introduces our probabilistic model that infers keyword expectation and trains the target model simultaneously. \n Question: What type of classifiers are used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-27d8911867a94885bbb9e4e57bcccd51",
            "input": "WikiSmall consists of ordinary and simplified sentences from the ordinary and simple English Wikipedias, which has been used as benchmark for evaluating text simplification BIBREF17 , BIBREF18 , BIBREF8 . The training set has 89,042 sentence pairs, and the test set has 100 pairs. WikiLarge is also from Wikipedia corpus whose training set contains 296,402 sentence pairs BIBREF19 , BIBREF20 . WikiLarge includes 8 (reference) simplifications for 2,359 sentences split into 2,000 for development and 359 for testing. \n Question: what are the sizes of both datasets?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-aa7dba9fb862435e8d419f7c94f91d53",
            "input": "The regressor and classifier used Support Vector Regression (SVR) and Support Vector Classification (SVC) respectively. The bottom section of Table TABREF26 shows the results of several variants of the neural architecture. The table includes a neural regressor (NNR) and a neural classifier (NNC).  \n Question: What approaches without reinforcement learning have been tried?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-9b466e978e564d47b720ba782f51e481",
            "input": "Inspired by BIBREF12, we integrate in this paper a boundary assembling step into the state-of-the-art LSTM model for Chinese word segmentation, and feed the output into a CRF model for NER, resulting in a 2% absolute improvement on the overall F1 score over current state-of-the-art methods. \n Question: What state-of-the-art deep neural network is used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-e315a29b152f492c93bbe4bc5faedc0e",
            "input": "We have created a dataset of discharge summaries and nursing notes, all in the English language, with a focus on frequently readmitted patients, labeled with 15 clinical patient phenotypes believed to be associated with risk of recurrent Intensive Care Unit (ICU) readmission per our domain experts (co-authors LAC, PAT, DAG) as well as the literature. BIBREF10 BIBREF11 BIBREF12 \n Question: How many different phenotypes are present in the dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-0d2df9a94998431ea417174cbcfaad48",
            "input": "For the language modeling evaluation, we also evaluate a baseline without knowledge distillation (termed NoKD), with a model parameterized identically to the distilled student models but trained directly on the teacher model objective from scratch. For downstream tasks, we compare with NoKD as well as Patient Knowledge Distillation (PKD) from BIBREF34, who distill the 12-layer BERTBASE model into 3 and 6-layer BERT models by using the teacher model's hidden states. \n Question: What state-of-the-art compression techniques were used in the comparison?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-c6f380356b044e8593101f9418fdb747",
            "input": "DeepMine is publicly available for everybody with a variety of licenses for different users. It was collected using crowdsourcing BIBREF4. The data collection was done using an Android application. Each respondent installed the application on his/her personal device and recorded several phrases in different sessions.  \n Question: how was the speech collected?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-4333957cfc1542038dd2afd544cf33b7",
            "input": "To increase the variety of corpus, we selected 4 English corpora and 4 Mandarin corpora in addition to the low resource language corpora. \n Question: Do they test their approach on large-resource tasks?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-3bf578faedab43e69297a862b4708e61",
            "input": "We considered UD1.2 corpora for the following 16 languages: Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish and Swedish. \n Question: which languages are explored?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-1301b2a2d8224bfdacb4cb13066c9f8f",
            "input": "Given the annotated tweets, we wanted to ascertain the distribution of: types of offensive language, genres where it is used, the dialects used, and the gender of users using such language. As the figure shows, sports and politics are most dominant for offensive language including vulgar and hate speech. As for dialect, we looked at MSA and four major dialects, namely Egyptian (EGY), Leventine (LEV), Maghrebi (MGR), and Gulf (GLF). Figure FIGREF14 shows that 71% of vulgar tweets were written in EGY followed by GLF, which accounted for 13% of vulgar tweets. MSA was not used in any of the vulgar tweets. As for offensive tweets in general, EGY and GLF were used in 36% and 35% of the offensive tweets respectively. Unlike the case of vulgar language where MSA was non-existent, 15% of the offensive tweets were in fact written in MSA. For hate speech, GLF and EGY were again dominant and MSA consistuted 21% of the tweets. This is consistent with findings for other languages such as English and Italian where vulgar language was more frequently associated with colloquial language BIBREF24, BIBREF25. Regarding the gender, Figure FIGREF15 shows that the vast majority of offensive tweets, including vulgar and hate speech, were authored by males. Female Twitter users accounted for 14% of offensive tweets in general and 6% and 9% of vulgar and hate speech respectively. Figure FIGREF16 shows a detailed categorization of hate speech types, where the top three include insulting groups based on their political ideology, origin, and sport affiliation. Religious hate speech appeared in only 15% of all hate speech tweets. \n Question: How did they analyze which topics, dialects and gender are most associated with tweets?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-085f2d9286054ccd941224b89e7811e4",
            "input": "We extract data from the WMT'14 English-French (En-Fr) and English-German (En-De) datasets.  \n Question: Are this techniques used in training multilingual models, on what languages?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-db8d2d7eca3d4643aa4060ada96893df",
            "input": "For the proposed model, we denote INLINEFORM0 parameterized by INLINEFORM1 as a neural-based feature encoder that maps documents from both domains to a shared feature space, and INLINEFORM2 parameterized by INLINEFORM3 as a fully connected layer with softmax activation serving as the sentiment classifier. We have left the feature encoder INLINEFORM0 unspecified, for which, a few options can be considered. In our implementation, we adopt a one-layer CNN structure from previous works BIBREF22 , BIBREF4 , as it has been demonstrated to work well for sentiment classification tasks. \n Question: What is the architecture of the model?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-47160111aefb476da9a8d34c04dbdf19",
            "input": "Our desiderata for the task collection were: sufficient diversity, existence of fairly large datasets for training, and success as standalone training objectives for sentence representations.\n\nMulti-task training setup \n Question: Which model architecture do they for sentence encoding?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-2fbb9d9c5bbe472ba28ec214f8071f29",
            "input": "The evaluation of chatbots remains an open problem in the field. Recent work BIBREF25 has shown that the automatic evaluation metrics borrowed from machine translation such as BLEU score BIBREF26 tend to align poorly with human judgement. Therefore, in this paper, we mainly adopt human evaluation, along with perplexity, following the existing work. \n Question: Is some other metrics other then perplexity measured?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-750f9e2df3384902bb8289e9380df87e",
            "input": "The baseline model BIBREF3 is implemented with a recurrent neural network based encoder-decoder framework. \n Question: Do they compare against Noraset et al. 2017?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-a19108ccf9fa46858cde34dcf4dcdc0c",
            "input": "Depechemood is a lexicon-based emotion detection method gathered from crowd-annotated news BIBREF24. Drawing on approximately 23.5K documents with average of 500 words per document from rappler.com, researchers asked subjects to report their emotions after reading each article. They then multiplied the document-emotion matrix and word-document matrix to derive emotion-word matrix for these words.  \n Question: How id Depechemood trained?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-c36612645171493cb8b2dbc102c958a7",
            "input": "The input document/summary that may have unordered sentences is processed so that it will have sentences clustered together. To test our approach, we jumble the ordering of sentences in a document, process the unordered document and compare the similarity of the output document with the original document. To structure an unordered document is an essential task in many applications. It is a post-requisite for applications like multiple document extractive text summarization where we have to present a summary of multiple documents. It is a prerequisite for applications like question answering from multiple documents where we have to present an answer by processing multiple documents. \n Question: What is an unordered text document, do these arise in real-world corpora?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-f0900aa30be349eba74fae3800c55b9b",
            "input": "Conclusion \n Question: What is triangulation?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-faff22cef695412285248597a777c61d",
            "input": "For the other techniques, we extract paragraphs containing any word from a predetermined list of LGTBQ terms (shown in Table TABREF19) \n Question: How do they identify discussions of LGBTQ people in the New York Times?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-c54a1bda0c3348098c76700c2b6be16f",
            "input": "Throughout this article, we use two different embedding spaces. The first is the widely used representation built on GoogleNews BIBREF8 . The second is taken from BIBREF2 , and was trained on a Reddit dataset BIBREF9 . \n Question: Which embeddings do they detect biases in?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-30f05520a40d46b6b599cd29e696ca1b",
            "input": "Nonetheless, the main caveat of this basic pre-training is that the source encoder is trained to be used by an English decoder, while the target decoder is trained to use the outputs of an English encoder — not of a source encoder. \n Question: Is pivot language used in experiments English or some other language?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-db85b1f20a044fac8430c9e99c24ef45",
            "input": " For a given query $q = \\langle s, r, ? \\rangle $ , we identify mentions in $S_q$ of the entities in $C_q \\cup \\lbrace s\\rbrace $ and create one node per mention. This process is based on the following heuristic:\n\nwe consider mentions spans in $S_q$ exactly matching an element of $C_q \\cup \\lbrace s\\rbrace $ . Admittedly, this is a rather simple strategy which may suffer from low recall.\n\nwe use predictions from a coreference resolution system to add mentions of elements in $C_q \\cup \\lbrace s\\rbrace $ beyond exact matching (including both noun phrases and anaphoric pronouns). In particular, we use the end-to-end coreference resolution by BIBREF16 .\n\nwe discard mentions which are ambiguously resolved to multiple coreference chains; this may sacrifice recall, but avoids propagating ambiguity. \n Question: How did they detect entity mentions?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-aec1f5ae63c14d88aa88032de99067b1",
            "input": "Various multilingual extensions of NMT have already been proposed in the literature. The authors of BIBREF18 , BIBREF19 apply multitask learning to train models for multiple languages. Zoph and Knight BIBREF20 propose a multi-source model and BIBREF21 introduces a character-level encoder that is shared across several source languages. In our setup, we will follow the main idea proposed by Johnson et al. BIBREF22 . The authors of that paper suggest a simple addition by means of a language flag on the source language side (see Figure 2 ) to indicate the target language that needs to be produced by the decoder. This flag will be mapped on a dense vector representation and can be used to trigger the generation of the selected language. The authors of the paper argue that the model enables transfer learning and supports the translation between languages that are not explicitly available in training. \n Question: What neural machine translation models can learn in terms of transfer learning?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-a4bbf55c0c5e4c909cfb9ea117a3c378",
            "input": "As this corpus of annotated patient notes comprises original healthcare data which contains protected health information (PHI) per The Health Information Portability and Accountability Act of 1996 (HIPAA) BIBREF16 and can be joined to the MIMIC-III database, individuals who wish to access to the data must satisfy all requirements to access the data contained within MIMIC-III. To satisfy these conditions, an individual who wishes to access the database must take a “Data or Specimens Management” course, as well as sign a user agreement, as outlined on the MIMIC-III database webpage, where the latest version of this database will be hosted as “Annotated Clinical Texts from MIMIC” BIBREF17. This corpus can also be accessed on GitHub after completing all of the above requirements. \n Question: Is this dataset publicly available for commercial use?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-f90466879ecf46c484932616302302a1",
            "input": "As in the work of E. Tong et al. ( BIBREF9 ), we pre-train word embeddings using a skip-gram model BIBREF4 applied to unlabeled data from escort ads, however, we go further by analyzing the emojis' embeddings and thereby expand the trafficking lexicon. \n Question: Do they use pretrained word embeddings?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-8b89c6d5693c4eca80942aaf732dfb88",
            "input": "We focus here on the Europarl domain, for which we have ample data in several languages, and use as in-domain training data the Europarl corpus BIBREF5 for two translation directions: English INLINEFORM0 German and English INLINEFORM1 French. When measuring out-of-domain performance, we will use the WMT newstest 2014. or French we use samples from News-Commentary-11 and Wikipedia from WMT 2014 shared translation task, as well as the Multi-UN BIBREF9 and EU-Bookshop BIBREF10 corpora. For German, we use samples from News-Commentary-11, Rapid, Common-Crawl (WMT 2017) and Multi-UN (see table TABREF5 ).  \n Question: what dataset is used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-822d8cd3ba7e41ef971943f02e257acc",
            "input": "Compared with 2015, our QA-SRL gold yielded 1593 annotations, with 989 core and 604 adjuncts, while theirs yielded 1315 annotations, 979 core and 336 adjuncts.  \n Question: How much more coverage is in the new dataset?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-1c221f7e54cd4f41bc26d69cb22176dc",
            "input": "Even though this corpus has incorrect sentences and their emotional labels, they lack their respective corrected sentences, necessary for the training of our model. In order to obtain this missing information, we outsource native English speakers from an unbiased and anonymous platform, called Amazon Mechanical Turk (MTurk) BIBREF19, which is a paid marketplace for Human Intelligence Tasks (HITs). We use this platform to create tasks for native English speakers to format the original incorrect tweets into correct sentences. Some examples are shown in Table TABREF12. The dataset used to evaluate the models' performance is the Chatbot Natural Language Unerstanding (NLU) Evaluation Corpus, introduced by Braun et al. BIBREF20 to test NLU services. It is a publicly available benchmark and is composed of sentences obtained from a German Telegram chatbot used to answer questions about public transport connections. The dataset has two intents, namely Departure Time and Find Connection with 100 train and 106 test samples, shown in Table TABREF18. Even though English is the main language of the benchmark, this dataset contains a few German station and street names. \n Question: Do they report results only on English datasets?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-0e5c563d93c041d392490b8591b84811",
            "input": "Evaluation is done using the development set, consisting of 22 documents and 1112 PIE candidates, and the test set, which consists of 23 documents and 1127 PIE candidates. For each method the best set of parameters and/or options is determined using the development set, after which the best variant by F1-score of each method is evaluated on the test set.\n\nSince these documents in the corpus are exhaustively annotated for PIEs (see Section SECREF40), we can calculate true and false positives, and false negatives, and thus precision, recall and F1-score. \n Question: Are PIEs extracted automatically subjected to human evaluation?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-eb635230c6f9410b935afb3cf3859be9",
            "input": "Before designing this NLP toolkit, we conducted a survey among engineers and identified a spectrum of three typical personas. \n Question: How do the authors evidence the claim that many engineers find it a big overhead to choose from multiple frameworks, models and optimization techniques?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-63b6a947404e46139c3e4aebf00efe7f",
            "input": "To train our model, we generated a dataset of 20,000 demonstrated 7 DOF trajectories (6 robot joints and 1 gripper dimension) in our simulated environment together with a sentence generator capable of creating natural task descriptions for each scenario. In order to create the language generator, we conducted an human-subject study to collect sentence templates of a placement task as well as common words and synonyms for each of the used features. By utilising these data, we are able to generate over 180,000 unique sentences, depending on the generated scenario. To test our model, we generated 500 new scenario testing each of the three features to identify the correct target among other bowls.  \n Question: Does proposed end-to-end approach learn in reinforcement or supervised learning manner?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-2ec08f06c57644d7920d846e45525178",
            "input": " Inspired by the line of work reported in these studies, we propose to use modified objective functions for a different purpose: learning more interpretable dense word embeddings. By doing this, we aim to incorporate semantic information from an external lexical resource into the word embedding so that the embedding dimensions are aligned along predefined concepts. This alignment is achieved by introducing a modification to the embedding learning process. In our proposed method, which is built on top of the GloVe algorithm BIBREF2 , the cost function for any one of the words of concept word-groups is modified by the introduction of an additive term to the cost function.  \n Question: What is the additive modification to the objective function?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-aff4c8e6277640edaba54b2a4b534f71",
            "input": "We test the existing debiasing approaches, CDA and REG, as well but since BIBREF5 reported that results fluctuate substantially with different REG regularization coefficients, we perform hyperparameter tuning and report the best results in Table TABREF12 . \n Question: which existing strategies are compared?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-4600cde056da4b09844c2dc5547ad25a",
            "input": "Inspired by the psycholinguistic studies BIBREF5 , BIBREF4 , BIBREF7 proposed a semi-character based RNN (ScRNN) that processes a sentence of words with misspelled characters, predicting the correct words at each step. Let $s = \\lbrace w_1, w_2, \\dots , w_n\\rbrace $ denote the input sentence, a sequence of constituent words $w_i$ . Each input word ( $w_i$ ) is represented by concatenating (i) a one hot vector of the first character ( $\\mathbf {w_{i1}}$ ); (ii) a one hot representation of the last character ( $\\mathbf {w_{il}}$ , where $l$ is the length of word $w_i$ ); and (iii) a bag of characters representation of the internal characters ( $\\sum _{j=2}^{l-1}\\mathbf {w_{ij}})$ . ScRNN treats the first and the last characters individually, and is agnostic to the ordering of the internal characters. Each word, represented accordingly, is then fed into a BiLSTM cell. At each sequence step, the training target is the correct corresponding word (output dimension equal to vocabulary size), and the model is optimized with cross-entropy loss. \n Question: What is a semicharacter architecture?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-d1119174116744798630ce079f6790af",
            "input": "To help the human representative quickly determine the cause of the escalation, we generate a visualization of the user's turns using the attention weights to highlight the turns influential in the escalation decision. \n Question: Do they explain model predictions solely on attention weights?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-2197b234273947318d13407f7fca359a",
            "input": "Pre-trained English NER model We construct the English NER system following BIBREF7 . This system uses a bidirectional LSTM as a character-level language model to take context information for word embedding generation.  For pre-trained English NER system, we use the default NER model of Flair. \n Question: Which pre-trained English NER model do they use?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-472695848b2f402f8eedc13e65b6bafc",
            "input": "Accordingly, marcheggiani2017 presented a neural model putting syntax aside for dependency-based SRL and obtain favorable results, which overturns the inherent belief that syntax is indispensable in SRL task BIBREF11 . \n Question: Are there syntax-agnostic SRL models before?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-5d49e525a8294bb2aa4299f823b7d98a",
            "input": "In this section we discuss the state of the art on conversational systems in three perspectives: types of interactions, types of architecture, and types of context reasoning. ELIZA BIBREF11 was one of the first softwares created to understand natural language processing.  Right after ELIZA came PARRY, developed by Kenneth Colby, who is psychiatrist at Stanford University in the early 1970s. A.L.I.C.E. (Artificial Linguistic Internet Computer Entity) BIBREF12 appeared in 1995 but current version utilizes AIML, an XML language designed for creating stimulus-response chat robots BIBREF13 . Cleverbot (1997-2014) is a chatbot developed by the British AI scientist Rollo Carpenter.  \n Question: What is the state of the art described in the paper?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-c84ae15ec26e420bbf3b2e8d8a7a28a3",
            "input": "Starting with this list, we can locate the profile page for a user, and subsequently extract additional information, which includes fields such as name, email, occupation, industry, and so forth. We also generate two maps that delineate the gender distribution in the dataset. Our dataset provides mappings between location, profile information, and language use, which we can leverage to generate maps that reflect demographic, linguistic, and psycholinguistic properties of the population represented in the dataset. \n Question: Which demographic dimensions of people do they obtain?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-028279c9b05b4bb98c8c13e44472600f",
            "input": " For PHASE-ONE, we randomly shuffled and divided the data (1913 signals from 14 individuals) into train (80%), development (10%) and test sets (10%). In PHASE-TWO, in order to perform a fair comparison with the previous methods reported on the same dataset, we perform a leave-one-subject out cross-validation experiment using the best settings we learn from PHASE-ONE. \n Question: How many electrodes were used on the subject in EEG sessions?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-4a1b0e109c844deb9cb4da3d74446453",
            "input": "For each object, $O_i$ in the input Open IE tuple $(S; P; O_1; O_2 \\ldots )$ , we add a triple $(S; P; O_i)$ to this table. \n Question: Are the OpenIE extractions all triples?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-02407108ff67428185694648054355d0",
            "input": "We decided to use sentences involving at least one race- or gender-associated word. The sentences were intended to be short and grammatically simple. We also wanted some sentences to include expressions of sentiment and emotion, since the goal is to test sentiment and emotion systems. \n Question: What criteria are used to select the 8,640 English sentences?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-c28ead29f4dc4e658b0500c380670b2f",
            "input": "We used the relation classification dataset of the SemEval 2010 task 8 BIBREF8 . It consists of sentences which have been manually labeled with 19 relations (9 directed relations and one artificial class Other). 8,000 sentences have been distributed as training set and 2,717 sentences served as test set. \n Question: Which dataset do they train their models on?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-466d149ef4f44f088bc5e48f3803504e",
            "input": "Then, we created test sets with varying levels of perturbation operations - $\\lbrace 20\\%,40\\%,60\\%\\rbrace $. \n Question: Are recurrent neural networks trained on perturbed data?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-aa98bd07e2a74d89ba23820cae19a618",
            "input": "GhostVLAD is an extension of the NetVLAD approach, which we discussed in the previous section. GhostVLAD works exactly similar to NetVLAD except it adds Ghost clusters along with the NetVLAD clusters. So, now we will have a K+G number of clusters instead of K clusters. The Ghost clusters are added to map any noisy or irrelevant content into ghost clusters and are not included during the feature aggregation stage, as shown in Figure 1 (Right side). \n Question: What is the GhostVLAD approach?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-4af00810d23d46a6968449753f8155bc",
            "input": "We evaluate our approaches on two public Chinese span-extraction machine reading comprehension datasets: CMRC 2018 (simplified Chinese) BIBREF8 and DRCD (traditional Chinese) BIBREF9. The statistics of the two datasets are listed in Table TABREF29. \n Question: Is this a span-based (extractive) QA task?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-561131d6c4c54deab972e19086dc1a58",
            "input": "Subsequently, an LSTM decoder generates the characters in the output word form using encoder states and an attention mechanism. \n Question: Does the model have attention?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-d37ad02c64714f3ba55579b3c16d8dfc",
            "input": "For this knowledge, we create a knowledge base by counting how many times a predicate-argument tuple appears in a corpus and use the resulted number to represent the preference strength. Specifically, we use the English Wikipedia as the base corpus for such counting. \n Question: What is the source of external knowledge?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-f77f9a6f4ed34030b3520dd8bff53f95",
            "input": "We started by answering always YES (in batch 2 and 3) to get the baseline performance. For batch 4 we used entailment. \n Question: What was the baseline model?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-b6534fed92464e45a7b43f9e2b190cd6",
            "input": "It has been shown that one can significantly increase the semantic information carried by a NER system when we successfully linking entities from a deep learning method to the related entities from a knowledge base BIBREF26 , BIBREF27 . Redirection: For the Wikidata linking element, we recognize that the lookup will be constrained by the most common lookup name for each entity.  \n Question: How do they combine a deep learning model with a knowledge base?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-9140cb82d089404da344881c9cf58cae",
            "input": " The unlabeled set of development data was used in the training of both the UBM and the i-vector extractor. \n Question: Do they single out a validation set from the fixed SRE training set?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-c20d584af48f4f7ba5cb3ab91218b469",
            "input": "First, we propose a class of recurrent-like neural networks for NLP tasks that satisfy the differential equation DISPLAYFORM0\n\nwhere DISPLAYFORM0\n\nand where INLINEFORM0 and INLINEFORM1 are learned functions. INLINEFORM2 corresponds to traditional RNNs, with INLINEFORM3 . For INLINEFORM4 , this takes the form of RNN cells with either nested internal memories or dependencies that extend temporally beyond the immediately previous hidden state.  \n Question: What novel class of recurrent-like networks is proposed?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-dc75f0dfc3094b77b5f2efa323535c8e",
            "input": "Embedding: We developed different variations of our models with a simple lookup table embeddings learned from scratch and using high-performance contextual embeddings, which are ELMo BIBREF11, BERT BIBREF16 and ClinicalBERT BIBREF13 (trained and provided by the authors). \n Question: What embeddings are used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-8b344a7e9cee4945b8e362592ab54f96",
            "input": "One of the major drawbacks of SCRF models is their high computational cost. In our experiments, the CTC model is around 3-4 times faster than the SRNN model that uses the same RNN encoder. To cut down the computational cost, we investigated if CTC can be used to pretrain the RNN encoder to speed up the training of the joint model. Figure 2 shows the convergence curves of the joint model with and without CTC pretraining, and we see pretraining indeed improves the convergence speed of the joint model. \n Question: Can SCRF be used to pretrain the model?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-36fc3fdec7ac41f781dcddea9e5eaece",
            "input": "However, all previous IE benchmarks BIBREF18 are too small to train neural network models typically used in QA, and thus we need to build a large benchmark. Therefore, we build a large scale benchmark named QA4IE benchmark which consists of 293K Wikipedia articles and 2M golden relation triples with 636 different relation types. We manually find 148 relations which can be projected to a WikiData relation out of 2064 DBpedia relations. \n Question: Was this benchmark automatically created from an existing dataset?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-e15de20f469f44b8b9919c6a9745528a",
            "input": "This step entails counting occurrences of all words in the training corpus and sorting them in order of decreasing occurrence. As mentioned, the vocabulary is taken to be the INLINEFORM0 most frequently occurring words, that occur at least some number INLINEFORM1 times. It is implemented in Spark as a straight-forward map-reduce job. \n Question: Do they perform any morphological tokenization?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-3cf34d843f51435498b6c0c75224ce9a",
            "input": "We recruited 100 crowdworkers on Amazon Mechanical Turk (AMT) and measured completion times and accuracies for typing randomly sampled sentences from the Yelp corpus.  \n Question: How many participants were trying this communication game?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-6f55905aa9644693bb81f7249c70938d",
            "input": "We constructed our seed lexicon consisting of 15 positive words and 15 negative words, as shown in Section SECREF27.  \n Question: How big is seed lexicon used for training?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-c06168ad97a84f95a7a8d5b296146da4",
            "input": "The Block Zoo is an open framework, and more modules can be added in the future. Embedding Layer: Word/character embedding and extra handcrafted feature embedding such as pos-tagging are supported. Neural Network Layers: Block zoo provides common layers like RNN, CNN, QRNN BIBREF2 , Transformer BIBREF3 , Highway network, Encoder Decoder architecture, etc. Furthermore, attention mechanisms are widely used in neural networks. Thus we also support multiple attention layers, such as Linear/Bi-linear Attention, Full Attention BIBREF4 , Bidirectional attention flow BIBREF5 , etc. Meanwhile, regularization layers such as Dropout, Layer Norm, Batch Norm, etc are also supported for improving generalization ability. Loss Function: Besides of the loss functions built in PyTorch, we offer more options such as Focal Loss BIBREF6 .\n\nMetrics: For classification task, AUC, Accuracy, Precision/Recall, F1 metrics are supported. For sequence labeling task, F1/Accuracy are supported. For knowledge distillation task, MSE/RMSE are supported. For MRC task, ExactMatch/F1 are supported. \n Question: What neural network modules are included in NeuronBlocks?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-eba6c8d724174e14843e0e8b6303246e",
            "input": "We pit our model against the following baselines: 1) SVM with unigram, bigram, and trigram features, which is a standard yet rather strong classifier for text features; 2) SVM with average word embedding, where a document is represented as a continuous representation by averaging the embeddings of the composite words; 3) SVM with average transformed word embeddings (the INLINEFORM0 in equation EQREF6 ), where a document is represented as a continuous representation by averaging the transformed embeddings of the composite words; 4) two mature deep learning models on text classification, CNN BIBREF3 and Recurrent Convolutional Neural Networks (RCNN) BIBREF0 , where the hyperparameters are based on their work; 5) the above SVM and deep learning models with comment information;  \n Question: What are the baselines?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-1ef37d0b0aa9415aa47448bdc68a79da",
            "input": "The SnapCaptions dataset is composed of 10K user-generated image (snap) and textual caption pairs where named entities in captions are manually labeled by expert human annotators (entity types: PER, LOC, ORG, MISC). \n Question: Which types of named entities do they recognize?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-7090ab6896844c5f8544bd880ce1f089",
            "input": "In the first task, participants were instructed to read the sentences naturally, without any specific task other than comprehension. Participants were told to read the sentences normally without any special instructions. \n Question: What is a normal reading paradigm?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-f35b322b986240ddb5aa0f65c7077c3b",
            "input": "We thus try to remove the language-specific information from the representations by centering the representations of sentences in each language so that their average lies at the origin of the vector space. \n Question: Are language-specific and language-neutral components disjunctive?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-504e3721ba0940d1b73e69363670ca20",
            "input": "The CRF module achieved the best result on the Thai sentence segmentation task BIBREF8 ; therefore, we adopt the Bi-LSTM-CRF model as our baseline.  \n Question: Which deep learning architecture do they use for sentence segmentation?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-aa9a07e1249f45e48d5373168691a56d",
            "input": "Moreover, because English does not mark grammatical gender, approaches developed for English are not transferable to morphologically rich languages that exhibit gender agreement BIBREF8 . \n Question: Why does not the approach from English work on other languages?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-9ddec1c934f44676bbcd762d2240b462",
            "input": "RKS approach proposed in BIBREF21, BIBREF22, explicitly maps data vectors to a space where linear separation is possible. The RKS method provides an approximate kernel function via explicit mapping. \n Question: What is the Random Kitchen Sink approach?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-f3118abddeea4dcab6448922d38629f4",
            "input": "To develop the dataset, we extracted 200 sentences from Sorani Kurdish books of grades one to three of the primary school in the Kurdistan Region of Iraq. We randomly created 2000 sentences from the extracted sentences. \n Question: How was the dataset collected?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-30efbc43cb99456f82ac459be82226c2",
            "input": " In this work, we consider words that can indicate the characteristics of the neighbor words as contextual keywords and develop an approach to generate features from the automatically extracted contextual keywords. \n Question: What contextual features are used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-83a2276c60f844a5a0d819b85b5b476a",
            "input": "To verify our assumption that target encoding and orthogonal regularization help to boost the diversity of generated sequences, we use two metrics, one quantitative and one qualitative, to measure diversity of generation. First, we simply calculate the average unique predictions produced by both INLINEFORM0 and INLINEFORM1 in experiments shown in Section SECREF36 .  In this example there are 29 ground truth phrases. Neither of the models is able to generate all of the keyphrases, but it is obvious that the predictions from INLINEFORM0 all start with “test”, while predictions from INLINEFORM1 are diverse. \n Question: How is keyphrase diversity measured?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-72e61ee205bd4a39917b50fbbcab872c",
            "input": "In order to make the corpus collection easier and faster, we adopted a semi-automatic procedure based on sequential neural models BIBREF19, BIBREF20. Instead, since tokens are transcribed at morpheme level, we split Arabish tokens into characters, and Arabic tokens into morphemes, and we treated each token itself as a sequence. Our model learns thus to map Arabish characters into Arabic morphemes. With this model we automatically transcribed into Arabic morphemes, roughly, 5,000 additional tokens, corresponding to the second annotation block.  Manual transcription plus a \n Question: How does the semi-automatic construction process work?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-c1f4bd164e954345b29539ab07d8a470",
            "input": "Vanilla ST baseline: The vanilla ST BIBREF9 has only a speech encoder and a decoder. It is trained from scratch on the ST-TED corpus.\n\nPre-training baselines: We conduct three pre-training baseline experiments: 1) encoder pre-training, in which the ST encoder is initialized from an ASR model; 2) decoder pre-training, in which the ST decoder is initialized from an MT model; and 3) encoder-decoder pre-training, where both the encoder and decoder are pre-trained. The ASR model has the same architecture with vanilla ST model, trained on the mixture of ST-TED and TED-LIUM2 corpus. The MT model has a text encoder and decoder with the same architecture of which in TCEN. It is first trained on WMT data (out-of-domain) and then fine-tuned on in-domain data.\n\nMulti-task baselines: We also conduct three multi-task baseline experiments including one-to-many setting, many-to-one setting, and many-to-many setting. In the first two settings, we train the model with $\\alpha _{st}=0.75$ while $\\alpha _{asr}=0.25$ or $\\alpha _{mt}=0.25$. For many-to-many setting, we use $\\alpha _{st}=0.6, \\alpha _{asr}=0.2$ and $\\alpha _{mt}=0.2$.. For MT task, we use only in-domain data.\n\nMany-to-many+pre-training: We train a many-to-many multi-task model where the encoders and decoders are derived from pre-trained ASR and MT models. Triangle+pre-train: BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 proposed a triangle multi-task strategy for speech translation. Their model solves the subnet waste issue by concatenating an ST decoder to an ASR encoder-decoder model. Notably, their ST decoder can consume representations from the speech encoder as well as the ASR decoder. For a fair comparison, the speech encoder and the ASR decoder are initialized from the pre-trained ASR model. The Triangle model is fine-tuned under their multi-task manner. \n Question: What are the baselines?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-3d45c95f3413483bb46bed0c31307223",
            "input": "Results ::: CoinCollector\nIn this setting, we compare the number of actions played in the environment (frames) and the score achieved by the agent (i.e. +1 reward if the coin is collected). In Go-Explore we also count the actions used to restore the environment to a selected cell, i.e. to bring the agent to the state represented in the selected cell. This allows a one-to-one comparison of the exploration efficiency between Go-Explore and algorithms that use a count-based reward in text-based games. Importantly, BIBREF8 showed that DQN and DRQN, without such counting rewards, could never find a successful trajectory in hard games such as the ones used in our experiments. Figure FIGREF17 shows the number of interactions with the environment (frames) versus the maximum score obtained, averaged over 10 games of the same difficulty. As shown by BIBREF8, DRQN++ finds a trajectory with the maximum score faster than to DQN++. On the other hand, phase 1 of Go-Explore finds an optimal trajectory with approximately half the interactions with the environment. Moreover, the trajectory length found by Go-Explore is always optimal (i.e. 30 steps) whereas both DQN++ and DRQN++ have an average length of 38 and 42 respectively. In case of Game CoinCollector,  In CookingWorld, we compared models in the three settings mentioned earlier, namely, single, joint, and zero-shot. In all experiments, we measured the sum of the final scores of all the games and their trajectory length (number of steps). Table TABREF26 summarizes the results in these three settings. Phase 1 of Go-Explore on single games achieves a total score of 19,530 (sum over all games), which is very close to the maximum possible points (i.e. 19,882), with 47,562 steps. A winning trajectory was found in 4,279 out of the total of 4,440 games. This result confirms again that the exploration strategy of Go-Explore is effective in text-based games. Next, we evaluate the effectiveness and the generalization ability of the simple imitation learning policy trained using the extracted trajectories in phase 1 of Go-Explore in the three settings mentioned above. In this setting, each model is trained from scratch in each of the 4,440 games based on the trajectory found in phase 1 of Go-Explore (previous step). As shown in Table TABREF26, the LSTM-DQN BIBREF7, BIBREF8 approach without the use of admissible actions performs poorly. One explanation for this could be that it is difficult for this model to explore both language and game strategy at the same time; it is hard for the model to find a reward signal before it has learned to model language, since almost none of its actions will be admissible, and those reward signals are what is necessary in order to learn the language model. As we see in Table TABREF26, however, by using the admissible actions in the $\\epsilon $-greedy step the score achieved by the LSTM-DQN increases dramatically (+ADM row in Table TABREF26). DRRN BIBREF10 achieves a very high score, since it explicitly learns how to rank admissible actions (i.e. a much simpler task than generating text). Finally, our approach of using a Seq2Seq model trained on the single trajectory provided by phase 1 of Go-Explore achieves the highest score among all the methods, even though we do not use admissible actions in this phase. However, in this experiment the Seq2Seq model cannot perfectly replicate the provided trajectory and the total score that it achieves is in fact 9.4% lower compared to the total score achieved by phase 1 of Go-Explore. Figure FIGREF61 (in Appendix SECREF60) shows the score breakdown for each level and model, where we can see that the gap between our model and other methods increases as the games become harder in terms of skills needed. In this setting the 4,440 games are split into training, validation, and test games. The split is done randomly but in a way that different difficulty levels (recipes 1, 2 and 3), are represented with equal ratios in all the 3 splits, i.e. stratified by difficulty. As shown in Table TABREF26, the zero-shot performance of the RL baselines is poor, which could be attributed to the same reasons why RL baselines under-perform in the Joint case. Especially interesting is that the performance of DRRN is substantially lower than that of the Go-Explore Seq2Seq model, even though the DRRN model has access to the admissible actions at test time, while the Seq2Seq model (as well as the LSTM-DQN model) has to construct actions token-by-token from the entire vocabulary of 20,000 tokens. On the other hand, Go-Explore Seq2Seq shows promising results by solving almost half of the unseen games. Figure FIGREF62 (in Appendix SECREF60) shows that most of the lost games are in the hardest set, where a very long sequence of actions is required for winning the game. These results demonstrate both the relative effectiveness of training a Seq2Seq model on Go-Explore trajectories, but they also indicate that additional effort needed for designing reinforcement learning algorithms that effectively generalize to unseen games. \n Question: How better does new approach behave than existing solutions?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-3d27d1d7af13488194b220466852ddf5",
            "input": "The dataset comprises a total of 353 conversations from 40 speakers (11 nurses, 16 patients, and 13 caregivers) with consent to the use of anonymized data for research. In Section SECREF16 , we build templates and expression pools using linguistic analysis followed by manual verification. \n Question: How big is their created dataset?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-c59aa864964e422da73003a458fb8bee",
            "input": "The challenge is addressed as follows: given a natural language input sequence describing the scene, such as a piece of a story coming from a transcript, the goal is to infer which action is most likely to happen next. \n Question: Do they literally just treat this as \"predict the next spell that appears in the text\"?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-a81ff0ec21ba4eceb82a602145178906",
            "input": "For the evaluation of performance of the proposed method on the NLI task, SNLI BIBREF22 and MultiNLI BIBREF23 datasets are used. We use Quora Question Pairs dataset BIBREF24 in evaluating the performance of our method on the PI task. In evaluating sentiment classification performance, the Stanford Sentiment Treebank (SST) BIBREF25 is used. \n Question: Which datasets were used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-f141eaaa9ec6479cb9838c8ce476d204",
            "input": "Paraphrases can be obtained by translating an English string into a foreign language and then back-translating it into English.  \n Question: It looks like learning to paraphrase questions, a neural scoring model and a answer selection model cannot be trained end-to-end. How are they trained?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-4eef215ef7564145857e9d9660e29637",
            "input": "The text and image encodings were combined by concatenation, which resulted in a feature vector of 4,864 dimensions. This multimodal representation was afterward fed as input into a multi-layer perceptron (MLP) with two hidden layer of 100 neurons with a ReLU activation function. \n Question: Is the dataset multimodal?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-6a006e2c4fee4575ad528f4a028c2371",
            "input": " Our final survey based classification results showed an accuracy of 96% in detecting PTSD and mean squared error of 1.2 in estimating its intensity given we have four intensity, No PTSD, Low Risk PTSD, Moderate Risk PTSD and High Risk PTSD with a score of 0, 1, 2 and 3 respectively.  \n Question: How is the intensity of the PTSD established?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-79b328fc6819404f99623e9b4f5ed0b2",
            "input": "Dataset Quality Analysis ::: Inter-Annotator Agreement (IAA)\nTo estimate dataset consistency across different annotations, we measure F1 using our UA metric with 5 generators per predicate. Dataset Quality Analysis ::: Dataset Assessment and Comparison\nWe assess both our gold standard set and the recent Dense set against an integrated expert annotated sample of 100 predicates.  Dataset Quality Analysis ::: Agreement with PropBank Data\nIt is illuminating to observe the agreement between QA-SRL and PropBank (CoNLL-2009) annotations BIBREF7.  \n Question: How was quality measured?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-b974ee3a98844b0cb08bfc9938133c82",
            "input": "Active learning methods can generally be described into two parts: a learning engine and a selection engine BIBREF28 . The learning engine is essentially a classifier, which is mainly used for training of classification problems. The selection engine is based on the sampling strategy, which chooses samples that need to be relabeled by annotators from unlabeled data. Then, relabeled samples are added to training set for classifier to re-train, thus continuously improving the accuracy of the classifier. In this paper, a CRF-based segmenter and a scoring model are employed as learning engine and selection engine, respectively. \n Question: How does the active learning model work?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-35979f8e841046faa93b8b1b0596e3fb",
            "input": "Knowledge Base Question Answering (KBQA) systems answer questions by obtaining information from KB tuples BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 . \n Question: What is te core component for KBQA?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-fccba6925203450c925e4836199bb41e",
            "input": "Despite this, we found that the training of our embeddings was not considerably slower than the training of order-2 equivalents such as SGNS. Explicitly, our GPU trained CBOW vectors (using the experimental settings found below) in 3568 seconds, whereas training CP-S and JCP-S took 6786 and 8686 seconds respectively. \n Question: Do they measure computation time of their factorizations compared to other word embeddings?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-9e0816dbba9941dd878b6232848b33d1",
            "input": "Using the distribution of the individual words in a category, we can compile distributions for the entire category, and therefore generate maps for these word categories.  \n Question: Do they build a model to automatically detect demographic, lingustic or psycological dimensons of people?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-0dda27a4c07f4acbbf673ecbbdf5c0f7",
            "input": "Furthermore, the gains in WER over the baseline are significantly larger for the Density Ratio method than for Shallow Fusion, with up to 28% relative reduction in WER (17.5% $\\rightarrow $ 12.5%) compared to up to 17% relative reduction (17.5% $\\rightarrow $ 14.5%) for Shallow Fusion, in the no fine-tuning scenario. \n Question: What metrics are used for evaluation?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-450dbdb7e248453c89004e2adc1c249c",
            "input": "Given that there is no public dataset available with financial intents in Portuguese, we have employed the incremental approach to create our own training set for the Intent Classifier. We have created domain-specific word vectors by considering a set 246,945 documents, corresponding to of 184,001 Twitter posts and and 62,949 news articles, all related to finance . \n Question: What datasets are used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-2cf77952e72c459bbbbfd70e62647da0",
            "input": "In its general form, a type-logical grammar consists of following components: \n Question: Does Grail accept Prolog inputs?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-70daa9610fcd4f1a88590f6c3b0f24c5",
            "input": "The baseline is a hierarchical phrase-based system BIBREF29 with a 4-gram language model, with feature weights tuned using MIRA BIBREF30 . Therefore, an appropriate additional baseline would be to mark translation rules with these indicator functions but without the scores, akin to identifying rules with phrases in them (Baseline + SegOn). \n Question: Which translation systems do they compare against?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-f34728b550494426b48d23fd7021795a",
            "input": "BIBREF17: The Lemming model is a log-linear model that performs joint morphological tagging and lemmatization.  \n Question: What were the non-neural baselines used for the task?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-e5b8e241f2f1489d903d789d04ec500e",
            "input": " We then train a paragraph vector model using the Document to Vector (Doc2Vec) framework BIBREF7 on the whole set (13 million) of preprocessed text records, although training on smaller sets (1 million) also produces good results. \n Question: Which text embedding methodologies are used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-7b58d35e0d18467b89803415aa40f340",
            "input": "Another approach for generating query and image representations is treating images as a black box. Given the statistics of our dataset (3B query, image pairs with 220M unique queries and 900M unique images), we know that different queries co-occur with the same images. Intuitively, if a query $q_1$ co-occurs with many of the same images as query $q_2$ , then $q_1$ and $q_2$ are likely to be semantically similar, regardless of the visual content of the shared images. Thus, we can use a method that uses only co-occurrence statistics to better understand how well we can capture relationships between queries. \n Question: Could you learn such embedding simply from the image annotations and without using visual information?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-f062606b9cac424e80c3f5a0c2acea4e",
            "input": "We evaluate our model on a publicly available dataset, KARA ONE BIBREF17 , composed of multimodal data for stimulus-based, imagined and articulated speech state corresponding to 7 phonemic/syllabic ( /iy/, /piy/, /tiy/, /diy/, /uw/, /m/, /n/ ) as well as 4 words(pat, pot, knew and gnaw). The dataset consists of 14 participants, with each prompt presented 11 times to each individual.  \n Question: What data was presented to the subjects to elicit event-related responses?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-dddca544e12d4236bd3918e571cd5471",
            "input": "Each game's video ranges from 30 to 50 minutes in length which contains image and chat data linked to the specific timestamp of the game. \n Question: What is the average length of the recordings?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-323cf1a9f61b4868a759905a1f75b279",
            "input": "To better analyze model generalization to an unseen, new domain as well as model leveraging the out-of-domain sources, we propose a new architecture which is an extension of the ARED model. In order to better select, aggregate and control the semantic information, a Refinement Adjustment LSTM-based component (RALSTM) is introduced to the decoder side. The proposed model can learn from unaligned data by jointly training the sentence planning and surface realization to produce natural language sentences.  \n Question: What is the difference of the proposed model with a standard RNN encoder-decoder?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-dcff49f9a86240909953a9466a2fc3d4",
            "input": "Despite its usefulness, linked entities extracted from ELS's have issues because of low precision rates BIBREF11 and design challenges in training datasets BIBREF12 . First, the extracted entities may be ambiguous. Second, the linked entities may also be too common to be considered an entity.  \n Question: Why are current ELS's not sufficiently effective?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-ef2dba1bd654461092c152d52714d224",
            "input": "Similarly the agents utterances can be clustered to identify system responses. However, we argue that rather than treating user utterances and agents responses in an isolated manner, there is merit in jointly clustering them. There is adjacency information of these utterances that can be utilized to identify better user intents and system responses. \n Question: Do they study frequent user responses to help automate modelling of those?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-0f061cc0bf4441b7a8c090fc0a7452f5",
            "input": "For instance, we observed that winter words were together and far away from summer ones. Week days were grouped as well and far from week-end days. \n Question: What geometric properties do embeddings display?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-6949c3806f894ba2b4197d3640dd6d40",
            "input": "we take the CORD-19 dataset BIBREF2, which contains over 45,000 scholarly articles, including over 33,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses. We develop sentence classification methods to identify all sentences narrating radiological findings from COVID-19.  We conduct experiments to verify the effectiveness of our method. From the CORD-19 dataset, our method successfully discovers a set of clinical findings that are closely related with COVID-19. \n Question: How large is the collection of COVID-19 literature?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-539967809eb94376b85b7dc91c2e78a6",
            "input": "For the competition, we provided 1,000 labeled dialogues from each dataset for training, and 240 unlabeled dialogues from each dataset for evaluation.  \n Question: How large is the first dataset?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-b343521232824bd28b749abd579d2151",
            "input": "Since we do not have an end-to-end dataset, the generated English poem may not work well with Shakespeare style transfer as shown in Figure FIGREF12 for \"Starry Night\" with a low average content score. This happens when the style transfer dataset does not have similar words in the training set of sentences. A solution would be to expand the style transfer dataset, for a better representation of the poem data. \n Question: What limitations do the authors demnostrate of their model?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-03de46ae6ba14c73a8b394fd93f2f1de",
            "input": "The annotation process was a trial-and-error, with cycles composed of annotation, discussing confusing entities, updating the annotation guide schematic and going through the corpus section again to correct entities following guide changes. \n Question: Did they experiment with the corpus?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-0942dcbd6bbb4d11bc0495cf710c1946",
            "input": "Three annotators, A1-A3, mark each tweet in the Dataset H as drunk or sober. \n Question: Is the data acquired under distant supervision verified by humans at any stage?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-cf14adc859ff48fc9bf04b5065d01433",
            "input": "One distinguishing feature of educational topics is their breadth of sub-topics and points of view, as they attract researchers, practitioners, parents, students, or policy-makers. We assume that this diversity leads to the linguistic variability of the education topics and thus represents a challenge for NLP. \n Question: What challenges do different registers and domains pose to this task?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-57a2f7eac44d4c14b844462befe4f96f",
            "input": "We leveraged an existing, annotated Twitter dataset that was constructed based on a hierarchical model of depression-related symptoms BIBREF12 , BIBREF13 . The dataset contains 9,473 annotations for 9,300 tweets. Each tweet is annotated as no evidence of depression (e.g., “Citizens fear an economic depression\") or evidence of depression (e.g., “depressed over disappointment\").  \n Question: Do they evaluate only on English datasets?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-b3b5fef853044286a33be0797f90dfc3",
            "input": "The data set we evaluate on in this work is WMT English-French NewsTest2014, which has 380M words of parallel training data and a 3003 sentence test set. The NewsTest2013 set is used for validation. \n Question: Do they only test on one dataset?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-6dc4fccd0d564b8d95d1d937a39b68fc",
            "input": "Our Transformer Transducer model architecture has 18 audio and 2 label encoder layers. Every layer is identical for both audio and label encoders. The details of computations in a layer are shown in Figure FIGREF10 and Table TABREF11. All the models for experiments presented in this paper are trained on 8x8 TPU with a per-core batch size of 16 (effective batch size of 2048). \n Question: Does model uses pretrained Transformer encoders?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-712386ff075344e5a68cdca2122b93eb",
            "input": " 2) Answers are formulated by domain experts with legal training.  \n Question: Are the experts comparable to real-world users?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-fcc158e896d648968f5e08e4273a9473",
            "input": "We introduce a new model for difficulty prediction combining learned representations induced via a pre-trained `universal' sentence encoder BIBREF6 , and a sentence encoder learned from scratch for this task. In total, this resulted in 57,505 and 2,428 sentences in the train and test set abstracts, respectively. \n Question: How much data is needed to train the task-specific encoder?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-110729b53a224e428dd169488008b266",
            "input": "According the distribution of the sentiment score, the sentiment on Tesla is slightly skewed towards positive during the testing period. \n Question: Do the authors give any examples of major events which draw the public's attention and the impact they have on stock price?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-6e127b04403d49f6955dbe67ad89f988",
            "input": "Thus, SIM has many fewer parameters than existing dialogue state tracking models. To compensate for the exclusion of slot-specific parameters, we incorporate better feature representation of user utterance and dialogue states using syntactic information and convolutional neural networks (CNN).  \n Question: How do they prevent the model complexity increasing with the increased number of slots?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-44be8d15929c4806ac3f0c85b5e66afa",
            "input": "The term `propaganda' derives from propagare in post-classical Latin, as in “propagation of the faith\" BIBREF1, and thus has from the beginning been associated with an intentional and potentially multicast communication; only later did it become a pejorative term. It was pragmatically defined in the World War II era as “the expression of an opinion or an action by individuals or groups deliberately designed to influence the opinions or the actions of other individuals or groups with reference to predetermined ends\" BIBREF2. \n Question: How is \"propaganda\" defined for the purposes of this study?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-31061cad51c54039b65400140129acd4",
            "input": "We model the ASP placement task as a successor of the AEP task. For all the `relevant' news entity pairs, the task is to determine the correct entity section. Each section in a Wikipedia entity page represents a different topic. For example, Barack Obama has the sections `Early Life', `Presidency', `Family and Personal Life' etc. Article-Section Ground-truth. The dataset consists of the triple INLINEFORM0 , where INLINEFORM1 , where we assume that INLINEFORM2 has already been determined as relevant. We therefore have a multi-class classification problem where we need to determine the section of INLINEFORM3 where INLINEFORM4 is cited.  \n Question: How do they determine the exact section to use the input article?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-fc0a9c2909b84a08b05ae76f746ceff5",
            "input": "The current configuration of Aristo comprises of eight solvers, described shortly, each of which attempts to answer a multiple choice question. To study particular phenomena and develop solvers, the project has created larger datasets to amplify and study different problems, resulting in 10 new datasets and 5 large knowledge resources for the community.\n\nThe solvers can be loosely grouped into:\n\nStatistical and information retrieval methods\n\nReasoning methods\n\nLarge-scale language model methods Over the life of the project, the relative importance of the methods has shifted towards large-scale language methods. The field of NLP has advanced substantially with the advent of large-scale language models such as ELMo (BID6), ULMFit (BID37), GPT (BID38), BERT (BID7), and RoBERTa (BID8). These models are trained to perform various language prediction tasks such as predicting a missing word or the next sentence, using large amounts of text (e.g., BERT was trained on Wikipedia + the Google Book Corpus of 10,000 books). They can also be fine-tuned to new language prediction tasks, such as question-answering, and have been remarkably successful in the few months that they have been available. We apply BERT to multiple choice questions by treating the task as classification: Given a question $q$ with answer options $a_{i}$ and optional background knowledge $K_{i}$, we provide it to BERT as:\n\n[CLS] $K_i$ [SEP] $q$ [SEP] $a_{i}$ [SEP] The AristoBERT solver uses three methods to apply BERT more effectively. First, we retrieve and supply background knowledge along with the question when using BERT. This provides the potential for BERT to “read” that background knowledge and apply it to the question, although the exact nature of how it uses background knowledge is more complex and less interpretable. Second, we fine-tune BERT using a curriculum of several datasets, including some that are not science related. Finally, we ensemble different variants of BERT together. \n Question: Is Aristo just some modern NLP model (ex. BERT) finetuned od data specific for this task?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-891d91ce19034759a3a599f82d9ce6fe",
            "input": "This work also contributes a new dataset of INLINEFORM0 pairs of free-form natural language instructions and high-level navigation plans. This dataset was collected through Mechanical Turk using 100 simulated environments with a corresponding topological map and, to the best of our knowledge, it is the first of its kind for behavioral navigation. The dataset opens up opportunities to explore data-driven methods for grounding navigation commands into high-level motion plans. While the dataset was collected with simulated environments, no structure was imposed on the navigation instructions while crowd-sourcing data. Thus, many instructions in our dataset are ambiguous. Moreover, the order of the behaviors in the instructions is not always the same. For instance, a person said “turn right and advance” to describe part of a route, while another person said “go straight after turning right” in a similar situation. The high variability present in the natural language descriptions of our dataset makes the problem of decoding instructions into behaviors not trivial. See Appendix A of the supplementary material for additional details on our data collection effort. \n Question: Did the collection process use a WoZ method?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-d3d706be770e4f689e71b14b18a749a2",
            "input": "We use OpenNMT BIBREF24 as the implementation of the NMT system for all experiments BIBREF5 . PBMT-R is a phrase-based method with a reranking post-processing step BIBREF18 . Hybrid performs sentence splitting and deletion operations based on discourse representation structures, and then simplifies sentences with PBMT-R BIBREF25 . SBMT-SARI BIBREF19 is syntax-based translation model using PPDB paraphrase database BIBREF26 and modifies tuning function (using SARI). Dress is an encoder-decoder model coupled with a deep reinforcement learning framework, and the parameters are chosen according to the original paper BIBREF20 . \n Question: what state of the art methods did they compare with?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-bc95b1e999d84e95aa538d86863999f3",
            "input": "In future work we also intend to add these types of studies to the ERP predictions.\n\nDiscussion \n Question: What datasets are used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-dc86069c17264dc28a26c0cf537ff1d2",
            "input": "convertible.pl: implementing DCG rules for 1st and 3rd steps in the three-steps conversion, as well as other rules including lexicon. \n Question: What DCGs are used?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-39cbd57ba24d4bf2806a409e5f5d7114",
            "input": "BIBREF12 found in a Chinese corpus that the word label \"End\" has a better performance than \"Begin\". This motivates us to carry out a backward greedy search over each sentence's label sequence to identify word boundaries. If two words segmented in a sentence are identified as nouns, and one word is immediately before the other, we assemble their boundaries, creating a new word candidate for entity recognition. This strategy has the advantage to find named entities with long word length. It also reduces the influence caused by different segmentation criteria. \n Question: What boundary assembling method is used?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-8ee7df8d99c345aabfc9bbf4dc881aaa",
            "input": "We use the following cheap ways to generate pseudo-source texts:\n\ncopy: in this setting, the source side is a mere copy of the target-side data. copy-marked: another way to integrate copies without having to deal with OOVs is to augment the source vocabulary with a copy of the target vocabulary.  copy-dummies: instead of using actual copies, we replace each word with “dummy” tokens.  \n Question: what data simulation techniques were introduced?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-975a5c955d6a48b09cccc9a4c4a1cf6c",
            "input": "We conducted experiments on the Amazon reviews dataset BIBREF9, which is a benchmark dataset in the cross-domain sentiment analysis field. This dataset contains Amazon product reviews of four different product domains: Books (B), DVD (D), Electronics (E), and Kitchen (K) appliances. Each review is originally associated with a rating of 1-5 stars and is encoded in 5,000 dimensional feature vectors of bag-of-words unigrams and bigrams.\n\nExperiment ::: Dataset and Task Design ::: Binary-Class.\nFrom this dataset, we constructed 12 binary-class cross-domain sentiment analysis tasks: B$\\rightarrow $D, B$\\rightarrow $E, B$\\rightarrow $K, D$\\rightarrow $B, D$\\rightarrow $E, D$\\rightarrow $K, E$\\rightarrow $B, E$\\rightarrow $D, E$\\rightarrow $K, K$\\rightarrow $B, K$\\rightarrow $D, K$\\rightarrow $E. Following the setting of previous works, we treated a reviews as class `1' if it was ranked up to 3 stars, and as class `2' if it was ranked 4 or 5 stars. For each task, $\\mathcal {D}_S$ consisted of 1,000 examples of each class, and $\\mathcal {D}_T$ consists of 1500 examples of class `1' and 500 examples of class `2'. In addition, since it is reasonable to assume that $\\mathcal {D}_T$ can reveal the distribution of target domain data, we controlled the target domain testing dataset to have the same class ratio as $\\mathcal {D}_T$. Using the same label assigning mechanism, we also studied model performance over different degrees of $\\rm {P}(\\rm {Y})$ shift, which was evaluated by the max value of $\\rm {P}_S(\\rm {Y}=i)/\\rm {P}_T(\\rm {Y}=i), \\forall i=1, \\cdots , L$. Please refer to Appendix C for more detail about the task design for this study.\n\nExperiment ::: Dataset and Task Design ::: Multi-Class.\nWe additionally constructed 12 multi-class cross-domain sentiment classification tasks. Tasks were designed to distinguish reviews of 1 or 2 stars (class 1) from those of 4 stars (class 2) and those of 5 stars (class 3). For each task, $\\mathcal {D}_S$ contained 1000 examples of each class, and $\\mathcal {D}_T$ consisted of 500 examples of class 1, 1500 examples of class 2, and 1000 examples of class 3. Similarly, we also controlled the target domain testing dataset to have the same class ratio as $\\mathcal {D}_T$. \n Question: Which sentiment analysis tasks are addressed?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-52a11cdaf9454aeabc395cf63094c56b",
            "input": "Modern Standard Arabic (MSA) and Classical Arabic (CA) have two types of vowels, namely long vowels, which are explicitly written, and short vowels, aka diacritics, which are typically omitted in writing but are reintroduced by readers to properly pronounce words. We show that our model achieves a case ending error rate (CEER) of 3.7% for MSA and 2.5% for CA. For MSA, this CEER is more than 60% lower than other state-of-the-art systems such as Farasa and the RDI diacritizer, which are trained on the same dataset and achieve CEERs of 10.7% and 14.4% respectively.  \n Question: what are the previous state of the art?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-37e8671fa48d42dfad0b11cc4b959fd9",
            "input": "We construct three datasets based on IMDB reviews and Yelp reviews. The IMDB dataset is binarised and split into a training and test set, each with 25K reviews (2K reviews from the training set are reserved for development). For Yelp, we binarise the ratings, and create 2 datasets, where we keep only reviews with $\\le $ 50 tokens (yelp50) and $\\le $200 tokens (yelp200). \n Question: What datasets do they use?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-5bc15bac432d47ce94ddcb8154554d7d",
            "input": "We employ precision, recall, F1 and accuracy for evaluation metrics. \n Question: what evaluation metrics are reported?",
            "output": [
                "Extractive"
            ]
        },
        {
            "id": "task462-bde59c8bd1584c9aaeee0a7806ee86cc",
            "input": "In this paper, we have shown that both human and machine translation can alter superficial patterns in data, which requires reconsidering previous findings in cross-lingual transfer learning. \n Question: Does the professional translation or the machine translation introduce the artifacts?",
            "output": [
                "Yes-no"
            ]
        },
        {
            "id": "task462-98ac706fc89a4531b08036feaa31b05e",
            "input": "Results form Table TABREF31 may give an idea that INLINEFORM0 is just an scaled INLINEFORM1 . While it is true that they show a linear correlation, INLINEFORM2 may produce a different system ranking than INLINEFORM3 given the integral multi-reference principle it follows. However, what we consider the most profitable about INLINEFORM4 is the twofold inclusion of all available references it performs. First, the construction of INLINEFORM5 to provide a more inclusive reference against to whom be evaluated and then, the computation of INLINEFORM6 , which scales the result depending of the agreement between references. We showed how INLINEFORM0 is an inclusive metric which not only evaluates the performance of a system against all references, but also takes into account the agreement between them.  \n Question: What makes it a more reliable metric?",
            "output": [
                "Abstractive"
            ]
        },
        {
            "id": "task462-317bb80841a847e4807df5874e88ebf1",
            "input": "The goal of an language model is to assign meaningful probabilities to a sequence of words. Given a set of tokens $\\mathbf {X}=(x_1,....,x_T)$, where $T$ is the length of a sequence, our task is to estimate the joint conditional probability $P(\\mathbf {X})$ which is\n\nwere $(x_{1}, \\ldots , x_{i-1})$ is the context. An Intrinsic evaluation of the performance of Language Models is perplexity (PPL) which is defined as the inverse probability of the set of the tokens and taking the $T^{th}$ root were $T$ is the number of tokens We propose an approximation of the joint probability as,\n\nThis type of approximations has been previously explored with Bi-directional RNN LM's BIBREF9 but not for deep transformer models. We therefore, define a pseudo-perplexity score from the above approximated joint probability. \n Question: How is pseudo-perplexity defined?",
            "output": [
                "Abstractive"
            ]
        }
    ],
    "Instance License": [
        "CC BY 4.0"
    ]
}