Question: How does the LISA algorithm choose which layers' parameters to freeze at each iteration?

Question: How does the LISA algorithm choose which layers' parameters to freeze at each iteration?

The LISA algorithm samples layer indices through a uniform distribution and freezes the parameters of the sampled layers.


Question: Among the baseline TTE approaches for the code-based task on Celiac Disease, which method achieves the best performance, excluding the new method proposed in this paper?

Your answer should be a python strings about the approach name. YOU MUST USE THE EXACT NAME FROM THE PAPER.


Question: Is there a dialogue dataset where a speaker's utterance is grounded in their persona, consisting of image-text pairs representing their episodic memories?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there a dialogue dataset where a speaker's utterance is grounded in their persona, consisting of image-text pairs representing their episodic memories?

MPCHAT: Towards Multimodal Persona-Grounded Conversation


Question: Is there research that investigates embedding multi-bit data into watermarks to improve resilience to text corruption, particularly aimed at safeguarding keywords and syntactic elements from modification?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there research that investigates embedding multi-bit data into watermarks to improve resilience to text corruption, particularly aimed at safeguarding keywords and syntactic elements from modification?

Robust Multi-bit Natural Language Watermarking through Invariant Features


Question: Is there any work that allows large numbers of model outputs to be encoded and compared by causal language models in a single forward pass?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any work that allows large numbers of model outputs to be encoded and compared by causal language models in a single forward pass?

EEL: Efficiently Encoding Lattices for Reranking


Question: What is the pipeline used to construct the dataset in the experiment of the paper "PepRec: Progressive Enhancement of Prompting for Recommendation"?

The pipeline to extract high quality justifications from raw user reviews consists of three steps: 1. Annotating a set of review segments with binary labels, i.e., to determine whether they are 'good' or 'bad' justifications. 2. Training a classifier on the annotated subset and applying it to distantly label all the review segments to extract 'good' justifications for each user and item pair. 3. Applying fine-grained aspect extraction for the extracted justifications, and building user personas and item profiles.

Question: What is the pipeline used to construct the dataset in the experiment of the paper "PepRec: Progressive Enhancement of Prompting for Recommendation"?


Question: In the direct preceding work of H2O, are there any metrics tested other than perplexity?


Question: In the paper proposing Voila-A, which baselines do the main experimental results demonstrate that Voila-A outperform?

Your answer should be a single python list of strings, each string is a baseline name.e.g. ["baseline1_name", "baseline2_name"]. Note that for the names, the abbreviation is enough.


Question: According to the author, what are the three main types of current dialogue evaluation catalogs?

Your answer should be a sentence stating the three main types.

Question: According to the author, what are the three main types of current dialogue evaluation catalogs?

Reference-based, reference-free and reference-assisted


Question: Is there any dataset that contains minimally-contrasting social situations that lead to different decisions about which behaviors are appropriate in that situation?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any dataset that contains minimally-contrasting social situations that lead to different decisions about which behaviors are appropriate in that situation?

NORMBANK: A Knowledge Bank of Situational Social Norms


Question: Which paper found that using common character encodings and ciphers, or even just convincing the model that it is not communicating in natural language, can bypass the safety guardrails of large models?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper found that using common character encodings and ciphers, or even just convincing the model that it is not communicating in natural language, can bypass the safety guardrails of large models?

GPT-4 IS TOO SMART TO BE SAFE: STEALTHY CHAT WITH LLMS VIA CIPHER


Question: Several information extraction and natural language processing tools were used in the anchor paper. What is the major limitation of prior works that the entity and relationship extractor used in the anchor paper was designed to solve?

Your answer should be brief text explaining the limitation solved by the extractor's design.

The major limitation of prior works is that they ignore the interrelation between spans (pairs).

Question: Several information extraction and natural language processing tools were used in the anchor paper. What is the major limitation of prior works that the entity and relationship extractor used in the anchor paper was designed to solve?


Question: According to the RAV paper, up to 2019, which method performs the best overall on evidence retrieval? Additionally, what's that method's FEVER score with 1 sentence selected for the subtask of recognizing textual entailment?

Your answer should be a Python list of 2 elements, the first is a string, the name of the method, and the second is a float, rounding to 2 decimal places.


Question: What are the model architecture parameters (dim, n_layers, head_dim, hidden_dim, n_heads, n_kv_heads) of the automatic evaluation model used in the aspect evaluation section of the paper?

Your answer should be a python dictionary with the following keys: dim, n_layers, head_dim, hidden_dim, n_heads, n_kv_heads, e.g., {"dim": 768, "n_layers": 6, "head_dim": 64, "hidden_dim": 3072, "n_heads": 12, "n_kv_heads": 12}.


Question: Which research paper leverages event structure information from Abstract Meaning Representation (AMR) graphs to aid in recognizing causal relations between events?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which research paper leverages event structure information from Abstract Meaning Representation (AMR) graphs to aid in recognizing causal relations between events?

Semantic Structure Enhanced Event Causality Identification


Question: What are the four types of compression methods used for BERT in the paper? Which one of them can be combined with the other three model compression methods? Can you describe the sketch of this method's procedure?

Your answer should be a brief text containing the four compression methods used for BERT in the paper and the method can be combined with other three methods with its procedure's sketch.

The four compression methods used for BERT in the paper are Knowledge Distillation, Pruning, Quantization, and Vocabulary Transfer. Among them, Vocabulary Transfer can be combined with the other three model compression methods. The sketch of this method's procedure can be described as follows: First, the vocabulary is constructed on the in-domain data, then an embedding is assigned to each token, transferring information from the pre-trained representations of the general-purpose language model.

Question: What are the four types of compression methods used for BERT in the paper? Which one of them can be combined with the other three model compression methods? Can you describe the sketch of this method's procedure?


Question: What is the distribution ratio of data sources for the toxicity ratings dataset used in the paper?

Your answer should be a python dictionary about the data sources and their distribution ratio(between 0 and 1, rounded to 2 decimal places), e.g., {"data_source_1": 0.49, "data_source_2": 0.51}. YOU MUST USE THE EXACT NAMES FROM THE PDF WITHOUT CHANGING THE CAPITALIZATION.


Question: How much higher is the percentage of "食品#品质" in Figure: Aspect category distributions(Test Set for Subtask3) in the Overview of the SIGHAN 2024 paper than the percentage of "食品#品质" in Figure: training data category distributions of HITSZ-HLT?

Your answer should be a python float number with three decimal places, and the answer should be in the range of [0, 1]


Question: In the paper titled "TeamShakespeare at SemEval-2023 Task 6: Understand Legal Documents with Contextualized Large Language Models", which model is based on when designing the model for NER task? For this base model, on which tasks is it evaluated in the main experiment of the source paper?

Your answer should be a single python list like ["base_model_name",["task1","task2",...]].Note that for the task name, you should use the full name, not the abbreviation.

Entity Typing
Relation Classification
Named Entity Recognition
Cloze-style Question Answering
Extractive Question Answering


Question: What are the main differences between the backbone model used in the main experiments of the paper "VISION-BY-LANGUAGE FOR TRAINING-FREE COMPOSITIONAL IMAGE RETRIEVAL" and the standard Transformer?

The input of the backbone model is a sequence of flattened 2D patches x_p\in \mathbb{R}^{N\times (P^2 \cdot C)}.

We prepend a learnable embedding to the sequence of embedded patches (z^0_0 = x_{class}), whose state at the output of the Transformer encoder (z^0_L) serves as the image representation y.

Question: What are the main differences between the backbone model used in the main experiments of the paper "VISION-BY-LANGUAGE FOR TRAINING-FREE COMPOSITIONAL IMAGE RETRIEVAL" and the standard Transformer?


Question: According to the paper "Geometric Deep Learning Grids, Groups, Graphs, Geodesics, and Gauges", what are the three main flavours of GNN layers? What's the relationship between GTs and GNN?

Your answer should be a python list of two elements, teh first one is alist of three flavours, and the second one is a python strings.

Question: What's the relationship between GTs and GNN?


Question: Is there a paper that uses similarity scores to check knowledge in diffusion models

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there a paper that uses similarity scores to check knowledge in diffusion models

Multilingual Conceptual Coverage in Text-to-Image Models


Question: What is the detailed procedure of progressive learning followed in the instruction tuning section of this paper?

The detailed procedure of progressive learning used in Orca 2 starts with LaMA-2-7B or LLaMA-2-13B checkpoint and finetunes it on the train split of FLAN-v2 dataset for one epoch. Then they train on 5 million ChatGPT data from Orca 1 for 3 epochs, and train on the combination of 1 million GPT-4 data from Orca 1 and Orca 2's 817K data for 4 epochs.

Question: What is the detailed procedure of progressive learning followed in the instruction tuning section of this paper?


Question: How many hours and separate recordings are contained in the STT4SG-350 corpus?

Your answer should be s single python list of two integers, e.g. [200, 5674]. The first integer represents hours, the second integer represents recordings. Note that you should use exact numbers, not approximations.


Question: In the architecture of MolMIM, which encoder is used? What's the core idea of the designing of this encoder?

Your answer should be a single python list of two strings, like ["encoder_name","sentences_about_core_idea"]

The core idea of this encoder is to introduce a small set of latent units that form an "attention bottleneck." Inputs must pass through this bottleneck, which solves the quadratic scaling problem of all-to-all attention in classical Transformers and decouples network depth from input size, enabling the construction of very deep models. The encoder iteratively attends to inputs, focusing its limited capacity on the most relevant parts based on previous steps. To compensate for the lack of explicit structures, position and modality-specific features are associated with each input element (e.g., pixels or audio samples). These features can be learned or constructed using high-fidelity Fourier features, effectively tagging input units with a high-fidelity representation of position and modality.

Question: What's the core idea of the designing of this encoder?


Question: In the benchmark C-LAP uses for image observations evaluation, Offline DV2 performs the best in which environment under mixed setting?

Your answer should be a python string, the name of the environment WITHOUT ANY explanation.


Question: What dataset does this paper("titled What Makes it Ok to Set a Fire? Iterative Self-distillation of Contexts and Rationales for Disambiguating Defeasible Social and Moral Situations") use for evaluation? According to its source paper, how many structured annotations does this dataset collect in total?

Your answer should be a single python list, the first element is the string of the dataset name, the second element is an integer number.


Question: I'm interested in the method used by the UCB1-FLAD paper for dataset format transformation, and I would like to contribute to that method. Where can I find the guidelines?

Your answer should be a Python string, the website URL starting with "https://", as given in the paper.


Question: Among existing Math and STEM QA datasets, which dataset includes theorem except TheoremQA? In which paper is this dataset introduced?

Your answer should be a list of two strings, the first element is the name of the dataset, and the second element is the title of the source paper.

Measuring mathematical problem solving with the math dataset


Question: In the paper that proposes TRL model which manages to surpass ChatGPT on some category on average according to the QATCH paper, what's the formula of the total loss in detail?

Your answer should be a string, the formula in LaTeX format. Note that you should expand the three parts of the total loss.

\mathcal{J}_{\text{CS}} = \frac{1}{|Columns|} \sum_{co \in Columns} \text{CE}(p_{col}^{(co)}, \mathbb{1}_{co=col}) + \frac{1}{|Cells(col)|} \sum_{c \in Cells(col)} \text{CE}(p_s^{(c)}, \mathbb{1}_{c \in C}) + \alpha \left(-\log p_a(op_0)\right)

Question: What's the formula of the total loss in detail?


Question: In the Comparisons experiment of the Metaworld benchmark in the paper "Prediction with Action: Visual Policy Learning via Joint Denoising Process", which benchmarks performed the best on easier tasks and harder tasks, respectively (excluding the method proposed in this article)?

Your answer should be a python list of strings, first element is the best performing benchmark on easier tasks, and the second element is the best performing benchmark on harder tasks. YOU MUST USE THE EXACT ABBREVIATION AS IN THE PAPER.


Question: In the experimental results on StepGame of this paper, which one of the three PLMs gains the best performance when k=4? Please provide the name of the model. And what is this model's github link?

Your answer should be a Python list of 2 strings, the name of the model, and the github link of this model.


Question: Which dataset is GrailQAbility dataset derived from? How many literals are there in the source dataset?

Your answer should be a single python list like ["dataset_name",100].The first element of the list is a string and the second is an integer number.


Question: Which website can I find the conversation video between Sheldon and Leonard?

Your answer should be a pure text string starting with "https". DO NOT INCLUDE ANY OTHER INFORMATION OR CONTEXT IN THE ANSWER.


Question: In the smallest dataset that the given paper applies, how is the mutual information calculated?

Your answer should be a string, the formula of mutual information in LaTeX format.

\mathbb{I}[k, w] =- \sum_{k=0}^{N} \langle \text{Bin}(k | f^w(x), N) \rangle \log [\langle \text{Bin}(k | f^w(x), N) \rangle]+ \sum_{k=0}^{N} \langle \text{Bin}(k | f^w(x), N) \log [\text{Bin}(k | f^w(x), N)] \rangle

Question: How is the mutual information calculated?


Question: Which numerical reasoning paper first published a dataset that considers different types of size of numbers and their representations in arithmetic questions?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which numerical reasoning paper first published a dataset that considers different types of size of numbers and their representations in arithmetic questions?

FERMAT: An Alternative to Accuracy for Numerical Reasoning


Question: Which object detector does this paper(titled "Can VLMs Play Action Role-Playing Games? Take Black Myth Wukong as a Study Case") use to assist the VLMs in better extracting useful information?On which datasets is this object detector pre-trained?

Your answer should be a list like ["detector_name", ["dataset1","dataset2",...]], where detector_name is the name of the object detector and ["dataset1","dataset2",...] is a list of dataset names(abbreviation) on which the object detector is pre-trained.


Question: Is there any paper that uses data collected from the Dark Web, specifically onion domains, to pretrain a language model?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any paper that uses data collected from the Dark Web, specifically onion domains, to pretrain a language model?

DarkBERT: A Language Model for the Dark Side of the Internet


Question: Which one is newer among the public datasets used in the main experiment of this paper("Query Routing for Homogeneous Tools: An Instantiation in the RAG Scenario")? Which link may I refer to to get this dataset?

Your answer should be a python list like ["string1", "string2"]. The first element should be the name(abbrievation) of the dataset. The second element should be a string, representing the link to the dataset.


Question: What are the parameters of the base models used in the paper "PsychoLex: Unveiling the Psychological Mind of Large Language Models", including Layers, Model Dimension, FFN Dimension, Attention Heads and Key/Value Heads?

Your answer should be a nested dictionary, e.g., {'base_model1': {'Layers': 12, 'Model Dimension': 768, 'FFN Dimension': 3072, 'Attention Heads': 12, 'Key/Value Heads': 12}}


Question: In the paper "Distributional Scaling Laws for Emergent Capabilities", what empirical evidence supports the RASP-Generalization Conjecture in the context of transformers' length generalization?

Your answer should be a sentence explaining the empirical evidence.

Question: In the paper "Distributional Scaling Laws for Emergent Capabilities", what empirical evidence supports the RASP-Generalization Conjecture in the context of transformers' length generalization?

The authors of "What Algorithms can Transformers Learn? A Study in Length Generalization" provide empirical evidence by demonstrating that transformers exhibit strong length generalization on tasks such as parity and addition when these tasks can be represented by short RASP programs. They show that by leveraging the RASP framework, transformers can generalize to sequences longer than those seen during training, supporting the conjecture that the simplicity of the RASP representation correlates with successful length generalization.


Question: What tasks were proposed in the paper of the dataset used in the experiment of the paper?

Your answer should be a python list of tasks, e.g. ["task1", "task2", ...]. YOU MUST USE THE EXACT NAMES FROM THE PDF WITHOUT CHANGING THE CAPITALIZATION.

Grounding Span Prediction
Agent Response Generation


Question:
In the paper titled "ON-TRAC consortium systems for the IWSLT 2023 dialectal and low-resource speech translation tasks", which dataset is used when training the translation model X $
ightarrow$ FR, EN? Which institute is this dataset released by?

Your answer should be a single python list like ["dataset_name","institute_name"].Note that you don't have to indicate the language pair in the dataset name.


Question: In the results of topic modeling through LDA, which keyword is the most frequent among all the topics? How many times did it appear? The models' performance was measured using BERT-score. Which conference did the BERT-score authors present BERT-score at?

Your answer should be a brief text containing the most frequent keyword with its frequency and the conference name.

The most frequent keyword is 'Missing' and it appeared in 3 topics. The BERT-score's authors presented BERT-score at the Conference on ICLR 2020.

Question: In the results of topic modeling through LDA, which keyword is the most frequent among all the topics? How many times did it appear? The models' performance was measured using BERT-score. Which conference did the BERT-score authors present BERT-score at?


Question: Which datasets were used in the specific private text domain for experiments with the document-level machine translation framework in the paper "Granularity is crucial when applying differential privacy to text: An Investigation for Neural Machine Translation"?

Your answer should be a python list, e.g., ['dataset1', 'dataset2', ...]. YOU MUST USE THE EXACT NAMES OF THE DATASETS, RATHER THAN ABBREVIATIONS OR ALIASES.

Business Scene Dialogue Corpus
ClinSPEn-CC


Question: List the names of the baselines used in the paper "On the Compositional Generalization in Versatile Open-domain Dialogue", along with the titles of papers that proposed these baselines.

Your answer should be a Python list of baseline-title pair, e.g., [["baseline1", "title1"] , ["baseline2", "title2"], ...]. YOU MUST USE THE EXACT TEXT FROM THE PAPER AND THE FULL TITLE TEXT OF THE PAPERS.

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion

Prefix-Tuning: Optimizing Continuous Prompts for Generation

Combining modular skills in multitask learning


Question: In the biography generation task that the Self-RAG paper applied, what're the five categories of freqValue?

Your answer should be a Python list of 5 strings, the 5 categories of freqValue.

Very rare
Rare
Medium
Frequent
Very frequent


Question: According to "Casting Hybrid Digital-Analog Training into Hierarchical Energy-Based Learning", how can the delta of energy function be used in the EP gradient of EBM and BP gradient of FF module?

Your answer should be in fluential English.

The EP gradient uses the difference of the second-order gradient of energy function.

The BP gradient uses the difference of the fourth-order gradient of energy function.

Question: According to "Casting Hybrid Digital-Analog Training into Hierarchical Energy-Based Learning", how can the delta of energy function be used in the EP gradient of EBM and BP gradient of FF module?


Question: From which three aspects do the authors evaluate the unlearned model?

Your answer should be a python list, each element is a string.

Performance on the Forget Set
Performance on the Retain Set
Performance on General Downstream Tasks

Question: From which three aspects do the authors evaluate the unlearned model?


Question: In the paper "Strengthened Symbol Binding Makes Large Language Models Reliable Multiple-Choice Selectors", on which dataset does PIF method reach its highest accuracy? In the paper where that dataset is proposed, which LLM performed the best, and how to account for its performance?

Your answer should be a Python list of three strings. The first string indicating the full name of the dataset, the second indicating the name of the LLM that performed the best, and third indicating the reason. e.g. ["dataset", "LLM", "reason"].

To understand the performance of BERT-LARGE, we analyzed 100 examples from the development set (Table 6). We labeled examples with categories (possibly more than one per example) and then computed the average accuracy of the model for each category. We found that the model does well (77.7% accuracy) on examples where surface clues hint to the correct answer. Examples that involve negation or understanding antonyms have lower accuracy (42.8%), similarly to examples that require factoid knowledge (38.4%). Accuracy is particularly low in questions where the correct answer has finer granularity compared to one of the distractors (35.4%), and in cases where the correct answer needs to meet a conjunction of conditions, and the distractor meets only one of them (23.8%).

Question: How to account for BERT-Large's performance on CSQA?


Question: Which paper introduced the human-evaluated timeliness metric for misinformation detection?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper introduced the human-evaluated timeliness metric for misinformation detection?

Human-in-the-loop Evaluation for Early Misinformation Detection: A Case Study of COVID-19 Treatments


Question: What are some evaluation benchmarks for LLM privacy at inference time, targeted towards model input and NOT the training data.

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: What are some evaluation benchmarks for LLM privacy at inference time, targeted towards model input and NOT the training data.

CAN LLMS KEEP A SECRET? TESTING PRIVACY IMPLICATIONS OF LANGUAGE MODELS VIA CONTEXTUAL INTEGRITY THEORY


Question: According to this survey, what're the three most recent decoder-only LLMs for NL2Code? How many programming languages do their training datasets each contain?

Your answer should be a Python dictionary of 3 key-value pairs, where each key is a string and each value is an integer.


Question: Is there any work that attacks language models in dialogue generation?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any work that attacks language models in dialogue generation?

White-Box Multi-Objective Adversarial Attack on Dialogue Generation


Question: I read two papers called Chain of Ideas and Nova, and they both involve the evaluation of the characteristics of ideas, such as quality, novelty, and so on. I want to know the specific measurement aspects of these two papers.

The Chain of Ideas paper evaluates the novelty, significance, clarity, feasibility, and expected effectiveness of ideas. The Nova paper evaluates the quality, novelty, and diversity of ideas.

Question: I read two papers called Chain of Ideas and Nova, and they both involve the evaluation of the characteristics of ideas, such as quality, novelty, and so on. I want to know the specific measurement aspects of these two papers.


Question: In figure 2, what is the next step to take after generating the supportive logical forms?Please use the name presented in this figure.And then for this step, what's the criteria held in this paper?

Your answer should be a list of two strings, the first element is the name of the next step, and the second element is sevaral sentences about the criteria.

The templates should break up the generation of a complicated question into a step by step process.

The templates should clearly identify the subcomponent in a logical form that requires LLMs to focus on for each step.

Question: For this step, what's the criteria held in this paper?


Question: Which paper first proposed extracting the pair of target and stance from sentences?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper first proposed extracting the pair of target and stance from sentences?

A New Direction in Stance Detection: Target-Stance Extraction in the Wild


Question: In the baseline construction of the experiment of TrojText, which models are employed? Where can I get the code or data of these models?

Your answer should be a single python list like [["model_name1","model_name2",...],["https://github.com/a/b","https://github.com/c/d",...]].Note that you should choose the most concise way to express the name of the model.


Question: Is there a paper that uses the tree structure of math equations in autoregressive language models?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there a paper that uses the tree structure of math equations in autoregressive language models?

Tree-Based Representation and Generation of Natural and Mathematical Language


Question: Is there any paper about style transfer for stories?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any paper about style transfer for stories?

StoryTrans: Non-Parallel Story Author-Style Transfer with Discourse Representations and Content Enhancing


Question: Are there any examples of using dense phrase retrieval systems in the automatic curation of entity dictionaries?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Are there any examples of using dense phrase retrieval systems in the automatic curation of entity dictionaries?

Automatic Creation of Named Entity Recognition Datasets by Querying Phrase Representations


Question: Which paper first studied differential privacy for in-context learning to prevent prompt leakage attacks?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper first studied differential privacy for in-context learning to prevent prompt leakage attacks?

PRIVACY-PRESERVING IN-CONTEXT LEARNING FOR LARGE LANGUAGE MODELS


Question: Is there any paper that explores and annotates the effectiveness of using testimonials or anecdotes in discussions?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any paper that explores and annotates the effectiveness of using testimonials or anecdotes in discussions?

StoryARG: a corpus of narratives and personal experiences in argumentative texts


Question: Could you recommend a contemporary research paper that has advanced natural language watermarking quality through algorithmic methods?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Could you recommend a contemporary research paper that has advanced natural language watermarking quality through algorithmic methods?

Robust Multi-bit Natural Language Watermarking through Invariant Features


Question: What's the framework proposed in this paper("Global Learning with Triplet Relations in Abstractive Summarization")? Which framework is it similar to? In the source papers of these two frameworks, how many same datasets are they experimented on?

Your answer should be a python list like ["string1", "string2", integer]. The first element should be a string, representing the name(abbrievation) of the framework proposed in this paper. The second element should be a string, representing the name(abbrievation) of the similar framework. The third element should be an integer, representing the number of common datasets experimented on.


Question: What is the difference between using two open-source LLMs in the experiments of the paper "ARE LARGE LANGUAGE MODELS BAYESIAN? A MARTINGALE PERSPECTIVE ON IN-CONTEXT LEARNING"?

Sliding Window Attention. Mistral-7B exploits the stacked layers of a transformer to attend information beyond the window size W . The hidden state in position i of the layer k, h_i, attends to all hidden states from the previous layer with positions between i-W and i.

Rolling Buffer Cache. A fixed attention span means that mistral-7B can limit our cache size using a rolling buffer cache. The cache has a fixed size of W , and the keys and values for the timestep i are stored in position i mod W of the cache. As a result, when the position i is larger than W , past values in the cache are overwritten, and the size of the cache stops increasing.

Pre-fill and Chunking. Mistral-7B can pre-fill the (k, v) cache with the prompt. If the prompt is very large, we can chunk it into smaller pieces, and pre-fill the cache with each chunk. For this purpose, we can select the window size as our chunk size. For each chunk, mistral-7B thus need to compute the attention over the cache and over the chunk.

Question: What is the difference between using two open-source LLMs in the experiments of the paper "ARE LARGE LANGUAGE MODELS BAYESIAN? A MARTINGALE PERSPECTIVE ON IN-CONTEXT LEARNING"?


Question: In order to improve Parrot's abilities, which method proposed in the paper "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" is used to train the model? Which method is compared with it under distribution shifts?

Your answer should be a Python list of two strings, answering the two questions respectively. You must use abbreviations as given in the papers.


Question: Is there any paper that employs code LLMs to iteratively generate and refine code with execution results to improve the performance?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any paper that employs code LLMs to iteratively generate and refine code with execution results to improve the performance?

Self-Edit: Fault-Aware Code Editor for Code Generation


Question: In section 4, what research quesitions do the authors aim to answer?

Your answer should be be plein text DIRECTLY FROM THE PDF.

In this section, we aim to answer three research questions: (RQ1) How do different medical generation models perform under our evaluation? (RQ2) How is the evaluation quality of DOCLENS compared to existing metrics? (RQ3) How is the evaluation quality of DOCLENS computed with open-source evaluators compared to proprietary ones?

Question: In section 4, what research quesitions do the authors aim to answer?


Question: Which paper combines the advantages of different frameworks for grammar error correction (GEC) and achieves good performance?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper combines the advantages of different frameworks for grammar error correction (GEC) and achieves good performance?

TemplateGEC: Improving Grammatical Error Correction with Detection Template


Question: What are the institutions of the first author and corresponding author of this paper?

Your answer should be a Python list of length 2 containing the institution names respectively, e.g., ["first_author_institute", "corresponding_author_institute"]. If there are multiple first authors or corresponding authors, please replace the corresponding institution name with a name list, e.g., [["first_author1_institute", "first_author2_institute", ...], ["corresponding_author1_institute", "corresponding_author2_institute", ...]].

The University of Sydney
The Hong Kong Polytechnic University


Question: In Experiment Section of the paper, what is the overall framework of the baseline model achieving the second best BERTScore on the dataset LOCOMO?

Your answer should be a python strings about the detailed overall framework of the baseline model.

The overall framework of MemoChat pipeline is a memorization-retrieval-response loop inner thinking. Very different from traditional methods that retrieve directly on these accumulated dialogues, the chatbot will automatically builds and updates a structured on-the-fly memo, storing past dialogues in categories. Then, the retrieval is conducted over all recordings according to their topics and summaries

Question: In Experiment Section of the paper, what is the overall framework of the baseline model achieving the second best BERTScore on the dataset LOCOMO?


Question: Which category of website takes up the most proportion in the dataset MC2?

Your answer should be a phrase indicating the category DIRECTLY FROM THE PDF WITHOUT ANY MODIFICATION OR EXPLANATION.


Question: According to the paper that proposes the pixel-level similarity metrics that the LG-VQ paper employs, what's the metrics' value, given that it's a gray scale image 0f 8 bits, with MSE=1?

Your answer should be a float, rounding to 2 decimal places.


Question: What molecular representation learning paper introduced a benchmark that focuses on learning over thermodynamically-accessible conformer ensembles across diverse molecular properties and chemical reactions?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: What molecular representation learning paper introduced a benchmark that focuses on learning over thermodynamically-accessible conformer ensembles across diverse molecular properties and chemical reactions?

Learning Over Molecular Conformer Ensembles: Datasets and Benchmarks


Question: Which paper showed that social relationships were helpful for identifying inappropriate messages?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper showed that social relationships were helpful for identifying inappropriate messages?

Your spouse needs professional help: Determining the Contextual Appropriateness of Messages through Modeling Social Relationships


Question: What paper mitigates language model sampling errors due to the softmax bottleneck?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: What paper mitigates language model sampling errors due to the softmax bottleneck?

CLOSING THE CURIOUS CASE OF NEURAL TEXT DEGENERATION


Question: What is the reason why Soft MoE cannot be applied to autoregressive models currently?

Question: What is the reason why Soft MoE cannot be applied to autoregressive models currently?

Soft MoE cannot be applied to autoregressive models because it needs to merge all input tokens together, which would break the causality requirement between past and future tokens that must be maintained during training in autoregressive models.


Question: Who is the first author of WebArena? How many papers of his/hers are cited in the paper? Which are they?

Your answer should be a Python list of 3 elements. The first one is a string serving as the first author name of WebArena. The second one is an interger indicating the number of self-referenced papers. The third one is a string list storing the titles of self-referenced papers.

Language models of code are few-shot commonsense learners

Hierarchical prompting assists large language model on web navigation

Execution-based evaluation for open-domain code generation

Hierarchical control of situated agents through natural language

Show me more details: Discovering hierarchies of procedures from semi-structured web data


Question: Calculate the increase in throughput when the batch size increases from 24 to 64 for H2O (20%) at a sequence length of 2048+2048 on A100 GPU.

Your answer should be a Python float number rounded to 1 decimal place. e.g. 20.3


Question: Whether the code and data of this paper are publicly available or not?

Your answer should be a simple "yes" or "no" WITHOUT PUNCTUATION OR EXPLANATION.


Question: Which paper first studied the efficiency robustness of multi-exit language models?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper first studied the efficiency robustness of multi-exit language models?

Dynamic Transformers Provide a False Sense of Efficiency


Question: Which paper shows that in instruction tuning, the instructions can be compressed to small supporting sets of words that provide useful information?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper shows that in instruction tuning, the instructions can be compressed to small supporting sets of words that provide useful information?

Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learning


Question: Is there any paper that reveals annotation problems in cross-lingual summarization caused by decomposing the task into translation and summarization?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any paper that reveals annotation problems in cross-lingual summarization caused by decomposing the task into translation and summarization?

Revisiting Cross-Lingual Summarization: A Corpus-based Study and A New Benchmark with Improved Annotation


Question: The experiment section of this paper introduces a new metric, S^2MATCH, what is the main difference between it and the metric used in Appendix D? Answer with one formula.

You only need to provide one definition formula of S2MATCH in Python strings. You don't need to explain the formula or variables.

softMATCH = 1 - d(\mathbf{x} ,\mathbf{y} )

Question: The experiment section of this paper introduces a new metric, S^2MATCH, what is the main difference between it and the metric used in Appendix D? Answer with one formula.


Question: What baselines (excluding MLP) are used in the paper proposing the model zoo in the paper?

Your answer should be a python list about the exact names of the baselines, e.g., ['baseline1', 'baseline2', ...]. YOU MUST USE THE EXACT NAMES FROM THE PDF WITHOUT CHANGING THE CAPITALIZATION.


Question: What is the shape of $W$ in Equation (3)? And what about $W_l$ and $W_v$ in Equation (5)?

Your answer should be a sentence describing the shapes of $W$, $W_l$ and $W_v$ in detail.

The shape of $W$ in Equation (3) is $\mathbb{R}^{d_s \times d_l}$, where $d_s$ is the dimension of the vision features, and $d_l$ is the dimension of the language features. The shapes of $W_l$ and $W_v$ in Equation (5) are both $\mathbb{R}^{d \times 1}$.

Question: What is the shape of $W$ in Equation (3)? And what about $W_l$ and $W_v$ in Equation (5)?


Question: For the VQA DOC task, what are the optimal values of alpha and beta?

Your answer should be a python list of two numbers


Question: Which model did both anchor_pdfs use for experiments?

Your answer should be a python string, and it should be the model name.


Question: Is there any paper that address attacks on code models by leveraging the semantic information of the source code through attention scores, while also guaranteeing that the generated adversarial examples can always be compiled successfully?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any paper that address attacks on code models by leveraging the semantic information of the source code through attention scores, while also guaranteeing that the generated adversarial examples can always be compiled successfully?

DIP: Dead code Insertion based Black-box Attack for Programming Language Model


Question: In the paper that proposes GLPFT, which training dataset is larger? In that dataset, what's the format of the data?

Your answer should be a Python list of 2 elements, the first is the name of the dataset, the second is the format of the data in "A-B pairs" format, as given in the paper. e.g. ["MMLU", "question-answer pairs"].


Question: Is there any paper that uses token-level loss to enhance sentence-level embedding learning?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any paper that uses token-level loss to enhance sentence-level embedding learning?

Dual-Alignment Pre-training for Cross-lingual Sentence Embedding


Question: What is the first paper that uses the generalized linear model to analyze multi-neural spike train data?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: What is the first paper that uses the generalized linear model to analyze multi-neural spike train data?

ONE-HOT GENERALIZED LINEAR MODEL FOR SWITCHING BRAIN STATE DISCOVERY


Question: What are the most important optimizations of transformer network in this paper?

Your answer should be a Python list of text strings, with each element being one important optimization that this paper proposes, e.g., ["optimization 1", "optimization 2", ...].

Removal of the one-to-one mapping constraint among queries and keys in multiple subspaces

Allowing each query to attend to multiple keys

Introduction of inner-subspace interaction and cross-subspace interaction to encourage consensus among heads

Question: What are the most important optimizations of transformer network in this paper?


Question: Which model uses Llama2-7B as the LLM base model in Table 1 in paper 'DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs' and in this model's original paper, how many models are compared in Table 4 in total?

Your answer should be a Python list of 2 strings, the name of the model, and the number of compared models.


Question: Regarding the dataset ROCKS used in the anchor paper, it contains ratings assessed by 20 annotators for each of the 12 pictures of a given rock type. How does its experimental setup ensure the objectivity and fairness of the ratings, specifically how do subjects use consistent scale values?

Your answer should be a python string that explains the detailed experimental setup.

To promote the use of consistent scale values across subjects, anchor pictures were displayed along with scale values on the computer screen throughout each rating session. One anchor picture corresponded to the lowest rating (e.g, the very darkest rock), a second anchor picture corresponded to the highest rating (e.g, the very lightest rock), and a third anchor corresponded to a rock that we judged to be roughly average on the rated dimension (e.g., a rock of average darkness/lightness). The anchors and scale values were displayed at the bottom of the screen throughout the rating session to ensure the objectivity and fairness of the ratings.

Question: Regarding the dataset ROCKS used in the anchor paper, it contains ratings assessed by 20 annotators for each of the 12 pictures of a given rock type. How does its experimental setup ensure the objectivity and fairness of the ratings, specifically how do subjects use consistent scale values?


Question: On what devices is StreamVoice trained?

Your answer should be a plein text directly from the PDF without explanation.

StreamVoice is trained using 8 V100 GPUs with a batch size of 7 utterances per GPU for 700k steps.

Question: On what devices is StreamVoice trained?


Question: Is there any paper that utilizes graph structure to model conversation history?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any paper that utilizes graph structure to model conversation history?

History Semantic Graph Enhanced Conversational KBQA with Temporal Information Modeling


Question: In which comparisons of models did the two papers reach similar conclusions?

Your answer should be a python list of several strings. The string should be language model name.


Question: For retrieval evaluation, what metrics applied by MTEB are not used by LocalRQA?

Your answer should be a python list of strings, each string is the name of a metric, as given in the MTEB paper.


Question: In the paper proposing SG-USM for task-oriented dialogues, which baseline method performs the best across all datasets, excluding SG-USM itself?

Your answer should be a python strings. YOU MUST USE THE EXACT ABBREVIATION AS IN THE PAPER. DO NOT USE FULL NAMES.


Question: How was the data for the latest text classification tasks used in this paper collected?

The data collection methodology is to create each sentence pair by selecting a premise sentence from a preexisting text source and asking a human annotator to compose a novel sentence to pair with it as a hypothesis.

Question: How was the data for the latest text classification tasks used in this paper collected?


Question: Which paper makes sure that the questions used in the paper are all from real users that are genuinely curious about a specific topic or concept?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper makes sure that the questions used in the paper are all from real users that are genuinely curious about a specific topic or concept?

CREPE: Open-Domain Question Answering with False Presuppositions


Question: To evaluate the AdaLoRA algorithm, which model is used for natural language understanding and question answering? Can you give me the relevant github link of this model?

Your answer should be a single python list of two strings, like ["model_name","https://github.com/a/b"].Note that in the model name, you should use "-" between the series name and the size, for example, "modelx-small".


Question: Regarding the construction of the Edit Matrix, which paper's findings does this study reference and follow? What are the affiliations of the authors of the cited work?

Your answer should be a brief text containing the cited paper's name and the authors' affiliations.

The cited paper is 'Incomplete Utterance Rewriting as Semantic Segmentation' and its authors are from the School of 'Computer Science and Engineering, Beihang University, China' and 'Microsoft Research, Beijing, China'.

Question: Regarding the construction of the Edit Matrix, which paper's findings does this study reference and follow? What are the affiliations of the authors of the cited work?


Question: What limitations do large language models have in evaluating information-seeking question answering?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: What limitations do large language models have in evaluating information-seeking question answering?

Evaluating Open-Domain Question Answering in the Era of Large Language Models


Question: What paper first associate the modeling frequency with input human skeletons under the NeRF framework?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: What paper first associate the modeling frequency with input human skeletons under the NeRF framework?


Question: Among the SR algorithms chosen in the paper "Expression Sampler as a Dynamic Benchmark for Symbolic Regression", which one performs the best on SRBench, considering the R-squared test.

Your answer should be a string, the abbreviation of the algorithm as given in the paper.


Question: The training dataset used in the paper "Semiparametric Token-Sequence Co-Supervision" is filtered to 42932 instances, then what's the original size of this dataset?


Question: How does Multi-DYLE combine the three different losses as the objective of training?

Your answer should be single formula in latex format, extracted from the specified pdf.

$$\mathcal{L}=\lambda_{g}\mathcal{L}_{g e n}+\lambda_{o}\sum_{j=1}^{M}\mathcal{L}_{o r a c l e}^{(j)}+\lambda_{c}\mathcal{L}_{c o n s i s t}$$

Question: How does Multi-DYLE combine the three different losses as the objective of training?


Question: Which dataset performs better on the LLaMA model, PRM800K or Math-Shepherd? In the source paper of PRM800K, which methods are compared with PRM?

Your answer should be a python list of two items. The first item is a python string. The second item is a python list of strings.


Question: Which paper investigates the influence of the diversity of source tasks on the performance of target tasks in prompt tuning using CrossFit?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper investigates the influence of the diversity of source tasks on the performance of target tasks in prompt tuning using CrossFit?

Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning?


Question: Which paper first proposed a cross-domain language model to automatically generate much labeled data for a unlabeled target domain?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper first proposed a cross-domain language model to automatically generate much labeled data for a unlabeled target domain?

Cross-Domain Data Augmentation with Domain-Adaptive Language Modeling for Aspect-Based Sentiment Analysis


Question: What does the special symbol $\overline{\mathcal{V}}$ mean in the proposed CFIC decoding strategy?

Your answer should be concise text string highlighting the meaning of the symbol.

Question: What does the special symbol $\overline{\mathcal{V}}$ mean in the proposed CFIC decoding strategy?

The special symbol $\overline{\mathcal{V}}$ denotes a token set containing each sentence's prefix, and the sentence prefix serves as an position identifier to facilitate the identification of the starting point of a supporting passage within the source context.

The answer must mention the restricted token set composed of sentence prefixes instead of the whole vocabulary.

The answer should also explain what the sentence prefixes come from, that is they are derived from the source context.


Question: In the algorithm applied to find the minimum norm interpolant, when does the first step stop?

Your answer should be a Python string, the formula of the end condition in LaTeX format.

\frac{\max_{0 \leq i \leq n+1} \left| \sqrt{s_i} - \sum_{j=0}^n \alpha_j s_i^j \right|}{\min_{0 \leq i \leq n+1} \left| \sqrt{s_i} - \sum_{j=0}^n \alpha_j s_i^j \right|}< 1.001

Question: What's the end condition?


Question: Which family of model generally perform the best for the event conceptualization task

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which family of model generally perform the best for the event conceptualization task

CAT: A Contextualized Conceptualization and Instantiation Framework for Commonsense Reasoning


Question: In the event extraction section of the paper "Is It Safe to Tell Your Story? Towards Achieving Privacy for Sensitive Narratives", how were the dependency parses of the stories obtained?

Stanza pipeline is used to process the stories and obtain dependency parses.It parses each sentence for its syntactic structure, where each word in the sentence is assigned a syntactic head that is either another word in the sentence, or in the case of the root word, an artificial root symbol. A Bi-LSTM-based deep biaffine neural dependency parser is implemented. We further augment this model with two linguistically motivated features: one that predicts the linearization order of two words in a given language, and the other that predicts the typical distance in linear order between them to significantly improve parsing accuracy.

Question: In the event extraction section of the paper "Is It Safe to Tell Your Story? Towards Achieving Privacy for Sensitive Narratives", how were the dependency parses of the stories obtained?


Question: What is the composition of the training dataset in the paper?

Your answer should be a python strings about the dataset.

WebVid-2M consists of 2.5M video-text pairs. The data was scraped from the web following a similar procedure to Google Conceptual Caption. The dataset consists of manually generated captions, that are for the most part well formed sentences. And the captions are aligned with the video and describe visual content.

Question: What is the composition of the training dataset in the paper?


Question: Among three single-agent baselines in table one, which performs best on Damped Spring?

Your answer should be a python string. YOU MUST USE THE EXACT NAME FROM THE PDF WITHOUT CHANGING THE CAPITALIZATION.


Question: In the CLAP paper, which three challenges for patchwork learning are mentioned?

Your answer should be a Python list of 3 strings, each string is a challenge.

- statistical heterogeneity; the multimodal data of local clients are typically non-i.i.d. The model may fail to adapt the learned dependencies to target clients;

- modality combination heterogeneity; local clients can have various modality combinations. The learned dependencies from Xi is hard to be used for other clients with different combinations Xj;

- modality combination vulnerability; the learned imputation method could be vulnerable to the modality combinations and the imputation quality significantly varies for two similar combinations;

Question: In the CLAP paper, which three challenges for patchwork learning are mentioned?


Question: In "MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic", what is the main advantage of "MetaGPT" that the authors claim to have over "AdaMerging"? Also, what is the most significant difference between the experiment settings of the papers which proposed these two methods?

Your answer should be brief text answering the 2 questions with separate sentences.

The authors claim that MetaGPT is more computational efficient than AdaMerging. MetaGPT only experiments on NLP tasks, while AdaMerging only experiments on vision tasks.

Question: In "MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic", what is the main advantage of "MetaGPT" that the authors claim to have over "AdaMerging"? Also, what is the most significant difference between the experiment settings of the papers which proposed these two methods?


Question: In the paper Zhengyuan Liu as the first writer published in ACl 2023 that is related to stance detection, which corpus dataset was only used in evaluation, not in training?

Your answer should be a string of the corpus's name without any explanation and anyother word.


Question: In the EGraFFBench paper, Equiformer's hyperparameter setting resembles that in the original Equiformer paper on MD17 dataset. Specifically, in the setting in the original paper that is the closest to the EGraFFBench setting, what's the value of L_{max}?


Question: Which model, among GPT3.5, GPT-4, Llama-7B, and Mistral-7B, experiences the largest drop in overall accuracy from the Conversation History Task to MMLU AA, relative to zero-shot performance?

Your answer should be a single model name, without any other text. The answer should be one of the following: GPT3.5, GPT-4, Llama-7B, or Mistral-7B.


Question: Which subtask of task3 of SemEval-2023 does the paper(titled "BERTastic at SemEval-2023 Task 3: Fine-Tuning Pretrained Multilingual Transformers- Does Order Matter?" perform the experience on?

Your answer should be a string describing this subtask. Note that you should not include other subtasks of task3 of SemEval-2023.

subtask 2 (ST2): framing detection. given a news article, identify one or more frames used in the article from a pool of 14 generic framing dimen sions (introduced in Card et al. (2015)): Economic,  Capacity and resources, Morality, Fairness and  equality, Legality, constitutionality and jurispru dence, Policy prescription and evaluation, Crime  and punishment, Security and defense, Health and  safety, Quality of life, Cultural identity, Public  opinion, Political, External regulation and reputa tion. This is a multi-class multi-label classification task at the article level.

Question: Which subtask of task3 of SemEval-2023 does the paper(titled "BERTastic at SemEval-2023 Task 3: Fine-Tuning Pretrained Multilingual Transformers- Does Order Matter?" perform the experience on?


Question: Both as hybrid digital twins, what's the advantage of HDTwinGen over PINN-based Med-Real2Sim?

Your answer should be in well-formed item list.

HDTwinGen injects domain knowledge from LLM rather than real human expert, which reduces the reliance of expensive and heavy expert labour and make the framework easily generalized to new domains.

HDTwinGen's pipeline can automatically optimize both model parameters and model structures, in constrast Med-Real2Sim only optimizes model parameters and its structure relies on human's a priori knowledge.

Question: Both as hybrid digital twins, what's the advantage of HDTwinGen over PINN-based Med-Real2Sim?


Question: Is there any works that explores how to achieve balance between representativeness and diversity in chosen samples for few-shot data selection?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any works that explores how to achieve balance between representativeness and diversity in chosen samples for few-shot data selection?

Cold-Start Data Selection for Better Few-shot Language Model Fine-tuning: A Prompt-based Uncertainty Propagation Approach


Question: What paper evaluated the ability of visual few-shot learning models to do in-context learning?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: What paper evaluated the ability of visual few-shot learning models to do in-context learning?


Question: In the dataset proposed by the authors, how many states are there per game?

Your answer should be a floating-point number with one decimal place.


Question: What's the original annotation process drawn by the paper named "Where Do People Tell Stories Online?Story Detection Across Online Communities?"?

Your answer should be a single string indicating the original annotation process.

All annotations were carried out by a single co-author after multiple rounds of discussions and the creation of a set of annotation guidelines. To calculate the expected inter-annotator agreement rate, a second co-author independently annotated a random sample of five texts at the end of the annotation process, using only the annotation guidelines for reference.

Question: What's the original annotation process drawn by the paper named "Where Do People Tell Stories Online?Story Detection Across Online Communities?"?


Question: which paper first focuses on addressing the over-smoothing issue for sentence embedding?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: which paper first focuses on addressing the over-smoothing issue for sentence embedding?

Alleviating Over-smoothing for Unsupervised Sentence Representation


Question: What is the original dataset size (including all data splits) for each shared dataset used in both the works ABEX and MinPrompt according to its original papers?

Your answer should be a Python dictionary with the keys being the names (case-sensitive) of the shared datasets and the integer values being the corresponding dataset sizes.


Question: In the paper "Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture", according to Figure 1, which two dimensions are mixed using Monarch matrices? In the source paper of Monarch matrices, what training settings can Monarch matrices be used for?

Your answer should be a python list containing two items. The first item is a python list with two strings. The second item is a python list with an indefinite number of strings.

E2E training
S2D training
D2S fine-tuning


Question: What do formula (2) to formula (4) mean?

Your answer should be a brief summarization of the meaning of these formulas, and you do not need to introduce these formulas one-by-one.

Formula (2) to (4) describe the self-contrastive training (SELFCONT) algorithm, which is designed to mitigate the repetition problem in language models. SELFCONT modifies the training process by adding a penalty to the output of the current model \(f_{	heta_1}\) when it predicts a repetitive token that the premature checkpoint \(f_{	heta_0}\) also predicts. The penalty is controlled by the weight \(w\), which is only active when the true next token is non-repetitive but the premature model predicts it as repetitive. This encourages the model to learn more complex patterns and reduce its tendency to generate repetitive text.

Question: What do formula (2) to formula (4) mean?


Question: Which dataset(s) is dataset SAFECONV derived from? Give me their github link(s).

Your answer should be a single python list like [["dataset1","dataset2",...],["https://github.com/a/b","https://github.com/c/d",...]].Note that you should retain the size in the dataset name if available.


Question: Which speech encoder does the paper("Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversations") choose to extract universal paralinguistic and prosody embeddings?In the proposal of this encoder, which datasets are used in both pre-training and downstream tasks?

Your answer should be a list like ["encoder_name", ["dataset1", "dataset2",...]].


Question: In the paper "Training Trajectories of Language Models Across Scales" (anchor_pdf), figure 17 and 18 can be used to provide a supportive reasoning on the conclusion, what is it? And is it giving a similar conclusion as the paper "Are Emergent Abilities of Large Language Models a Mirage?" did?

Your answer should be a Python list of two elements, where the first element is the supportive reasoning on the conclusion provided in figure 17 and 18, and the second is a boolean value indicating whether it is giving a similar conclusion as the paper "Are Emergent Abilities of Large Language Models a Mirage?" did.

Question: In the paper "Training Trajectories of Language Models Across Scales" (anchor_pdf), figure 17 and 18 can be used to provide a supportive reasoning on the conclusion, what is it? And is it giving a similar conclusion as the paper "Are Emergent Abilities of Large Language Models a Mirage?" did?

Models get its abilitities on different tasks in a relative smooth way, instead of a emerging way, yes.


Question: Which paper enables interactive semantic parsing by training an error correction model with simulated human feedback instead of human annotations?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper enables interactive semantic parsing by training an error correction model with simulated human feedback instead of human annotations?

Learning to Simulate Natural Language Feedback for Interactive Semantic Parsing


Question: For each test split in Figure 5, provide the name and type of the website with the highest step success rate.

Your answer should be a Python dictionary. e.g. {"split1": ["web1", "type1"], "split2": ["web2", "type2"], ...}. YOU MUST USE THE EXACT WORDS FROM PDF WITHOUT CHANGING CAPITALIZATION.


Question: Which paper proposed a learning-based data augmentation method for improving compositional generalization of language models?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper proposed a learning-based data augmentation method for improving compositional generalization of language models?

Learning to Substitute Spans towards Improving Compositional Generalization


Question: Which was the first paper to explore the online adaptation of neural MT metrics for use during the inference stage?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which was the first paper to explore the online adaptation of neural MT metrics for use during the inference stage?

Test-time Adaptation for Machine Translation Evaluation by Uncertainty Minimization


Question: According to Figure 1, with the increasing of the number of few-shot training samples, which setting keeps getting a better score?

Your answer should be the name of the setting appearing in the legend of the image.


Question: In FewRel's 10-way 5-shot setting, what is the maximum decrease of AOD+ROD between ConPL and AGCKD across all task indexes?

Your answer should be a positive floating-point number with two decimal place.


Question: Which vision-language model paper in 2023 developed techniques that reduce input tokens to improve model inference speed?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which vision-language model paper in 2023 developed techniques that reduce input tokens to improve model inference speed?

PuMer: Pruning and Merging Tokens for Efficient Vision Language Models


Question: How to better attract readers to news articles by generating personalized headlines?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: How to better attract readers to news articles by generating personalized headlines?


Question: For each category of PQA in terms of the form of provided answers, from what aspects does the author analyze it?

Your answer should be a Python list, where each element is a string representing an aspect DIRECTLY FROM THE PDF. Note that the aspects are the same for each category. e.g. ["aspect1", "aspect2", ...]

Problem Definition
Datasets & Evaluation Protocols
Methods
Pros and Cons


Question: How much data do the author use in total in million for the main experiment conducted on WMT17 ZhEn?

Your answer should be a Python float, rounding to 1 decimal places.


Question: On which model does the paper(titled "Making Large Language Models Better Reasoners with Orchestrated Streaming Experiences") conduct the most analysis experiments?Is there any other size of parameters for this model?

Your answer should be a single python list formatted like ["model_name", ["10B","20B",...]].The first element of the list is a string representing the name of the model, the second element of the list is a list representing other size of params(Note that you shouldnot include the size already employed in the paper).


Question: Which paper employs a two-stage approach in generative models to tackle ABSA tasks across various domains?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper employs a two-stage approach in generative models to tackle ABSA tasks across various domains?

Bidirectional Generative Framework for Cross-domain Aspect-based Sentiment Analysis


Question: Which paper first attempts to take potential dependencies among same-level labels into account in Hierarchical Text Classification?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper first attempts to take potential dependencies among same-level labels into account in Hierarchical Text Classification?

Peer-Label Assisted Hierarchical Text Classification


Question: Which is the first multimodal model combining text and speech transformers trained without labelled text-speech pairs?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which is the first multimodal model combining text and speech transformers trained without labelled text-speech pairs?

Introducing Semantics into Speech Encoders


Question: What are the Low resource languages in INDICGENBENCH?

Your answer should be a Python list, where each element is a string representing a language. e.g. ["language1", "language2", ...]

Awadhi Haryanvi Tibetan Garhwali Konkani Chhattisgarhi Rajasthani Maithili Manipuri Malvi Marwari Santali Bodo


Question: What's the total number of augmented training samples across all datasets used in the MINPROMPT work?

Your answer should be a single integer number.


Question: According to the author, how does Cross Entropy contribute to miscalibration?

Question: According to the author, how does Cross Entropy contribute to miscalibration?

According to the author, Cross Entropy contributes to miscalibration by encouraging the model to assign a probability of 1 to the ground-truth token and 0 to all other tokens. This leads to overestimating the probability of the correct token (over-confidence) and underestimating the probabilities of the other tokens (under-confidence), causing miscalibration during the fine-tuning process.


Question: Is there any paper that applies symbolic distillation on black-box generalist language models to harvest high-quality counterfactual data for out-of-distribution generalization?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any paper that applies symbolic distillation on black-box generalist language models to harvest high-quality counterfactual data for out-of-distribution generalization?

DISCO: Distilling Counterfactuals with Large Language Models


Question: What is the accuracy of the base model used in the experiment in the paper "Protecting Privacy in Classifiers by Token Manipulation" on the RACE test set?

Your answer should be a python float with one decimal places.


Question: The methods proposed in the two anchor PDFs have similarities. For the following statements in VoiceFlow: ["Duration adapter", "y", "\|u_\theta(x_t, y, t) - (x_1 - x_0)\|^2"], which statements in Reflow-TTS correspond closely to them respectively?

Your answer should be a Python list containing three strings, arranged in the same order as in the question.

Question: For the formula "\|u_\theta(x_t, y, t) - (x_1 - x_0)\|^2", which statement in Reflow-TTS corresponds closely to it?


Question: In the PPTC benchmark paper, among the works that focus on LLMs' tool-use ability to generate APIs for solving user instructions, which one doesn't apply AST accuracy?

Your answer should be a string, the name of the method or model.


Question: Are there any papers that study whether you can identify if a LLM has been instructed to hide some information?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Are there any papers that study whether you can identify if a LLM has been instructed to hide some information?

HOW TO CATCH AN AI LIAR: LIE DETECTION IN BLACK-BOX LLMS BY ASKING UNRELATED QUESTIONS


Question: In the paper "MultiTabQA: Generating Tabular Answers for Multi-Table Question Answering", dataset SPIDER was used in the training process. What were the inputs and outputs in the original design of SPIDER, and how did the authors of this work adapt the dataset for their task?

Your answer should be brief text regarding the inputs and outputs of SPIDER in the two works.

The original design of SPIDER is a text-to-SQL dataset. The inputs are natural language queries, and the model should output corresponding SQL queries. In the anchor paper, the authors use the SQL queries as inputs, and train models to output SQL execution result tables.

Question: In the paper "MultiTabQA: Generating Tabular Answers for Multi-Table Question Answering", dataset SPIDER was used in the training process. What were the inputs and outputs in the original design of SPIDER, and how did the authors of this work adapt the dataset for their task?


Question: what is the exact performance drop when the diffusion model is removed in the MMWHS MR-to-CT UDA setting?

Your answer should be float number rounded to 1 decimal place.


Question: Which language performs better on old sense IDs compared to new sense IDs during experiments?

Your answer should be a string of a language name.


Question: In the S4WM paper's 3D environment, what's the episode length for the largest setting?


Question: How to calculate the Word-pair Representation matrix in Figure 4?

Your answer should be a list of formulas, representing the calculation of Word-pair Representation matrix.

The Word-pair Representation matrix in the described model is calculated through Conditional Layer Normalization (CLN). Here are the formulas representing the calculation:

1. **Conditional Layer Normalization (CLN) for word-pair representation:**
   $r_{i,j} = \text{CLN}(h_i, h_j) = \gamma_i \odot \left( \frac{h_j - \mu}{\sigma} \right) + \lambda_i$

2. **Scale factor $\gamma_i$ and shift factor $\lambda_i$ with additional contextual information:**
   $\gamma_i = W_\gamma h_i + b_\gamma$
   $\lambda_i = W_\lambda h_i + b_\lambda$

3. **Mean $\mu$ and standard deviation $\sigma$ of $h_j$:**
   $\mu = \frac{1}{d} \sum_{k=1}^{d} h_{jk}$
   $\sigma = \sqrt{\frac{1}{d} \sum_{k=1}^{d} (h_{jk} - \mu)^2}$

Question: How to calculate the Word-pair Representation matrix in Figure 4?


Question: How many models are evaluated in WebArena? What's the success rate of the most powerful model?

Your answer should be a Python list of 2 elements. The first one is an integer. The second one is a float rounded to 2 decimal places.


Question: What research advances are incorporated into the generative language model that used to generate associations in different languages in the SeeGULL Multilingual paper?

Your answer should be a python list of several strings.

Architectural and objective improvements

Question: What research advances are incorporated into the generative language model that used to generate associations in different languages in the SeeGULL Multilingual paper?


Question: What is the limitation of this work proposed by the authors themselves?

The authors have only explored applying the approach to encoder models, leaving room for applications on decoder models.

Despite the variety of existing contrastive learning methodologies, this work still adheres to utilizing the contrastive learning objectives provided by the tasks.

Question: What is the limitation of this work proposed by the authors themselves?


Question: According to the paper, how to combine the two losses presented in figure 2?

Your answer should be a single formula in latex format extracted from the paper.

\begin{array}{l}\mathcal{L}_{(	ext {multi })}\left(\phi_{(s h)}, \phi_{1}, \phi_{2}
ight)= \delta_{1} \mathcal{L}_{s a}\left(\phi_{(s h)}, \phi_{1}
ight)+\delta_{2} \mathcal{L}_{c l}\left(\phi_{(s h)}, \phi_{2}\right)\end{array}

Question: According to the paper, how to combine the two losses presented in figure 2?


Question: Is there any paper that investigates backdoor attacks across various types of tasks, not limited to classification, in language models?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any paper that investigates backdoor attacks across various types of tasks, not limited to classification, in language models?

Multi-target Backdoor Attacks for Code Pre-trained Models


Question: In the situation between 2 agents and 2 arms, how many unique matching pairs will incur linear regret with non-incentive-aware learning algorithms?


Question: Can we learn to represent an image with arbitary numbers of tokens?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Can we learn to represent an image with arbitary numbers of tokens?

SparseFormer: Sparse Visual Recognition via Limited Latent Tokens


Question: When the number of retrieved pairs is chosen empirically to be in the range of 3 to 5 for this data, which caption group of the testing set performs the best overall?

Your answer should be a python strings about the name of the caption group. YOU MUST USE THE EXACT NAME FROM THE PAPER.


Question: What baselines are used in this paper? Note that a baseline is counted only once, even if different variants are provided based on it.

Your answer should be a python list of strings, every element of the list is the name of a baseline directly mentionned in this paper.

NP-SpanBERT
QA-SRL Parser
TNE-Parser
Mistral


Question: How many LLMs does the authours test in the experiment part? And how many of the LLMs are openly accessible?

Your answer should be a python list of two integer. The first one is the number of the LLMs, and the second one is the number of the openly accessible LLMs of them.


Question: Is there any paper exploring real speakers and thus performing multimodal emotion recognition task?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any paper exploring real speakers and thus performing multimodal emotion recognition task?

A Facial Expression-Aware Multimodal Multi-task Learning Framework for Emotion Recognition in Multi-party Conversations


Question: What are the core parameters L, H, A, and the total number of parameters(params) of the base model of the classifier in this paper?

Your answer should be a python dictionary with keys 'L', 'H', 'A', and 'params', and the corresponding value should be a number, e.g., {'L': 1, 'H': 1, 'A': 1, 'params': 1}.


Question: Is there any paper that utilizes Gaussian processes to analyze the vulnerability of text-conditioned generative models?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any paper that utilizes Gaussian processes to analyze the vulnerability of text-conditioned generative models?

Query-Efficient Black-Box Red Teaming via Bayesian Optimization


Question: Considering all the methods tested in the experiment section of the paper, which LLM performs the worst on the Jailbreak Success Rate metric?

Your answer should be a python strings about the name of the LLM model. YOU MUST USE THE EXACT NAME FROM THE PAPER.


Question: How many tools does the proposed CodeAgent integrate into its framework, and which one is the most useful based on its ablation study?

Your answer should be a Python list of length 2. The first one is an integer number, and the second one is a text string of the tool name.


Question: Which paper found that mutual learning benefits multlingual models?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper found that mutual learning benefits multlingual models?

Towards Higher Pareto Frontier in Multilingual Machine Translation


Question: When discussing "Engagingness" in the paper, what is the definition of engagingness with interestingness from prior research?

The definition of engagingness should engage the partner in conversation, such as presenting an interesting factTherefore, an engaging response y should provide high volume of information that acknowledges both the history x to engage the partner and the context c which we assume contains relevant facts. This naturally leads to the following metric definition: ENGAGINGNESS (\mathbf{y} , \mathbf{x} , \mathbf{c} ) = sum (align(\mathbf{y} \to [\mathbf{x} , \mathbf{c} ])), where we concatenate the history x and knowledge context c, and measure the extent of response y's acknowledgement of the information.

Question: When discussing "Engagingness" in the paper, what is the definition of engagingness with interestingness from prior research?


Question: Name a paper which proposes a probabilsitic formulation of retrosynthesis.

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Name a paper which proposes a probabilsitic formulation of retrosynthesis.

RETRO-FALLBACK: RETROSYNTHETIC PLANNING IN AN UNCERTAIN WORLD


Question: Which dataset used in this paper consists of 14K open-domain English conversations with a total of 80K question-answer pairs? I want to use this dataset for my research. Can you provide me with the github link of this dataset?

Your answer should be a Python list of 2 strings, the name of the dataset, and the github link of this dataset.


Question: What stages does training of Med-Real2Sim comprise?

Your answer should be a string list consisting of the training stages in order.

physics-informed pretext task
physics-guided finetuning


Question: How much percent does VerifiNER improve the F1 score of the three baseline models on average on GENIA?

Your answer should be a Python float rounded to two decimal places WITHOUT ANY PUNCTUATION OR EXPLANATION. e.g. 21.30


Question: Is there a Chinese hate speech paper that constructs an insulting lexicon while building the dataset?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there a Chinese hate speech paper that constructs an insulting lexicon while building the dataset?

Facilitating Fine-grained Detection of Chinese Toxic Language: Hierarchical Taxonomy, Resources, and Benchmarks


Question: Which paper proposes a memory-efficient optimizer considering the confidence of each update during the optimization?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper proposes a memory-efficient optimizer considering the confidence of each update during the optimization?

CAME: Confidence-guided Adaptive Memory Efficient Optimization


Question: Considering both benefits and costs, what is the best size of generation pool for the proposed method?

Your answer should be a single integer number.


Question: Why can I find the exact dataset utilized by the LEGO-Prover paper?

Your answer should be a string, the URL as given in the paper without "https://", "http://" or "www.", e.g. "google.com"


Question: When it comes to Empirical and Certified Robustness, On which dataset and which poison rate the accuracy on Benign Samples of the method proposed by the paper is closest to the accuracy on Benign Samples of no defence situation?

Your answer should be a single python list, the first element is the dataset name, the second element is a float number rounded to 1 decimal place.


Question: What paper showed first that one can build a fully differentiable mixture of experts layer with no increase in time complexity?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: What paper showed first that one can build a fully differentiable mixture of experts layer with no increase in time complexity?


Question: What is the iAA(inter-annotator agreement when AttentionXML) value of the correctly predicted results?

Your answer should be a floating-point number with two decimal place.


Question: How many more turns per dialogue are there in MMDU Benchmark than in MMDialog?

Your answer should be an integer, the difference of average turns rounded to integer.


Question: Using only task-level prompts or using only example-specific prompts, which is better on the Multi-Domain test set?

Your answer should be a single string, either "task-level" or "example-specific".


Question: Which model did the authors use to fine-tune the pre-detector $g_{\phi}$ and conflict disambiguator $g_{\psi}$? How many times more params does ELMo have than this model?

Your answer should be a python list of two elements. The first element is a string of a name of a model, and the second element is a python float with two decimal places.


Question: Which Dataset has the most classes according to Table 1?

Your answer should be a single string of the Dataset's name.


Question: Which paper first applied the chain-of-thought technique in the text summarization field?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper first applied the chain-of-thought technique in the text summarization field?

Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Method


Question: What paper first proposed a robust perceptual similarity metric with certificates?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: What paper first proposed a robust perceptual similarity metric with certificates?

LIPSIM: A PROVABLY ROBUST PERCEPTUAL SIMILARITY METRIC


Question: Is there a paper that uses an app for a popular tabletop game to gather real transcripts of gameplay with concrete values for players' and monsters' health?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there a paper that uses an app for a popular tabletop game to gather real transcripts of gameplay with concrete values for players' and monsters' health?

FIREBALL: A Dataset of Dungeons and Dragons Actual-Play with Structured Game State Information


Question: What are the sources of the forecasting questions in the datasets used in the experimental section of the paper "AUTOCAST++: ENHANCING WORLD EVENT PREDICTION WITH ZERO-SHOT RANKING-BASED CONTEXT RETRIEVAL"?

Your answer should be a python list of strings, e.g., ["source1", "source2"].

Metaculus
Good Judgment Open
CSET Foretell


Question: Among the several bias presented in the paper named "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", which one is considered and discussed in the paper named"Humans or LLMs as the Judge? A Study on Judgement Bias"? After the discussion, is this bias the main influencial factor in the paper(I mean the latter)?

Your answer should be a single list, the first string is the bias name, the second string is bool, e.g., ["Verbosity bias", false]


Question: I believe introducing synthetic text-to-SQL data into model fine-tuning will benefit the final accuracy of the text-to-SQL task. But will this also help the model to generalize better on other general and board tasks?

Your answer should be a single boolean value (`True` or `False`) indicating whether the synthetic data will improve the performance on universal tasks, not only text-to-SQL.


Question: What's the evaluation baseline used in the paper titled "Generative Adversarial Training with Perturbed Token Detection for Model Robustness"? What's the contributions that makes this baseline different from existing adversarial datasets?

Your answer should be a python list of two strings, the first element is the baseline name(the abbrievation format is enough), and the second element is the contributions.

Comprehensive Coverage:AdvGLUE is able to cover as many adversarial linguistic phenomena as possible.

Systematic Annotations: this is the first work that performs systematic and comprehensive evaluation and annotation over 14 different textual adversarial examples.

General Compatibility:  AdvGLUE covers the widely-used GLUE tasks and creates an adversarial version of the GLUE benchmark to evaluate the robustness of language models

High Transferability and Effectiveness: AdvGLUE has high adversarial transferability and can effectively attack a wide range of state-of-the-art models.

Question: What's the contributions that makes this baseline different from existing adversarial datasets?


Question: What is a paper studying data being collected in bundles in reinforcement learning ?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: What is a paper studying data being collected in bundles in reinforcement learning ?

Sample-Efficiency in Multi-Batch Reinforcement Learning: The Need for Dimension-Dependent Adaptivity


Question: According to the paper that proposed the second smallest baseline applied in the MetaGPT paper, what's the difference in pass rate between the best and the worst method, under single-line infilling setting?

Your answer should be a float between 0 and 100, rounding to one decimal place.


Question: What is the architecture of the transformer-based classifier used in the paper "A Two-Model Approach for Humour Style Recognition"?

The student DistilBERT has the same general architecture as BERT.

The token-type embeddings and the pooler are removed while the number of layers is reduced by a factor of 2.

Question: What is the architecture of the transformer-based classifier used in the paper "A Two-Model Approach for Humour Style Recognition"?


Question: Is there a single GNN model that can inductively generalize to any knowledge graph?;What is the method to generalize knowledge graph reasoning to graphs with new entities and relations?;Is there a foundation model for knowledge graphs that does not learn embeddings for each node and relation type?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there a single GNN model that can inductively generalize to any knowledge graph?;What is the method to generalize knowledge graph reasoning to graphs with new entities and relations?;Is there a foundation model for knowledge graphs that does not learn embeddings for each node and relation type?

TOWARDS FOUNDATION MODELS FOR KNOWLEDGE GRAPH REASONING


Question: Which large model series were used to evaluate the prototype of the BBH dataset when it was proposed in the paper "Tree of Problems: Improving structured problem solving with compositionality"?

Your answer should be a python list of string, e.g., ['model1', 'model2']. YOU MUST USE THE ABBREVIATIONS PROVIDED IN THE PAPER.

BIG-G
BIG-G sparse
PaLM
Gopher
Chinchilla
T0


Question: Is there any paper that leverages syntactic rules to explicitly guide text generation?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any paper that leverages syntactic rules to explicitly guide text generation?

Explicit Syntactic Guidance for Neural Text Generation


Question: What paper mitigates the vocabulary size limitation when pretraining multilingual masked language models using a contrastive loss?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: What paper mitigates the vocabulary size limitation when pretraining multilingual masked language models using a contrastive loss?

Headless Language Models: Learning without Predicting with Contrastive Weight Tying


Question: In the main evaluation results of the different baselines in this paper, both of which use CodeLLaMA as the base model, which one performs better? In the paper introducing this model, aside from the datasets mentioned in this paper, what other in-domain datasets are used?

Your answer should be a python list, the first element is the name of the baseline model, and the following elements are the in-domain datasets used in the paper, e.g.,["baseline_model_name", "dataset1", "dataset2", ...]. YOU MUST USE THE EXACT NAMES FROM THE PDF WITHOUT CHANGING THE CAPITALIZATION.


Question: Is there commonsense reasoning dataset which generates diverse sentences to describe the relation between concepts?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there commonsense reasoning dataset which generates diverse sentences to describe the relation between concepts?

DimonGen: Diversified Generative Commonsense Reasoning for Explaining Concept Relationships


Question: Which datasets are used in the experiments of the paper proposing the previous best RGB-based method that are not used in this paper?

Your answer should be a python list about the names of the datasets, e.g., ['dataset1', 'dataset2']. YOU MUST USE THE EXACT NAMES FORM THE PAPER.


Question: According to Table 3, on which single-task and on which metric, no multi-task model can outperform the corresponding single-task model?

Your answer should be a Python list of two strings. The first string is the name of the task, and the second string is the name of the metric.


Question: What's the second method discussed in the "Related Work" of the paper that proposes CoMeDi? In the work that proposed that method, how many fewer sabotages per game in average did the method propsed by the authors make than their baseline, under "Non-repulser" condition?

Your answer should be a Python list of 2 elements, the first is a string, the name of the method, and the second is a float rounding to 2 decimal places, the difference of sabotages per game in average.


Question: Provide a brief introduction to the task in the SuperGLUE benchmark that were not used in the paper "CUSTOMIZABLE COMBINATION OF PARAMETER-EFFICIENT MODULES FOR MULTI-TASK LEARNING".

ReCoRD is a multiple-choice QA task. Each example consists of a news article and a Cloze-style question about the article in which one entity is masked out. The system must predict the masked out entity from a list of possible entities in the provided passage, where the same entity may be expressed with multiple different surface forms, which are all considered correct. Articles are from CNN and Daily Mail.

Question: Provide a brief introduction to the tasks in the SuperGLUE benchmark that were not used in the paper "CUSTOMIZABLE COMBINATION OF PARAMETER-EFFICIENT MODULES FOR MULTI-TASK LEARNING".


Question: How many more examples each model are used in the experiment of "LLM Evaluators Recognize and Favor Their Own  Generations" than in "Benchmarking Cognitive Biases in Large Language Models as Evaluators"?


Question: What are the detailed hyperparameters of the sentence-level scorer in Fine-grained Evaluation System of this paper?

Your answer should be a python dictionary about the hyperparameters, e.g. {"hyperparameter1": "value1", "hyperparameter2": "value2", ...}. YOU MUST USE THE EXACT NAMES FROM THE PDF WITHOUT CHANGING THE CAPITALIZATION.


Question: The paper("BLM-s/lE: A structured dataset of English spray-load verb alternations for testing generalization in LLMs") use two pre-trained models for experiment. For the newer one, what is its name and based on what task is it pre-trained?

Your answer should be a single python list, the first element is the name of the model, the second element is the task name it is pre-trained on.


Question: In the introduction of this paper, five tracks are mentioned. What are the detailed definitions of Track 2 and Track 3?

Your answer should be a python strings about the detailed definition of Track2 and Track3.

The formulation of this track2 is to predict, for each essay, Batson's empathic concern ("feeling for someone") and personal distress ("suffering with someone") scores. Teams are expected to develop models that predict the empathy score for each essay (self-report data from the essay writer). Both empathy and distress scores are real values between 1 and 7. Empathy score is an average of 7-point scale ratings, representing each of the following states (warm, tender, sympathetic, softhearted, moved, compassionate); distress score is an average of 7-point scale ratings, representing each of the following states (worried, upset, troubled, perturbed, grieved, disturbed, alarmed, distressed). These are state measures: measures that vary within people across time. The formulation of track3 is to predict, for each essay, one or more emotion labels from the following Ekman's six basic emotions (sadness, joy, disgust, surprise, anger, or fear) as well as neutral, and we also added hope.

Question: In the introduction of this paper, five tracks are mentioned. What are the detailed definitions of Track 2 and Track 3, and how do they differ from each other?


Question: Which paper first constructed a structured knowledge base to interconnect different human social roles and attributes?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper first constructed a structured knowledge base to interconnect different human social roles and attributes?

PEACOK: Persona Commonsense Knowledge for Consistent and Engaging Narratives


Question: According to the DreamLLM paper, how many evaluated Text2Image Specialists it failed to beat in MS-COCO?


Question: How many samples are there in SeeClick's general vision-language instruction-following data?

Your answer should be a single number rounding to thousands, e.g. 15000.


Question: Which paper first applied the chain of thought concepts in 3D localization problem?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper first applied the chain of thought concepts in 3D localization problem?

COT3DREF: CHAIN-OF-THOUGHTS DATA-EFFICIENT 3D VISUAL GROUNDING


Question: What do you think is the biggest advantage of Variator compared to baseline models as shown in table 1 in the article 'Variator: Accelerating Pre-trained Models with Plug-and-Play Compression Modules'? Please briefly describe the working principle of LTP, which is one of the baseline models.

The biggest advantage of Variator is its minimal additional parameters. LTP works by adaptively removing unimportant tokens based on attention scores, pruning tokens below a learned threshold for each layer.

Question: What do you think is the biggest advantage of Variator compared to baseline models as shown in table 1 in the article 'Variator: Accelerating Pre-trained Models with Plug-and-Play Compression Modules'? Please briefly describe the working principle of LTP, which is one of the baseline models.