Question: Is there an existing dataset of images with alt-text that also includes the text the image was originally posted with?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there an existing dataset of images with alt-text that also includes the text the image was originally posted with?

ALT-TEXT WITH CONTEXT: IMPROVING ACCESSIBILITY FOR IMAGES ON TWITTER


Question: Which paper first proposed to combine pretrained masked language models (BERT) and discrete diffusion language models?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper first proposed to combine pretrained masked language models (BERT) and discrete diffusion language models?

DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models


Question: On the ALFWorld dataset experiments, how much did the success rate improve when the authors used their method compared to the original baseline model?

Your answer should be a floating-point number with one decimal place.


Question: What're the three types of agents in IBSEN and which agent involves the usage of database?

Your answer should be a Python list of 2 elements. The first element is a Python list of 3 elements, containing the names of the three types of agents in IBSEN. The second element is a string, indicating the name of the agent that involves the usage of database. e.g. [["agent1", "agent2", "agent3"], "agent"].


Question: What is the main innovation in the distillation methods employed by the models in the experimental section of the article "BEYOND UNIFORM SCALING: EXPLORING DEPTH HETEROGENEITY IN NEURAL ARCHITECTURES"?

The authors add a new token, the distillation token, to the initial embeddings. It interacts with other embeddings through self-attention, and is output by the network after the last layer. Its target objective is given by the distillation component of the loss. The distillation embedding allows the model to learn from the output of the teacher, as in a regular distillation, while remaining complementary to the class embedding.

Question: What is the main innovation in the distillation methods employed by the models in the experimental section of the article "BEYOND UNIFORM SCALING: EXPLORING DEPTH HETEROGENEITY IN NEURAL ARCHITECTURES"?


Question: Compared to vanilla ViT-Base, how much relative accuracy degradation does ECP-ViT result in on ImageNet? How much relative latency save does ECP-ViT obtained?

Your answer should be a list of two floats rounded to 2 decimal places. Both floats should be in [0, 100] as percentages. The first float is the accuracy degradation and the second float is the latency reduction.


Question: Is there any paper that constructs augmented training data based on the entity-to-entity correlations?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any paper that constructs augmented training data based on the entity-to-entity correlations?

PeerDA: Data Augmentation via Modeling Peer Relation for Span Identification Tasks *


Question: What's the original form of the metrics "w(S) = \frac{1}{2} \mathbb{E} \sup_{x, y \in S} \langle g, x - y \rangle"?

Your answer should be a Python string, the formula in LaTeX format.

w(K) := \mathbb{E} \sup_{u \in K - K} \langle g, u \rangle

Question: What's the original form of Gaussian mean width?


Question: According to the paper that proposes the Subspace Identification Guarantee model, which method is used to estimate the label distribution from the target domain $p_{\hat{\mathbf{y}}}$? What's the formula of the loss they reweight using the distribution? In the paper that proposes the aforementioned method, as the dataset size increases, when the proposed method first surpasses the baseline method on MNIST with $\alpha=1.0$?

Your answer should be a Python list of 3 elements, the first is a string, the abbreviation of the method, the second is a string, the formula in LaTeX format, and the last is an integer, the approximate dataset size. Note that for the third sub-question, you don't need to figure out the exact value, just provide the approximate value that appears on the horizontal axis of the figure.

Question: What's the formula of the loss they reweight using the estimated distribution?


Question: In the MetaMath paper, a bootstraping method A is utilized in Example 3.4. In the paper that proposes the method A, which baseline method surpasses the proposed method A in AQuA under some specific setting?

Your answer should be a string, the name of the baseline method.


Question: What does formula (3) in this paper mean?

Your answer should be a Python strings of the detailed explanation of the formula.

This formula is to compute dynamic parameters and transform the less important dynamic parameters into input-agnostic static parameters. Specifically, a mask $M_i$ is utilized to indicate whether the i-th element of $\hat{\Theta}$ is dynamic or static. $M_i = 1$ means the i-th element of $\hat{\Theta}$ is dynamic so we update it through the dynamic function W with input x and dynamic factors $\Theta$. Otherwise, $\hat{\Theta}$ remains the same.

Question: What does formula (3) in this paper mean?


Question: In the experiments of the paper "SPZ: A Semantic Perturbation-based Data Augmentation Method with Zonal-Mixing for Alzheimer's Disease Detection" on the ADReSS challenge dataset, which single model method and ensemble method performed the best, excluding the method proposed in this paper?

Your answer should be a python list, with the first element being the best single model method and the second element being the best ensemble method. YOU MUST USE THE ABBREVIATIONS IN THE TABLE.


Question: Which paper did a comprehensive survey of the code large language model (code LLMs)?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper did a comprehensive survey of the code large language model (code LLMs)?

Large Language Models Meet NL2Code: A Survey


Question: What are the meanings of $h_i, r_i, t_i$ in Equation (1)?

Your answer should be a precise sentence describing the meanings of $h_i, r_i, t_i$ as given in the paper.

$h_i, r_i, t_i$ represent the $i$-th dimension in the head, relation, and tail representations respectively.

Question: What are the meanings of $h_i, r_i, t_i$ in Equation (1)?


Question: Is there any paper leverages knowledge distillation of language models for textual out-of-distribution detection or anomaly detection?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any paper leverages knowledge distillation of language models for textual out-of-distribution detection or anomaly detection?

Multi-Level Knowledge Distillation for Out-of-Distribution Detection in Text


Question: Which paper studies how current retrieval systems handle queries which contain multiple constraints?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper studies how current retrieval systems handle queries which contain multiple constraints?

QUEST: A Retrieval Dataset of Entity-Seeking Queries with Implicit Set Operations


Question: In the experiment of the paper that proposed knowledge card, a model is used as the component denoted by a cube with a question mark in the overview figure. What're the training hyperparameters of this model according to the paper that proposed it?

Your answer should be a paragraph, the training hyperparameters of the model.

We train Codex using the same learning rate as the corresponding GPT model, with a 175 step linear warmup and cosine learning rate decay. We train for a total of 100 billion tokens, using the Adam optimizer with $\beta_1 = 0.9$, $\beta_2 = 0.95$, $\epsilon = 10^{-8}$, and a weight decay coefficient of 0.1.

Question: What're the training hyperparameters of CodeX?


Question: According to Table 1, how many times is the average number of tokens for the dataset with the highest average number of tokens versus the one with the least average number of tokens?

Your answer should be a floating-point number with two decimal places.


Question: I would like to utilize the datasets introduced in the "DeakinNLP at BioLaySumm" paper. Could you tell me in which format were the papers in the datasets retrieved from each data source?

Your answer should be a string, the name of the format, e.g. JSON, HTML, MARKDOWN.


Question: Is there a paper that applies large language models to visual Raven's Progressive Matrices?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there a paper that applies large language models to visual Raven's Progressive Matrices?

In-Context Analogical Reasoning with Pre-Trained Language Models


Question: Give me a paper proposing to circumvent a single-truth target in training generative language models.

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Give me a paper proposing to circumvent a single-truth target in training generative language models.

Soft Alignment Objectives for Robust Adaptation of Language Generation


Question: Are there sequential learning guarantees for configuring a linear system solver under a distributional assumption on the systems' target vectors?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Are there sequential learning guarantees for configuring a linear system solver under a distributional assumption on the systems' target vectors?

LEARNING TO RELAX: SETTING SOLVER PARAMETERS ACROSS A SEQUENCE OF LINEAR SYSTEM INSTANCES


Question: What method was adopted in the paper developing DisGAT and SpkGAT for ERC to integrate these two modules?

A mutual cross-attention was adopted.  The computation process is formulated as follows: A_1 = softmax(H^{Dis}W_1(H^{Spk})^T ); A_2 = softmax(H^{Spk}W_2(H^{Dis})^T ); H^{Dis'} , H^{Spk'} = A_1H^{Spk}, A_2H^{Dis}

Question: What method was adopted in the paper developing DisGAT and SpkGAT for ERC to integrate these two modules?


Question: In the NeurIPS paper, mentioned in the RestoreAgent paper, that utilizes uniquely designed prompts to guide the network, what formula can the module in light yellow in Figure 3 be summaried as?

Your answer should be a string, the formula in LaTeX format.

\mathbf{P} = \text{Conv}_{3\times3} \left( \sum_{c=1}^{N} w_c \mathbf{P}_c \right), \quad w = \text{Softmax} \left( \text{Conv}_{1\times1} (\text{GAP}(\mathbf{F}_1)) \right)

Question: In the PromptIR paper, the PGM process is summarized as?


Question: How to faithfully and explicitly measure the helpfulness of human explanations to language models during finetuning and inference?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: How to faithfully and explicitly measure the helpfulness of human explanations to language models during finetuning and inference?

Are Human Explanations Always Helpful? Towards Objective Evaluation of Human Natural Language Explanations


Question: Which paper systematically examed the input mismatch between training and sampling in diffusion models

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper systematically examed the input mismatch between training and sampling in diffusion models

ELUCIDATING THE EXPOSURE BIAS IN DIFFUSION MODELS


Question: How many datasets are generated in the source paper of the dataset mainly used in the paper named "Steering Llama 2 via Contrastive Activation Addition"?


Question: What paper first uses decoupled workers in distributed RL system?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: What paper first uses decoupled workers in distributed RL system?

SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores


Question: According to the main body of the paper "Bayesian low-rank adaptation for large language models", which method is statistical significant on BoolQ with ECE metrics? In that method, how is the hyperparameter k selected?

Your answer should be a Python list of 2 strings, the first is the full name of the method, and the second is the formula in LaTeX format.

Question: How is the hyperparameter k selected?


Question: What success rate does MapGPT(with GPT-4V) achieve on the validation unseen set of the R2R dataset?

Your answer should be a single float, rounded to 1 decimal place.


Question: In the proxy task of exploring lexical semantics in the paper "Fantastic Semantics and Where to Find Them: Investigating Which Layers of Generative LLMs Reflect Lexical Semantics", how many instances are there in total?


Question: In terms of WER values with ASR across the six different methods tested in the paper, how much higher is DD2 compared to NV1?

Your answer should be a single float number, rounded to 3 decimal places.


Question: Which paper first conducted the positioned error test for the MAUVE metric?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper first conducted the positioned error test for the MAUVE metric?

On the Blind Spots of Model-Based Evaluation Metrics for Text Generation


Question: Which work proposes an approach to improve candidate responses in the smart reply task by directly optimizing the metric to ensure that a response is selected by the user?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which work proposes an approach to improve candidate responses in the smart reply task by directly optimizing the metric to ensure that a response is selected by the user?

Model-Based Simulation for Optimising Smart Reply


Question: According to Figure 4, when generating Token 11, which tokens will the cache preserve, and what positions will be assigned to them?

Your answer should be a Python list containing two sublists. The first sublist should list the tokens that the cache preserves. The second sublist should contain the positions assigned to each corresponding token. Example format: [[0, 1, 2], [0, 1, 2]].


Question: Among the dataset used for experimentation in the TEXTEE paper and the "Small Models, Big Insights" paper, how many were proposed in 2022?


Question: Which paper proposed the integration of human translators' considerations, such as length control, rhyme type control and suggestion, and enhancing compatibility between translation output and unseen melodies, into the design of machine translation models when translating lyrics?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper proposed the integration of human translators' considerations, such as length control, rhyme type control and suggestion, and enhancing compatibility between translation output and unseen melodies, into the design of machine translation models when translating lyrics?

Songs Across Borders: Singable and Controllable Neural Lyric Translation


Question: What is the main design of Auto-GUI framework from the aspects of the encoder, interaction, and decoder?

Your answer should be a Python list of text strings, with each element being one core stage of this framework, you"d better use the origin text, e.g., ["stage 1", "stage 2", ...].

Encoding: Acquire encoded features from both vision and language inputs. The vision input is encoded by a frozen vision encoder, the language input is encoded by a language encoder.

Interaction: The encoded vision and language representations are integrated by a single-head self-attention network and a gated fusion

Decoding: The fused representation is fed to the decoder to generate a chain of future action plans. The target predictions consist of a chain of future action plans and the current action prediction separated by specific prompts

Question: What is the main design of Auto-GUI framework?


Question: In the paper "Leveraging Behavioral Cloning for Representation Alignment in Cross-Domain Policy Transfer", the Portable Latent Policy (PLP) method is introduced. In Figure 5 depicting the alignment scores, how many PLP and its variant methods exhibit a P2P-medium accuracy that exceeds the P2P-obs-medium accuracy?


Question: Except WKM, which method performs the best on WebShop? Whether the two methods' papers use the same evaluation datasets or not?

Your answer should be a Python list of two strings, the first is the abbreviation of the method, the second is either `true` or `false`.


Question: In the paper that develops KUCB-RL, which model-free algorithm applies weakly communicating MDP assumption? What's the algorithm's main contribution in the online setting, regarding the assumption?

Your answer should be a Python list of two strings, the first string is the name of the algorithm, and the second string is its main contribution.

Our algorithm is the first computationally efficient model-free method with \tilde{O}(\sqrt{T}) regret for weakly communicating MDPs.

Question: What's the algorithm's main contribution in the online setting, regarding the assumption?


Question: What're the related domains of this paper according to related works?

Your answer should be a Python list of strings where each string is a related domain. e.g. ["domain1", "domain2", ...]

Prompt-based Learning
Lexical Relation Classification


Question: In terms of experimental results when unigrams are used for evaluation, which model gets the highest F1-score among Mbase, Mclf, Mcxt and Mclfcxt? What's its added module compared with Mbase according to figure 2?

Your answer should be a list of two strings, the first element is the name of the model(chosen from Mbase, Mclf, Mcxt and Mclfcxt), and the second element is the name of the added module presented in figure 2.


Question: Which paper first aggregates statements to represent political actors and learns the mapping from languages to representation via pre-training?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper first aggregates statements to represent political actors and learns the mapping from languages to representation via pre-training?

UPPAM: A Unified Pre-training Architecture for Political Actor Modeling based on Language


Question: Which core NLP problem is mentioned in the paper "Are Machines Better at Complex Reasoning? Unveiling Human-Machine Inference Gaps in Entailment Verification", and what is it usually structured as?

Your answer should be a python list of two elements. The first one is a core NLP problem name and you should use abbreviation as given in the papers. The second one is a python strings, describing how the problem is usually structured.

It is usually structured as a two or three class classification problem.

Question: what is NLI usually structured as?


Question: In the paper "Are Emergent Abilities in Large Language Models just In-Context Learning?", token edit distance is introduced as an additional evaluation metric, what's the purpose of doing so?

Your answer should be a sentence explaining the purpose of introducing token edit distance as an additional evaluation metric in the context of evaluating emergent abilities in large language models.

Question: In the paper "Are Emergent Abilities in Large Language Models just In-Context Learning?" (anchor_pdf), token edit distance is introduced as an additional evaluation metric, what's the purpose of doing so?

To align to the findings in the paper "Are Emergent Abilities of Large Language Models a Mirage?" that unproper metrics can mislead to the emergent abilities phenomenon.


Question: Is there a tool that can automatically segment speech and the corresponding text transcriptions, to obtain a finer grained alignment?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there a tool that can automatically segment speech and the corresponding text transcriptions, to obtain a finer grained alignment?

CMOT: Cross-modal Mixup via Optimal Transport for Speech Translation


Question: Which paper shows assessment of training instabilities at different levels for language models?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper shows assessment of training instabilities at different levels for language models?

Measuring the Instability of Fine-Tuning


Question: Among the previous methods applied in the FunCoder's experiments on open-source models, which one was proposed later? Additionally, which datasets was applied in the evaluation of that method, but not in FunCoder?

Your answer should be a Python list of 2 elements, the first is the name of the method, the second is a python list of strings, the name of the datasets.


Question: Which domain in the GRBench dataset has the most number of questions?

Your answer should be a single string representing the domain name.


Question: According to the results, in which interval of attack budget does the ASR of SCTS saturate? Note that the interval has been indicated directly in the text.

Your answer should be a python list of two float rounded to 2 decimal places, e.g [0.35, 0.40]


Question: Are there any papers that construct convolutional networks which are equivariant with respect to non-compact/non-abelian Lie groups?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Are there any papers that construct convolutional networks which are equivariant with respect to non-compact/non-abelian Lie groups?

LIE GROUP DECOMPOSITIONS FOR EQUIVARIANT NEURAL NETWORKS


Question: On average, how many steps does a solution have in the training set of PRM800K, and how many solutions are provided per question?

Your answer should be a Python list containing two floating-point numbers, each rounded to two decimal places. The first number represents the steps per solution, and the second number represents the solutions per question. Example format: [1.23, 4.56].


Question: What research exists on incorporating knowledge graphs into language models to improve their complex question-answering capabilities?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: What research exists on incorporating knowledge graphs into language models to improve their complex question-answering capabilities?

Knowledge Graph-augmented Language Models for Complex Question Answering


Question: In the paper that introduces TMID dataset, which pretrained model gets the best F1 score after fine-tuning on TMID dataset? And when pretraining this model, what percentage of tokens were used in the Self-Supervised Blank Infilling task?

Your answer should be a single python list containing two strings, the first element of the list is the pretrained model's name, and the second element of the list is the percentage as an integer, for example, '56%'.


Question: Is there any paper that applies curriculum learning to various NLG tasks without depending on specific metrics?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any paper that applies curriculum learning to various NLG tasks without depending on specific metrics?

In-sample Curriculum Learning by Sequence Completion for Natural Language Generation


Question: According to the tables about Zero-shot performance, what is the range of accuracy of OPT-125M in different task tests (considering the data tested in all papers)?

Your answer should be a python list of two floats, rounding to one decimal place, e.g. [12.1, 23.4].


Question: From which subset of ExHVV was MemeMQACorpus chosen, and why? How many questions were selected? Also, provide the changes in each role-label for the chosen subset.

Your answer should be a Python list of 4 elements. The first element is the subset's name. The second element is the reason why the author chose this subset. The third element is an integer, denoting the number of questions chosen. The fourth element is a Python dict, containing role-labels and their corresponding changes, where each change is calculated as the new count minus the old count. e.g. ["answer1", "answer2", 3, {"role1": -2, "role2": 3, ...}]

This domain choice is based on diversity in the entity distribution across different roles compared to the other subset (on Covid-19) of ExHVV dataset.

Question: Why was MemeMQACorpus chosen from the US Politics subset of ExHVV?


Question: How many words are in the train splits of the oldest dataset used by GeNE to evaluate language modeling?


Question: Is there any paper that proposes a set of criteria to comprehensively evaluate generated conversations?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any paper that proposes a set of criteria to comprehensively evaluate generated conversations?

Modeling What-to-ask and How-to-ask for Answer-unaware Conversational Question Generation


Question: For Llama2 on DialogSum, which newly proposed module contributes more to the improvement of performance? How do the authors further explain why that module works?

Your answer should be a Python list of 2 elements, where the first element is the FULL NAME of the module that contributes more to the improvement of performance, and the second element is a string, explaining why that module works. e.g. ["module", "explanation"].

In our main experiments, we use dialogue as the condition for generating the final summary in the FG. To explore the effect of using such condition, we run experiments without using it, where the results are ported in Table 10. It clearly shows that, compared with the models without using the entire dialogue, our approach is able to generate better summaries, which emphasizes the contribution of the entire dialogue, for the reason that it provides global or environmental information to guide FG identifying useful content produced by the experts.

Question: How do the authors further explain why Fusion Generation (FG) works?


Question: Is there any paper that proposes a new multimodal video dataset that image-level multimodal models do not work well?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any paper that proposes a new multimodal video dataset that image-level multimodal models do not work well?

Revealing Single Frame Bias for Video-and-Language Learning


Question: According to the paper that enhances traditional input transformations by mixing the input image with images from other categories to create admixed images, what're the related adversarial attacks?

Your answer should be a Python list of abbreviations of the attacks.


Question: In the agent "MO-DDN" (Multi-object Demand-driven Navigation), what is the basic success rate for a specific demand instruction DI?

Your answer should be a formula string in latex format.

SR_b=\frac{1}{N}\sum_{i=1}^N\max_{s_b\in So_b}\frac{\sum_{o\in FL}\mathbb{1}_{o\in s_b}}{Len(s_b)}

Question: In the agent "MO-DDN" (Multi-object Demand-driven Navigation), what is the basic success rate for a specific demand instruction DI?


Question: What are the top 3 most main CONCEPTNET relations in the commonsense reasoning task dataset used in the paper "Can LLMs Learn From Mistakes? An Empirical Study on Reasoning Tasks"?

Your answer should be a python list. YOU MUST USE THE EXACT NAMES OF THE RELATIONS AS THEY APPEAR IN THE PAPER.


Question: On which labeled dataset did the metric AMR not reduce to Acc? On that dataset, which model performs best on the metric AMR?

Your answer should be a Python list of three elements, the first element is the name of the labeled dataset, the second and third element is the model family and the variant of the model. e.g. ["answer1", "answer2", "answer3"].


Question: What is the difference between Equation (1) and Equation (2)?

Your answer should be text describing the difference.

Equation (1) is used in single-vector retrieval models. It calculates the similarity score $s(q, d)$ as the dot product between the encoded vector of the query $v_q = \eta_Q(q)$ and the encoded vector of the document $v_d = \eta_D(d)$. In this method, the entire query and the entire document are each represented by a single vector, and their similarity is determined by the cosine of the angle between these two vectors. This approach does not consider token-level interactions, as all token embeddings are pooled into a single vector before the similarity score is computed.

Equation (2) is used in multi-vector retrieval models, specifically in ColBERT. It calculates the similarity score by considering the interaction between each token in the query and each token in the document. The similarity score $s(q, d)$ is defined as $s(q, d) = \sum_{i=1}^{N} \max_{j} v_{qi}^T v_{dj}$, where $v_{qi}$ and $v_{dj}$ are the last-layer contextualized token embeddings of BERT for the i-th token in the query and the j-th token in the document, respectively. This operation, known as MaxSim, exhaustively compares each query token to all document tokens, effectively capturing the most relevant token-level interactions.

Question: What is the difference between Equation (1) and Equation (2)?


Question: Which paper utilized MMD flows with Riesz kernels to solve Bayesian inverse problems?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper utilized MMD flows with Riesz kernels to solve Bayesian inverse problems?

Posterior Sampling Based on Gradient Flows of the MMD with Negative Distance Kernel


Question: Which subtask in NADI 2024 was the CUFE paper related to?

Your answer should be a python string. The string should be "Subtask 1", "Subtask 2" and so on.


Question: The article "DISTINGUISHED IN UNIFORM: SELF-ATTENTION VS. VIRTUAL NODES" employed certain datasets from the recent LRGB collection paper. Please specify which benchmarking datasets included in that paper were not utilized in this study.

Your answer should be a python list of strings, e.g., ["dataset1", "dataset2"].


Question: In the EQA-MX dataset, which task takes the smallest proportion? In that task, which output appears the most?

Your answer should be a Python list of 2 strings, where the first element is the abbreviation of the task and the second element is the most frequent output in that task.


Question: In the paper that introduces Agent-as-a-Judge and proposes a dataset called DevAI, which evaluation method is the primary comparison target to Agent-as-a-Judge? And in this evaluation method's original paper, which LLM performs the best in Consistency?

Your answer **must** be a single python list containing two strings, the first element of the list is the method's name, the second element of the name of the best model in Consistency.


Question: In the experiment section of this paper, it is proposed that research shows one evaluation method is better. What desired criteria are these conclusions based on?

Your answer should be a python list about the criteria, e.g. ["criterion1", "criterion2"]. YOU MUST USE THE EXACT NAMES FROM THE PDF WITHOUT CHANGING THE CAPITALIZATION.

Metric Monotonicity
Metric Separability
Metric Linearity
Metric Time Efficiency
Metric Accuracy
Size Robustness
Imbalance Robustness


Question: Which paper examined the scalability of instruction-tuning with respect to Mixture of Expert models?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper examined the scalability of instruction-tuning with respect to Mixture of Expert models?

Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models


Question: What is the difference between DICE and TAILO as regard to unlabeled data and what are the two steps of training a discriminator c(s)?

DICE regards all unlabeled data as negative samples whereas TAILO use Positive-Unlabeled (PU) learning to train the discriminator. Training a discriminator c(s) consists two steps: 1)Training another discriminator c'(s) that identifies safenegative samples, and 2) formal training of c(s)

Question: What is the difference between DICE and TAILO as regard to unlabeled data and what are the two steps of training a discriminator c(s)?


Question: Among the tested models, which model performs best on code problems?

Your answer should be a python string of the name of the model.


Question: Which evaluation method is used in the paper against gold standards, despite having a low correlation with human judgments according to various studies?

Your answer must be ONE string of the evaluation method's name.


Question: How many datasets are evaluated in the work that the "BIG-Bench Mistake" paper follows to generate each step separately?


Question: Which paper introduce a DRO (distribution robust optimization) like training objective for doing adversarial training without constructing adversarial samples.

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper introduce a DRO (distribution robust optimization) like training objective for doing adversarial training without constructing adversarial samples.

DSRM: Boost Textual Adversarial Training with Distribution Shift Risk Minimization


Question: On which downstream tasks did the authors experiment with their method, and by how much did it improve compared to the best existing methods?

Your answer should be a Python dictionary, where the keys represent the downstream tasks on which the authors conducted experiments, and value is the numerical part of a percentage (between 0 and 100, rounded to 1 decimal place), indicating the improvement compared to the best existing method.. e.g. {"task1": 1.9, "task2": 3.5, ...} .


Question: Could you recommend research that assesses how well language learning models, such as ChatGPT, perform in creating reading comprehension tasks for educational software?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Could you recommend research that assesses how well language learning models, such as ChatGPT, perform in creating reading comprehension tasks for educational software?

Evaluating Reading Comprehension Exercises Generated by LLMs: A Showcase of ChatGPT in Education Applications


Question: Which paper uses the latent diffusion model for the first time to solve offline reinforcement learning problems based on the sequential modeling framework?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper uses the latent diffusion model for the first time to solve offline reinforcement learning problems based on the sequential modeling framework?

Efficient Planning with Latent Diffusion


Question: In the existing deblurring dataset compared in GS-Blur paper, that contains both real and synthetic data, what's the average noise level estimated?

Your answer should be a float, rounding to 4 decimal places.


Question: Which paper introduces the R-GCN technique into document-level joint entity and relation extraction?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper introduces the R-GCN technique into document-level joint entity and relation extraction?

A Novel Table-to-Graph Generation Approach for Document-Level Joint Entity and Relation Extraction


Question: I want to replicate the experiment in this paper. Please list all the datasets and baselines that I should prepare.

1. For datasets, this paper select five challenging logical reasoning benchmarks: (1)LogiQA (2)ProofWriter (3)FOLIO (4)PrOntoQA (5)LogicalDeduction(LD). 2. For baselines, (1)Standard prompting (2)Chain-of-Thought(CoT) (3)Chain-of-Thought with Self-Consistency(CoT-SC) (4)Selection-Inference(SI) (5)LAMBADA (6)Tree-of-Thought(ToT) (7)Cumulative Reasoning(CR).

Question: I want to replicate the experiment in this paper. Please list all the datasets and baselines that I should prepare.


Question: What are the advantages of the transformer-based architecture proposed in the paper?

Your answer should be a python strings about the obvious advantages.

The Rough Transformer is independent of the sequence length of input data.

The Rough Transformer is robust to irregular sampling

The Rough Transformer decreases the memory and computational bottleneck inherent to the vanilla Transformer.

Question: What are the advantages of the transformer-based architecture proposed in the paper?


Question: Among the datasets of the benchmark that collects CHIP-CDN, what are the evaluation metrics they applied?

Your answer should be a Python list of strings, containing the names of the evaluation metrics.


Question: What are the two loss functions in Equation (8) means?

Your answer should a list with two items, representing the meaning of the first and the second loss function respectively.

$\mathcal{L}_{y}(\theta)$: This term refers to the loss associated with the text prediction task. It is part of the multi-task learning strategy, where the model is trained to predict the textual output, such as the transcription or translation of the input speech.

$\mathcal{L}_{z}(\theta)$: This term corresponds to the loss for the acoustic unit prediction task. Acoustic units are discrete representations of the speech signal, and this loss helps the model learn to generate these units, which are then used to synthesize the translated speech.

Question: What are the two loss functions in Equation (8) means?


Question: What's the baseline used in the experiment?

Your answer should be a plein text DIRECTLY FROM THE PDF.

We employ the released PLATO-v1 model, a pre-trained dialogue generation model based on UniLM, for our experiment.

Question: What's the baseline used in the experiment?


Question: How to calculate final loss function(Loss) in this paper?

Your answer should be a python string, which is a formula in latex format to calculate a parameter.

\textit{Loss} = -\frac{1}{N} \sum_{i=1}^{N} \left( w_p y_i \log(\hat{y}_i) + w_n (1 - y_i) \log(1 - \hat{y}_i) \right) -rac{1}{N} \sum_{i=1}^{N} \log \left( \frac{\exp(s(y_i)/\tau)}{\sum_j \exp(s(y_j)/\tau)} \right)

Question: How to calculate final loss function(Loss) in this paper?


Question: I would like to reproduce the experiments of KnowGPT, could you please provide me with the websites of the datasets applied in the experiment?

Your answer should be a Python list of 3 strings, the websites. Note that you should provide the original URL as given in the papers that proposed the datasets.


Question: On cmudog, which one among the linguistics operators appears the most frequently? What's its distribution?

Your answer should be a Python list of 2 elements. The first element is the linguistic operator"s name, and the second element is its disrtibution in percent in string format. e.g. ["answer", "5%"].


Question: On which datasets this study surpassed the SOTA?

Your answer should be a Python list of dataset, e.g., ["dataset1", "dataset2", ...]. YOU MUST USE THE EXACT TEXT FROM THE PAPER.

DailyDialog
CMU_DoG
LIGHT
EmpathicDialogue
Wizard of Wikipedia
CommonsenseDialog


Question: What does each record in the dataset used by the event-based knowledge editing benchmark in the paper "EVEDIT: Event-based Knowledge Editing for Deterministic Knowledge Propagation" include?

Your answer should be a python list of strings. YOU MUST USE THE EXACT NAMES FROM THE PAPER, RATHER THAN MATHEMATICAL SYMBOLS OR ABBREVIATION.

requested rewrite
paraphase prompts
neighborhood prompts
generation prompts
reference texts


Question: What manipulation operations does the manipulation network in this paper allow? For the add network, how is the training loss of relation defined?

Your answer should be a single python list, the first element of the list is a list of the strings of manipulation operations, the second element of the list is a string of the formula in latex format, e.g. [["operation1", "operation2"], "l_{\text {relation }}=..."]

\ell_{	ext {relation}}=-\log \left(p\left(h_{\mathrm{r}}\left(\widetilde{o}_{	ext {new }}, o_{i}
ight)=r
ight)
ight)

Question: For the add network, how is the training loss of relation defined?


Question: What dataset does this paper(titled "Identifying Conspiracy Theories News based on Event Relation Graph") use for training? How many event coreference chains does this dataset contain?

Your answer should be a python list of two strings, the first element is the dataset name(abbrievation), and the second element is an integer number.


Question: In the paper that proposes the only comparable interactive theorem prover applied as a baseline by AIPS, where are the evaluation samples chosen from?

Your answer should be the a raw text from the papers.

We perform experiments on theorems from "Mathematics in Lean" : a book for beginners to formalize and prove mathematical theorems in Lean. It has 233 theorem proving exercises, covering topics from sets and functions to topology, calculus, and measure theory. For evaluation, we randomly selected 50 theorems, and their proofs have 5.52 tactics on average.

Question: In the paper that proposes LeanCopilot, where are the evaluation samples chosen from?


Question: Find the NLP paper that focuses on dialogue generation and introduces advancements in the augmentation of one-to-many or one-to-one dialogue data by conducting augmentation within the semantic space.

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Find the NLP paper that focuses on dialogue generation and introduces advancements in the augmentation of one-to-many or one-to-one dialogue data by conducting augmentation within the semantic space.

DialoGPS: Dialogue Path Sampling in Continuous Semantic Space for Data Augmentation in Multi-Turn Conversations


Question: In figure 3, there are three types of losses. What is the function for the first loss?

Your answer should be a formula in latex format extracted from the paper.

\mathcal{L}_{	ext {topk }}=\sum_{i} \sum_{j \in N N_{k}(i)} \sum_{m} \mid \operatorname{Sim}\left(c e_{i}, c e_{j}
ight)- \operatorname{Sim}\left(f\left(c e_{i}
ight)[: m], f\left(c e_{j}
ight)[: m]
ight) \mid

Question: In figure 3, there are three types of losses. What is the function for the first loss?


Question: Please give me the github link of this work.

Your answer should be a single string of the github link.


Question: Could you recommend a dataset paper which presents relation extraction performance on translated data and compare it to English data?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Could you recommend a dataset paper which presents relation extraction performance on translated data and compare it to English data?

MultiTACRED: A Multilingual Version of the TAC Relation Extraction Dataset


Question: In Figure 3, what can we infer from comparing the performance with training data generated by self-training (ST) versus without it?

Your answer should be a Python strings about the conclusion from comparing the performance with training data generated by self-training (ST) versus without it.

Figure 3 shows that self-training is beneficial for each of the languages. The improvement is particularly strong when the teacher model was based on a very small amount of data.

Question: In Figure 3, what can we infer from comparing the performance with training data generated by self-training (ST) versus without it?


Question: What does formula (1) mean in the Methodology section?

Your answer should be a python strings of the detailed explanation of the formula.

The idea of this formula is decomposing token representations into their constituent vectors based on vector-based approaches. Decompose the i-th token representation in layer l into elemental vectors attributable to each of the N input tokens. So we can compute the norm of the attribution vector of the k-th input to quantify its total attribution to xi.

Question: What does formula (1) mean in the Methodology section?


Question: How does the system efficiently adapt to completely unfamiliar opponent policies during deployment, while still maintaining performance with known policies?

Question: How does the system efficiently adapt to completely unfamiliar opponent policies during deployment, while still maintaining performance with known policies?

The system adapts to completely unfamiliar opponent policies by collecting and accumulating opponent trajectory data in an Opponent-Collecting Window (OCW), which is then sampled and stitched together using the GetOnD function for in-context learning. For known policies, the system quickly re-engages suitable responses by leveraging previously accumulated trajectories, ensuring both fast adaptation to familiar policies and effective extrapolation for unfamiliar ones.


Question: Which institution funded the model with worse debiasing ability in Post-hoc in "A Parameter-Efficient Multi-Objective Approach to Mitigate Stereotypical Bias in Language Models"?

Your answer should be a python string of the institution name. You should use full name.


Question: What is the first paper that theoretically studies training neural networks under small initialization?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: What is the first paper that theoretically studies training neural networks under small initialization?

Early Neuron Alignment in Two-layer ReLU Networks with Small Initialization


Question: Is there any paper that attempts to evaluate the similarity of meaning representations without using annotated data?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any paper that attempts to evaluate the similarity of meaning representations without using annotated data?

Evaluate AMR Graph Similarity via Self-supervised Learning


Question: In which four directions can the author's model be trained?

Your answer should be python list, each element of the list is a string like 'A-to-A', 'A-to-B'.

word-to-word word-to-definition definition-to-definition definition-to-word


Question: In the two papers that updated ToMi according to the SimTom paper, what are the other datasets used to evaluate the models, besides ToMi?

Your answer should be a Python list of strings, containing the name of the datasets.


Question: Which vision-language model can demonstrate that visual grounding could facilitate efficient language acquisition? (OctoBERT)

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which vision-language model can demonstrate that visual grounding could facilitate efficient language acquisition? (OctoBERT)

World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Models


Question: For the strongest baseline mentioned in "TIES-Merging: Resolving Interference When Merging Models", which benchmark and what tasks were used for NLP in the paper which proposed it?

Your answer should be a python dictionary with the keys "benchmark" and "tasks". The value for "benchmark" should be a string and the value for "tasks" should be a list of strings.


Question: According to the paper that proposes ExpressivityArena, what's the notable example that uses human feedback to manually evaluate the model? In that arena, which model has a 0.72 win-rate against llama-2-7b-chat at the time when that paper was written?

Your answer should be a Python list of 2 strings, the name of the example, and the name of the model as given in the paper.


Question: In the paper "Quantized Local Independence Discovery for Fine-Grained Causal Dynamics Learning in Reinforcement Learning", which method performs the best on ID states with full-chain setting? In the paper that proposed that method, where does the -inf in Fig. 3 come from?

Your answer should be a Python list of 2 strings, the first is the abbreviation of the method, the second is the reason why -inf appears in Fig. 3.

Certain features are masked to $-\infty$ according to a binary map $M_j$.

Question: Where does the -inf in Fig. 3 come from?


Question: What are the datasets used in the experiments and what are their respective durations? Where can I get these datasets?

Your answer should be a Python list of 2 elements. The first element is a Python dictionary containing dataset names and respective durations (in hours, rounded to 1 decimal place), and the second element is a Python string containing the answer to the last question, e.g. [{"dataset1": 2.1, "dataset2": 10.8, ...}, "answer"]

The article indicates that these are internal datasets, thus they may not be currently accessible.

Question: Where can I get these datasets?


Question: How much higher is the best EX score in the paper "Synthetic SQL Column Descriptions and Their Impact on Text-to-SQL Performance" than that in the original dataset applied? Assuming that both experiments are taken under no knowledge setting.

Your answer should be a float between 0 and 1, rounding to 4 decimal places.


Question: How does the latest Wellness Descriptions dataset used in the paper address the ambiguity issue in the task of annotating text for wellness dimensions?

Your answer should be a python strings concisely summarizing the method.

They address the ambiguity issue by providing a set of perplexity guidelines. The major perplexity guidelines are as follows. 1. Presence of Multiple Aspects: Social media posts often express feelings due to various reasons. To address this, specific text spans contributing to the more focused health consequences can be identified and associated with corresponding wellness dimensions. 2. Annotation Ambiguity: Although there may be multiple aspects in the post, we must consider the holistic aspect as per the experts' opinion. 3. Reading between the lines: The text may contain implicit or subtle hints that suggest a particular wellness dimension. But clear and meaningful words that suggest one of the four wellness dimensions should be annotated accordingly.

Question: How does the latest Wellness Descriptions dataset used in the paper address the ambiguity issue in the task of annotating text for wellness dimensions?


Question: In the experiments presented in the paper "Fast and Efficient Speech Enhancement with Variational Autoencoders" evaluating the performance for speech enhancement, which method demonstrates the best performance apart from the proposed framework itself?

Your answer should be a python strings. YOU MUST USE THE EXACT ABBREVIATION AS IN THE PAPER.


Question: Among the diverse interactive domains used to test SOFT-SC, which one is the first parallel interactive text-based and embodied environment?

Your answer should be a python string, and it should be a diverse interactive domain name.


Question: What linguistic property does this paper investigate?

Your answer should be an English string.

Question: What linguistic property does this paper investigate?


Question: Is there a decoder-only language model that does not use a tokenizer and operates on raw text bytes?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there a decoder-only language model that does not use a tokenizer and operates on raw text bytes?

ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Models


Question: What paper first used the technique of prompt engineering to generate adversarial prompts that can fool LLMs into making wrong predictions in prompt-based learning?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: What paper first used the technique of prompt engineering to generate adversarial prompts that can fool LLMs into making wrong predictions in prompt-based learning?

AN LLM CAN FOOL ITSELF: A PROMPT-BASED ADVERSARIAL ATTACK


Question: Are there datasets and benchmarks available for measuring LLM graph reasoning abilities?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Are there datasets and benchmarks available for measuring LLM graph reasoning abilities?

TALK LIKE A GRAPH: ENCODING GRAPHS FOR LARGE LANGUAGE MODELS


Question: How much higher ASRs do user cases with high content freedom yield, compared to those with low content freedom?

Your answer should be a floating point numbers with one decimal places.


Question: What research has been conducted on incorporating visual data into the text summarization process?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: What research has been conducted on incorporating visual data into the text summarization process?

Summary-Oriented Vision Modeling for Multimodal Abstractive Summarization


Question: In the paper that DRAGIN follows the most in term of template prompt, which models are utilized as the CoT generator of the proposed retriever?

Your answer should be a string, giving the detailed names of the models, as proposed in the reference paper.

We experiment with OpenAI GPT3 (code-davinci-002) and Flan-T5 of different sizes as its CoT generator.

Question: In the paper that DRAGIN follows the most in term of template prompt, which models are utilized as the CoT generator of the proposed retriever?


Question: For dataset MultiDialog, what's the number of dialogues, the number of turns, total length in hours, number of speakers, and the name of source dataset of dialogue scripts? Please adopt the statistics the most accurate you can find in the paper.

Your answer should be a python list,of which the elements are in the following order: number of dialogues(int), number of turns(int), total length in hours(float, rounded to 2 decimal places), number of speakers(int), and the name of source dataset of dialogue scripts(str), every element of the list is an int or float or string representing the relevant dataset information.


Question: According to the DeepKKT paper, which method performs the best on CIFAR10 with 1 generated image per class under 50-shot setting? Additionally, in the paper that proposes that method, what's the latest dataset evaluated?

Your answer should be a Python list of 2 elements, the abbrevation of the method and the name of the dataset.


Question: According to Figure 1, which net has the best generalization ability?

Your answer should be a single string of the net's name.


Question: Which paper first proposed to only update some original weights of self-attention layers in parameter-efficient fine-tuning?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper first proposed to only update some original weights of self-attention layers in parameter-efficient fine-tuning?

HiFi: High-Information Attention Heads Hold for Parameter-Efficient Model Adaptation


Question: How many programming languages in The Stack are selected in the code dataset used for hypernetwork training in the paper?


Question: Which paper proposes to use rewriting based approaches to defending against adversarial attacks in text classification?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper proposes to use rewriting based approaches to defending against adversarial attacks in text classification?

Don't Retrain, Just Rewrite: Countering Adversarial Perturbations by Rewriting Text


Question: Which evaluation metric demonstrates the greatest improvement in the finetuned model proposed in this paper compared to GPT baseline?

Your answer should be a Python string, which is the name of the evaluation metric DIRECTLY FROM THE PDF.


Question: Which paper first introduced document content as an intermediate generation target and utilized textual document identifiers in generative retrieval?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper first introduced document content as an intermediate generation target and utilized textual document identifiers in generative retrieval?

TOME: A Two-stage Approach for Model-based Retrieval


Question: Which article first proposed shuffled-group-whitening to solve the problem of sentence representation learning?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which article first proposed shuffled-group-whitening to solve the problem of sentence representation learning?

WhitenedCSE: Whitening-based Contrastive Learning of Sentence Embeddings


Question: How much does the average performance of the model improve on WMT'19 test sets by replacing one of example-specific prompts with a task-level prompt?

Your answer should be a single float number ranging from 0 to 100 and rounded to 2 decimal places, representing the subtraction result.


Question: Are there any papers that build dense retrievers with mixture-of-experts architecture where each expert is responsible for different types of queries?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Are there any papers that build dense retrievers with mixture-of-experts architecture where each expert is responsible for different types of queries?

Chain-of-Skills: A Configurable Model for Open-Domain Question Answering


Question: Which PLM method achieve the best bias score in the experiment?

Your answer should be a single string representing the PLM method.


Question: According to the paper "Whose Preferences? Differences in Fairness Preferences and Their Impact on the Fairness of AI Utilizing Human Feedback", both the papers "Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation" and "Designing Toxic Content Classification for a Diversity of Perspectives" adopted standard analysis methods. Then which variable's impact on experimental data is considered in all three papers?

Question: According to the paper "Whose Preferences? Differences in Fairness Preferences and Their Impact on the Fairness of AI Utilizing Human Feedback", both the papers "Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation" and "Designing Toxic Content Classification for a Diversity of Perspectives" adopted standard analysis methods. Then which variable's impact on experimental data is considered in all three papers?


Question: What shortcoming does REV overcome? and how?

Your answer should be a paragraph describing the shortcoming that REV overcomes and how it overcomes it, based on the content of the paper.

The REV metric overcomes the shortcoming of existing free-text rationale evaluation methods that focus primarily on how well a rationale helps predict a label. Traditional metrics often fail to assess the new information a rationale provides beyond what is already present in the input or label.

REV addresses this shortcoming by quantifying the additional, label-relevant information in a rationale, using an information-theoretic approach. It evaluates rationales along two dimensions: (1) Support for the Label: Whether the rationale helps predict the intended label; (2) New Information Contribution: How much unique information the rationale adds, beyond the input and label, to justify the prediction

Question: What shortcoming does REV overcome? and how?


Question: What's the Type of the Pattern "Character Role Play" in jailbreak prompts? How can we make role-playing models more responsible in the RoleLLM paper?

Your answer should be a python list of two elemments. The first is a python string of a type name, and the second one is a list of several measures.

Implement Advanced Moderation Tools.
Bias Detection and Mitigation.
Transparency in Development and Use.
Feedback Mechanism.

Question: How can we make role-playing models more responsible in the Jailbreaking paper?


Question: What are the advantages and disadvantages of MUX-PLMs mentioned in the paper?

Your answer should be a string list of advantages and disadvantages.

Significant Throughput Improvement: MUX-PLMs leverage data multiplexing to process multiple inputs in a single forward pass, resulting in a dramatic increase in throughput.

Comparable Performance to PLMs: Despite the increased throughput, MUX-PLMs maintain performance close to traditional PLMs on downstream tasks.

Model Generalizability: MUX-PLMs can be fine-tuned like traditional PLMs for various downstream tasks, such as text classification and named entity recognition.

Performance-Throughput Trade-off: As the number of multiplexed inputs (N) increases, the throughput improves further, but the performance may slightly degrade.

Model Size Limitations: While the paper demonstrates the effectiveness of MUX-PLMs across different model sizes, the model size still impacts performance and throughput. Larger models may offer better performance but with a potentially smaller throughput improvement compared to smaller models.

Data Sampling Strategy: The paper employs a random data sampling strategy, but a more sophisticated approach could potentially enhance performance, for example, clustering similar instances based on similarity metrics and multiplexing them.

Question: What are the advantages and disadvantages of MUX-PLMs mentioned in the paper?


Question: Which domain in the GRBench dataset does not have any hard questions?

Your answer should be a single string representing the domain name.


Question: How many samples are there in total in the dataset used by MIDGARD for Task 2 evaluation?

Your answer should be an integer, the number of samples.


Question: In section 3, the author provides an exemplary event description. List the features in the example that correspond to the semantic roles discussed in the following paragraph.

Your answer should be a Python dictionary where the keys are the semantic roles and the values are the features that correspond to the roles. e.g. {"semantic_role1": "feature1", "semantic_role2": "feature2", ...}

soldiers


Question: Which national project supports both the BeamAggR and SpikeVoice paper?

Your answer should be a python string of the project name. You should use full name as given in the papers and don't add "the" before the project name.


Question: Which languages are included in the evaluation dataset used in the paper?

Your answer should be a list of languages, e.g., ["Language1", "Language2"].


Question: If one would like to train (or evaluate) a helpful assistant agent that can converse with humans while the humans traverse an environment, which work has the most suitable resource?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: If one would like to train (or evaluate) a helpful assistant agent that can converse with humans while the humans traverse an environment, which work has the most suitable resource?

SIMMC-VR: A Task-oriented Multimodal Dialog Dataset with Situated and Immersive VR Streams


Question: What is the adversarial dataset used for the PI task in this paper, and where are the source sentences of this dataset from?

The adversarial dataset used for the PI task in this paper is PAWS. Source sentences are drawn from both the Quora Question Pairs (QQP) corpus and Wikipedia.

Question: What is the adversarial dataset used for the PI task in this paper, and where are the source sentences of this dataset from?


Question: Can you find a dataset that shows LLM-based evaluation may not be reliable enough?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Can you find a dataset that shows LLM-based evaluation may not be reliable enough?

EVALUATING LARGE LANGUAGE MODELS AT EVALUATING INSTRUCTION FOLLOWING


Question: Among the two baselines introduced in the experiment setting of the paper "Fine-tuning Language Models for Factuality", which one performs better on Medical QA? That baseline was evaluated on which dataset in the paper that proposed it?

Your answer should be a Python list of 2 elements, the first is the abbreviation of the baseline, and the second is the name of the dataset.


Question: What is the lowest accuracy score achieved by RawNet2 on a dataset used in the paper "Reliability Estimation for Synthetic Speech Detection" but not in "SAMO: Speaker Attractor Multi-Center One-Class Learning For Voice Anti-Spoofing"?

Your answer should be a Python float, rounded to 2 decimal places.


Question: Which model gains the highest acc in the Table 1 under the dataset of XNLI in the paper 'InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training', and in the original paper of this model, how many layers the author designes in the base size?

Your answer should be a Python list of 2 strings, the name of the model, and the number of layers of this model's base size.


Question: What is the original form of the metric formula used in the anchor paper for the test split of the BabyLM shared task dataset?

Your answer should be one formula in LaTeX format without explanation.

Question: What is the original form of the metric formula used in the anchor paper for the test split of the BabyLM shared task dataset?

\mathrm{PLL}_{\mathrm{orig}}(S) := \sum_{t=1}^{n} \mathrm{log}~P_{\mathrm{MLM}}(s_t~|~S_{\setminus t})


Question: *Could you suggest a dataset with legally or ethically contentious content, and labels for acceptable and non-acceptable questions.

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: *Could you suggest a dataset with legally or ethically contentious content, and labels for acceptable and non-acceptable questions.

SQUARE: A Large-Scale Dataset of Sensitive Questions and Acceptable Responses Created Through Human-Machine Collaboration


Question: According to the paper that proposed acceptance rate and relative performance ratio, what's the most direct metrics for model evaluation?

Your answer should be a string, the formula of the metrics in LaTeX format.

\mathcal{Q}^{\mathrm{d}} = \{ \mathcal{Q}_1^{\mathrm{d}}, \mathcal{Q}_2^{\mathrm{d}}, \cdots, \mathcal{Q}_K^{\mathrm{d}} \}, \quad \mathcal{Q}_{\text{ave}}^{\mathrm{d}} = \frac{1}{K} \sum_i \mathcal{Q}_i^{\mathrm{d}},

Question: What's the formula for Distributed Absolute Performance?


Question: Which paper used both automatically generated and manual templates with word tuples to adapt language models from one timestamp to another?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper used both automatically generated and manual templates with word tuples to adapt language models from one timestamp to another?

Learning Dynamic Contextualised Word Embeddings via Template-based Temporal Adaptation


Question: Is there any paper that previously proposed to control a risk using prediction sets, based on the literature in conformal prediction?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any paper that previously proposed to control a risk using prediction sets, based on the literature in conformal prediction?


Question: In formula (3) of the paper, how is I_{container} calculated?

Your answer should be a python strings about the calculation approach and formula of I_{containe}.

Question: In formula (3) of the paper, how is I_{container} calculated?

I_{container} is calculated by an overflow sensitivity training method to help the Decoder to be more sensitive when facing edge range pixel value. The purpose of the design is that when decoding pixel values in the edge range, the feature extraction is only related to the template amplitude and has nothing to do with the positivity. Specifically, the approach is to add conditions to the additive operations. If the pixel value after superposition overflows, the addition operation of the pixel is converted into a subtraction operation. The formula is as follows: I_{container}= \begin{cases} I_{host}-I{additive}, overflow \ I_{host}+I{additive}, otherwise \end{cases}


Question: Which knowledge graph completion method focuses on reducing memory usage by pruning features?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which knowledge graph completion method focuses on reducing memory usage by pruning features?

GreenKGC: A Lightweight Knowledge Graph Completion Method


Question: Which paper highlights the need for leveraging all available resources, including dictionaries, machine translation systems, and language learners, to construct NLP data in low-resource languages?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper highlights the need for leveraging all available resources, including dictionaries, machine translation systems, and language learners, to construct NLP data in low-resource languages?

Rethinking Annotation: Can Language Learners Contribute?


Question: What is the main difference between the paper retrieval methods of this two papers?

In Chain of Ideas, the paper retrieval method is based on the citation relationship. While in SCIPIP, the paper retrieval method is based on the literature database it constructed.

Question: What is the main difference between the paper retrieval methods of this two papers?


Question: Which paper trains on linear regression to hypothesize how fine-tuning affects language models?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper trains on linear regression to hypothesize how fine-tuning affects language models?

UNDERSTANDING CATASTROPHIC FORGETTING IN LANGUAGE MODELS VIA IMPLICIT INFERENCE


Question: Which paper formally defines the problem of model selection in llm agent for multi-modal reasoning?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper formally defines the problem of model selection in llm agent for multi-modal reasoning?

TOWARDS ROBUST MULTI-MODAL REASONING VIA MODEL SELECTION


Question: According to Figure 1,compared with CoT prompt,what is the advantage of QAP?

Your answer should be a sentence that clearly mention the advantage of QAP

QAP explains the question in its own way.

QAP more likely leads to a correct answer.

Question: According to Figure 1,compared with CoT prompt,what is the advantage of QAP?


Question: What is the average length of essays used in this study?

Your answer should be a float number with 1 decimal place.


Question: In the TARA dataset, which tool is evaluated the most in test set?

Your answer should be a Python string, the name of the tool.


Question: Is there any research paper that can extract attributes from both a predefined label set and the surrounding context?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any research paper that can extract attributes from both a predefined label set and the surrounding context?

AtTGen: Attribute Tree Generation for Real-World Attribute Joint Extraction


Question: In the dataset that the GeMQuAD paper used for Spanish labeled examples, which language has the longest paragraph in average in tokens?

Your answer should be the full name of the language, e.g. English


Question: Is there a paper illustrating that pre-trained transformers from LLMs can be used to encode visual information in a wide range of scenarios?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there a paper illustrating that pre-trained transformers from LLMs can be used to encode visual information in a wide range of scenarios?

FROZEN TRANSFORMERS IN LANGUAGE MODELS ARE EFFECTIVE VISUAL ENCODER LAYERS


Question: How much faster is OFU-MLogB compared to MNL-UCB in the multinomial logistic bandit experiment?

Question: How much faster is OFU-MLogB compared to MNL-UCB in the multinomial logistic bandit experiment?


Question: For the latest two selected ML datasets of vision modality in Croissant, where did the raw images come from?

Your answer should be a single word, the name of the website where the images came from.


Question: Is there any paper that utilizes masked language modeling to defend against word-level adversarial attacks?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any paper that utilizes masked language modeling to defend against word-level adversarial attacks?

RMLM: A Flexible Defense Framework for Proactively Mitigating Word-level Adversarial Attacks


Question: What motivates the author to propose this paper?

The author is motivated by the idea that different sensors, though separated by modality or time, capture correlated information about the same underlying phenomena. This motivates using an information-theoretic approach to learn shared representations across multi-view data and disentangle common latent factors.

Question: What motivates the author to propose this paper?


Question: Is there such a reading comprehension dataset in understanding a snippet from a long story book, while it requires to integrate the necessary long history texts before the snippet to full understand it?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there such a reading comprehension dataset in understanding a snippet from a long story book, while it requires to integrate the necessary long history texts before the snippet to full understand it?

Personality Understanding of Fictional Characters during Book Reading


Question: There are 12 datasets examined with code-davinci-002 and 2 datasets have a large accuracy gap. What is the average performance difference of these two datasets while using instruction fine-tuned model?

Your answer should be a float number with 4 decimal places between 0 and 1.


Question: According to the paper that proposed BooookScore, another paper analysed the disadvantage of the only existing public dataset for book-length summarization. In that analysis paper, which book published in the 21st century is among the books where GPT-4 performs the best on name cloze?

Your answer should be a Python string, the title of the book.


Question: Explain the reasoning behind formula (3) in the paper.

Your answer should be a python strings about the concise reasoning behind formula (3) in the paper.

Question: Explain the reasoning behind formula (3) in the paper.

Formula (3) is designed to force the student's predictions of CutMixed spectrograms to be consistent with the CutMixed teacher's predictions of original samples, which is to encourage the model to alleviate excessive temporal dependency and make it less vulnerable to varying contexts while making predictions.


Question: According to Table 2, what is the difference of the percentage of over 80% agreement threshold for both concessive and causal relations between the first and second iteration on the English dataset?

Your answer should be a percentage with two decimal places, indicating the difference in proportion.


Question: What paper is the first to prove finetuned LLM can be a reliable judge?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: What paper is the first to prove finetuned LLM can be a reliable judge?

PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization


Question: What new efficient pre-training method was used in the pre-training process of the model used to compute embedding representations in the Document Similarity section of the paper?

Your answer should be a python strings concisely summarizing the method.

They explored training a system to solve the potentially easier proxy task of predicting only which text as a whole is paired with which image and not the exact words of that text.

Question: What new efficient pre-training method was used in the pre-training process of the model used to compute embedding representations in the Document Similarity section of the paper?


Question: What datasets are used in the paper proposing the baseline for the experiments in this paper?

Your answer should be a python list, each element is the name of the dataset used, e.g.,["dataset1", "dataset2", ...]. YOU MUST USE THE EXACT NAMES FROM THE PDF WITHOUT CHANGING THE CAPITALIZATION.


Question: Which paper first used structural information for coherence modeling?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper first used structural information for coherence modeling?

Modeling Structural Similarities between Documents for Coherence Assessment with Graph Convolutional Networks


Question: Which model achieves the best performance in the experimental results of MQUAKE-T in Figure 2? Considering the GPU VRAM consumption of the model from the previous question, which models from ENN and KE have similar consumption?

Your answer should be a single python list containing two strings, the first element of the list is the model's name, the second element of the name of the model that has similar GPU VRAM consumption of with the first element.


Question: What are the two key technologies used in the process of augmenting trajectory-level data proposed in this paper?

Augment trajectories using a partial noising with forward process and a denoising framework with amplified return guidance.

Question: What are the  two key technologies used in the process of augmenting trajectory-level data proposed in this paper?


Question: Which two independent sources of variance do the models performing sentiment classifications have to cope with?

Your answer should be a python strings of the two independent sources of variance.

cultural expressions of sentiment and errors in automatic translations

Question: Which two independent sources of variance do the models performing sentiment classifications have to cope with?


Question: How to calculate three important parameters that appear in the second part of Figure 2?

Your answer should be a Python list of three string elements, every element is a formula in latex format to calculate a parameter.

Question: How to calculate three important parameters that appear in the second part of Figure 2?


Question: Where can I get the code of GreenKGC method?

Your answer should be the url of the code of GreenKGC method.


Question: The study employs two main methods to analyze linguistic features: LIWC and BERT. What are the advantages of the two methodes respectively?

Your answer should be a python list of two strings.every string describes clearly the advantages of one methode.

Interpretability of LIWC: LIWC categories have clear psychological meanings, making it easier to interpret the analysis results.

Low computational cost of LIWC: LIWC analysis only requires simple word counting, which has a low computational cost.

Ease of use of LIWC: LIWC dictionaries are easy to obtain and use, and the operation is simple.

Contextual understanding of BERT: The BERT model can capture the contextual information of words, thus understanding the meaning of words more accurately.

Stronger feature representation ability of BERT: The BERT model can generate richer feature vectors, capturing more complex linguistic features.

Better prediction performance of BERT: In some tasks, the prediction performance of the BERT model may be better than that of the LIWC model.

Question: The study employs two main methods to analyze linguistic features: LIWC and BERT. What are the advantages of the two methodes respectively?


Question: What is the collection process of the latest mainstream KBQA dataset used in the paper "Augmenting Reasoning Capabilities of LLMs with Graph Structures in Knowledge Base Question Answering"?

Question: What is the collection process of the latest mainstream KBQA dataset used in the paper "Augmenting Reasoning Capabilities of LLMs with Graph Structures in Knowledge Base Question Answering"?

Canonical logical form generation. We leverage the logical form generation algorithm from Su et al.  It first traverses the KB ontology to generate graph-shaped templates that only consist of classes, relations, and functions, and then ground certain nodes to compatible entities to generate logical forms in their meaning representation called graph query.

Canonical question annotation. Each validated canonical logical form is annotated with a canonical question by a graduate student, which is then cross-validated by another student to ensure its fidelity and fluency.

Crowd-powered paraphrasing. We use Amazon Mechanical Turk to crowdsource paraphrases of the canonical question. The crowdsourcing framework with automated quality control mechanisms that contains three task: Paraphrasing, Cross-validation and Entity surface form mining.

Grounding and sampling. We do controlled sampling to generate the final questions: From the pool of logical forms and paraphrases associated with the same canonical logical form, we sample one from each pool at a time to generate a question. Start with uniform weights and each time a logical form or paraphrase is selected, its weight is divided by rho_l and rho_p , respectively. We set rho_l to 2 and rho_p to 10 to enforce more linguistic diversity. Finally, we randomly replace entity surface forms with the ones mined in Task 3 (if there is any).


Question: Which paper studies the concept of enhancing the coverage of a selective prediction system by re-attempting the questions on which it was not sufficiently confident.

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper studies the concept of enhancing the coverage of a selective prediction system by re-attempting the questions on which it was not sufficiently confident.

Post-Abstention: Towards Reliably Re-Attempting the Abstained Instances in QA


Question: To generate NSFW images, which technique is used in the paper titled "Erasing Undesirable Concepts in Diffusion Models with Adversarial Preservation"?Please tell me the full name, not the abbreviation.

Your answer should be a single string of the full name of the technique.


Question: According to Table 1, which baselines this paper used has the highest average score? In which paper is this method proposed? And in which conference was this paper published?

Your answer should be a python list with three items, the first item is the name of baseline reaching the highest average score according to Table 1, the second item is the name of paper where this method proposed, and the third item is the abbreviation of the conference name where this paper was published.

A contrastive framework for neural text generation


Question: Is there a paper that uses Explainable AI techniques to investigate how language models represent the expression of morality?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there a paper that uses Explainable AI techniques to investigate how language models represent the expression of morality?

What does a Text Classifier Learn about Morality? An Explainable Method for Cross-Domain Comparison of Moral Rhetoric


Question: In the main experiment of "AN LLM CAN FOOL ITSELF: A PROMPT-BASED ADVERSARIAL ATTACK", what high-level tasks do the five tasks used belong to, as categorized in the original paper?

Your answer should be a python dictionary with the following format: {'task1': 'task1_category', 'task2': 'task2_category', 'task3': 'task3_category', 'task4': 'task4_category', 'task5': 'task5_category'}. YOU MUST USE THE EXACT NAMES OF THE CATEGORIES IN THE PAPER AND THE ABBREVIATION OF THE TASKS.


Question: In terms of evaluation results on SuperGLUE using RoBERTaBASE, on subtask ReCoRD, which model(s) achieve the overall best results? Additionally, which model(s) perform the best among the two-stage MTL models?

Your answer should be a python dict(without \n) containing two keys "overall best" and "two-stage MTL best", each value of the two keys is a python string list. e.g.{"overall best":["modelname1"],"two-stage MTL best":["modelname2","modelname3"]}


Question: Which paper contains quantitative results demonstrating taking VQ tokens as inputs is inferior to pixel images for dense recognition tasks?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper contains quantitative results demonstrating taking VQ tokens as inputs is inferior to pixel images for dense recognition tasks?

ADDP: Learning General Representations for Image Recognition and Generation with Alternating Denoising Diffusion Process


Question: For the PLM with the lowest number of synset candidates under synset retrieval in the paper "Predicate Sense Disambiguation for UMR Annotation of Latin: Challenges and Insights", what negative impact will it have if it randomly blocks a certain percentage of input tokens during pre-training?

It creates a mismatch between pre-training and fine-tuning.

Question: For the PLM with the lowest number of synset candidates under synset retrieval in the paper "Predicate Sense Disambiguation for UMR Annotation of Latin: Challenges and Insights", what negative impact will it have if it randomly blocks a certain percentage of input tokens during pre-training?


Question: In the paper that introduced the latest dataset used by RetinaQA, what innovation related to F1 was also applied in the evaluation of RetinaQA?

Your answer should be a paragraph, describing the innovation on F1.

In regular answer evaluation (R), we compare the predicted answer (which could be NA) with the gold answer in the modified KB, as usual. Specifically for unanswerability, we also consider lenient answer evaluation (L), where we account for the gold answer in the original (ideal) KB as well, and also give credit to models which are able to recover this answer, perhaps via inference.

Question: What innovation related to F1 was also applied in the evaluation of RetinaQA?


Question: To enhance scalability and effectiveness, which Ensemble Model does this paper("MINERS : Multilingual Language Models as Semantic Retrievers") choose? In the source paper, which datasets is this framework tested on?

Your answer should be a python list like ["string1", ["string2", "string3", ...]]. The first element should be a string, representing the name of the Ensemble Model. The second element should be a list of strings, representing the names of the datasets tested on.For these names, abbrievation is enough.


Question: In the framework used in the paper "ADAPTIVE DEEP SPIKING NEURAL NETWORK WITH GLOBAL-LOCAL LEARNING VIA BALANCED EXCITATORY AND INHIBITORY MECHANISM", what are the ANN2SNN conversion functions?

Your answer should be a python list of strings, e.g., ['function1', 'function2']. YOU MUST USE THE EXACT FUNCTION NAMES AS IN THE PAPER.


Question: What work proposes a model to learn a latent regular cell complex from data?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: What work proposes a model to learn a latent regular cell complex from data?

From Latent Graph to Latent Topology Inference: Differentiable Cell Complex Module


Question: What paper considers sensitive data issue when prompting large language model APIs?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: What paper considers sensitive data issue when prompting large language model APIs?

Enhancing Small Medical Learners with Privacy-preserving Contextual Prompting


Question: Is there any paper that aligns speech and text embeddings better than CTC training?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any paper that aligns speech and text embeddings better than CTC training?

WACO: Word-Aligned Contrastive Learning for Speech Translation


Question: For the specific dataset where CLiCoTEA does not outperform all models, in which languages does this occur?

Your answer should be a Python list of string elements, every element is the abbreviation of a langugage mentioned in the paper, e.g. ["AR", "ES", "FR", ...].


Question: According to the experiment result, How much better does the GPT-2 model perform on Task A compared to the CNN-BiLSTM model in terms of F1-score?

Your answer should be a single python float, rounded to 2 decimal places.


Question: The anchor paper mentioned a service which uses the methodology proposed in this paper. What is the name of the service? On which page of the paper can I find graphic information on this service?

Your answer should be a python list with 2 elements, the first element being the name of the service, and the second element being the page number. The first element should be a string and the second element should be an integer. Use the exact name of the service from the paper without changing CAPITALIZATION.


Question: According to the paper, which dataset also use a retrieval-based system for relevant files selecting?In that dataset, how many lines do the codebase contain on average?

Your answer should be a python list of 2 elements, the first is the name of the dataset, and the second is the number of lines in thousands, rounding to the nearest integer. e.g. ["MMMU", 9]


Question: Which paper is among the earliest to train on extensive collection of signing video and subtitle pairs available from online platforms?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper is among the earliest to train on extensive collection of signing video and subtitle pairs available from online platforms?

Gloss-Free End-to-End Sign Language Translation


Question: In the paper that proposed the dataset used by ReFIR for the evaluation of the second experimental setup, which image restoration method was also proposed? Additionally, what metric applied to evaluate that method was not applied in ReFIR?

Your answer should be a Python list of 2 strings, the name of the method and the name of the metric.


Question: In the experiment result of the paper named "EIT: Enhanced Interactive Transformer", which model gets the highest RG-L on the summarization task? For the source paper of this model, what issue about Neural network-based methods for abstractive summarization does this paper want to address?

Your answer should be a single python list, the first element is the string of the model name, the second element is the string of the issue.e.g.["EIT","Neural network-based methods for abstractive summarization often neglect..."].

Neural network-based methods for abstractive summarization produce outputs that are more fluent than other techniques, but which can be poor at content selection.

Question: For the source paper of this model, what issue about Neural network-based methods for abstractive summarization does this paper want to address?


Question: Among the datasets proposed in the Introduction section of the paper "Towards Robust Temporal Reasoning of Large Language Models via a Multi-Hop QA Dataset and Pseudo-Instruction Tuning", which one has the least Q-A pairs?

Your answer should be a single string, the name of the dataset as given in the Introduction section.


Question: According to MSAD paper's review, which is the largest VAD benchmark datasets with multiple domain before? Additionally, in that dataset, which are the two scenarios with the most videos?

Your answer should be a Python list of 2 elements, the first is the name of the dataset, the second is a list of 2 strings, the full name of the scenarios.


Question: How were the data samples selected for the dataset used in the SFT of the paper "FACTALIGN: Long-form Factuality Alignment of Large Language Models"?

Question: How were the data samples selected for the dataset used in the SFT of the paper "FACTALIGN: Long-form Factuality Alignment of Large Language Models"?

A score-first, diverse-aware data selection strategy, denoted as \pi_{DEITA} were proposed to combine complexity, quality, and diversity measures.

The strategy incorporates a new evol score s that combines complexity and quality by multiplying the complexity score c with the quality score q as s := c * q. For multi-turn dialogues, we calculate this score for each turn, summing them to obtain the final score for the entire conversation. Next, we sort all samples in X using s, yielding the sorted pool X^* = (x^*_1, x^*_2, ..., x^*_n), where x^*_0 represents the sample with the highest evol score. Beginning with S^1_{\pi_{DEITA}} = (x^*_0), we iteratively select data from X^*/S^1_{\pi_{DEITA}} one by one, following the REPR FILTER strategy, and discard redundant samples for S^1_{\pi_{DEITA}}.


Question: Which paper first derived online occupany estimation technique to get sqrt(T) bound for reinforcement learning in adversarial linear MDP?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper first derived online occupany estimation technique to get sqrt(T) bound for reinforcement learning in adversarial linear MDP?

Towards Optimal Regret in Adversarial Linear MDPs with Bandit Feedback


Question: How do the authors split the dataset for the experiments?

For the empirical modeling analysis and performance benchmarking, we randomly split the dataset into 3 sets: train (70%), dev (5%), and test (25%) sets, while ensuring both domains (fashion and furniture) have the same split distributions.

Question: How do the authors split the dataset for the experiments?


Question: On how many datasets was COSA evaluated for its performance in Object Discovery and Composition?


Question: Is there any paper that tries to investigate LLMs' capabilities in solving elliptical constructions by using a test-dataset based on the psycolinguistic notion of Thematic Fit?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any paper that tries to investigate LLMs' capabilities in solving elliptical constructions by using a test-dataset based on the psycolinguistic notion of Thematic Fit?

We Understand Elliptical Sentences, and Language Models Should Too: A New Dataset for Studying Ellipsis and its Interaction with Thematic Fit


Question: Among StudentEval, HumanEval and MBPP, which one has the most test cases per problem?

Your answer should be a string, indicating the name of the dataset.


Question: Both two papers, "Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization" and "Composing Parameter-Efficient Modules with Arithmetic Operation" focus on arithmetics on parameter-efficient modules. Which setting did the first paper mainly focus on, while the second paper didn't?

Your answer should be brief text on the setting mainly focused on only by the first paper.

Cross-lingual transfer on summarization tasks.

Question: Both two papers, "Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization" and "Composing Parameter-Efficient Modules with Arithmetic Operation" focus on arithmetics on parameter-efficient modules. Which setting did the first paper mainly focus on, while the second paper didn't?


Question: Between the two agent-based method that are explicitly introduced in Related Work section of LLM-DP, which one is not applied as a baseline? Why not?

Your answer should be a Python list of two strings, the first is the name of the method, the second is the reason why it's not applied.

Voyager is an agent specialized in Minecraft, who is not capable of datasets like Alfworld.

Question: Why Voyager is not applied as a baseline in the LLM-DP paper?


Question: On the HEML sentence level with AUC metric, which baseline outperforms MIND on specific conditons?Is it the best variant according to the paper that proposed that baseline?

Your answer should be a Python list of two strings. The first string is the name of the baseline (with variant) that outperforms MIND, as proposed in the anchor PDF. The second string is either `true` or `false`.


Question: Is there a paper that utilizes the characteristics of human evolutionary knowledge to guide language models in generating scientific ideas?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there a paper that utilizes the characteristics of human evolutionary knowledge to guide language models in generating scientific ideas?

Exploring and Verbalizing Academic Ideas by Concept Co-occurrence


Question: Are there any large-scale and open-source text simplification datasets dealing with long passages?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Are there any large-scale and open-source text simplification datasets dealing with long passages?

SWIPE: A Dataset for Document-Level Simplification of Wikipedia Pages


Question: What is the relationship between the two papers in terms of datasets?

The dataset of paper "What does Parameter-free Probing Really Uncover?" is based on the dataset of paper "Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT".

Question: What is the relationship between the two paper (in anchor_pdf) in terms of datasets?


Question: Is there any paper that studies a teacher AI inferring mental states of a student role in a role-playing game setup using reinforcement learning?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Is there any paper that studies a teacher AI inferring mental states of a student role in a role-playing game setup using reinforcement learning?

I Cast Detect Thoughts: Learning to Converse and Guide with Intents and Theory-of-Mind in Dungeons and Dragons


Question: According to Table 1, on how many mathematical task test sets was Rho-1 tested?


Question: According to the paper "Full-Atom Peptide Design with Geometric Latent Diffusion", which model has the lowest success rate on PepBDB? In the paper that proposes the model, how is the overview of the proposed method given in form?

Your answer should be a Python list of 2 elements, the first is the name of the model, the second is a Python list of formula in LaTeX format. e.g. ["method", ["formula_1", ..., "formula_n"]]

\mathcal{G}_A = \text{SI}(\{s_i\}_{i \in \mathcal{V}_A, i \notin \mathcal{V}_P}), i \in \mathcal{V}_A

\mathcal{G}_S = \text{SP}(\mathcal{G}_E, \mathcal{G}_P)

h_i, X_i = \text{AME}(\mathcal{G}_E, \mathcal{G}_S, \mathcal{G}_A), i \in \mathcal{V}_E \cup \mathcal{V}_S \cup \mathcal{V}_A

p_i = \text{Predict}(h_i), i \in \mathcal{V}_P

\tilde{X}_i = \text{Dock}(\mathcal{G}_A, \mathcal{G}_S), i \in \mathcal{V}_A

Question: In the paper that proposes the model, how is the overview of the proposed method given in form?


Question: Which paper proposes the two-stage training method, i.e., task-specific fine-tuning and cross-domain pre-training, to train an open-domain dialogue evaluator using the self-collected dataset.

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper proposes the two-stage training method, i.e., task-specific fine-tuning and cross-domain pre-training, to train an open-domain dialogue evaluator using the self-collected dataset.

RADE: Reference-Assisted Dialogue Evaluation for Open-Domain Dialogue


Question: Which paper about parameter-efficient finetuning first proposes to feed the pretrained weight instead of the activation to an adapter?

Your answer should be the title of the paper WITHOUT ANY EXPLANATION.

Question: Which paper about parameter-efficient finetuning first proposes to feed the pretrained weight instead of the activation to an adapter?

Parameter-Efficient Fine-Tuning without Introducing New Latency