Research Papers

A curated collection of research papers that I find interesting, have read, or plan to read in the future.

---

2017

• Attention is All You Need

• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

• Proximal Policy Optimization Algorithms

• Deep reinforcement learning from human preferences

---

2018

• Deep contextualized word representations

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

• Improving Language Understanding by Generative Pre-Training

• SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

• Universal Language Model Fine-tuning for Text Classification

---

2019

• Language Models are Unsupervised Multitask Learners

• RoBERTa: A Robustly Optimized BERT Pretraining Approach

• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

• BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

• XLNet: Generalized Autoregressive Pretraining for Language Understanding

• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

• Generating Long Sequences with Sparse Transformers

---

2020

• Reformer: The Efficient Transformer

• Longformer: The Long-Document Transformer

• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

• Big Bird: Transformers for Longer Sequences

• Language Models are Few-Shot Learners

• Rethinking Attention with Performers

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

• Measuring Massive Multitask Language Understanding

• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

• ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

• Scaling Laws for Neural Language Models

---

2021

• RoFormer: Enhanced Transformer with Rotary Position Embedding

• Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM

• Transcending Scaling Laws with 0.1% Extra Compute

• Improving language models by retrieving from trillions of tokens

• LoRA: Low-Rank Adaptation of Large Language Models

• Scaling Language Models: Methods, Analysis & Insights from Training Gopher

• Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

---

2022

• EFFICIENTLY SCALING TRANSFORMER INFERENCE

• Fast Inference from Transformers via Speculative Decoding

• Training Compute-Optimal Large Language Models

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

• Training language models to follow instructions with human feedback

• BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

• Emergent Abilities of Large Language Models

• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

• DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale

• Red Teaming Language Models with Language Models

• Holistic Evaluation of Language Models

• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

• Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

---

2023

• Voyager: An Open-Ended Embodied Agent with Large Language Models

• Universal and Transferable Adversarial Attacks on Aligned Language Models

• Tree of Thoughts: Deliberate Problem Solving with Large Language Models

• WizardLM: Empowering Large Language Models to Follow Complex Instructions

• DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales

• GPT-4 Technical Report

• Mistral 7B

• Direct Preference Optimization: Your Language Model is Secretly a Reward Model

• PaLM 2 Technical Report

• LIMA: Less Is More for Alignment

• Mamba: Linear-Time Sequence Modeling with Selective State Spaces

• Visual Instruction Tuning

• Textbooks Are All You Need II: phi-1.5 technical report

• Qwen Technical Report

• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

• LLaMA: Open and Efficient Foundation Language Models

• Llama 2: Open Foundation and Fine-Tuned Chat Models

• Efficient Memory Management for Large Language Model Serving with PagedAttention

• QLoRA: Efficient Finetuning of Quantized LLMs

• Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment

• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

• Generative Agents: Interactive Simulacra of Human Behavior