# Research Papers

***

### 2017 <a href="#id-2017" id="id-2017"></a>

* [ ] [Attention is All You Need](https://arxiv.org/abs/1706.03762)
* [ ] [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://arxiv.org/abs/1701.06538)
* [ ] [Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347)
* [ ] [Deep reinforcement learning from human preferences](https://arxiv.org/abs/1706.03741)

***

### 2018 <a href="#id-2018" id="id-2018"></a>

* [ ] [Deep contextualized word representations](https://arxiv.org/abs/1802.05365)
* [ ] [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805)
* [ ] [Improving Language Understanding by Generative Pre-Training](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf)
* [ ] [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://arxiv.org/abs/1808.06226)
* [ ] [Universal Language Model Fine-tuning for Text Classification](https://arxiv.org/abs/1801.06146)

***

### 2019 <a href="#id-2019" id="id-2019"></a>

* [ ] [Language Models are Unsupervised Multitask Learners](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)
* [ ] [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://arxiv.org/abs/1907.11692)
* [ ] [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://arxiv.org/abs/1910.01108)
* [ ] [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://arxiv.org/abs/1910.13461)
* [ ] [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://arxiv.org/abs/1906.08237)
* [ ] [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/abs/1909.08053)
* [ ] [Generating Long Sequences with Sparse Transformers](https://arxiv.org/abs/1904.10509)

***

### 2020 <a href="#id-2020" id="id-2020"></a>

* [ ] [Reformer: The Efficient Transformer](https://arxiv.org/abs/2001.04451)
* [ ] [Longformer: The Long-Document Transformer](https://arxiv.org/abs/2004.05150)
* [ ] [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://arxiv.org/abs/2006.16668)
* [ ] [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401)
* [ ] [Big Bird: Transformers for Longer Sequences](https://arxiv.org/abs/2007.14062)
* [ ] [Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)
* [ ] [Rethinking Attention with Performers](https://arxiv.org/abs/2009.14794v4)
* [ ] [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/abs/1910.10683)
* [ ] [Measuring Massive Multitask Language Understanding](https://arxiv.org/abs/2009.03300)
* [ ] [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://arxiv.org/abs/1910.02054)
* [ ] [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators](https://arxiv.org/abs/2003.10555)
* [ ] [Scaling Laws for Neural Language Models](https://arxiv.org/abs/2001.08361)

***

### 2021 <a href="#id-2021" id="id-2021"></a>

* [ ] [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://arxiv.org/abs/2104.09864)
* [ ] [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://arxiv.org/abs/2104.04473)
* [ ] [Transcending Scaling Laws with 0.1% Extra Compute](https://arxiv.org/abs/2210.11399)
* [ ] [Improving language models by retrieving from trillions of tokens](https://arxiv.org/abs/2112.04426)
* [ ] [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685)
* [ ] [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://arxiv.org/abs/2112.11446)
* [ ] [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://arxiv.org/abs/2201.11990)

***

### 2022 <a href="#id-2022" id="id-2022"></a>

* [ ] [EFFICIENTLY SCALING TRANSFORMER INFERENCE](https://arxiv.org/pdf/2211.05102)
* [ ] [Fast Inference from Transformers via Speculative Decoding](https://arxiv.org/abs/2211.17192)
* [ ] [Training Compute-Optimal Large Language Models](https://arxiv.org/abs/2203.15556)
* [ ] [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/abs/2201.11903)
* [ ] [Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155)
* [ ] [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://arxiv.org/abs/2211.05100)
* [ ] [Emergent Abilities of Large Language Models](https://arxiv.org/abs/2206.07682)
* [ ] [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://arxiv.org/abs/2205.14135)
* [ ] [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://arxiv.org/abs/2305.13245)
* [ ] [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://arxiv.org/abs/2108.12409)
* [ ] [DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale](https://arxiv.org/abs/2207.00032)
* [ ] [Red Teaming Language Models with Language Models](https://arxiv.org/abs/2202.03286)
* [ ] [Holistic Evaluation of Language Models](https://arxiv.org/abs/2211.09110)
* [ ] [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://arxiv.org/abs/2210.17323)
* [ ] [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://arxiv.org/abs/2206.04615)

***

### 2023 <a href="#id-2023" id="id-2023"></a>

* [ ] [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://arxiv.org/abs/2305.16291)
* [ ] [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://arxiv.org/abs/2307.15043)
* [ ] [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://arxiv.org/abs/2305.10601)
* [ ] [WizardLM: Empowering Large Language Models to Follow Complex Instructions](https://arxiv.org/abs/2304.12244)
* [ ] [DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales](https://arxiv.org/abs/2308.01320)
* [ ] [GPT-4 Technical Report](https://arxiv.org/abs/2303.08774)
* [ ] [Mistral 7B](https://arxiv.org/abs/2310.06825)
* [ ] [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://arxiv.org/abs/2305.18290)
* [ ] [PaLM 2 Technical Report](https://arxiv.org/abs/2305.10403)
* [ ] [LIMA: Less Is More for Alignment](https://arxiv.org/abs/2305.11206)
* [ ] [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/abs/2312.00752)
* [ ] [Visual Instruction Tuning](https://arxiv.org/abs/2304.08485)
* [ ] [Textbooks Are All You Need II: phi-1.5 technical report](https://arxiv.org/abs/2309.05463)
* [ ] [Qwen Technical Report](https://arxiv.org/abs/2309.16609)
* [ ] [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://arxiv.org/abs/2308.12966)
* [ ] [LLaMA: Open and Efficient Foundation Language Models](https://arxiv.org/abs/2302.13971)
* [ ] [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://arxiv.org/abs/2307.09288)
* [ ] [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180)
* [ ] [QLoRA: Efficient Finetuning of Quantized LLMs](https://arxiv.org/abs/2305.14314)
* [ ] [Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment](https://arxiv.org/abs/2312.12148)
* [ ] [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://arxiv.org/abs/2307.08691)
* [ ] [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://arxiv.org/abs/2306.00978)
* [ ] [Generative Agents: Interactive Simulacra of Human Behavior](https://arxiv.org/abs/2304.03442)
