Worthwhile Research for building SOTA LLMs (Jan 2024 Recap)

5 directions we think will stand the Test of Time: Search, Synthetic Data, Subquadratic LLMs, MoEs and Merges... Also: our usual highest-signal recap of top items for the AI Engineer from Jan 2024!