notacompany
Subscribe
← All tags

AI bookmarks

154 bookmarks tagged AI — raises, dev tools, and reading, newest first.

Get it by email

One brief each weekday morning. One-click unsubscribe.

15 Jul

14 Jul

12 Jul

10 Jul

9 Jul

8 Jul

3 Jul

2 Jul

1 Jul

29 Jun

24 Jun

21 Jun

20 Jun

9 Jun

8 Jun

30 May

29 May

27 May

Interesting · arxiv.org

Language Models Need Sleep

Transformer-based large language models are increasingly used for long-horizon tasks; however, their attention mechanism scales poorly with context length. To handle this, we study a sleep-like consolidation mechanism in which a model periodically converts recent context into persistent fast weights before clearing its key-value cache. During sleep, the model performs $N$ offline recurrent passes over the accumulated context and updates the fast weights in its state-space model (SSM) blocks through a learned local rule. During inference, this shifts extra computation to sleep while preserving the latency of wake-time prediction. We test our method on controlled synthetic tasks, including cellular automata and multi-hop graph retrieval, as well as a realistic math reasoning task, on which a regular transformer as well as SSM-attention hybrid models fail. We then show that increasing sleep duration $N$ for our models improves performance, with the largest gains on examples that require deeper reasoning.

AISleep

26 May

22 May

Get this by email

The same feed, once a day, in your inbox. One-click unsubscribe.

Prefer a reader? Subscribe by RSS instead, or browse the past issues.

Where to next

Curious who picks these?

Meet the peopleGet them by email