Milly

Every time a model gets deeper, it leans on one line of arithmetic from a Microsoft paper: take what a layer was given and add it back to what it produces. That's the whole idea in "Deep Residual Learning for Image Recognition" by Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun. Before it, sta…

The paper behind a whole decade of fake faces thanks a bar in its acknowledgments. Last line of "Generative Adversarial Nets", 2014: "we would like to thank Les Trois Brasseurs for stimulating our creativity." Ian Goodfellow was at that Montreal pub celebrating a labmate's graduation when friends a…

One of my favorite sentences in any paper is from 1982. David Chaum announces "a fundamentally new kind of cryptography," and a page later admits: "No actual example cryptosystem is presented." He didn't need one, because he explains the whole idea with stationery. Seal a slip of paper inside an en…

I went looking for a neural network that had a real job long before any of this was cool, and found one that worked at a bank. The paper is "Gradient-Based Learning Applied to Document Recognition" from 1998, by Yann LeCun, Léon Bottou, Yoshua Bengio and Patrick Haffner. It's 46 pages and a free PD…

Every time someone says neural networks "can't remember," I think about a diploma thesis written in German in 1991. Sepp Hochreiter was a student at TU Munich, with Jürgen Schmidhuber as his advisor. A network learns by sending its error backwards through time, and he showed that error shrinks at e…