Recent Developments in Deep Learning

There's a great Google Tech Talk by Geoff Hinton that serves as a slightly technical introduction to Deep Learning if anyone's interested: ▶ this video

Have you seen the online courses? link (From one of the authors of this paper!) link link Prof. Hinton's videos are very watchable: link ▶ this video

There's a very good Google Tech Talk by Geoff Hinton (who has worked closely with Dahl on a lot of this research and developed some of the key algorithms in this field) that explains how to build deep belief networks using layers of RBMs: link That video focuses on handwritten digit recognition, but it's great for understanding the basics. There's a second Google Tech Talk video from a few years later that talks directly about phoneme recognition as well: ▶ this video

That must have been a long time back. Having lived through the time when web pages were served through CGI and mobile phones only existed in movies, when SVMs where the new hotness in ML and people would write about how weird NNs were, I feel like I've seen a lot more concrete progress in the last few decades than this year. This year honestly feels quite stagnant. LLMs are literally technology that can only reproduce the past. They're cool, but they were way cooler 4 years ago. We've taken big ideas like "agents" and "reinforcement learning" and basically stripped them of all meaning in order to claim progress. I mean, do you remember Geoffrey Hinton's RBM talk at Google in 2010? [0] That was absolutely insane for anyone keeping up with that field. By the mid-twenty teens RBMs were already outdated. I remember when everyone was implementing flavors of RNNs and LSTMs. Karpathy's character 2015 RNN project was insane [1]. This comment makes me wonder if part of the hype around LLMs is just that a lot of software people simply weren't paying attention to the absolutely mind-blowing progress we've seen in this field for the last 20 years. But even ignoring ML, the world's of web development and mobile application development have gone through incredible progress over the last decade and a half. I remember a time when JavaScript books would have a section warning that you should never use JS for anything critical to the application. Then there's the work in theorem provers over t

I think the top priority should be how a hierarchy of weak AI agents can form strong AI with the ability to rewrite its own notions and not become unstable. I've seen very little about this, except for a video where (as I recall) a multilayer neural network can be trained while holding certain connections at a constant value. That way the network can be trained to do different tasks and then put back into those modes by setting those training connections to various values during runtime: ▶ this video Eventually another neural net could be trained to know which values to hold to get different behaviors. It sounds like one of the goals is to reuse logic for similar tasks, and eventually be able to adapt in novel situations like our brains do. I believe there was also a bit about how time figured into all of this, but I haven’t watched it for a while. Anyway, I’m thinking that a lot of what we’ve learned about version control systems like git could be used to let an AI brainstorm in a sandbox and replace its own code as it improves, and always be able to revert or merge if needed. Throw in some genetic programming and decent lisp machines with 1000 cores and we could really have something. When I think about this sort of “real work”, as opposed to the daily minutia our industry gets mired in, it makes me sad inside.

Something really interesting must be happening with AI at Google, because in the past few months both Ray Kurzweil (the best-known proponent of the singularity) and Geoff Hinton (the crazy-talented individual who invented deep-belief networks using interconnected "restricted Boltzman machines"[1]) have joined the company. -- [1] For an overview of deep belief networks, see these videos: link , ▶ this video , and link

Hinton's team (SuperVision) uses an interesting 'dropout' technique. He gave a Google Tech Talk on this back in June. link And an older talk that covers some of what a deep convolutional net is: ▶ this video

If you enjoyed Geoff Hinton's talk you will probably find the theano 'deep learning' library to be of use. Still undergoing quite a lot of iteration but powerful and you get to run your stuff on the GPU for added fun. link . Incidentally Hinton gave another google tech talk in 2010 ▶ this video .

I would remove one maybe both muches. I agree that in general SVMs are better but they are also every bit as black box. Deep belief networks though, are on another level. literally, svms are depth 2, dbn's are depth unbounded, i think. ▶ this video