The current AI boom has more to do with NVIDIA, and the popularity of computer gaming giving us GPU compute, than who was using neural networks back in 1990's.
More specifically, it was really AlexNet, the 2012 ImageNet entry, running on two NVIDIA GTX 580's, that highlighted the practicality and utility of running large scale neural nets on affordable hardware. CUDA had been released in 2006, but cuDNN (the CUDA library for neural nets) didn't come out until 2014 - after AlexNet had already kickstarted the demand.
What followed from AlexNet was a few years of intense competition on the ImageNet benchmark, and larger and larger/deeper neural nets (CNNs), which gave rise to a lot of the algorithms and concepts still used today such as residual connections (originally from ResNet), ADAM (training algorithm), ReLU/etc, normalization, dropout, etc... all the fundamentals that made building large neural nets possible.
Schmidhuber's continual reminding everyone that he was working on neural nets back in the 1990s is beyond tiresome. Yes, he should have been recognized alongside Hinton/Bengio/LeCun as one of the pioneers, but time for him to get over it.
TU Munich and Nipkow, Makarius et.al. are also at the center of the influential Isabelle theorem prover. TU Munich is cool :-)
jcattle
There's this crowd on HN which is very vocal against academia. From what I've seen, the main points are that academia isn't efficient, most of the science coming out of academia is useless and that the whole system is just a waste of taxpayers money. Instead, what is often argued, all good research is done in private labs. Then pointing to SpaceX, Moderna, OpenAI, Google, etc.
And while it is very true that often the research coming out of Academia is useless, what is always neglected are the roots of the research done in private labs.
When Jürgen Schmidhuber and team published their work on Neural Nets back in 1991 it was also useless. Unless you had a supercomputer and very, very deep pockets you were not going to do anything with what came out of their lab.
But still, 30 years later here we are, standing on top of the shoulders of this useless research.
show comments
trashburger
This article, too, was originally discovered by Jürgen Schmidhuber in 1991!
trilogic
> it is easy to forget that the foundations of this trillion-dollar industry were laid down over 30 years ago in Munich
Yes is very easy to forget, cause the trillion is not being made in Europe. If it was really conceived in Munich (like the maps that got stolen also), it show how incompetent is Europe to keep it´s technology and protect European companies.
It is painful to read this article.
show comments
cold_harbor
worth separating: LSTM (Hochreiter & Schmidhuber 1997) is ironclad and widely cited. the transformer attention priority claims are far shakier. conflating them is how Schmidhuber undermines himself
This sort of seems like a pattern in CS - someone creates something and then it blows up 20 or 30 years later when the world is ready for it.
gillesjacobs
Which work has more value: the abstract description of a catalogue of potential model architectures or their validated application trained on real data?
In the Schmidhuber case their is 20 years and a chain of countless other works in between the two.
throwa356262
Hot take:
The real root of the current AI boom is a master thesis from university of Toronto.
The thesis demonstrated that neural networks much longer than before could be trained by simply having a random fraction of the neurons excluded during forward and back propagation.
That's how we got practical deep neural networks. Without that we would still be in AI winter.
jacknews
Surely the roots, if we skip over the early preceptron work', are in backpropagation and Hinton, and the work going on at Edinburgh and elsewhere in the 80s.
Indeed I remember buying a set of three conference-papers-as-books around that time, titled Artificial Neural Networks .. proceedings of the whatever the conference was.
No doubt Schmidhuber made important contributions, but I see him pop up claiming to be the 'root' of it all every couple of years.
show comments
jongjong
It's crazy to think that if Elon Musk hadn't mentioned Schmidhuber, most people would have no idea.
It's nauseating how all the researchers who happened to work for big tech got tons of media coverage but Schmidhuber and his team were getting zero coverage yet they made massive contributions. I bet there are many others not mentioned.
Nobody even knows about Frank Rosenblatt. It's insane how distorted our perception of innovation is.
Even science has been corrupted. It makes one doubt every story we're told about who invented what.
show comments
storus
Instead of focusing on the future, EU is busy rewriting history to please some eccentric researcher that claims he invented it all.
show comments
greenavocado
Schmidhuber will NEVER stop trying to aggressively preserve his relevance and its endlessly amusing. Good for him.
sagex
I believe invention of Transformers and especially Attention mechanism do have influence from past research but its not definitely only the Schmidhuber's work. Said that, if we remove the papers mentioned by Schmidhuber from history, I am quite certain that there will be no influence in the discovery of Transformers, hence his works can not be the root. He has to grow up and accept that work and equations can appear similar, looking at inverse squared law and saying Newton stole that from someone is being dishonest.
The current AI boom has more to do with NVIDIA, and the popularity of computer gaming giving us GPU compute, than who was using neural networks back in 1990's.
More specifically, it was really AlexNet, the 2012 ImageNet entry, running on two NVIDIA GTX 580's, that highlighted the practicality and utility of running large scale neural nets on affordable hardware. CUDA had been released in 2006, but cuDNN (the CUDA library for neural nets) didn't come out until 2014 - after AlexNet had already kickstarted the demand.
What followed from AlexNet was a few years of intense competition on the ImageNet benchmark, and larger and larger/deeper neural nets (CNNs), which gave rise to a lot of the algorithms and concepts still used today such as residual connections (originally from ResNet), ADAM (training algorithm), ReLU/etc, normalization, dropout, etc... all the fundamentals that made building large neural nets possible.
Schmidhuber's continual reminding everyone that he was working on neural nets back in the 1990s is beyond tiresome. Yes, he should have been recognized alongside Hinton/Bengio/LeCun as one of the pioneers, but time for him to get over it.
Also see Schmidhuber's take on the Hinton + Hopfield Nobel prize: https://people.idsia.ch/~juergen/physics-nobel-2024-plagiari...
TU Munich and Nipkow, Makarius et.al. are also at the center of the influential Isabelle theorem prover. TU Munich is cool :-)
There's this crowd on HN which is very vocal against academia. From what I've seen, the main points are that academia isn't efficient, most of the science coming out of academia is useless and that the whole system is just a waste of taxpayers money. Instead, what is often argued, all good research is done in private labs. Then pointing to SpaceX, Moderna, OpenAI, Google, etc.
And while it is very true that often the research coming out of Academia is useless, what is always neglected are the roots of the research done in private labs.
When Jürgen Schmidhuber and team published their work on Neural Nets back in 1991 it was also useless. Unless you had a supercomputer and very, very deep pockets you were not going to do anything with what came out of their lab.
But still, 30 years later here we are, standing on top of the shoulders of this useless research.
This article, too, was originally discovered by Jürgen Schmidhuber in 1991!
> it is easy to forget that the foundations of this trillion-dollar industry were laid down over 30 years ago in Munich
Yes is very easy to forget, cause the trillion is not being made in Europe. If it was really conceived in Munich (like the maps that got stolen also), it show how incompetent is Europe to keep it´s technology and protect European companies.
It is painful to read this article.
worth separating: LSTM (Hochreiter & Schmidhuber 1997) is ironclad and widely cited. the transformer attention priority claims are far shakier. conflating them is how Schmidhuber undermines himself
https://en.wikipedia.org/wiki/J%C3%BCrgen_Schmidhuber
This sort of seems like a pattern in CS - someone creates something and then it blows up 20 or 30 years later when the world is ready for it.
Which work has more value: the abstract description of a catalogue of potential model architectures or their validated application trained on real data?
In the Schmidhuber case their is 20 years and a chain of countless other works in between the two.
Hot take:
The real root of the current AI boom is a master thesis from university of Toronto.
The thesis demonstrated that neural networks much longer than before could be trained by simply having a random fraction of the neurons excluded during forward and back propagation.
That's how we got practical deep neural networks. Without that we would still be in AI winter.
Surely the roots, if we skip over the early preceptron work', are in backpropagation and Hinton, and the work going on at Edinburgh and elsewhere in the 80s.
Indeed I remember buying a set of three conference-papers-as-books around that time, titled Artificial Neural Networks .. proceedings of the whatever the conference was.
No doubt Schmidhuber made important contributions, but I see him pop up claiming to be the 'root' of it all every couple of years.
It's crazy to think that if Elon Musk hadn't mentioned Schmidhuber, most people would have no idea.
It's nauseating how all the researchers who happened to work for big tech got tons of media coverage but Schmidhuber and his team were getting zero coverage yet they made massive contributions. I bet there are many others not mentioned.
Nobody even knows about Frank Rosenblatt. It's insane how distorted our perception of innovation is.
Even science has been corrupted. It makes one doubt every story we're told about who invented what.
Instead of focusing on the future, EU is busy rewriting history to please some eccentric researcher that claims he invented it all.
Schmidhuber will NEVER stop trying to aggressively preserve his relevance and its endlessly amusing. Good for him.
I believe invention of Transformers and especially Attention mechanism do have influence from past research but its not definitely only the Schmidhuber's work. Said that, if we remove the papers mentioned by Schmidhuber from history, I am quite certain that there will be no influence in the discovery of Transformers, hence his works can not be the root. He has to grow up and accept that work and equations can appear similar, looking at inverse squared law and saying Newton stole that from someone is being dishonest.