The required compute seems a bit high: "We trained all CIFAR-10 models on 1xB200 GPU, and all ImageNet 64×64 models on 8xB200 GPUs. The largest CIFAR-10 model uses 20 B200 hours to train, and the largest ImageNet 64×64 model uses 640 B200 hours"
20 B200 hours for CIFAR-10 seems like a lot...
andybak
When I first learned about computer science at the age of 11 or so (and in 1982 or so) the first page of the text book put digital and analogue computers on what seemed to be an equal footing. And then proceeded to ignore the latter for the rest of the book. Apart from a few notable exceptions ( https://en.wikipedia.org/wiki/Phillips_Machine ) I've often wondered about analogue computing.
show comments
tugdual
I love this ! Used to work at Rain AI on training neural networks in unconventional hardware - people often that computers don't necessarily have to be electronic digital - there is a whole domain dedicated to creating machines that can apply certain mathematical operations faster or more efficiently than their electronic counter parts. I created this site to try create a classification of that space:
Really interesting - if I understood the article correctly, they're simulating this on conventional hardware, so in order to get the proposed benefits, it would need to be implemented in some other electronic medium.
vessenes
Very cool. I’m reminded of Wolfram’s pitch that neural nets are a search through the very broad computational complexity of the function space they describe; he did a little work to show that you could find similar behavior in other function spaces. These oscillators are yet again a different function space, and its cool they can be harnessed in this way.
The question of what physical / electronic phenomena is the most efficient yet large enough function space to be used for inference is a really good one to think about. I have no suggestions.
WhitneyLand
It’s not clear to me how this would ever be practical since it seems dependent on n^2 scaling.
You’ve got to wonder when you have an image generation demo why would you possibly have 64 x 64 pixel output as your demo?
If I’m understanding this properly to generate a 4K image, you need like 5 trillion point to point connections on the chip. Even if power use from the oscillators is zero that’s going to be an issue.
show comments
ainch
This method is cool and the post explains it well. It would, however, be good to get more detail on the energy efficiency they flag as their motivation: is this model actually more energy efficient than the comparators they highlight?
show comments
dimatura
Very cool work - refreshing to see a of different approach. I learned about Kuramoto oscillators many years ago from a book called Sync, by Steven Strogatz, which I highly recommend.
italiansolider
Readers care, this requires a nice amount of physics knowledge to really understand. Not too advanced but still, physics.
NopIdoN
> However, the trade-off with our approach is that it requires a more complex loss that operates given only generated samples.
foax
This kind of reminds me of DCT in lossy image compression, but in reverse.
_def
Not at all related but still reminds me a bit of FM synthesis
fusionadvocate
Is this somewhat related to reservoir computing?
show comments
OutOfHere
Can this even make an image having more than one "class"? Can it make an image of an astronaut riding a horse on the moon?
show comments
luciana1u
finally, a way to generate images that's slower AND worse. progress.
The required compute seems a bit high: "We trained all CIFAR-10 models on 1xB200 GPU, and all ImageNet 64×64 models on 8xB200 GPUs. The largest CIFAR-10 model uses 20 B200 hours to train, and the largest ImageNet 64×64 model uses 640 B200 hours"
20 B200 hours for CIFAR-10 seems like a lot...
When I first learned about computer science at the age of 11 or so (and in 1982 or so) the first page of the text book put digital and analogue computers on what seemed to be an equal footing. And then proceeded to ignore the latter for the rest of the book. Apart from a few notable exceptions ( https://en.wikipedia.org/wiki/Phillips_Machine ) I've often wondered about analogue computing.
I love this ! Used to work at Rain AI on training neural networks in unconventional hardware - people often that computers don't necessarily have to be electronic digital - there is a whole domain dedicated to creating machines that can apply certain mathematical operations faster or more efficiently than their electronic counter parts. I created this site to try create a classification of that space:
https://computers.tugdual.fr/
Really interesting - if I understood the article correctly, they're simulating this on conventional hardware, so in order to get the proposed benefits, it would need to be implemented in some other electronic medium.
Very cool. I’m reminded of Wolfram’s pitch that neural nets are a search through the very broad computational complexity of the function space they describe; he did a little work to show that you could find similar behavior in other function spaces. These oscillators are yet again a different function space, and its cool they can be harnessed in this way.
The question of what physical / electronic phenomena is the most efficient yet large enough function space to be used for inference is a really good one to think about. I have no suggestions.
It’s not clear to me how this would ever be practical since it seems dependent on n^2 scaling.
You’ve got to wonder when you have an image generation demo why would you possibly have 64 x 64 pixel output as your demo?
If I’m understanding this properly to generate a 4K image, you need like 5 trillion point to point connections on the chip. Even if power use from the oscillators is zero that’s going to be an issue.
This method is cool and the post explains it well. It would, however, be good to get more detail on the energy efficiency they flag as their motivation: is this model actually more energy efficient than the comparators they highlight?
Very cool work - refreshing to see a of different approach. I learned about Kuramoto oscillators many years ago from a book called Sync, by Steven Strogatz, which I highly recommend.
Readers care, this requires a nice amount of physics knowledge to really understand. Not too advanced but still, physics.
> However, the trade-off with our approach is that it requires a more complex loss that operates given only generated samples.
This kind of reminds me of DCT in lossy image compression, but in reverse.
Not at all related but still reminds me a bit of FM synthesis
Is this somewhat related to reservoir computing?
Can this even make an image having more than one "class"? Can it make an image of an astronaut riding a horse on the moon?
finally, a way to generate images that's slower AND worse. progress.
I didn't really understand anything...lgtm