Seeing Beyond the Brain: Masked Modeling Conditioned Diffusion Model for Human Vision Decoding. Using Latent Diffusion to improving the image quality of brain-to-image generation. Because… why not?
Have you ever dreamed about some of the Amazing technology we could see in the Future it was a sci-fi film fan I know That I love to see all those various Gadgets attached to people’s brains to Read their very minds or perhaps the Reflection of a face from an eyeball to Help solve a crime wouldn’t it be Awesome to have something just like that Maybe we could attach software such as Stable diffusion and use our brain Signals with some sort of headset or Maybe you could be in virtual reality to Create a Holodeck of the future and then Simply change your environment based on Your own thoughts how much fun would That be or perhaps as an artist just to Be able to create an image rather than Using text you could just do mind to Image brain to image how but all this is Fantasy isn’t it isn’t it well I think Fantasy may be just a little closer than It was before thanks to this paper Seeing beyond the brain masked modeling Conditioning diffusion model for human Vision decoding as you can see they’ve Got the visual stimulus that has a brain Encoding which is then turned into an Fmri pattern and decoded to be a Reconstructed image so what the person’s Looking at can be reconstructed that’s Just ridiculous isn’t it or is it well I Mean the code is open source and they’ve Got everything here so you can just
Download and run it I haven’t because I Don’t have something capable of creating An fmri but never mind perhaps in the Future we’ll have some way to upload our Dreams as we can see from this overview Here we’ve got a masked brain modeling An MBM and an ldm conditioning by fmri Latent I mean look at that I don’t Understand anything about brain modeling But I certainly trust that it’s all Correct they’ve got two data sets there So if you want to download that and They’ve got the pre-trained models so You can indeed set up your Condor Environment and there it is the Environment set up so if you’ve got Condor installed you can just create Your new conda environment there Download the data and the checkpoints And then run on one of the pre-trained Encoders now obviously the most fun you Could have would be to fine tune it on Your own fmri data but I don’t have a Specialized dream Booth wait could that Be a thing you could go into a dream Booth and then just record your own Brain things and then you’d have your Own pattern and you could go home and Train your own models anyway maybe in The future but what else have we got Here we’ve got some more pictures and This is what we can see here the ground Truth and their model is very very good It’s certainly in the same sort of
Domain as you can see there with the Bear and the dog it’s fairly similar as With all the other pictures so as you Can see it may not be quite as Futuristic as we want but it is still Rather a good step forward in the image Quality when you’re taking an image from Somebody’s brain and then reconstructing It so what sort of Technology do you Think this could lead to certainly be Interesting to see won’t it Thank you