Researchers at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) have developed an imaging technique that lets a person reach in and "touch" objects in videos.

Called Interactive Dynamic Video (IDV), the tech uses traditional cameras and algorithms so it can look at the tiny, almost invisible vibrations of an object to create video simulations users can virtually interact with.

Like Us on Facebook

IDV has many potential uses. For example, filmmakers could use IDV to produce new kinds of visual effects. IDV can help architects determine if buildings are structurally sound.

IDV can also go a step beyond the ability of the Pokemon Go to drop virtual characters into real-world environments by actually enabling virtual objects (including Pokemon) to interact with their environments in specific, realistic ways, like bouncing off the leaves of a nearby bush.

"This technique lets us capture the physical behavior of objects, which gives us a way to play with them in virtual space," said CSAIL PhD student Abe Davis, who will publish the work this month for his final dissertation.

"By making videos interactive, we can predict how objects will respond to unknown forces and explore new ways to engage with videos."

Davis used IDV on videos of a variety of objects, including a bridge, a jungle gym and a ukulele. With a few mouse-clicks, he showed he can push and pull the image, bending and moving it in different directions. He even demonstrated how he can make his own hand appear to telekinetically control the leaves of a bush.

"If you want to model how an object behaves and responds to different forces, we show that you can observe the object respond to existing forces and assume that it will respond in a consistent way to new ones," said Davis, who also found the technique even works on some existing videos on YouTube.

He outlined the technique in a paper he published earlier this year with PhD student Justin G. Chen and professor Fredo Durand.

This work was supported by the National Science Foundation and the Qatar Computing Research Institute. Chen also received support from Shell Research through the MIT Energy Initiative.

"The ability to put real-world objects into virtual models is valuable for not just the obvious entertainment applications, but also for being able to test the stress in a safe virtual environment, in a way that doesn't harm the real-world counterpart," said Davis.

You can view a YouTube video of IDV here.

TagsMIT, Computer Science and Artificial Intelligence Laboratory, Interactive Dynamic Video, IDV, Abe Davis