Look at your keyboard. It's roughly rectangular (or maybe you have one of those curvy ergonomic ones), but it's not just a rectangle, is it?
You can flip it over and see the little feet that prop it up at an angle, small holes for screws, maybe a sticker or two. On the front, each key protrudes from the surface, and they probably have a slight curve on the top of each one that you barely notice unless you look right at it.
If your keyboard sat in the background of a TV show, you wouldn't notice those little details. You wouldn't notice them if it sat in the background of a VR environment either, so I think you'd be annoyed if that keyboard sitting in the background were the reason your VR experience stuttered as it tries to render every little detail.
There are scanning tools already out there which can look at a real room and digitise it. Now we have VR to take a digital room and make it look real. The missing link is taking that massive raw scan with too much detail everywhere, and cut out the minutiae so your computer won't faint when it tries to process it. At the moment, this usually involves getting an artist to sit down and retopo - remake the topology/model so it captures the general shape more efficiently.
Now it looks like Microsoft has decided to throw their weight behind this problem by acquiring Simplygon, which is a state-of-the-art automatic retopo/model simplification tool. It's cool and it works, and according to this article, they're pointing that tool at AR and VR.
My prediction? Look forward to seeing more scans of real-world things in VR and AR!
Simplygon takes 3D models in a number of formats, and reduces the volume of data used to describe them by taking out some of the detail -- somewhat like reducing the size of a JPEG image file by increasing the level of compression while leaving the resolution unchanged. That means the models can be rendered more rapidly or using less powerful hardware, something that will help Microsoft with the "3D for everyone" vision it outlined last October at the launch of Windows 10 Creators Update. The company is pushing hard into the markets for virtual reality, with a US$300 consumer VR headset for PCs due out as early as March, and augmented reality, where it is seeking to build an ecosystem around Hololens, its enterprise-oriented stand-alone AR headset.