The State of Immersive Reality in 2018
After a busy 2017 for the industry Thomas Ffiske gives his annual review of everything immersive.
In late December of 2017, a Moscow resident suffered fatal wounds after playing in virtual reality (VR). According to reports, the man tripped and crashed into a glass table, suffering fatal wounds and dying there due to loss of blood. As far as we are aware of, this is the first ever VR-related death recorded.
This comes over three years after Denny Unger, of Cloudhead Games, predicted that the first VR death will come from horror games: “When the commercial version comes out, somebody is going to scare somebody to death – somebody with a heart condition or something like that. It is going to happen. Absolutely.”
News like this always coincides with the development of new technologies, and mainstream VR is no exception. A variety of ‘firsts’ have been achieved over the last few years, shouted by companies to differentiate themselves from an ever-changing market. They are inevitable, and are markers of progress.
In my overview The State of VR in 2017, I argued that we are in the ‘wild west’ of VR development, where developers are given the freedom to develop many kinds of technologies buoyed by interested investors. This is still true – across healthcare, marketing, videogames, retail, education and more, new developers are cropping up to throw their cyber hat into the ring.
I also argued that we were going through an ‘immersive reality revolution,’ where VR, augmented reality (AR) and mixed reality (MR) are cropping up to capitalise on a trend towards immersion as the next step for interactive gameplay, learning and development. The State of VR in 2017 was a false title, as people were using VR as an all-encompassing title for a trend which spans beyond headsets. This year, with the rise of explicitly MR machines (such as Magic Leap One), the time feels right to use ‘immersive reality.’
2018 is set to be an interesting year, building on the successes of last year. Jeremy Dalton of PwC agrees: “We’ve built a strong foundation this year and launched some great content. We’ve also learnt a lot about what works in VR, and this motivates us to produce better and richer content that will appeal to everyone.” A strong and fertile base feeds growth, and we can expect some exciting software in 2018.
More importantly, there would be a more direct link with returnable results. Dr Adrian Leu, CEO of INITION, argues that it will become easier to link the technology with tangible results. “Measurable biometrics will offer a good understanding of people’s and the impact of different content. This will provide invaluable data for optimising content, plus a feedback loop for perfecting and personalising experiences in real-time, so driving stronger ROI for VR brand experiences.”
These are interesting times, and there will be a huge variety of HMDs and software titles launching over the next 12 months. They will be disruptive, innovative, and may in many cases fail. That said, it will be safe to say that 2018 will not be boring.
Working Through The Hardware: HMDs, Standalones And Mobile
The HMD market can be compared to our current generation of games consoles. Sony, Microsoft and Nintendo are targeting living rooms to become the go-to console for the family, using a variety of tactics which are remarkably similar to HTC, Oculus and the PlayStation VR. Exclusive titles ensure people buys the respective headset, such as Lone Echo for the Oculus Rift – while it restricts consumer choice, it helps to build an install base for the relevant brand.
Like the close of 2016, Oculus and HTC are still head-to-head in terms of applicability room-scale interactions. As the heavyweights of the competition, fists will start flying in 2018 as they build on their solidified reputations to usurp one another. Oculus has an excellent head start with its Oculus Touch controls and its lowered price. The PlayStation VR, while weaker than the other two, has sold over a million headsets over the year, thanks in part to its lower costs and install-base of PS4s which can handle the high power required for VR.
The key learning from 2017 is that it is a price elastic market; a decrease in costs leads to an increase in sales, with hardware fidelity being less of a factor for the consumer. With the standalone headsets being pushed in 2018 / 2019, sales may spike upwards depending on the costs of the hardware. One tactic which may be risked is undercutting the competition by selling the hardware near the costs of production, lowering the profit margins of the hardware but receiving returns with software downloads. The Facebook-funded Oculus is in a good position to do this. In any case, a price war is expected.
In terms of mobile headsets, the Samsung Gear VR remains a strong contender with its included controller and (relatively) low price accompanied with the phones. The system is comparable to the updated Google Daydream View, which also has a respectable line-up and a nice range of phones which are compatible with the system.
Then there are a wide range of headsets which allow people to slip on the headset and play, such as the Google Cardboard. They are rudimentary, though functional and work with some high-quality apps. At the moment the market is saturated with plastic headsets which promise stellar VR, but offer little more than the Google Cardboard in terms of actual function. While they can be considered a rip-off, their suffusion in the market still helps people at least try mobile VR for the first time.
In terms of new headsets – the market is likely to be absolutely suffused with new tech which, for the normal and navigating consumer, it may be difficult to pinpoint the one to choose.
Oculus is set to mimic Boeing’s business strategy of offering multiple services to capitalise on different areas. 2018 will have several headsets come through, and an exciting one is Oculus Santa Cruz (pending to be released in 2018 or beyond). A standalone headset with controllers and solid experiences is set to revolutionise the market, and people will be following the headset closely. The same goes for Oculus Go, a lower-end version which will bring more people into the industry.
Google is doing the same by expanding the Daydream project and collaborating with Lenovo standalone headsets. A previous collaboration with HTC has been cancelled as the Vive Focus only launches in the Chinese market – potentially to win the Asian market before moving west with its product. Coupled with the Vive Wave development platform, which promises to make it easier to develop games and import to the Vive, the company is set to make waves beyond the Western shores.
Microsoft and Magic Leap will dip their toes into MR, though in different angles. Microsoft has partnered with Lenovo, HP, and Samsung, and offer versions based on integrated and dedicated graphics cards. Magic Leap will be launching Magic Leap One, and it is currently difficult to pinpoint further details until more is revealed later this year. Characteristic of the company, very little is known at the moment.
Altogether, there will be an onslaught of headsets appearing in 2018. While 2017 was a battle between several large companies in mobile and HMDs, the market will now open further with MR and standalone headsets – a fresh market set to rupture how people will experience immersive reality. My prediction is that Oculus Go and Oculus Santa Cruz will be taking the headlines for the New Year.
VR filmmaking is a very different process compared to traditional TV and cinema, and with it comes the baggage of learning new techniques for filming. Last year I described 2016 as the ‘Wild West’ period of the industry where there are fewer rules and filmmaking techniques were less established. The new, somewhat untraveled land forced some filmmakers to learn new techniques on the fly, adapting to the situations they want to portray.
One of the people researching the field is Alia Sheikh, a Senior Development Producer in BBC Research and Development, who experiments to find out what works and what doesn’t. In a 360 degree field, a viewer can look wherever they like; Alia’s technical research covers the psychological techniques broadcasters can use to subtly push viewers in the right direction.
To Alia, these subtle ways of directing the viewer can work in a number of ways: a voice to the left of the scene; the people at the centre of the scene pointing to the left; or a person walking across the scene towards the left, among many other dynamic prods. The same effect can be done with light – like moths to a flame, viewers tend to follow the source of light over dark.
All of these ideas have been lifted from traditional theatre techniques, and with eyetracking software Alia noticed how effective they can be for viewers in 360°. Even then, Alia found that some people still tend to dart around the scene left and right, not focusing on a particular area for a long time. These viewers like to scan their surroundings as a scene plays along, rather than stare at one area for a long time. Others might choose one area of interest and remain fixed on it until very strongly directed to shift attention. People don’t stop being highly individual.
Scene transitions can still happen, and they do not need to be complicated. Fade to black then fading into a new scene still works, for example, and this matches the use of lighting effects in theatrical productions. Not unexpectedly, spinning the camera into a new scene tends to make people feel nauseated, however if footage is sped up fast enough the motion blur effects disappear and turn into a more acceptable ‘blinking’ effect which surprisingly sometimes works for transitions.
Level of viewer engagement can be affected by the amount the viewer can be included in the scene. Alia created a scene when two people were having an argument, with the viewer listening in, and then an almost identical scene except in this case the people arguing attempted to involve the viewer. There was far less engagement when the viewer remained unacknowledged. Although in the second example the actors in the scene attempted to draw the viewer into the scene, the viewer was not able to affect the scene in any way. But when the viewer was given a name, and brought into the argument – with continual side glances from the arguers towards the viewer – attention spiked upwards. The Viewer became a part of the argument, not just a passive listener. Alia calls this as the ‘illusion of interaction’
Another factor is distance from the camera, and how it affects the watcher. Alia staged a fight between two people, and situated the camera 2ft, 3ft, and 4ft away from the fight. What Alia found was that the viewer’s level of comfort is somewhat contingent to the distances of real life. If a fight was two feet away viewers felt overwhelmed, whereas if a fight was 4ft away they felt safer watching the conflict. Distance in this case affected the emotional impact of the scene and the ‘correct’ distance for the camera from the actors depended on the desired emotional state for the viewer as well as taking into account that the low resolution of current 360° systems makes it hard to see people clearly if they are too far away.
The team found that changing the distance of scene objects from the camera also allows them to very convincingly play with size using traditional perspective effects. Working with the Familia De La Noche team on their 2016 production of Gullivers Travels, Alia and Trainee R&D Engineer Sam Nicholson, found that it was possible to approximate a ‘giant’ talking to ‘Lilliputians’. The giant effect was created by placing one actor on a stepladder and having him lean forward over the 360° camera which had been placed close between his hands. The Lilliputians were created by having the actors a few meters from the camera, looking artificially ‘up’ at the giant (who himself exaggerating his gaze downwards). There is a perception that a 360° camera perspective is somehow a more ‘honest’ representation of the world, as the entire panorama can be viewed and there is no hidden area ‘behind’ the camera. However these tests do show that the camera can, as was always the case, very effectively lie.
When the team went to the Edinburgh Fringe and did experiments filming street performers, they found the same conclusions – camera position is vital. There is a perfect middle between performer and audience where a viewer can watch the performance without being part of a crowd, and therefore be in a good position to observe both. This position only really makes sense if it is either important for the viewer to be able to turn away from the performer and observe the crowd for some reason, or if being situated at the front of the crowd would offer a poor view of the performance. Once the action is even a few meters away, on current playback hardware, detail is lost, and in that instance it is preferable to move the camera closer to the performance at the expense of feeling ‘part of’ an audience.
For one-on-one performances however, where the viewer is the only audience member, more interesting camera positions afford themselves in one case for example, having a performer juggle over the camera to make the viewer look up and feel a sense of peril, as well as provide an alternative view. In another example, a performer thrusted a flaming sword near the camera and gave a cheeky smile, a view that is closer to what the viewer would experience during a one-on-one interaction. These performances were made accessible via BBC R&D’s WebVR player, which was coded in Javacript and used on the BBC TASTER site to give an impression of the variety of street performances on show at the Edinburgh Festivals. Crucially, having a bespoke player meant that content did not have to be made available via an external platform, and the way that audiences navigated around the player (either on headset or non-headset device) could be customised to the content on offer.
From arguing friends to talking giants to dancers at the Fringe – at the heart of it all is narrative. No good story works solely on structure – it needs a tale to weave it together. And after a lot of tinkering on her side, Alia came away with three key questions that any 360° experience producer needs to be able to answer:
- Who is the audience? What is their role?
- How do you want to make them feel?
- What do you want them to know / understand?
Overall, Alia is excited for the future. ‘The language of filmmaking is still there, it’s just in a different dialect.’
 All the notes are from an interview with Alia Sheikh, 9 October 2017
What We Can Learn From IR Porn
The following chapter explores explicit content which may make some readers uncomfortable. While the recent revolution in VR draws many important points from the pornography industry, this chapter is skippable should the reader choose to move on.
It’s hardly the most approachable of topics around any dinner table. It’s a social taboo of sorts, one which isn’t broached much by people. Importantly though, it’s a strong area of development.
Let’s start with the numbers. There is absolutely no doubt that IR pornography has a significance level of investment and activity. Corey Price, VP of Pornhub, has his own opinions on this: ”Virtual reality is rapidly advancing and transforming some of the most intimate areas of our lives. While there seems to be a seismic shift towards VR, probably because it’s still so nascent, we still anticipate regular desktop/laptop/mobile browsing on our site to continue to prosper. In fact, we combined the best of both worlds just recently and announced a free VR category on Pornhub chock full of VR videos for our fans to experience at no cost. All they need is a headset and they can give it a try. And while it may mimic real life sexual interaction, we all know it won’t replicate it exactly. Nothing will. Not yet at least….”
Whatever your stance is on the topic, there is no doubt that there is a humongous amount of interest in the area – and companies are willing to spill the cash for branded content.
However, camera angles are not subjective. Like producing a movie, the camera shot is important to get a full grasp of the situation, whether it is a broad shot of the scene, or a POV shot to show the story through a first-person perspective. The shot has a purpose, and a bad shot would ruin the experience for the viewer. The shot is key for the experience, so what changes in the process?
The first is immersion. Adding a first-person perspective bestows the responsibility to the viewer, even if they are not directly enacting the action. For example, let’s say you put on a headset and you find yourself as a man receives a blowjob. You are not just a person who is watching a POV video of a blowjob; you are receiving a blow job from a first person perspective. You are not watching a man pull a woman’s hair and potentially hurting her; you are the man pulling the hair yourself. And in extreme cases, you’re not watching someone drop hot wax on a partner; you are the one dropping it, one by one.
For many, this would be an uncomfortable experience. To be placed in the shoes of a person who would be making actions you personally may not feel comfortable with is a new experience entirely. That is natural. In much the same way we would click through videos to find the right one, we would love on from topics which are particularly uncomfortable.
Jennifer Lyon Bell of Blue Artichoke Films has her own thoughts on the development on the industry: “As with so many new technologies, adult VR is the wedge by which many new consumers get acquainted with new technologies. Interest in adult VR is going to continue to help drive VR market penetration. But my personal hope is to see lots of interesting variety in VR erotic media, and that’s not happening in full force yet, at least not as much as me and many other viewers would like.
“We need even more platforms that will accept adult content. We also need lots more general consumer education about VR, how VR works, and the different technologies for enjoying VR media. I’m frankly surprised at how few people, even tech-savvy ones, understand VR. Given how fun it was to create erotic VR myself, I don’t mind trying to take on some of that burden to help show people how unique and engaging a VR erotic experience can be, and helping them find the right gadgets to do it.”
It is debatable whether this sort of pornography would supplant web browsing porn, in much the same way that magazines were replaced with the internet. I would argue no. Internet porn has won because it is easy to access, easy to browse, and easy to find the right content. This depends on the country you live in, but even in countries with more rigid laws it is easier to access internet pornography than a magazine showing the same content. IR porn is another barrier which may turn people off, and web browsing content would always be king.
What I can imagine is that it will become its own ‘major’ umbrella category to fill a particular niche when browsing videos. A tick mark of sorts to find videos which fulfil your criteria, but in 360° video format. This wouldn’t be a subcategory in of itself, to the same level as red heads or amateur; this would be a major kind which defines the type of video content further. This would make the most sense as it is so drastically different from any other sport of video.
In the future, there will likely be several reports on whether VR pornography can be linked to cheating, or if it is linked to violent behaviour. Like with many new trends before, a natural progression of a particular trend will become unfamiliar, a target for the media to use. This happened with videogames and violence, where commenters suggested there is a causal link between playing violent videogames and causing violence; this has since been debunked as the principle factor to these attacks tended to be either poor upbringing or mental health issues. In my mind IR pornography will have the same turmoil, discussion, and eventual acceptance, like many other topics before it.
Education And Training
From learning difficult engineering skills to teaching children, utilising immersive capabilities for education brings tangible results. As mentioned by Piotr Baczynski, the CEO at Immersion, issues that are incredibly hard to describe or present in 2D can be “experienced” in 3D and real scale. The research, as well as first practical trials, confirm that the effects of educating children through VR are
Piotr also brings a warning: “When speaking of education, we must not forget to mention museums, galleries and digital libraries, all of which are increasingly investing in VR expositions as a means to breaking boundaries in knowledge sharing and education. VR allows visitors to travel back in time and space, leaving them fully immersed in the topic. This learning through immersive experiences will revolutionise museum content over the next few years.”
If these projects were also brought into a 3D space, then it brings benefits if they’re rendered together. As Mark Miles of Render states, the add-on bonus here is that by digitising this data we are creating a de facto real-time content pipeline where multiple versions can be created near-instantaneously, without any re-editing or re-rendering. Not only does this further increase efficiencies, it also helps put companies firmly on the path towards the fourth industrial revolution.
Adoption is fast, with industries building a fast portfolio of devices. Dion Price at WaveOptics also recommends that technology needs to be wearable for further adoption, which is debatable though not false. This speed is good, and will likely continue through 2018.
As any cutting-edge medium rides a wave of hype, there are normally several corporate sharks who attempt to ride the stream, in a bid to gather attention for their products and services. In many cases, these collaborations were nothing more than a tact-on addition to a campaign, with little beyond ‘it is immersive’ to back its reasoning. Further, many PR companies sent stats and figures on the growth of VR or its place in the future based on polls, in a bid to hijack the news with very similar sounding but largely false news. In one case, a company merged VR and 360° video together to inflate the statistics, not drawing a distinction between the two very different mediums to improve its stats.
360° video will likely remain a domineering force for content production with its ease of access on social platforms. Blend Media, a pioneer in this sector, primarily uses 360° videos for campaigns, particularly the King of the Netherlands’ fiftieth birthday and the recent collaboration with MTV. This is unlikely to change in 2018 as HMDs remain expensive and largely inaccessible by the general public.
Yet the passing of time generates nuance, and 2018 looks ripe for great campaigns linked with immersion. This is, in part, linked to the suffusion of producers cropping up who are specialising in the technology. VirtualRealityMarketing.com states that, at least on the production side, there are a tonne of virtual volcano eruption of participants: “As VR becomes better, cheaper and mainstream this volcano will only erupt harder with each passing year.”
It is expected that immersive reality marketing is here to stay, at least in some form. In select cases, 360° videos on Facebook will be portrayed positively and used effectively, such as McDonalds early last year. The content is becoming cost-effective, and with that comes innovations. 2018 will have some slow transitions, but they will be interesting to follow.
E-commerce And Retail
Why browse with pictures on a website or visit a shop, when you can virtually see the products for yourself? The tactile feeling of interaction is vital for consumer choice. In a similar way in which retail companies are experimenting with how a house would look before it is built, companies are experimenting with choice on a virtual shelf as well as outside it. This falls into the realm of sci-fi as larger trends and demands draw the attention of brands, but it is always a fun topic to explore. When a market could add as much as £1 billion to retail, the financial benefits are ripe for reaping.
The ubiquitous trends in this sector are stores using AR to bring the product to the homes of consumers, to great effect. IKEA is an excellent example of using AR to place furniture in a home, showing what it would look like before a purchase – a clear connection with convenience and using the tech. Scott Lester of EyeKandy argues that Amazon’s AR View has set a new industry benchmark, transforming the technology from a ‘nice to have’ to a ‘must-have’ for retailers.
Yet Scott also warns, that, for most retailers, the costs of development are still too high. “With retail sales falling for the first time since 2013, allocating 2018 budget to AR development will be unrealistic for some retailers. As such, this year we’ll see a major shift in the deployment of the technology, with the arrival of white labelled solutions and AR-as-a-Service business models.
“Plug-in applications offering point-and-place technology remove the need for costly R&D and HR spend, and the emergence of monthly subscription plans means retailers can scale and spend in line with a retail market which will continue to fluctuate. These technologies will lower the barrier to entry for launching an AR app, enhance consumer engagement, and help elevate retailers to a competitive position alongside market innovators.”
With the rise of using AR (at least for larger companies, analytics tools will likely develop as well to complement the analytics. Back in the realm of VR, users can explore scenes however they choose; understanding their interactions through heat map data provides insights into audience behaviour. Yulio’s tech allows for the production of heat maps of attention, which add context to feedback – creators will better understand what clients were looking at when they made comments. In any case, some slow but steady progress is expected in this area.
Of all the areas to focus on, VR and healthcare will always be a favourite. It is one area where it can make a significant and important impact on someone’s life, and genuinely become a positive and healing force for those who train under it.
One reason why it is so useful is that it can block out reality. As Dr. Wendy Powell of IEEE summarises, it can create a safe and controlled environment to practice activities that might be dangerous, impractical or unpleasant otherwise. For example, someone recovering from a stroke can practice judging scenarios and surroundings. Or a traumatised soldier can be safely exposed to PTSD triggers, desensitising under trained supervision. In short, there is absolutely no doubt that more and more immersive tech will be used for healthcare
Another area explored by Wendy is rehabilitation, as the company is currently working with amputees who can get severe pain in the missing limb (phantom pain). Tricking the brain into thinking the limb is still there using VR can allow a patient to move it around and interact as if it was really there thereby reducing pain.
Like in 2017, this year will be another year of strong and steady growth, particularly among researchers dabbling with the technology and pushing boundaries. Though the benefits will not be obvious for a little longer, the technology is here to stay.
Funding And Investment
Passion for any project flickers and dies without fuel, and funding goes a long way to see a project reach its end. With something as intangible as immersive reality, and the difficulties of showing the tech as easily as movie or gaming trailers, it has previously been difficult to get the funds. With 2018, the focus is shifting.
Sam Watts of Make Real is largely hopeful of the future of funding, especially for smaller producers. In his view, external funding will continue to be a necessary part of content creation throughout 2018 as smaller studios struggle to self-fund titles based upon previous sales revenue alone, thanks to a still relatively small userbase and hardware market size. In short, there are still funding opportunities available but expectations and requirements to achieve them are nearer to more traditional gaming in that either a strong new IP or existing IP is needed, with a fleshed out vertical slice of gameplay beyond a simple prototype made from asset store components, a structured development plan, delivery timelines and considered deployment terms.
Aurelien Simon, Head of Immersive at Digital Catapult, has his own thoughts: “Investment isn’t all about funding – and 2018 needs to be the year that we channel energy into creating a holistic environment for VR innovation. Organisations developing immersive technologies, VR included, need the support, space, technology and time to take risks and see where their ideas take them.
“This is why programmes such as Digital Catapult’s CreativeXR are instrumental to advancing immersive in the UK. Targeted at the creative industries, CreativeXR offers those looking to develop immersive technologies access to the expertise and facilities that they need to take these further.”
2018 Onwards – What Next?
Taken together, the future of immersive reality in 2018 is in several different pieces, each with strands of specialisation.
Oculus is largely hopeful for the future. Jason Rubin, Head of Content at Oculus, said: “2018 is set to be a great year for us and the VR industry in general. We’ve built a strong foundation this year and launched some great content. We’ve also learnt a lot about what works in VR, and this motivates us to produce better and richer content that will appeal to everyone.”
One area expected to develop, though not see much mainstream use for now, is haptic feedback. Companies like Teslasuit allow for feeling through virtual worlds – something which TVs do not allow for, and brings immersive technologies several steps above its 2D counterparts. With developments in virtual embodiment, this field is expected to make some good progress as experimenters test the limits of how users can ‘feel’ around their environment.
For Igloo Vision, the aim is to make immersive content much more shareable. The big business driver for a company like Igloo has always been the quality and availability of VR and 360° content. It’s becoming much easier and cheaper to make. The creators are getting better at it all the time. And, as more content does become available, more people will appreciate the need to make it sharable.
In any case, the future of immersive reality looks bright. Developments have taken a keen edge, and while 2018 is not expected to break any records, it’s set to make another hefty dent in the immersive world.