Choosing a game engine to develop XR content is a significant commitment and will impact every aspect of your development practices. The advantages of popular game engines such as Unity, Unreal Engine, CryEngine, Amazon Lumberyard are the following:
a large community for support, tutorials and like-minded developers; this is Unity's biggest advantage with its large community and wealth of resources available online.
support for multiple deployment targets, making it easy to build applications for most VR ecosystems like Oculus and Steam. CryEngine is the most limited of these engine, only supporting Oculus Rift as XR platform.
support for most newly released XR peripherals, gadgets, and hardware. SDKs are mostly released for Unity and Unreal Engine.
Your choice of game engine largely depends on your prior experience, your project requirements (e.g. deployment target, compatibility with hardware and peripherals), your preference of marketplaces (e.g. Unity AssetStore vs. Unreal Marketplace), programming languages, emphasis on graphical quality, or any other of the differentiating features of each engine. Don't get sucked into the endless arguments and hatred on various social media platforms that place one engine over another. It all comes down to personal preference and project requirements. Choose wisely!
3D models are the nuts and bolts of your XR content. You can either create models yourself by using tools such as Blender, Maya, Modo, Rhino, SketchUp, ZBrush or you can purchase 3D models from various marketplaces like TurboSquid, SketchFab, CreativeMarket, or CGTrader. Each game engine also has marketplaces that offer a variety of 3D models among other assets. The most notable of these marketplaces are the Unity AssetStore, Unreal Engine Marketplace, and CryEngine Marketplace. Moreover, there are some 3D modeling tools that allow you to create models within game engines without the need for external programs. An example of this is Unity's ProBuilder which has its primary use for prototyping and level design.
There are a few things to consider when creating or purchasing 3D models for XR content:
Optimization is key: look for "game-ready", "game-optimized", or "low-poly" models or when modelling yourself, keep a (polygon) budget in mind for how complex your models can be. It is absolutely crucial to have rock-solid performance at 90 frames per second for most XR applications to avoid stutters/lag that can cause nausea and discomfort for your users.
User interactions are a key contributor to make your XR environment engaging. If you place a cupboard in your scene that cannot be opened, it will feel awkward for users that expect a certain degree of realism. When modeling or purchasing items, watch out for separately modeled parts that can be interacted with, e.g. drawers, doors, steering wheels. For example, there are many houses and rooms available for purchase on the Unity AssetStore, but only very few contain separately modeled meshes for most objects, allowing XR users to realistically interact with the environment. If in doubt, contact the artist before making any purchase. If you already own 3D models that need to be separated into several parts, try this Unity Asset (Mesh Cutter), it’s fantastic!
There is one additional way to create 3D models: photogrammetry. By capturing photos of an object from many angles and optionally with laser scans, objects can be mapped and converted into realistic, detailed 3D models. This can be done for objects and persons alike. A few resources that describe photogrammetry can be found here or here and a nice summary video can be found here.
Characters can greatly enhance the quality and user engagement of any XR scene, but they also add a lot of complexity and can cause many problems. Creating characters from scratch is fairly complex and requires modeling/sculpting, texturing, rigging/skinning, and animating using any of the major 3D modelling programs.
More affordable options are available for designing 3D characters in applications such as DAZ Studio, Morph3D, Adobe Fuse, iClone Character Creator, Autodesk Character Generator, MakeHuman, and Manuel Bastioni Lab. We created a full comparison of 3D characters and design tools on our YouTube channel and have a full playlist on 3D characters.
You should apply similar caution to your 3D character purchases as with any other 3D model. Characters should be optimized for real-time rendering and labeled "game-ready" or "low-poly". Moreover, you should look for "rigged" characters unless you plan to add a rig, a character's bone structure needed for animating the 3D model, to the character yourself. Ready-posed characters are much less flexible and can usually only remain in this one pose, making them only suitable as static backdrops for (architectural) visualizations. You should also look at the character's rig. If you need to animate the character's fingers or toes, make sure the rig contains the appropriate bones. Lastly, some characters also contain blendshapes which are parts of the body that can be animated. Blendshapes are often used to create a character's facial expressions or to lipsync with dialogue.
Some of the more difficult aspects of finding suitable 3D characters for our projects were the availability of different age groups and ethnicity of characters as well as realistic (non-cartoon/non-fantasy) clothing items that conform to the various body shapes of characters. Also, a consistent art style is quite important, making it necessary to find artists or marketplaces that have multiple characters available. There are many nuances to successfully implement human and non-human characters in XR applications. If you have any specific questions, head over to our YouTube channel or contact us here.
Textures are an essential part to XR development, giving life and identity to objects. While characters and 3D models almost always use textures, sometimes you just need an extra texture for a terrain, some rocks, signs, a skybox or a variety of other things.
A good first start is to search Google Images for textures or photos, just make sure you select the appropriate usage rights to modify or use the images in your application or game. Read more about image usage rights here.
Next, there are several texture "ecosystems" out there that represent a set of tools and approaches to using textures. There are Allegorithmic's Substances, Quixel's Megascans and Marmoset Toolbag as the main competitors, the first two offering large libraries of realistic, seamless textures and all three offering tools to create, edit, and export textures for yourself. Make sure to check out whether these toolsets are integrated in or compatible with your development tool (i.e. game engine) of choice to get their full benefits. For example, as of Unity 2018.2, you will need to use Allegorithmic's plugin in order to use Substances in Unity while in prior Unity versions, Substances worked out-of-the-box.
There is a fair number of marketplaces and texture-focused websites out there that allow you to download free or paid textures. The most notable of them is probably Textures.com (formerly known as CGTextures.com). It has a large selection of textures and with a free membership you can download at least a few low- to medium-resolution textures each day. For high-resolution textures you will have to subscribe and spend paid credits.
A lot of the textures that you can purchase online are scanned images of real-world materials and are often meant for a so-called PBR workflow - physically-based rendering. That is, these textures are meant to be rendered in a way that takes into account physically realistic properties of a material - that is, light interacts realistically with this material. With this PBR workflow in mind, a simple (diffuse) texture is not enough anymore, we also need a number of other "maps" that define how light interacts with the surface of the material. If you are interested in an in-depth view of PBR, have a look here and here or watch a video here.
Audio can have a strong impact on XR applications. The spatial nature of any XR application requires you to pay special attention to the placement of any audio source. Misplaced, misconfigured, or missing audio cues can easily ruin your users' immersion. For example, a lack of collision sounds will feel weird when you drop an object in a virtual scene or bash a virtual foe with a club. Even tiny details, footsteps, ambient noise, breathing sounds, and many others can greatly enhance the XR experience.
Let's start with some VR Audio tutorials that you can check out for more detailed explanations. These will cover general VR audio and so-called Spatializers, plugins that enhance the spatial audio experience in virtual environments - Unity Manual, Resonance Audio SDK, Oculus VR Audio guide, Steam Audio overview. All Unity Audio content can be found here. Also, Raywenderlich.com has several very good text-based audio tutorials for Unity (Intro & Audio Mixer).
For recording and processing audio, a very popular tool is Audacity - it's free, open-source and available here.
If you'd like to know more about audio hardware and recording audio, we have enjoyed the YouTube channel of Curtis Judd, covering all aspects of A/V recording.
There are surely many nuances and a lot of complexity to developing XR applications. Though, let us assure you that XR development has never been as accessible and achievable as it is now. VR and AR hardware is more abundant and affordable than ever and the entry point to development is as low as it can be - most tools and resources are available for free.
There are no dramatic differences between developing XR applications and regular non-XR applications such as games and simulations. Optimization is mission-critical to avoid lag and the discomfort and potential motion sickness that it can cause. This older Oculus blog post still contains a few good pieces of wisdom for you to follow and look out for.
Most XR hardware manufacturers make sure their devices are compatible with the game engines Unity and Unreal Engine. You will usually find SDKs for most devices, whether released by Facebook, Google, Microsoft, Valve, Apple or small startups via Kickstarter. This compatibility among devices and drivers is one of the major reasons why XR development has really taken off in the past few years.
Next, we want to highlight that XR development does not have to break the bank. While the usual costs for software licenses, XR hardware and the hourly cost of artists and developers apply, there's nothing stopping any hobbyist from downloading most tools and assets for free and getting started in the world of XR. We have created several videos on our YouTube channel that show first and easy steps in XR for the game engine Unity.
Lastly, it is important to us to stress how impactful XR is for a variety of industries. Our background is in clinical XR and research associated with clinical XR use - this is the perspective that most of the content on this site will have, but we would love to hear other perspectives and experiences from any industry - share your work, share your opinion.
Don't get distracted by the wild predictions of analysts, Gartner hype cycles or naysayers. XR applications are changing the lives of people in healthcare, education, and training. This alone, even without any gaming impact, is worth pursuing XR further, growing it at a slow and steady pace.
If you know of any content that is relevant to this page, please share it with us - we gladly add tutorials and resources to this list.