While developing Unity projects I have found it very helpful to capture gameplay videos at various points of development. These videos are very useful for documenting the progress and progression of each project and make great content for demo videos, blog posts, and professional portfolios. Most of these videos have been 2D screen captures, which work well in most scenarios, but fail to capture some of the immersive aspects of virtual reality (VR) applications. Two dimensional videos are also very underwhelming and often boring when being viewed from within a VR headset. So the solution for this was to learn how to capture 360 degree videos.
In this post I will cover the steps taken to capture and play 360 video in Unity for the S&DVR project using the Rock VR – VR Capture Unity package.
This post will cover the following sections
Capturing 360 Video
Setup:
First I imported the RockVR – VR Capture package into my unity project.
The RockVR – VR Capture Unity package can be downloaded at the following link:
Then I accessed the RockVR video capture prefab located in the Assets > RockVR > Video > Resources > Prefabs folder. The prefab is labeled “VideoCaptureCtrl”. Once I located the prefab I dropped it into the scene.
Next I positioned the VideoCaptureCtrl prefab where I wanted the video to be captured from.
Thats when I ran into issues . . .
The issue I ran into was related to VRTK. Since the S&DVR project is build to be playable on Oculus Quest, the project itself was targeted to Android. But since RockVR – VR Capture is designed to function on desktop, I had to tether my Quest to my PC and run it as if it were an Oculus Rift. The issue arose because when the project starts up VRTK uses different setups for various development environments. Although the project was targeted as an Android project, Unity uses steamVR to access the Quest when its tethered and acting as a Rift. So, VRTK hit crossover errors between accessing steamVR through an Android project. The fix for this was fairly simple, I just switched the target platform for the project in the “Build Settings” window of the Unity project. Switching the project over to a PC/Standalone project resolved the issue and allowed VRTK to start up with the correct steamVR linkages.
Use:
After this bug was resolved, using RockVR – VR Capture to record 360 video was very straight forward. Just hit “play” in the Unity editor, then in the game window click the “Start Capture” button to start recording. When finished recording, simply click the “Stop Capture” button and the video gets saved to the RockVR folder in the documents folder of the PC.
Playing 360 Video
(Skybox)
Setup:
Once I had a 360 video file I was happy with, it was now time to figure out how to render 360 video in Unity.
I was able to find a pretty straight forward guide at the following link: https://learn.unity.com/tutorial/play-360-video-with-a-skybox-in-unity#
However, in case that link stops working I will detail my steps below.
Playing 360 Video with a Skybox in Unity
The following steps will detail the setup of a 360 video in Unity by applying a Redner Texture to a Skybox.
Importing 350 Video files
If you already have 360 video files, you can drag them into the Unity Project window to import them into your project.
360 Video in 3D Space
In order to playback 360 Video in Unity, we will need at least three elements:
A Video Player, a Render Texture, and a Skybox Material.
In short, the Video Player “broadcasts” the video to a special Texture known as a Render Texture. This Render Texture is then used as a Texture for your Skybox Material, which allows the video to properly display as a panorama in your Scene.
Creating a Skybox Material
In the project window, right-click and select Create > Material. (The material can be named whatever you’d like). (Image below)
Select the Material you just created and, in the Inspector, set the Shader to Skybox > Panoramic. (Image below)
Creating a Render Texture
Now we need to create a Render Texture that the Skybox Material will use. In your Project window, right-click and choose Create > Render Texture. (Image below)
You can name the Render Texture as you see fit, but, to keep things consistent, append the name with _RT
Setting the Render Texture resolution
Now we need to set the Render Texture so it will play at the correct resolution. If you already know the video resolution, you can change the size parameter in the Render Texture to the correct dimensions. If you do not, you’ll need to find those dimensions in the Source Info panel.
To do this, select the video in the Project window and click the video title in the bottom of the Inspector, next to the play button. Select Source Info. The video resolution is located in the Pixels row. (Image below)
Enter the values of your Render Texture in the Size box, starting with frame width. (Image below)
Finally, select your 360 Video Material, and set the newly created Render Texture as the Texture. You can do this by either dragging your Render Texture onto the Texture field or by clicking the Select button and searching for your new Render Texture. (Image below)
Creating the Video Player
(I did not do this step for this specific implementation since the RockVR – VR Capture prefab VideoCaptureCtrl object already includes a video player component. To do this I had to drag the video clip I wanted to play into the Video Clip field on the Video Player component on the VideoCaptureCtlr object. I also had to drag the Render Texture into the Target Texture field on the VideoCaptureCtlr.) (Image below)
(If you’re using the RockVR – VR Capture prefab VideoCaptureCtrl object for this process then you can skip this step and jump ahead to the “Completeing the Skybox” step)
Now back to the normal instructions from the Unity documentation site.
To create a video player, create an empty GameObject and name it VideoPlayer.
Click your VideoPlayer GameObject and, in the Inspector window, click Add Component. In the search bar, type “Video” to find the VideoPlayer component. (Image below)
Completing the Skybox
Now you will need to tell the Video Player which video to play, and which Render Texture to target.
Leave the VideoPlayer GameObject selected in the Hierarchy window, and find your video in the Project window. Left-click and drag the video into the Video clip box in the Inspector window. (Image below)
Find the Render Texture and drag it into the Render Texture slot on the Video Player component. (Image below)
Setting the Skybox Material
The final step in this process is to tell your Skybox to use the Material you created at the beginning of this workflow.
Select Window > Rendering > Lighting Settings to access the settings for the Scene’s Skybox.
Drag your Skybox Material onto the Skybox Material slot under Environment. You can also click the widget to the right of the Skybox Material and search for your Material there. (Image below)
When you enter Play Mode, you should see the 360 Video appear in the viewport. If you want to test your 360 Video with movement, simply switch to Scene view and use the alt+left-click to orbit the camera around your Scene.
Playing 360 Video
(View Sphere)
Setup:
Now that the skybox 360 video player is running correctly, I wanted to try to setup a View Sphere version that can be placed around the scene. To do this we will build off of the steps above, as it will require most of the same elements.
Duplicate the Render Texture
In order to avoid ruining the now functioning skybox 360 video player we set up in the previous section, we are going to start this section by making a copy of the Render Texture we created in the previous sections. I recommend renaming this duplicate to avoid mix ups moving forward. You can name your duplicate whatever you’d like, however I named mine “Vid360_ViewSphere_RT”.
I also made a duplicate VidePlayer GameObject and disabled the original, once again to avoid ruining the functionality of the skybox video player. I’ll refer to this duplicate VideoPlayer as the ViewSphere VideoPlayer.
Assign the Render Texture
Next, we will set the ViewSphere Render Texture to the Target Texture field on the ViewSphere VideoPlayer.
Sphere Object
For this next step, I recommend finding or creating a sphere model/mesh that has inverted/reversed normals. This means a sphere model which treats the inside of the sphere as the front and vice versa, essentially an inside-out sphere. The next step is doable with a normal sphere GameObject, but it works better with an inverted normals sphere.
Create a Render Texture Material
In order to generate a material for the viewsphere, drag the Render Texture onto the sphere GameObject in the hierarchy. This will create and apply a material to the sphere that is made from the Render Texture.
Set the Base Map
Now we want to make sure the new material is displaying the Render Texture that will play the video. To do this, find the new material we created in the last step and select it in the project window. With it selected, find the Surface Inputs > Base Map field and drag the Render Texture you used to create this material into that field.
Depending on what type of sphere you are using (normal or inverted) you may have to change the Surface Options > Render Face dropdown selection. If you are using an inverted sphere, then set this dropdown to Front. If you are using a regular sphere, then set this dropdown to Back.
Inverted Video Fix
You may notice that when you enter playmode and the video starts playing on the new ViewSphere, that the video may appear mirrored or backward. The quick fix I found for this was to set the scale of the viewsphere’s X variable to be negative. Doing this reversed how the video was displayed and appeared to fix it, making the video appear correctly.
Trouble Shooting / Bug Hunt
Issue: Strange Recording Position/Tilt
Once I got all the basics of capturing and playing 360 video set up, it was time to start recording the actual content. I was able to record 360 video and play it back without problems, but I noticed that the video I captured was recorded from an odd location and tilted vantage point. I tried a couple variations on positioning the VideoCaptureCtrl GameObject where I wanted the video recorded from, but each time the recording came out with a different position and angle than intended. I noticed that when I entered playmode in the editor, the VideoCaptureCtrl object moved to an obscure position and rotation around the scene. I then tried to reposition the VideoCaptureCtrl object to where I wanted it after already in playmode. Although the recordings came out closer to what I wanted, they were still captured from a different position and vantage point than where I had moved it to before pressing “Start Capture”. This is when I discovered that the VideoCaptureCtrl object was tracking and mirroring the movements of my VR headset, but only when I had the headset on my head (and thus could no longer see my computer screen or the unity editor).
Solution:
The solution to the problem was to make a copy of the “360Capture” GameObject that is a child of the VideoCaptureCtrl. I then disabled the original 360Capture object and moved the copy in the hierarchy window so that it had no parent object. Then I positioned it where I wanted the video to be recorded from. (This step could be done with the original rather than a copy, but I decided to keep a version of the original 360Capture in case I wanted to use it for capturing any future recordings). I then renamed the 360Capture copy “360Capture – Stationary” to easily differentiate the copy from the original.
I then assigned the stationary copy to the “Video Captures” field on the Video Capture Ctrl script component on the VideoCaptureCtrl GameObject to make sure that the recording is captured from the new stationary 360Capture object.
Then to the primary change needed to fix the issue. Two fields need to be changed on the “Camera” component on the stationary 360Capture object. First change the “Target Display” field from “Display 2” to “Display 3”. The second change is to change the “Target Eye” field from “Both” to “None (Main Display)”.
Changing these two fields will allow you to record 360 video from a stationary camera placed anywhere you’d like around the scene.
Issue: Non 360 Youtube video
When I uploaded the captured 360 video file to youtube, it was not in the correct format. This prevented the viewer from panning in 360 degrees and instead presented the video as a flat and warped view.
Solution: Export a 360 video file
When editing the video file in Premiere Pro, I was able to inform the program that this was a 360 video. To do this I switched to the “Effects” tab, then on the right side of the screen I selected the “Effects” tab and typed “VR” into the search bar. I then selected the video clip I wanted to add an effect to and dragged the “VR Projection” effect onto the clip. This should change the view of the video clip to represent a 360 sphere instead of a warped flat 2D view.
When exporting the video file I made sure to set the file format to “H.264”, then below that in the “Video” tab I scrolled to the very bottom where there is a section labeled “VR”. Check the “Video is VR” box and select a Frame Layout type (Monoscopic, Over/Under, Side by Side). After making these changes, the exported video file was now recognized and processed correctly as a 360 video when uploaded to YouTube.
Additional Notes:
After the steps above I was able to upload a video that Youtube treated correctly as a 360 video, which allowed the viewer to click and drag to pan the view around the 360 video. However, the video did not display the field of view that I wanted, so only a small central portion of the video was visible at any given time. To fix this I found a very helpful tutorial at this link:
https://www.youtube.com/watch?v=F8tmjDf2ueU
First I changed the sequence format for the VR clip. To do this I selected the video clip I am working with, then open the “Sequence” dropdown from the top left menu and select the “Sequence Settings” option. At the bottom of the Sequence Settings window there is an section labeled “VR Properties” which contains a “Projection” dropdown field. This should be set to “Equirectangular”.
Next we can add a “Toggle VR Video Display” icon/widget to the viewing window (not sure if that’s the official title for the window, but the area you view the video that is currently being edited). At the bottom right corner of this window is a small “+” icon. Click this to open a window of additional icons/widgets that can be added to the tool bar. Find and drag the “Toggle VR Video Display” icon onto the toolbar and press the ok button. Activating this toggle will switch the video view from normal to VR.
Next we want to widen the view of the video by clicking the wrench icon in the bottom right of the window and selecting VR Video > Settings. In the settings window we want to change the “Monitor View Horizontal” option to 180 degrees.
Leave a Reply