Personal Exploration

Non-Human Avatar Development

February 17, 2023 by awf5212 Leave a Comment

A while back, while working on the IVAN project, I developed a simple non-human character for use in the experience’s tutorial. A small floating screen with a line that approximates soundwaves as it “speaks”. This character was meant to provide a focus point in the tutorial and experiment with using the current avatar tech I had been working on with a non-human character.

We’ve decided recently to begin earnestly developing this character into something that can be used in more experiences. As such, we opened things up to a group discussion on changes and ideas to explore with it, as well as naming the character. From our discussions a few names have come up, but I’m currently leaning on the name “Customizable Immersive Experience Liason – CIEL”. As part of the exploration, I took a lot of the ideas for changes and tried mixing and matching various ones onto new versions of CIEL as seen below. THese changes ranged from changing the antenna of CIEL to changing the screen to look more like an oscilloscope than it already does.

new CIEL concepting

After some more discussion, I also took a shot at a more drastically changed design from my colleague who made her own concept art. This one is more inspired by old Macintosh computers and even has some coloration design to it that we can modify as needed. Of course, all the experimental bits from before may also apply to this version of CIEL, so further testing and development is sure to follow.

alternate CIEL concept

Soon we’re hoping to have this new version of CIEL finalized so we can bring it into new projects going forward. One such project we specifically started this process for involves attempting to utilize AI chat programs within Unity and having a “talk” with an NPC character in an AR space. The issue with this project was that a humanoid character wouldn’t know how to properly emote from just AI text prompts, so we decided it’d be a perfect job for CIEL, who can speak without emoting and still come across as charming and appealing.

Microsoft Rocketbox Avatar Library Review

August 11, 2022 by awf5212 Leave a Comment

Recently I took a look at Microsoft’s publicly released “Rocketbox Avatar Library”. Developed and released specifically for research applications with a focus on VR, it seems like an excellet tool to add to the Center for Immersive Experiences catalogue. From my explorations, I’ve come to the conclusion that it is indeed something we’ll get use out of, though there are aspects of it that are limiting so it is not a wholesale solution or replacement for use making our own custom designed avatars. In this post, I’ll go into detail about the pros and cons of this library.

First, let’s talk about the structure itself. The Rocketbox collection comes with over 100 avatars ranging from various occupations, gender, race, age, etc. It even includes some animal avatars for use. Also included in the package is all the textures for the avatars, their source files (in .max format), and a large collection of animations for the humanoid avatars, images of each avatar for easy lookup, and scripts to assist in Unity and Unreal integration. Each human avatar also comes in two varieties: one using a full facial rig for facial animation and one using facial Blendshapes. This adds up to a very massive 26 GB repository of assets. Luckily, it is quite well organized making it easy to find the assets you’re looking for as needed. This means that for our uses, it’s best to just take what avatars we need on a project by project basis and create new isolated packages of them. This is required due to some other issues I’ll be going into anyways, so it works out in the end regardless.

Rocketbox Avatar Showcase

Moving onto the quality of the avatars themselves: they’re highly professional and very well made. All avatars are incredibly efficient in their polygons and come with high quality textures with color, normal and specular maps. They are also fully rigged with well weighted skeletons that allow for very simple and clean animation. As stated above, each humanoid also comes in two versions: face rig and Blendshape. There’s benefits to both of these styles depending on the use. Full face rigs allow for more detailed animation and programmatic nuances like allowing for face-tracking possibilities. On the other hand, Blendhapes allow for easier setup of automated systems like emotional overlays and talk loops. Salsa Lipsync, the plugin we’ve been using focuses on Blendshapes, so we’ll likely be using those ones for the most part.
The main issue with the models and rigs is the lack of customization options. The avatars are built in an incredibly efficient way, which includes making them all singular meshes. Their clothing is hard modeled into their bodies, not being able to be separated or swapped around like you can with custom made CC3 characters. This lack of customization also applies to their style: all the avatars have a sort of “realistic” look. Not quite an extreme level of realism like some CC3 characters, but convincing enough. The detail on them can be a bit uncanny in some lighting though. Changing their textures is certainly possible, though it would certainly be limited by the extents of the model, especially if they have specific work outfits. These issues make it clear that these characters aren’t really built for being the primary focus of an experience. Under close inspection they likely would come off as off-putting. But as background models or distant characters meant to fill in a virtual environment, they should work quite well. If a more “stylized” look is needed, we can also use similar techniques to modify their textures to what I did with high fidelity food models prior.

Showcase of Rocketbox characters in Unity

Moving on, let’s talk about implementation. As stated, the collection comes with scripts to assist in the transfer into Unity and Unreal. The Unity script is very welcome after my initial explorations in Maya of the models. Another issue with them that had me worried was their naming structures for the bones. It was very messy which could have had issues within Unity. However, library came with an editor script that specifically goes into the avatar’s skeleton and renames things as best suited for Unity. This made the automated processes of turning the avatars into Humanoid rigs smooth. As such, actually bringing them into a scene is a simple click and drag. A similar process of standardizing them to the characters I made before is also easily possible, with one exception I’ll go into in the cons below. Within Unity, even in URP, the characters look great. Opacity on hair and other map issues are all automatically resolved without issues from what I’ve seen. The animations included in the library are also easy to process in a similar way and apply to our character animation trees. They seem to be similar to the animations that can be found on Mixamo.
Cons on the implementation side involve some issues with the naming conventions of other aspects not addressed in the Unity Editor script, and the limitations of the animation library. While the skeleton is renamed and fixed up, there’s no standard “mesh” naming scheme among the avatars like there is in the CC3 characters. As such, if left alone we can’t share Blendshape animation details among them. This is a fairly simple fix however, as I just need to give them a similar name among them. This does still need to be it’s own animation set as opposed to the CC3 characters though due to the Blendshape nodes within the characters themselves being very different. With the animation library, a similar issue of lacking customization comes up. The animations are high quality, but also very high keyframed, meaning editing them is hard to do within Unity itself. Even in Maya there’s issues that can come up with trying to customize such premade animations. We’ll need to consider our options when it comes to what animations we want to apply to the characters. One last minor con is in how the characters use their eyes. While they do have eyeball joints, their eyes are semi-spheres rather than full spheres. Meaning with my eye tracking script, it is possible to turn their eyes so much the edge becomes visible. This could be fixed in a few ways if it ever becomes a real problem, so for now we won’t worry about it.

As a final positive note, I want to talk about their animal models a bit. The provided animals range vastly from multiple types of dogs, to chickens, to even camels. These characters obviously don’t have animations with them due to not using humanoid rigs. However, they are all still well rigged. The chicken for example has full bird wing joints going to wings hidden within the model. This allows for some animations where the wings come out that should be rather convincing when not examined closely. I can easily see many situations where some simple looping animations would be possible to make the animals fit into a virtual environment. I’m really excited for the potential with them.

Demonstration of animation on Rocketbox characters

Overall, the Microsoft Rocketbox Avatar Library is a fantastic tool. It will provide us with a lot of useful assets for fleshing out environments with characters. While they have limitations that keep them from being the main focus of an experience, they still provide plenty of utility as props within an experience. Using both the premade avatars from the Rocketbox collection and our custom made characters will give us many new avenues of exploration.

Interactive Heavy Metal

September 13, 2021 by awf5212 Leave a Comment

Last week I decided to spend an evening participating in a fascinating interactive experience, a digital metal concert. This particular event caught my attention for several reasons, but primarily it was due to the band involved being one that only exists in a video game that I’ve been playing for over 7 years.

Pentakill Lost Chapter Album

First, some background: League of Legends, developed by Riot Games, is one of the biggest e-sports in the world. The 5 vs 5 multiplayer online battle arena (MOBA) game has been continuing development for over a decade since it’s conception as an original IP based on the Warcraft mod, Defense of the Ancients. The game thrives through constant content updates including new playable characters, cosmetics and events. The cosmetics in particular are a major part of the game where players buy “skins” for the characters they enjoy playing that offer new thematic versions of those characters. For example, an ice witch character has a skin that turns her into a Tron-esque living program throwing shards of code instead of ice. These skins are often given to many champs to tie them all into one theme called a “skinline”.

One popular skinline from League of Legends is called Pentakill. Named after the act of a player wiping out the entire enemy team (getting 5 kills in a row, thus penta-kill), the skinline imagines several characters as a metal band, giving them each a role in the band and changing their weapons to match those instruments. The gravedigger Yorick replaces his shovel with a bass, the lich Karthus swaps his staff for a mic stand, the viking Olaf uses his hand-axes as drumsticks, etc. This skinline actually began 10 years ago, first releasing in August 2011. Even back then Riot chose to advertise this digital band through an actual musical release, coming out with 3 albums since their conception. Each album released with new additions to the band’s backstory and mythology alongside new skins to commemorate the events.

The latest album, Pentakill III: Lost Chapter, released with a major event that took place completely separate from the game League of Legends. Riot partnered with a website called “Wave XR” (wavexr.com) to host an interactive concert for the band. An animated spectacle reminiscent of the 1981 film Heavy Metal, with a storyline involving Pentakill saving their world of metal from an antagonistic force called the “dissonance”.

I had never heard of Wave XR, but from what I’ve gathered they have been making a business creating interactive digital events for performers on their unique streaming platform. Using a team of digital artists, they create unique visual environments and animations that wouldn’t be possible in a live concert. These experiences range from more story driven concepts like Pentakill’s to more abstract visualizer type demonstrations. In both regards, the big draw and meaning to it being XR lies in the interactive elements the audience can participate in.

When I first entered the stream when it aired on September 8th, I was presented with a message saying I was a devout of “The Immortal One”, one of three gods established in the Pentakill mythos. Every viewer was assigned one of these gods (The Deep One, The Mad One & The Immortal One) as a sort of team they’d be cheering on during the concert. While the gods themselves don’t play an important role in the event, various points of the concert filled musical riffs with moments pitting the teams against each other in order to “give favor to their gods”. For example, the first portion of this competition came when the dissonance was raining lightning on the Pentakill performance. A faceless audience member walked up to a large catapult and a prompt appeared saying to offer oneself to the fight. While the stand-in for the audience cheered people on, a button prompt appeared on the screen for viewers to repeatedly press and cheer on their side of the competition. Despite the simplicity of the action, I can’t deny I was wrapped up in trying to raise the bar of The Immortal One, curious as to what would happen. Unfortunately, it seemed The Mad One’s team had some more avid clickers as they won the trail (and actually most after). After the victor was decided, the previously nameless audience stand-in now gained a name above him in the purple color of The Mad One. Not just any name though, it was a viewer’s name. It seems a random person from the winning team was picked from the stream and got to have their name projected onto this figure as it fired itself from the catapult and into the lightning, exploding in a fiery purple blaze. The point of the competitions were actually about slightly changing the visuals of the event. Nothing that would affect the story as a whole, but enough for those of the team to take pride in their colors being put up.

These button mashing sections weren’t the only interactive aspects either. While the team dynamics created a sense of competition, the concert also integrated audience content directly at times. Text taken from the stream chat (likely curated and specifically picked) popped up in segments with the names attached. Some viewers went even farther, joining a special Zoom based room where they shared video of themselves watching the stream. These videos were also integrated to parts of the concert, floating in segments behind or around the performers, creating a live audience to this digital band. All of these aspects show a fascinating part of this experience: it truly is somewhat “live” despite being a digital event. I don’t know for sure, but my guess is that rather than a video, the event takes place in an interactive live rendering of the concert with sections set up for the showrunners to dynamically include outside inputs. They can gather inputs from the audience as the scripted portions of the performance goes on, then when it reaches the more interactive segments they display the gathered input for the audience to enjoy in a non-scripted fashion.

This became most clear in some major fanservice moments of the concert. Between the songs, some story events would take place to lay the groundwork for the next song. During some of these moments, a segment was set aside for a live motion-captured act. Known Riot developers donned mocap suits (shown as small picture in picture moments) and controlled the villain of the story directly for a bit to call out to popular League of Legends streamers that were watching at the moment. The reactions of these streamers having their names directly called out by the characters in the concert were very neat moments for the tens of thousands of fans of those streamers, spreading the hype of the concert even further.

With all these different interactive segments combined, the great visuals and the base established fanbase, this digital concert wound up being a ton of fun. Both for myself, and for others as I’ve looked into the reactions on various social media platforms. Overall, the Wave XR concert appears to be a huge success. I’m very glad I participated directly on their platform and got to see it all go down live. The various methods they used to drive up audience engagement got me thinking of how digital events in the future will progress. In particular, some aspects of the concert got me wondering how it’d be in a VR setting. VR concerts have occured before, but not with this kind of audience integration and concepting. On Wave XR’s page for the concert, they even included a portion of their FAQ to make it clear that this event was NOT a VR event, which implies they are aware how people are interested in the concept. Even outside of a VR space, just imagining the ways this kind of unique live digital experience could work in other forms of media is fascinating to consider. I may have been drawn to this experience due to my history with the game it was based on, but I left it hopeful as a developer of interactive experiences as to what future experiences could pull off.