We took some time recently for an experiment in digitizing a bit of Penn State’s campus through the use of point cloud data. By taking a lot of images of the area and putting them through a program, we generated a fairly large point cloud of one of the greenhouses on campus. From there, the point cloud was given to me as a blocky 3D mesh and I attempted to make a workable VR environment from it. There were a lot of issues in the process however which we need to keep in mind when going forward with future attempts.

To start, here’s one of the images of the greenhouse from the data gathering.

A few big issues stand out that came up from the choice of location. Firstly, the point cloud really did NOT like recreating the semi-transparent tarp the greenhouse is wrapped in. This may have also been due to the fact that the chosen area was simply too large. The resulting point cloud had some clear features present and captured the ground well enough, but the top part was almost entirely missing. A bigger issue that came up, however, was that we couldn’t get the color values of the point cloud to transfer over to Maya correctly. As a result, I ended up having to work off of an incomplete grey blob of a mesh. The point cloud was also absolutely massive in Maya, which would have been an issue if not for the high end computer I use for CIE work.

It wasn’t all bad, however. There was enough visible structure to get the most important thing down here: the shape of the area. Luckily, despite the size, the greenhouse is structurally a simple pattern of metal frames lined up with the tarp on top. Within the point cloud, even without color, you can clearly make out where the metal frames jut out from the rest of the data, which let me create the greenhouse’s “skeleton” fairly easily. Then it was a simple matter to put the tarp part on top and match the ground shape somewhat. This is the general benefit of using the point cloud as everything comes out at least proportionally right. The scale was a bit off but as long as I had one point of reference with a measurable value (i.e. a doorway) I could scale everything else up to match it.

After this I went in and did some refining to make the ground match the point cloud well enough. Since the plants couldn’t be discerned from the ground in the point cloud, I just followed a general shape with some grooves. I also added in more details from the structure of the greenhouse like the black cloth on the sides and some props (i.e. the boxes in the back and hoses going along the length). This was also a chance to try some simple plant modeling with the spinach present in the greenhouse. I created two leaf patterns and arranged them in two different setups. Once in Unity, I made prefabs of the plants and spread them all along the rows in somewhat random patterns. I also gave them a script to randomize the scale and rotation of each plant somewhat to create some more variation. At this point, most of what I was doing was based more on the real picture references we had rather than anything of the point cloud data. After all the material application and cleanup, the final result turned out pretty well. The overall proportions are very close to the original, though some aspects clearly didn’t transfer over cleanly.

Overall, this process has some promise but I think we need to pick our subjects a bit better. A higher quality point cloud would certainly provide better results, which would mean it’d be better to pick a smaller area that can have a finer detail count. Getting the colors into Maya would also help immensely as that would let me more easily differentiate parts of the cloud when it’s not super clear like with the ground vs the plants. The big takeaway should really be that point cloud data is only really good for expediting some initial setups within the modeling though. It only helps with getting the proportional placement of things quickly. Once those objects are placed, everything else was better handled via just looking at the direct reference photo.

We’ve started work on a variety of other projects requiring a few new approaches to aspects. First one I’ll be discussing here is our updated pipeline utilizing Character Creator 3 and modern URP Unity.

Creation aspects within CC3 haven’t changed too much since my post back in 2020 on the subject. We are using some new bases and outfits, but overall the creation process is the same, including clothing reduction and game-ready base model within CC3 once finished building the character up. What starts to change up is in our export. Overall a lot of the settings are the same: Clothed FBX export “To Unity 3D”, with or without LoD based on use case. Turns out we don’t have to include the T-Pose every time now as that seems to be set up properly regardless.

Within Unity is the real update. It seems over the last couple years Reallusion themselves dropped their personal pipeline support for newer versions of Unity. Instead, they released the plugin source code and some other members of the Reallusion forum took it upon themselves to develop a solution for the various Unity pipelines, including the Universal Render Pipeline we at CIE are focusing on. Luckily, the solution is set up in a convenient Git Package which we have become well versed in integrating to our projects. This new solution also comes with some unique aspects and options that we’ll be looking into in the future, but for now the main factors is the automatic Prefab setup that is similar to the old system. Unlike that system, this one easily supports multiple character imports, organizing them into a list for easy switching and applying the construction process. It also sets up all the materials for proper URP readability.

From there, cleanup on the models is fairly simple. First, a quick texture fix needed for these stylized models. For some reason the occlusion map on the eye comes in at 100%, causing the area around the eyeball to be darkened to the point it turns black. This may work for subtle eye movements where the eye doesn’t move much as it creates a nice shading effect, but with larger eye movements the forced shadow becomes obviously off. Turning this down or off is required for these characters at least.

Next is a new setup for the character animator to make sharing animations easier: the main body is given a “CharAnim” controller that contains the Humanoid rig animation data we’ve gone over before. This allows them all to use the same humanoid animations made by the Center or pulled from Mixamo. Back when first talking about this character pipeline I mentioned issues with the Blendshape animation not being able to share between characters. I didn’t realize it at the time, but the solution was quite simple: using a second animator entirely for Blendshapes. Every CC3 character is structured the same way, with a unique name on the top level but sharing the name of the pieces beneath, particularly “CC_Base_Body”. This is the skin of the character and where the Blendshape data is actually controlled. The reason I couldn’t share the animation data between things before was cause I was trying to do it from the same top level animator. As such, every character had a unique path to their Blendshapes (i.e. “Neurtal_Female_Base/CC_Base_Body” vs “Neurtal_Male_Base/CC_Base_Body”). By creating a second animator on the CC_Base_Body object itself, the path is now uniform, meaning every CC3 character can share Blendshapes as long as they all have those Blendshapes named the same way (which they do by default). So we can place things like blinking and mood shifting into this animator and connect it to the Dialogue System I developed to work with these previously.

Finally we apply the eye tracker setup I used previously. Take two Empty Objects from the positions of the eye joints and pull them forward. Create a new Empty Object under the facial joint so it remains connected when not tracking that is between the two eye objects. Place the eye objects under this tracker. Then simply apply Look At constraints to the eye Joints, pointing them at the original Empty Objects we dragged from their position. Make sure the eyes are facing forward in the Look At and the eyes should now track the Tracker as it moves. Then I put the simple “Eye Tracker” script I put together on the tracker object which creates a gizmo for easy visualizing and the option to stick the Tracker to a specific object in the scene (i.e. the Player Camera).

All of this means we now have a much more streamlined pipeline for getting characters into our projects. This was done based on one we’re working on which will involve interacting with stylized characters that use the same model but different skin tones, so we needed a nice process for developing nuance with iterations within the characters. Below is the WIP characters we developed to start with as we established this updated pipeline. We’ll share more about the project as it gets going, but for now, the system is progressing smoothly.

 

As part of our continuing research explorations with the College of Health and Human Sciences, we decided to run an experiment involving participant reactions to virtual models of food. This was to see if there was a notable difference in how a participant would react to a “high fidelity” virtual food and a “low fidelity” version of the same food. If the difference turns out to be not as substantial as we assumed, then it opens up more development options using lower fidelity models and unique stylization without worrying about these options affecting their reaction as to how it presents as food. To that end, I set up a pipeline to take high fidelity photogrammetry models we purchased and reduce their visual quality through removal of details and stylization.

The models were purchased through Turbosquid. We went with this model package from evermotion as it included a lot of high quality food objects that clearly used real photographs for the modeling and texturing to provide a good high quality standard to start with. We wanted the high and low fidelity versions to remain relatively close for the sake of comparison, and when it comes to something like that it is much easier to remove detail from something than it is to add it. This reduction process started with some experimentation in reducing the poly count of the objects and modifying the textures through various methods.

Poly reduction is an automatic process that can easily be applied to all sorts of models. The percentage some get decreased had to be adjusted due to some models being more or less poly dense. For example, the bread above was easily dropped through a 90% poly reduction as the original was fairly dense and the shape allowed for way fewer polygons while retaining relatively the same form. But when tried with some grape models in the package we bought, where each grape was a lower poly sphere, the reduction had to be reduced to around 50% or else the grapes began to just turn into cubes or triangles.

The texture reduction took a bit more processing. The first attempt was done by merely reducing the actual image quality of the texture. This effect somewhat works, but the result wasn’t quite as strong as we had hoped to make it appear “less detailed”. After a while, I found a good process in applying the Cutout filter in Photoshop. The filter settings had to be adjusted on a case by case basis with each food (some contained more colors or small details than others that were lost with lower Cutout settings) but overall the process of applying the Cutout option to each food texture worked perfectly to reduce the fidelity of the foods while retaining just enough visual distinction.

As one final step of obstruction to lower the visual fidelity of the food, a toon shader was applied within Unity. A toon shader is a material shader that takes lighting effects within the program and applies them along flat levels onto the texture. So rather than having smooth shading on the object, it is shaded in steps based on certain light thresholds. This creates a kind of “flat” look to the object when set up properly. Toon shaders also include an outline around the object to make it stand out distinctly. For this project, we used the Flat Kit package from Dustyroom, purchased through the Unity Asset Store. The result when paired with the lower poly models and cutout textures was a perfect balance of reducing the quality of the models while retaining just enough fidelity to tell they are the same food object. This pipeline provided us a ton of food options for this small study along with some tools we may use in the future if we desire projects using some more stylized setups.

Showcased below are some new avatars I’ve developed for CIE projects. We’re currently looking into the strengths and weaknesses between realistic and stylized avatars.

[Read more…]

I’ve been taking some time to go back to Character Creator 3 and really try to learn what it has to offer. To this end, I’m working through a lot of tutorials both on their official site and from other freelancers on Youtube. I started from the end of the pipeline though for the sake of some project deliverables we needed ASAP. A project required two generic characters with the same animations, so I started by finding out all I could about exporting characters from CC3 to Unity as efficiently as possible. From the tutorials, I was able to figure out a lot of the things I was missing from the export settings.

[Read more…]