The Heretic
Making of a real-time short film: digital humans and vfx characters
Welcome to the Making of The Heretic - a real-time short film made with Unity.
Unity Demo Team
2
Robert Cupisz
Tech Lead, Demo Team
Lasse Jon Fuglsang Pedersen
Sr. Software Engineer, Demo Team
We’re presenting the work of Unity Demo Team. In the past we’ve made demos such as The Blacksmith, Adam and Book of the Dead.
3
In the past demos, we needed to implement basic building blocks of higher-fidelity rendering: area lights, volumetrics, soft shadows, some post-processing.
4
The Heretic
A work in progress
Now the HD Render Pipeline and Post Processing are quite solid, and we were able to put much more effort into building on top of that foundation. Among other things, we put some effort into human faces and vfx-heavy characters.
Let’s watch a preview we’ve released so far at GDC.
5
Screening, 3:45
Constructing the digital face for ‘The Heretic’
Lasse Jon Fuglsang Pedersen
Sr. Software Engineer, Demo Team
Some numbers
~28k vertices
~50k triangles
318 blendshapes
212 frames of 4D animation
~7 seconds
Base texture data
4K maps for albedo, normal, cavity, thickness, etc.
Pose-driven texture data
48 activation masks (some idle)
16x3 additional 4K maps for albedo, normal, cavity
Composition
Skin
Composition
Skin
Eyes
Composition
Skin
Eyes
Teeth
Composition
Skin
Eyes
Teeth
Eyebrows
Composition
Skin
Eyes
Teeth
Eyebrows
Eyelashes
Composition
Skin
Eyes
Teeth
Eyebrows
Eyelashes
Stubble
Composition
Skin
Eyes
Teeth
Eyebrows
Eyelashes
Stubble
Tearline
Composition
We’re relying heavily on HDRP
Lots of nice things out-of-the-box
With some customizations
Shaders for skin, eyes, teeth, tearline
Aiming to get these into HDRP
Shading
Based on HDRP/StackLit
SSS
Secondary specular
Adds pose-driven features
E.g. add wrinkles when the character squints
Specular occlusion in cavities
Also modulates smoothness
Fresnel fade
Skin shader
Emulates two-layer material
Geometry defines upper layer
Virtual plane for iris
Evaluates lightloop twice
Specular from upper layer
Diffuse from bottom layer
Adds “Light transform” concept
Used to light the iris
Marker-driven occlusion
Anisotropic spherical gaussian
Eye shader
Emulates two-layer material
Geometry defines upper layer
Virtual plane for iris
Evaluates lightloop twice
Specular from upper layer
Diffuse from bottom layer
Adds “Light transform” concept
Used to light the iris
Marker-driven occlusion
Anisotropic spherical gaussian
Eye shader
Nicholas Brancaccio
Teeth shader
Based on HDRP/Lit
SSS
Marker-driven occlusion
Area of spherical polygon
6 markers on the lips
Project onto hemisphere
Compute area (spherical)
Use area to fudge occlusion
Also applied to tongue/interior
Different diffusion profile
Tearline
Done with local blur
Normals, smoothness
Skin and eye shaders aware
Write to normal buffer in prepass
Read back before shading
Mark region with custom decal
Writes “blur normals” bit
Separate pass does actual blur
Works only on stencilled region
Kernel shrinks to edge
Animation
Wanted to have realistic facial animation
Attempt to avoid uncanny valley
High granularity of motion
Decided to try using 4D capture and playback
4D meaning one 3D scan per frame
“Geometry-accurate to actor’s performance”
At least macro-level
Brings a whole bunch of technical challenges...
4D technical challenges
Raw 4D data
“Photo-realistic” skin geometry
Missing sections and fine features
Cleaning the data
Retopologize to match neutral
Transplant missing parts
Partial noise removal
Recovering fine features
Correlate 4D frames with facial rig
Pose-driven normals, cavity, etc.
4D data pipeline
Neutral + FACS
4D capture
Clean 4D
Combine
Snappers
snapperstech.com
russian3dscanner.com
http://ir-ltd.net/
Infinite Realities
Russian3DScanner
Actor
Facial rig
4D technical challenges
Raw 4D data
“Photo-realistic” skin geometry
Missing sections and fine features
Cleaning the data
Retopologize to match neutral
Transplant missing parts
Partial noise removal
Recovering fine features
Correlate 4D frames with facial rig
Pose-driven normals, cavity, etc.
What about other parts, like hair?
What needs to move with the skin?
Hair
Eyebrows
Eyelashes
Stubble
Decals
Tearline blur region
Markers
Markers on eyelids
Markers on lips
What needs to move with the skin?
Hair
Eyebrows points
Eyelashes points
Stubble points
Decals
Tearline blur region points
Markers
Markers on eyelids points
Markers on lips points
Skin-attachments
We need to resolve lots of points
Everything that moves with the skin
Skin-attachments
We need to resolve lots of points
Build list of local poses for each point
Find closest vertex
Local pose per incident triangle
Barycentric + distance to plane
Skin-attachments
We need to resolve lots of points
Build list of local poses for each point
Find closest vertex
Local pose per incident triangle
Barycentric + distance to plane
Resolve at runtime
Batch operation
No dependencies between points
Parallel operation
C# jobs and Burst
Skin-attachments
We need to resolve lots of points
Build list of local poses for each point
Find closest vertex
Local pose per incident triangle
Barycentric + distance to plane
Resolve at runtime
Batch operation
No dependencies between points
Parallel operation
C# jobs and Burst
~429k poses, ~73k points
Skin-attachments
Longer strands require a bit more
E.g. eyelashes
Need them to pivot around roots
Locate roots of strands / cards
Find closest vertex at root
Propagate to tip
Runtime resolve not affected
33
Concept art by Georgi Simeonov
Robert Cupisz
Tech Lead, Demo Team
Boston is the name of this robotic companion. These are concept drawings by our concept artist, Georgi.
34
Concept art by Georgi Simeonov
From the beginning it was pretty clear we wanted a mechanical, robotic creature. Built with tech not available today, very physical in nature and reconfigurable.
35
Initial exploration - hacked together, slow, not very controllable...
36
But enough to suggest it might be a good direction.
Boston forms
37
Bird
Let’s see what were all the various forms we needed to support so far:
The bird form
Boston forms
38
Bird
Boston forms
39
Bird
Tentacle
Boston forms
40
Bird
Tentacle
Tentacle - crawling
Boston forms
41
Bird
Tentacle
Tentacle - crawling
Boston forms
42
Bird
Tentacle
Tentacle - crawling
Tentacle - leaping
Boston forms
43
Bird
Tentacle
Tentacle - crawling
Tentacle - leaping
Gauntlet
Boston forms
44
Bird
Tentacle
Tentacle - crawling
Tentacle - leaping
Gauntlet
Dreamcatcher
Tentacle form
45
Needed precise animator control, grabbing things. Couldn’t allow the wires to just flow anywhere.
Tentacle form
46
Hermite spline
If we say the wire should go through chain joints and be controllably smooth, we can solve its shape with a cubic Hermite spline.
Tentacle form
47
Hermite spline
Offset joint position
Then offset joint positions to get a bundle of wires. The plane of the circle is the mirror plane of the two bones. The rotation can follow the bone (as in the gif) or be automatic.
Tentacle form
48
The bundle behaves quite well under bends and twists.
Tentacle form
49
Start/end trim
Noise
Radius variation curve
...
Some extra controls to add variation: noise; distribution (Fibonacci, random, edge); trim at the ends.
50
Tentacle form
Initial exploration experiment with one of the wires made as a chain of HDRP line lights (and with an emissive material).
No simulation
51
No persistent state or reliance on previous frame
Better directability
Instant preview, exploration
Scrubbable
So far we’ve managed to avoid simulation completely. And by simulation I mean the current frame being a function of the previous frame. Another way of looking at it is reliance on some persistent state. Simulations add another layer of difficulty to making a complex effect directable, and making Boston directable in its form and animation was our biggest problem.
On top of that simulations are not scrubbable - require prewarming or caching. So the self-imposed limitation we had was that if we just now the positions of all the controls and values of all the parameters, it’s possible to evaluate Boston’s shape.
Wire rendering
52
Pre-generate a static tube mesh, store segment id and vertex angle in pos.xy
CS: Animate a segmented line*
CS: Generate segment tangent frame
Render instantiated wire meshes
VS: vertex knows its mesh instance, and segment id and vertex angle; indexes into segment position and tangent frame CS output buffers
* only this stage knows about wire behaviours
This is how wire animation and rendering is organised. There’s a static tube mesh that gets generated on init. Remaining stages run every frame. First all the animation compute kernels run on a buffer of points of a given wire, across all wires. Next stage, based on these output segment positions, generates a tangent, normal and binormal for each segment, outputs to a buffer as well. Wire meshes are rendered as instantiated geometry.
The regular HDRP Lit shader has a modded vertex modification stage. Since each vertex knows which segment it belongs to, it can look up segment position and tangent frame in the buffers. Each vertex also knows at what angle in the tube cross-section should it be. Based on that and the normal/binormal pair, the vertices are exploded into the final deformed tube shape. The animation compute kernels are the only ones knowing anything about all the different wire behaviours.
Tentacle crawling form
53
Need a way to conform to surfaces, collide with obstacles, but keep moving forward along the directed path.
Tentacle crawling form
54
This is a pretty good example of conforming to the surface and forming interesting shapes while at it.
But you can also see on the backside of the column how the wire paths quickly degenerate into a single wire. Still useful in some cases.
Tentacle crawling form
55
A better example of the wires remembering to continue