SIGGRAPH 2010

SIGGRAPH 2010

Dmitry Andreev (AND)
dandreev@LucasArts.com
Real-time Frame Rate Up-conversion
for Video Games
or how to

Agenda

Stories, Ideas, Possibilities
Excitement, Inspiration
Better Games
Happy Players
…
Technical Details are Simplified
Only current real-time

Real-time
DEMO

Frame Rate Up-conversion

Natural Motion of Objects
In Software
AVISynth + MSU FRC
WinDVD (Trimension)
In

Motion Estimation

Block-matching
Diamond Search
Optical Flow
Phase Correlation
MAP/MRF
or take it from the MPEG

Video Frame Interpolation

Build Inner frames from outer frames
Using Motion Vectors
High-res estimation

Motion Vectors vs Velocities
Estimated vs Rendered
Motion Vectors Velocity Buffer

No Need for Motion Estimation

The Source of Motion is Known
Objects
Camera
Handle Special

Running at 30 fps

Game
Extra 16.6 ms
More advanced rendering techniques
Deferred / Screen-space,

Running at 30 fps

Game
Extra 16.6 ms
More advanced rendering techniques
Deferred / Screen-space,

Motion Eye-tracking (60 fps)

Observed @ 60Hz

Display A, B, C, ...
Temporal Functions
Display
Eye

0

1/60

Motion Eye-tracking (30 fps)

Observed @ 60Hz

Display A, A, B, B, C,

Motion Blur

Helps to Remedy “Flickering”, but…
Happens in the Eye…
Not the camera

Motion Blur In Games

The Idea is …

Render Velocity Buffer

The Idea is simple …

Render Velocity Buffer
Interpolate the Middle Frame

The Idea is very simple …

Render Velocity Buffer
Interpolate the Middle Frame
Present

Ideally (to be “free”)

Reuse as Much as Possible
Velocity buffers
Previous frames @

Typical Rendering Pipeline

Depth Buffer (~2-3 ms)
Shadows, Lighting (deferred), SSAO
Main Color Pass

Modified Rendering Pipeline

Depth Buffer (~2-3 ms)
Render Velocity / Interpolate / Schedule

One-frame Based Solution

Pros
No extra latency
Little extra memory
Cons
Issues with shadows and reflections
Manageable
Harder

Very Basic Interpolation

Pn-½ = f( Pn-1,-½·Vn )
Sample half way backwards
Previous

Original

I

Interpolated
Very basic interpolation

II

Very Basic Interpolation III

Based on Current Velocity Pn-½ = f( Pn-1, -½·Vn

Camera Motions (TFU2) I

3rd Person Camera

Translation

Camera Motions (TFU2) II

3rd Person Camera

Translation

Rotation

Artifact Minimization

Static Geometry
Velocity buffer filtering
Dynamic Geometry (Characters)
Render separately, re-render completely or

Two-frame Based Solution

Pros
Smoother shadows, alpha, reflections
More real data to handle overlaps
Cons
Extra

Modified Velocity Rendering

Re-render Characters
At interpolated positions
Use other Channels
To help with interpolation

Velocity Merging

Very Simple and Efficient
Vmerged = Vn-1 if ( Zn-1 +

Original

I

Interpolated
Velocity Merging

II

Character Removal I

Visual System’s Blind Spot Inspired

Character Removal I

Visual System’s Blind Spot Inspired
Synthesize the Interior
Assuming That
Horizontal remains horizontal
Vertical

Character Removal II

Generate Mask

Character Removal II

Generate Mask
Leak Neighboring Image Patches

Character Removal II

Generate Mask
Leak Neighboring Image Patches
Duplicate and offset Up

Character Removal II

Generate Mask
Leak Neighboring Image Patches
Duplicate and offset Up Down

Character Removal II

Generate Mask
Leak Neighboring Image Patches
Duplicate and offset Up Down Left

Character Removal II

Generate Mask
Leak Neighboring Image Patches
Duplicate and offset Up Down Left Right
Merge all layers

Character Removal III

Repeat N Times
Blur the Interior Slightly

original mask pass

Character Removal IV

Original Synthesized
3 passes
@ 640x360
XBox360 ~ 0.4ms
PS3 ~

Improved Interpolation

Based on Vmerged
Sample half way backwards
Previous real frame Pn-1
Synthesized frame

Original

I

Interpolated
Velocity Merging
and
Character Removal

II

Next Original
In case of One-frame based
Interpolation it is Unknown

III

Typical Flipping

Flip Normal Buffer
As soon as VBlank is received
Immediately, if VBlank

Modified Flipping

Flip Interpolated Frame Based on Time
Normal conditions
Frame rate drops

Custom flipping on XBox360

Bypass Direct3D Present with...
Asynchronous Swaps
Multiple front buffers
SwapCallback, VerticalBlankCallback
Manual

Real-time Implementation

One-frame Based
Interpolation of Dynamic Objects
Character removal
Performance @ 1280x720
XBox360 ~ 1.5

Motion Blur In TFU2

Current Solution
Alpha “friendly”, simple edge condition
Better than plain

Future Work

Transparency Customizations
Re-render lightsabers, HUD
Multi-layer interpolation
Up-scale from Lower Frame Rates
E.g. render

Acknowledgments
Szymon Swistun Ruslan Abdikeev Cory Bloyd Cedrick Collomb Axel Wefers