System Quality
Resume:
Remote Reality Demonstration
Terrance E. Boult
EECS Dept., 19 Memorial Drive West
Lehigh Univ., Bethlehem, PA 18015
******@****.******.***
Fax: 610-***-****
Contact Author: T.Boult
Submission category: Technical
Thirty Word Summary (not including title) :
Remote Reality Demonstration
Discussion/demonstration of system combining eld-of-view full-motion video with frame-rate warping
and a HMD to create a geometrically correct, model-free, immersive environment.
Thirty Word Summary (not including title) :
Remote Reality Demonstration
Discussion/demonstration of system combining eld-of-view full-motion video with frame-rate warping
and a HMD to create a geometrically correct, model-free, immersive environment.
Remote Reality Demonstration
Terrance E. Boult, Lehigh Univ.
******@****.******.***
Abstract mately $5K and the computing/HMD play-back system
was about $3K. The system uses a 233Mhx based com-
Remote Reality is an approach to providing an immer-
puter (running Linux) & real-time frame capture card.
sive environment via omni-directional imaging. The sys-
The system computes monocular CIF-resolution full-rate
tem can use a live video-feed from a remote location or
video (30 fps) (25fps PAL). This is reasonably well
can use recorded data and be remote in both space and
matched to the resolution of the Virtual I-O glasses HMD.
time. While less interactive than traditional VR, remote
The HMD has a built in head tracker which provides
reality has an important advantages: there is little to no
yaw,pitch an roll, with updates to the viewing direction
need for model building. In addition, the objects, the tex-
at 15-30fps. The system provides (digital) zoom and
tures and the motions are not just realistic, they are re-
color/contract manipulation. A mouse or joystick can
mote views of reality.
also be used for view slection. We are currently adding
System Overview GPS localization to the collection system to better sup-
The development of the RR system was made possi- port augmented reality applications.
ble by recent research, [Nayar-1997], which revolution- Because the camera is compressing a viewing hemi-
ized wide- eld of view imaging by introducing an omni- sphere (or larger), maintaining resolution and captured
directional sensor a system that images a full hemi- image quality is quite important. While the process
sphere while maintaining a single perspective viewpoint. scales to any size imager, the current system uses NTSC
Unlike sh-eye lenses, each image in the new system can (640x480) or PAL (756x568) video cameras. The spa-
&$ " !
%# pixels
be processed to generate geometrically correct perspec- tial resolution along the horizon is degree or 4.2
$$'
%(
tive images in any direction within the viewing hemi-
pixels
degree (5.1 for PAL).
sphere. Unlike Quicktime VR, it images the full viewing
hemisphere (or more) at once and can hence be used for
Omni-Directional Imaging
full motion video. The current system provides a viewing
The Omni-directional imager is an catadioptric system
volume of . Placing two systems back-to-back
(lens plus mirros), which combines an orthographic lens
(and using 2 cameras) would allow a true viewing sphere,
and a parabolic mirror. The axis of the parabolic mirror is
i.e. viewing.
paralled to the camera s optic axis. To see how it works,
let us trace rays backwards from the camera. Because the
lens is orthographic, these rays are parallel. By de nition,
rays parallel to the axis re ect off a parabolic surface at
an angle such that, if extended to lines, they would all
meet at the focus of the parabolic surface. (This is why
that point is called the focus). Thus we see that the focus
of the parabolic surface provides a single virtual view-
point which is enabling many novel applications, e.g. see
http://www.eecs.lehigh.edu/research/omni
This single virtual viewpoint is important because it
allows for consistent interpretation of the world in any
viewing direction. To generate a proper perspective im-
age from the paraimage we can place an imaging plane
in a viewing direction and sample image the location
where the rays through the focus would intersect that im-
age. The result can be captured in an inverse warping
table and computed very ef ciently. As the HMD turns,
the viewpoint is stationary and only the imaging plane
is moved providing for a consistent and smooth view vari-
The main components of the system are the omni-
ation.
directional camera, video recording systems, car mount-
ing bracket and a head-mounted-display (HMD). (See
[Nayar, 1997] S. K. Nayar. Catadioptric Omnidirectional Video
www.cyclovision.com for commercial Para-cameras with
Camera. Proc. of DARPA Image Understanding Workshop,
basic WindowsNT software.)
May 1997.
The current prototype system was designed for min-
imal cost while maintaining acceptably quality. In to-
tal, our current data collection system was approxi-
Generating a Perspective View
Camera The inverse warping table is generated by
)))
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
tracing rays back (via the virtual viewpoint)
)))
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
to the appropriate location in image space.
Lens
)))
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
)0 )0 )0
G
en
er
Omni-Cameras capture at
Captured Image ed
a 360x210 view in Im
Reflected Rays
the parabolic mirror ag
e
are parallel
with all rays having
a single point of
projection
Remote Reality via Omni-Directional Imaging
s on
ay ecti
t R ir
d en ry d
ci e
In ev
m
fro
For each video frame, the warping table
Ray paths is used to sample the captured image
if not
reflected
Figure 1: Omni-directional Imaging Geometry
Video Camera/Camcorder
Camcorder
Lens system (Orthographic)
12 12 12 12
Support rods
555
555
777
56 56 5
777
66
78 78 7
88
PC with FrameGrabber
Parabolic Mirror
HMD
AA
BABA
BB
Optional GPS/INS audio decoder
Car Mount with suction cups
999
@9@9@9
@@@
Optional GPS/INS audio or video encoder
Remote Reality Playback
Remote Reality Recorder
Playback System Diagram
Video Callback Tracker Callback Audio Callback
(optional)
30FPS Frame Capture Get HMD orientation change * Decode Audio GPS/INS signal
Figure 2: Remote Reality System Diagram
Update View/Warping tables
* Decode video GPS/INS signal
* Note at most one GPS/INS
encoding is used, not both.
Display routine
Use warp tables to generate 320x240 view from current frame
Use GPS/INS to overlay map/label information
Display on HMD
Hardware/Software needs:
The system needs PC s, at least K6233 in power, running Linux (though we might have nished the windows Port by July). We
use Virtual I-O glasses with head trackers. If needed I will bring my own PC s and 3 sets of Head Mounted Display.
During the presentation I will need a VGA compatible projector, hopefully one that can sync at at resolution of both 640x480 and
320x240. (I can tune to particular frequencies if it supports anything close).
I would be very, very, interested in taking part in the Creative Applications Lab. The sketch will explain how it works, but the
system needs the immersive nature of the HMD to appreciate what it can do. The participation would be a running Remote Reality
system using either video-taped tours (I ll bring a few) or a live camera setup somewhere at SIGGRAPH. Again I can bring my own
hardware (3 units) if necessary.
I would be interested in taking part in the SIGGRAPH TV Program. There are a number of things we could do with omni-
directional imaging. The simplest is that we could set up a Omnicamera web-tv site where remote viewers could each control a
virtual pan-tilt camera sitting somewhere at the SIGGRAPH conference. This uses a simple Java client and push technology so that
end users don t need anything special.
A second potential SIGGRAPH TV contribution would be to multi-cast on MBONE (or at a lower bandwidth on the general
Internet) an omnivideo and provide a streaming omni-movie-viewer that remote computers users could use to unwarp the image.
(This streaming viewer does not yet exist but is under development and should be ready by May.)
A nal component could be the development of a Remote Reality Video Tape capturing the the SIGGRAPH Experience. This
would involve collecting and editing the video. Trying to capture a normal eye-level view is very challenging because of the
nonstandard orientation of the camera and, more signi cantly, the super-hemispherical view which leaves the camera crew nowhere
to hide.
Contact this candidate