2.1

AR displays

Although an AR system may incorporate other senses than the sense of sight, vision is generally considered to be the main sense addressed by almost all AR systems. Thus a display which allows presenting some sort of computer generated content to a user is usually mandatory and fundamental for an AR system. Generally it is possible to categorize most systems into visible-see-through (VST) display or optical-see-through (OST) display based systems. Both principles have their specific advantages but also disadvantages.

OST displays allow a direct view of the real world. The superimposition is achieved via a translucent display. The natural view on the world remains mostly intact, which is usually preferable for critical applications like car navigation or surgery aids. With an OST display it is not trivial to cover bright real objects with darker virtual content [3]. OST based displays are more complicated due to the need for calibration, as every user has a different anatomy. The missing possibility to directly check the outcome and quality of the augmentation process, as the system has no direct feedback. It is rather the user that can evaluate the superimposition process and thus is usually required to perform a calibration process in the beginning. As image processing and tracking cannot perform in real-time, delay issues are common for OST displays. Fast head rotations or fast object movements lead to the problem that the virtual superimposed objects are not correctly aligned and are shown not at the correct position but where the real object was shortly before. The illusion of a coexisting real and virtual world quickly falls apart with an incorrect image registration.

VST displays use a video imaging system that captures the real environment, whose images are processed and superimposed before the user perceives them on some kind of a display system. The user is incapable to perceive the real world directly while using a VST display, which could raise possible security issues and hazards with certain applications. VST are better suited for Mediated Reality applications as it is easier to remove unwanted objects from a video image then try to occlude them with an OST display. Delays are also present but don’t represent such a big problem as the video stream and rendered overlay can be synchronized easily. With VST systems the user perceives the reality by the use of a video system, thus the resolution of the camera and the display determines the possible visual impression. It is reasonable to assume that no technology will recreate a natural visual impression in the near future and thus VST systems will lack realism in this field. In comparison to OST systems, where the system has no direct feedback, VST systems can use computer vision methods, independently of the used tracking technique, to analyze the video image directly and to verify the outcome of the superimposition, which gives them the possibility to automatically perform necessary corrections to the image registration process.

AR displays can also be further categorized by the following characteristics:

These characteristics generally derive from or rather relate to the type of the application. Mobile devices like smartphones or tablet computers with integrated cameras can be easily used as mobile VST AR systems as they include all necessary components to run AR applications. Those almost ubiquitous devices are fast enough to allow tracking, content rendering and image registration in real-time.

Head-mounted OST or VST devices are becoming smaller and smaller and are thus better suited for everyday use. On the other hand head-mounted devices are still uncommon and attract attention. So a breakthrough cannot be expected until the devices are as unobtrusive as regular glasses. Stationary AR displays are more common in industry projects or tech demos like e.g. Microsoft’s Holodesk, where a user sees through a stationary half transparent OST and can interact with real and virtual objects [4].

2.2

Content rendering and image registration

AR applications are usually meant to visualize some kind of data or information that is related to the real scenery. E.g. a car navigation system showing the planned route and highlighting important navigation details. Additionally the car computer system could emphasize important information (augmented vision), like the current speed, outdoor temperature and additional road users. On the other hand medical AR applications are meant to visualize completely different information. Thus it is clear that depending on the aim of the application, not only the setup and implementation but also the augmented content varies a lot.

For our virtual lens experiment application we needed to simulate simple lenses, light sources and a bunch of light rays passing through. Although our simulation was not very sophisticated it was sufficient to allow the students to understand the underlying physical principles. The same applies to the processed and visualized data for any AR system.

Generally the content needs to be visualized three-dimensionally in order to be superimposed perspectively correct on the real world view (Figure 2).

Figure 2

Visualization of a successful and failed image registration of a superimposed cube on top of an ARToolkit marker.

This so called image registration process depends heavily on the quality and precision of the tracking. For a realistic looking outcome it might also become necessary to not only consider the positional and orientational tracking. Light conditions, shadows and occlusion of real and virtual elements does matter as well to get a nice and realistic augmentation. Different focus planes of real and virtual elements could interfere with the user’s experience. With video based AR systems the resolution, color temperature and motion blur could become an issue. With optical-see-through displays transparency, occlusions and the contrast of bright and dark elements become a matter. All current stereoscopic head-mounted glasses lack realistic field of depth. Generally the content is presented on one fix depth layer, causing the vergence-accommodation conflict and leading to discomfort [5]. Current research tries to solve this conflict of vergence and accomodation by developing and utilizing volumetric, multi-focal-plane or light-field display technologies. Although single techniques prove to be suitable there are still plenty of unsolved problems [6] [7] [8] [9] [10] [11].

2.3

Tracking

In order to superimpose virtual onto real content, the AR systems needs to be able to match the users viewing perspective onto the scenery and the virtual content that shall be shown. The recognition of space and objects placed in the scenery appears to be trivial and natural to a human being, but is a complex task for a computer system. In order to superimpose realistically virtual images on the view of the real world, AR systems rely on various tracking methods. Tracking allows for a constant determination of the position and orientation of parts of the AR system, the user and/or objects in the environment in relation to each other. Depending on the AR application the requirements for the tracking system differ significantly, but generally the tracking should have the following ideal characteristics:

It is possible to categorize tracking systems by the means of their principal tracking concept into Outside-In and Inside-Out systems. E.g. a system that uses a marker placed on a head-mounted-display that is captured with one or multiple cameras placed in a room is categorized as Outside-In. The camera images are used to determine the position and orientation of the marker and thus the display on the users head. The objects are basically tracked from the “outside”.

Inside-Out systems reverse this principle. The objects are equipped with sensors that allow the objects to be tracked from the “inside”. E.g. the head-mounted-display could be equipped with camera sensors capturing markers that are placed within the room. Beside these basic categorization it is possible to further categorize tracking systems on the basis of the used physical principle. Tracking systems are based on:

In the field of AR especially the optical tracking methods are popular. Mobile handheld AR applications mostly rely on printed marker, GPS position, magnetic- and inertial-sensing. Recently Microsoft Hololens and Google Project Tango entered public beta phase, which are relying on markerless optical tracking methods [12] [13].

Positional and orientational tracking

The basic functionality of the LEAP motion is the ability to track a user’s hands. The position and orientation of the hand and its fingers are being tracked and can be directly mapped onto a virtual skeleton model. The SDK provides easy access to data of all fingers and its bones (Figure 5).

Figure 5

LEAP motion SDK: Skeletal tracking model [25].

It is therefore straight forward to anticipate using the tracking data to control a virtual representation of the user’s hand (Figure 6). The user’s hands can be visualized in VR applications, to increase the presence in the virtual world. In addition to the tracked data it is possible to fade-in the image data of the LEAP motion IR-camera.

Figure 6

Virtual and real hand of a user interacting with a VR demo [27].

Gesture recognition

To improve interaction possibilities the LEAP SDK is able to detect hand gestures. Gestures can be used to trigger events or can be combined with the positional and orientational tracking data. The SDK knows the following four basic gestures (Figure 7):

Figure 7

Basic LEAP motion gestures (f.l.t.r circle, swipe, key tap, screen tap) [28]

Further the LEAP motion software is able to detect pinching and grabbing gestures that can be used context-sensitive to directly manipulate the virtual elements. E.g. it is possible to grab, rotate and reposition objects. Two-handed pinch-gestures can be used to allow zoom actions, similar to the well-known zoom-pinch gestures on touchscreens, or to select and manipulate virtual dial- or slider-controllers.

These gestures are relatively simple and easy to learn. In order to be understood by the user they need to be explained first. It is considered good practice, to introduce the key concepts of the user interface with an interactive tutorial and to give the user in-app cues and hints. The current state of the application and the next possible actions should always be clear to a user. User interfaces should be designed to be forgiving in the case of operating errors. Possible errors should be made impossible or if not avoidable there should be an undo option.

The LEAP motion app “Blocks” demonstrates the design principles of a VR gesture based UI by the means of a simple game (Figure 8). The app incorporates all important elements for a pleasant user experience. A short interactive tutorial explains the interaction possibilities to the user. Immersed in the virtual world the user can see a virtual set of hands and is able to create and interact with virtual blocks. The UI is gesture based and visual cues allow the user to interact absolutely intuitively [31].

Figure 8

Screenshot from LEAP motion VR game Blocks demonstration [31].

Another very interesting project is “AR screen”. The application is also based on a VR HMD and a LEAP motion successor prototype called “Dragonfly”. Unlike the LEAP motion, the Dragonfly controller is able to capture high-resolution color images and thus converting a VR HMD into a VST AR HMD. The AR screen application demonstrates the extension of a common Windows desktop into AR (Figure 9) [32].

Figure 9

Screenshot from "AR screen" demonstration [32].

Tool tracking

The SDK also allows the tracking of pen-shaped tools. These tools can be used to perform arbitrary tasks in the air. Due to the uniformity of a pen it is not possible to track longitudinal rotations, although some people managed to work around that restriction (Figure 10).

Figure 10

LEAP motion tool tracking [29]

Unfortunately there is no official support yet that allows a tracking of custom defined objects that could be used as additional tools. This restriction probably arises due to the used algorithm, which prioritizes fast hand tracking but lacks the flexibility to detect custom objects [26]. Also the newest VR capable SDK doesn’t support tool tracking yet [29]. A short video from Michael Quigley demonstrates the tracking of a custom extended pen-shaped medical surrogate tool [30]. Interestingly enough the tool has been equipped with an extension that allows the additional tracking of longitudinal rotations of the tool.

A virtual representation of the tool can be seen on a regular computer display (Figure 11). More sophisticated simulations including more realistic interaction could help to visualize the human anatomy and to understand surgeries better. The same technique could be applied for various other educational simulators. A similar functionality can also be realized with VST AR systems and the current LEAP motion controller. The project VoxelAR (Figure 12) demonstrates a hybrid AR interface for finger-based interaction and occlusion detection. An additional webcam is used to capture a color image of a user’s hand. The system was used for physical rehabilitation [33].

Figure 11

Tracking of a pen-shaped tool with an extension [30].

Figure 12

VoxelAR hybrid AR interface [33].