This PDF file contains the front matter associated with SPIE-IS&T Proceedings Volume 7257, including the Title Page, Copyright information, Table of Contents, and Conference Committee listing.

The most recent video coding standard H.264 achieves excellent compression performances at many different bit-rates. However, it has been noted that, at very high compression ratios, a large part of the available coding resources is only used to code motion vectors. This can lead to a suboptimal coding performance. This paper introduces a new coding mode for a H.264-based video coder, using quantized motion vector (QMV) to improve the management of the resource allocation between motion information and transform coeffcients. Several problems have to be faced with in order to get an efficient implementation of QMV techniques, yet encouraging results are reported in preliminary tests, allowing to improve the performances of H.264 at low bit-rates over several sequences.

Adaptive warping methods for motion estimation and reconstruction at the receiver side have been considered recently for low-bit rate video coding. Such approaches are attractive because they are compatible with the existing MPEG and H.264/AVC standards. This paper follows in that path, introducing an improved method for low bit rate coding of video in which a forward-backward warping algorithm is employed. The base algorithm involves an integration of low and high spatial resolution frames where the low resolution frames are dense in time while the high resolution frames are sparse in time. The forward-backward warping algorithm presented in this paper improves the robustness of the original warping algorithm. Since conventional H.264/AVC is based on macroblocks, block artifacts often surface at low bit rates. The proposed approach tends to eliminate this problem at low bit rates, resulting in improved subjective quality and improved PSNR performance relative to H.264/AVC.

In this paper, a multihypothesis prediction scheme for inter frame video coding is proposed. Using a template matching algorithm, motion vectors are derived at the decoder side instead of explicitly coding the motion vectors into the bitstream. Therefore, higher numbers of hypotheses can be used in the averaging process at no additional coding cost. The proposed scheme has been implemented into the H.264/AVC reference software. Simulation results show bitrate reductions compared to H.264/AVC of 7.7% on average for the tested video sequences. It is shown that part of the performance gain is due to rounding effects in H.264/AVC sub-pixel interpolation which can be exploited in the averaging calculation of the proposed multihypothesis prediction. Experiments with an improved interpolation filter for both reference scheme and the proposed scheme still yield bitrate reductions of 4.7% on average.

Rate control is a capital issue in video coding. It allows a regulation of the bitrate out from the encoder, to cope with some network transmission or quality constraints. Scalable Video Coding emerged several years ago as an answer to the growing need of application-adaptable video streams. Although the interest of scalable video coding has been confirrmed by recent studies, it can not be used in practical contexts without proper rate control techniques. In this paper we present a new rate control scheme for scalable video, based on a simple yet attractive bitrate modelling framework called ρ-domain. Our scheme performs accurate rate control on spatial, temporal and quality scalabilities, while maintaining a constant PSNR. Inter layer prediction is also handled effectively.

In this paper, we consider the problem of video decoding for very high resolution image content. Our focus is on future applications, and our emphasis is on the specific problem of entropy decoding. Here, we introduce the concept of an "entropy slice" that partitions a bit-stream into units that can be individually entropy decoded without effecting the reconstruction process. This allows us to parallelize the entropy decoding process of an H.264/AVC decoder with little impact on coding performance. We compare the entropy slice technique to the standard H.264/AVC slice method to parallelization and observe that the proposed method improves coding efficiency. Specifically, compared to the standard slice method, results show an average bit-rate savings of 5.5%. As an additional contribution of this "entropy slice" concept, we also propose the use of a transcoder to convert an H.264/AVC compliant bit-stream to the parallelized entropy slice format. The transcoding operation has the desirable property of allowing highly parallel decoding of current, standards compliant material without affecting the reconstructed image data.

Detailed image structures and visual textures (of stochastic nature) in digital video are difficult to compress efficiently. At medium to low bit rates, texture flattening and blocking artifacts appear, even when using advanced video coding standards such as H.264/MPEG-4 AVC. In this paper, we propose video compression systems to compress stochastic textures by exploiting rank-reduction techniques. In this work, rank reduction is implemented by applying a singular value decomposition and selective transmission of the primary signal components as in principal component analysis. In the low bit-rate range, our implementation shows encouraging results compared to H.264/MPEG-4 AVC, not only in rate-distortion performance, but also in the improved visual quality of the reconstructed videos.

In contrast to conventional video coding, Wyner-Ziv video coders perform simple intra-frame encoding and complex inter-frame decoding using side information. An important process in Wyner-Ziv decoding is the reconstruction process. So far, two main reconstructions have been used: the Maximum a Posteriori (MAP) reconstruction and the Minimum Mean Square Error (MMSE) reconstruction. In this paper, we study the MAP and MMSE reconstructions for both Pixel-Domain and Transform-Domain Wyner-Ziv video coders. We also present a new reconstruction function for Pixel-Domain Wyner-Ziv video coders that adapts to the local accuracy of the side information. Experimental results show that our adaptive reconstruction provides improvements over both the MAP and the MMSE reconstructions.

For object analysis in videos such as in video surveillance systems, the preliminary segmentation step is very important. Many segmentation methods using static camera have been proposed in the last decade, but they all suffer in occurrance of object reflection especially on the ground, i.e. reflected regions are also segmented as foregrounds. We present a new method which detects the border between the real object and its reflection. Experiments show that an outstanding improvement of segmentation results are obtained by removing the reflection part of the over-segmented objects.

The continuous growth of imaging databases increasingly requires analysis tools for extraction of features. In this paper, a new architecture for the detection of traffic signs is proposed. The architecture is designed to process a large database with tens of millions of images with a resolution up to 4,800x2,400 pixels. Because of the size of the database, a high reliability as well as a high throughput is required. The novel architecture consists of a three-stage algorithm with multiple steps per stage, combining both color and specific spatial information. The first stage contains an area-limitation step which is performance critical in both the detection rate as the overall processing time. The second stage locates suggestions for traffic signs using recently published feature processing. The third stage contains a validation step to enhance reliability of the algorithm. During this stage, the traffic signs are recognized. Experiments show a convincing detection rate of 99%. With respect to computational speed, the throughput for line-of-sight images of 800×600 pixels is 35 Hz and for panorama images it is 4 Hz. Our novel architecture outperforms existing algorithms, with respect to both detection rate and throughput

This paper presents new methods for effi9;cient object tracking in video sequences using multiple features and particle filtering. A histogram-based framework is used to describe the features. Histograms are useful because have the property that they allow changes in the object appearance while the histograms remain the same. Particle filtering is used because it is very robust for non-linear and non-Gaussian dynamic state estimation problems and performs well when clutter and occlusions are present. Color histogram based particle filtering is the most common method used for object tracking. However, a single feature tracker loses track easily and can track the wrong object. One popular remedy for this problem is using multiple features. It has been shown that using multiple features for tracking provides more accurate results while increasing the computational complexity. In this paper we address these problems by describing an efficient method for histogram computation. For better tracking performance we also introduce a new observation likelihood model with dynamic parameter setting. Experiments show our proposed method is more accurate and more efficient then the traditional color histogram based particle filtering.

In this paper, we study a flexible framework for semantic analysis of human motion from a monocular surveillance video. Successful trajectory estimation and human-body modeling facilitate the semantic analysis of human activities in video sequences. As a first contribution, we propose a flexible framework that enables automatic analysis of human behavior and semantic events. It can be utilized in surveillance applications with four-level analysis results. The second contribution is the introduction of a 3-D reconstruction scheme for scene understanding. The total framework consists of four processing levels: (1) a pre-processing level including background modeling and multiple-person detection, (2) an object-based level performing trajectory estimation and posture classification, (3) an event-based level for semantic analysis and (4) a visualization level including camera calibration and 3-D scene reconstruction. Our proposed framework was evaluated and proved its effectiveness as it achieves a near real-time performance (6-8 frames/second).

In this paper, a new method of facial expression recognition is proposed for missing tensor data. In this method, the missing tensor data is estimated by facial expression synthesis in order to construct the full tensor, which is used for multi-factorization face analysis. The full tensor data allows for the full use of the information of a given database, and hence improves the performance of face analysis. Compared with EM algorithm for missing data estimation, the proposed method avoids iteration process and reduces the estimation complexity. The proposed missing tensor data estimation is applied for expression recognition. The experimental results show that the proposed method is performing better than only utilize the original smaller tensor.

We investigate the problem of video multicast in lossy networks using network coding and multiple description codes. The rate allocation for multiple descriptions can be optimized at the source to generate a scalable video bitstream such that the expected PSNR of the video at the receivers is maximized. We show that using network coding with multiple description codes can significantly improve the quality of video obtained at the receiver, in comparison to routing (with or without replication). Simulations show that as the loss rate increases, the improvement in the performance increases and in certain cases for loss rate of 0.20, the improvement can be as high as 3 to 3.5 dB when compared to routing with replication. Moreover, network coding obviates the need to construct multiple multicast trees for transmission, which is necessary in routing with replication.

A new approach is presented for obtaining feature matching based on the identified features from images. Features which are described by high-dimension vectors are first extracted from a set of reference images and stored in a database, then the correspondence between similar features from different images are established by introducing the notion of a Min-cost K-flow Problem (MKP), which consists in finding a min-cost flow subject to the constraint that the flow value is K. The similarity function, which characterizes these vector components, can avoid the errors that come from different metrics of vectors. Finally, the K-flow is checked to reject ambiguous correspondence bi-directionally and automatically in accordance with the ratio of the matching cost. Experiments on three image sets demonstrate encouraging results.

In this paper, we propose an approach to manage network resources for a Direct Sequence Code Division Multiple Access (DS-CDMA) visual sensor network where nodes monitor scenes with varying levels of motion. It uses cross-layer optimization across the physical layer, the link layer and the application layer. Our technique simultaneously assigns a source coding rate, a channel coding rate, and a power level to all nodes in the network based on one of two criteria that maximize the quality of video of the entire network as a whole, subject to a constraint on the total chip rate. One criterion results in the minimal average end-to-end distortion amongst all nodes, while the other criterion minimizes the maximum distortion of the network. Our approach allows one to determine the capacity of the visual sensor network based on the number of nodes and the quality of video that must be transmitted. For bandwidth-limited applications, one can also determine the minimum bandwidth needed to accommodate a number of nodes with a specific target chip rate. Video captured by a sensor node camera is encoded and decoded using the H.264 video codec by a centralized control unit at the network layer. To reduce the computational complexity of the solution, Universal Rate-Distortion Characteristics (URDCs) are obtained experimentally to relate bit error probabilities to the distortion of corrupted video. Bit error rates are found first by using Viterbi's upper bounds on the bit error probability and second, by simulating nodes transmitting data spread by Total Square Correlation (TSC) codes over a Rayleigh-faded DS-CDMA channel and receiving that data using Auxiliary Vector (AV) filtering.

We explore sensor scheduling strategies to maximize the operational lifetime of a user-centric image sensor network. Image sensors are deployed for gathering visual information over a monitored region. Users navigate within this region by specifying a time-varing desired viewpoint and the network responds with the requested visual data. By modeling the user's desired viewpoint in a probabilistic framework, we develop a stochastic formulation of the network lifetime and investigate the camera scheduling strategy that maximizes the expected value of network lifetime. By suitably abstracting the problem, we present a closed-form solution for the simplistic case when the monitored region is divided into two parts. Using asymptotic analysis, we then present a simple camera scheduling strategy for the general case that we conjecture to be optimal. Simulation results demonstrate a clear advantage of the proposed camera scheduling approach over previously considered alternatives.

Due to the limited display capacity of multiview/ automultiscopic 3D displays (and other 3D display methods which recreate lightfields), regions and objects and greater depths from the zero disparity plane appear aliased. One solution to this, namely prefiltering renders the scene visually very blurry. An alternative approach is proposed in this paper, wherein regions are large depths are identified in each view. The 3D scene points corresponding to these regions is rendered as 2D only. The rest of scene still retains parallax (hence the depth perception). The advantages are that both aliasing and blur are removed, and the resolution of such regions is greatly improved. A combination of the 2D and 3D visual cues still make the scene look realistic, and the relative depth information between objects in the scene is still preserved. Our method can prove to be particularly useful for the 3D video conference application, where the people in the conference will be shown as 3D objects, but the background will be displayed as a 2D object with high spatial resolution.

Reproducing a natural and real scene as we see in the real world everyday is becoming more and more popular. Stereoscopic and multi-view techniques are used for this end. However due to the fact that more information are displayed requires supporting technologies such as digital compression to ensure the storage and transmission of the sequences. In this paper, a new scheme for stereo image coding is proposed. The original left and right images are jointly coded. The main idea is to optimally exploit the existing correlation between the two images. This is done by the design of an efficient transform that reduces the existing redundancy in the stereo image pair. This approach was inspired by Lifting Scheme (LS). The novelty in our work is that the prediction step is been replaced by an hybrid step that consists in disparity compensation followed by luminance correction and an optimized prediction step. The proposed scheme can be used for lossless and for lossy coding. Experimental results show improvement in terms of performance and complexity compared to recently proposed methods.

Stereoscopic video is a practical and important manner for 3-D video applications, and robust stereoscopic video transmission over error-prone networks has posed a technical challenge for stereoscopic video coding. In this paper, we present a rate-distortion optimization algorithm with inter-view refreshment for error-resilient stereoscopic video coding. First, inter-view refreshment is proposed to suppress error propagations besides intra refreshment. Then, we propose an end-to-end distortion model for stereoscopic video coding which concurrently considers network conditions, inter-view refreshment, and error concealment tools. Finally, based on the proposed end-to-end distortion model, a rate-distortion optimization algorithm is presented to adaptively select inter-view, inter and intra coding modes for error-resilient stereoscopic video coding. Simulation results show that the proposed scheme has a superior transmission efficiency improvement for stereoscopic video coding.

We present a novel analysis of encoding latency for arbitrary multiview video coding prediction structures. This analysis allows us to compare the efficiency of multiview prediction structures beyond rate-distortion performance, i.e., it makes it possible to select those structures that are more amenable to low latency encoding. As a result, we have developed a general framework (models and tools) for the characterization of latency that can be used for arbitrary dependency structures. Finally, we have focused on the JMVM prediction structure with the aim of deriving new structures with similar rate distortion performance and lower latency. Experimental results help us illustrate the importance of taking into consideration encoding latency when selecting a prediction structure.

Depth maps are used in many applications, e.g. 3D television, stereo matching, segmentation, etc. Often, depth maps are available at a lower resolution compared to the corresponding image data. For these applications, depth maps must be upsampled to the image resolution. Recently, joint bilateral filters are proposed to upsample depth maps in a single step. In this solution, a high-resolution output depth is computed as a weighted average of surrounding low-resolution depth values, where the weight calculation depends on spatial distance function and intensity range function on the related image data. Compared to that, we present two novel ideas. Firstly, we apply anti-alias prefiltering on the high-resolution image to derive an image at the same low resolution as the input depth map. The upsample filter uses samples from both the high-resolution and the low-resolution images in the range term of the bilateral filter. Secondly, we propose to perform the upsampling in multiple stages, refining the resolution by a factor of 2×2 at each stage. We show experimental results on the consequences of the aliasing issue, and we apply our method to two use cases: a high quality ground-truth depth map and a real-time generated depth map of lower quality. For the first use case a relatively small filter footprint is applied; the second use case benefits from a substantially larger footprint. These experiments show that the dual image resolution range function alleviates the aliasing artifacts and therefore improves the temporal stability of the output depth map. On both use cases, we achieved comparable or better image quality with respect to upsampling with the joint bilateral filter in a single step. On the former use case, we feature a reduction of a factor of 5 in computational cost, whereas on the latter use case, the cost saving is a factor of 50.

A new fast mode decision (FMD) algorithm for multi-view video coding (MVC) is presented. One of the multiple views is encoded based on traditional methods, which provides a mode decision (MD) map, while encoding of the other views is based on the analysis of the homogeneity of the depth map. This approach reduces the burden of the rate-distortion (RD) motion analysis based on the availability of a depth map, which is assumed to be provided by the acquisition process. Although there is a slight decrease of performance in rate-distortion terms, there is a significant reduction in computational cost.

We consider the effect of depth-image compression artifacts on the quality of virtual views rendered using neighboring views. Such view rendering processes are utilized in new video applications such as 3D television (3DTV) and free viewpoint video (FVV). We first analyze how compression artifacts in compressed depth-images result in distortions in rendered views. We show that the rendering position error is a monotonic function of the coding error. For the scenario in which cameras are arranged with parallel optical axes, we further demonstrate specific properties of rendering position error. Exploiting special characteristics of depth-images, namely smooth regions separated by sharp edges, we investigate a possible solution to suppress compression artifacts by encoding depth-images with a recently published sparsity-based in-loop de-artifacting filter. Simulation results show that applying such techniques not only provides significantly higher coding efficiency for depth-image coding, but, more importantly, also improves the quality of rendered views in terms of PSNR and subjective quality.

The recently standardized Scalable Video Coding(SVC) extension of H.264/AVC allows bitstream scalability with improved rate-distortion efficiency with respect to the classical Simulcasting approach, at the cost of an increased computational complexity of the encoding process. So one critical issue related to practical deployment of SVC is the complexity reduction, fundamental to use it in consumer applications. In this paper, we present a fully scalable fast motion estimation algorithm that enables an excellent complexity performance.

This paper proposed a linear rate-distortion (RD) cost model for skip mode in H.264/MPEG-4 AVC. The proposed RD cost model is derived theoretically based on the quantization scheme of H.264, and the simulation results also verify that the proposed RD cost model can estimate the RD cost of skip mode accurately. Based on the proposed RD cost model, an early skip mode selection algorithm is provided. The proposed early skip mode selection method can terminate the mode selection process adaptively and works well for both low complexity and high complexity video. Experimental results show that the proposed early skip mode selection algorithm can save the encoding time up to 56% with negligible performance loss compared with the original reference encoder.

To acquire the optimal coding mode of each macroblock, the H.264/AVC encoder exhaustively calculates the ratedistortion cost for all available modes and chooses the minimum one as the best mode. Therefore, the mode decision process is very computationally demanding. To reduce the computation complexity of the rate-distortion cost, in this paper, we propose a novel rate estimation model for the mode decision in H.264/AVC. By modeling the transform coefficients with Generalized Gaussian distributions (GGD), a direct relationship between the magnitude and the information bits of the quantized transform coefficients is deduced. Based on this deduction, the weighted sum of quantized transform coefficients is proposed as an efficient bit-rate estimator of the residual blocks. Extensive experiments show that the proposed algorithm can save up to 30% of total encoding time with ignorable degradation in coding performance for both inter- and intra-mode decision.

The H.264/AVC1 standard of the Video Coding Experts Group (VCEG) and the Moving Pictures Experts Group (MPEG), also known as MPEG-4 AVC, achieves significant compression gain compared to its predecessors. Not only Inter yet also Intra coding has been greatly improved. Today VCEG encourages coding efficiency improvements through the KTA (Key Technical Area) software,² a collection of tools that improves the H.264/AVC standard, to prepare the next generation video codec. The work proposed in this paper has been designed in this context. An Intra coding scheme is proposed. A macroblock is split in 1D partitions to reduce the distance between the pixel to encode and its predictors. Three scan orders for the partitions are available: a raster scan, a bidirectional scan and a hierarchical scan. Predictors adapted to the shape and characteristics of the 1D partitions are defined and finally a 1D-DCT is applied to the residual signal. Experimental results report an average bitrate savings of 8.6% compared to the H.264/AVC standard (up to 19% on a particular sequence).

It is known that quality fluctuation has a major negative effect on visual perception. In previous work, we introduced a constant-distortion bit allocation method [1] for H.263+ encoder. However, the method in [1] can not be adapted to the newest H.264/AVC encoder directly as the well-known chicken-egg dilemma resulted from the rate-distortion optimization (RDO) decision process. To solve this problem, we propose a new two stage constant-distortion bit allocation (CDBA) algorithm with enhanced rate control for H.264/AVC encoder. In stage-1, the algorithm performs RD optimization process with a constant quantization QP. Based on prediction residual signals from stage-1 and target distortion for smooth video quality purpose, the frame-level bit target is allocated by using a close-form approximations of ratedistortion relationship similar to [1], and a fast stage-2 encoding process is performed with enhanced basic unit rate control. Experimental results show that, compared with original rate control algorithm provided by H.264/AVC reference software JM12.1, the proposed constant-distortion frame-level bit allocation scheme reduces quality fluctuation and delivers much smoother PSNR on all testing sequences.

This paper develops a new adaptive scanning methodology for intra frame scalable coding framework based on a subband/wavelet(DWTSB) coding approach for MPEG-4 AVC/H.264 scalable video coding (SVC). It attempts to take advantage of the prior knowledge of the frequencies which are present in different higher frequency subbands. We propose dyadic intra frame coding method with adaptive scan (DWTSB-AS) for each subband as traditional zigzag scan is not suitable for high frequency subbands. Thus, by just modification of the scan order of the intra frame scalable coding framework of H.264, we can get better compression. The proposed algorithm has been theoretically justified and is thoroughly evaluated against the current SVC test model JSVM and DWTSB through extensive coding experiments for scalable coding of intra frame. The simulation results show the proposed scanning algorithm consistently outperforms JSVM and DWTSB in PSNR performance. This results in extra compression for intra frames, along with spatial scalability. Thus Image and video coding applications, traditionally serviced by separate coders, can be efficiently provided by an integrated coding system.

Content-based image retrieval using a Scalable Vocabulary Tree (SVT) built from local scale-invariant features is an effective method of fast search through a database. An SVT built from fronto-parallel database images, however, is ineffective at classifying query images that suffer from perspective distortion. In this paper, we propose an efficient server-side extension of the single-view SVT to a set of multiview SVTs that may be simultaneously employed for image classification. Our solution results in significantly better retrieval performance when perspective distortion is present. We also develop an analysis of how perspective increases the distance between matching query-database feature descriptors.

This paper presents a system for automatic image selection for storytelling applications, like slideshows and photobooks, where targeting a specific image count is usually of high importance. A versatile image collection representation is introduced, which allows for automatic scalable selection in order to target a specific final image count, while preserving a good coverage of the event in order to maintain the storytelling potential of the selection. A hierarchical time clustering is presented, which is traversed at a specific hierarchy level in order to select images by alternating among all time clusters, and selecting the most relevant images in that cluster. The relevance ordering we use is based on a combination of features, namely, important people, smile detection, image appeal measures, and whether a nearduplicate of the image has already been selected. Once this Hierarchical Scalable Representation has been created, it can be reused to generate any target size selection. Two automatic image selection algorithms have been implemented, one that selects images from clusters with high average image relevance more frequently, and another one that selects images from larger clusters more frequently. The overall system has been used over the last year on several large image collections; the resulting selection was presented to their owners in the form of photo-books in order to get feedback, validating the presented approach.

In recent years, a new technique called High Dynamic Range (HDR) has gained attention in the image processing field. By representing pixel values with floating point numbers, recorded images can hold significantly more luminance information than ordinary integer images. This paper focuses on the realization of a lossy compression scheme for HDR images. The JPEG2000 standard is used as a basic component and is efficiently integrated into the compression chain. Based on a detailed analysis of the floating point format and the human visual system, a concept for lossy compression is worked out and thoroughly optimized. Our scheme outperforms all other existing lossy HDR compression schemes and shows superior performance both at low and high bitrates.

One major challenge in image compression is to efficiently represent and encode high-frequency structural components in images, such as edges, contours, and texture regions. To address this issue, we propose a scheme to learn local image structures and efficiently predict image data based on this structure information. We prove that, using singular value decomposition, we can find a small number of basis vectors whose linear combinations are able to closely approximate local image patches. By extrapolating these linear combination coefficients, we can efficiently predict neighboring pixels of the local image patch. We find that this structure learning and prediction procedure is very efficient for image regions with significant structural components. However, because of its large overhead and high computational complexity, its performance degrades in other image regions, such as smooth areas. Therefore, we propose a classification scheme to partition an image into three types of regions: structure regions, non-structure regions, and transition regions. Structure regions are encoded with structure prediction. With image in-painting, the non-structure and transmission regions are extended into a maximally smoothed image which can be efficiently encoded with conventional image compression methods, such as JPEG2000. Our experimental results demonstrate that the proposed method outperform the state-of-the-art JPEG2000 image compression.

In this work, we develop a new framework for image and video compression. In traditional frameworks, images are considered 2-D arrays of pixels and encoded using spatiotemporal prediction and transform. Here, we consider an image as an orderless set of atom units, called structurelets, which satisfy a set of imaging constraints. We use lossless image compression as an example to demonstrate how the image encoder and decoder can be designed. Our experimental results demonstrate that the proposed algorithm outputs existing methods for lossless image compression.

We investigate transform coding to efficiently store and transmit SIFT and SURF image descriptors. We show that image and feature matching algorithms are robust to significantly compressed features. We achieve near-perfect image matching and retrieval for both SIFT and SURF using ~2 bits/dimension. When applied to SIFT and SURF, this provides a 16× compression relative to conventional floating point representation. We establish a strong correlation between MSE and matching error for feature points and images. Feature compression enables many application that may not otherwise be possible, especially on mobile devices.

In this paper a multiple description coding (MDC) that uses more than one multiple description scalar quantizer (MDSQ) is presented. The MDSQ merging constraints for joint decoding of descriptions obtained using side quantizers from different MDSQs are presented. In this way, the bit streams from the side quantizers resulting in significantly different rate-distortion properties are jointly decoded as opposed to that of two bit streams of similar rate-distortion performances obtained using a single MDSQ. Rate distortion performances of joint decoding of different description combinations obtained from multiple MDSQs outperforms those obtained from a single MDSQ. Robustness of the proposed approach evaluated under packet erasure channel is higher than that of the conventional approach. In addition to high robustness, we also demonstrate how this scheme is extended to successive side quantizer bin merging that is used to achieve robust quality scalability within an MDC framework.

In this paper, an auto-regressive (AR) model is proposed to generate the side information for low-delay distributed video coding (DVC). The side information generation of current Wyner-Ziv (WZ) frame t consists of two forward AR interpolations. First, each pixel within the rebuilt frame t−1 is approximated as a linear combination of pixels within a spatial neighborhood along the motion trajectory within the rebuilt frame t−2. Applying the least mean square algorithm, the coefficient of the first forward AR model is derived. Secondly, the pixels within the rebuilt frame t−2 are approximated by the corresponding pixels within rebuilt frame t−1. And then the geometric symmetric property of the AR model is exploited to derive the coefficient of the second forward AR model. Finally, the side information is generated as the average of the interpolations obtained by the two forward AR interpolations. The experimental results have demonstrated that the proposed AR model can significantly improve the PSNR of the side information compared to existing motion extrapolation based approaches.

Packet scheduling is of great importance to the performance optimization of multi-session video streaming over wireless mesh networks. In this paper, we explore the time-varying characteristics of the input videos and propose a Content-and-Deadline-Aware Scheduling (CDAS) scheme for multi-session video streaming over wireless multi-hop networks. The basic idea of proposed scheduling scheme is to choose and transmit more packets with higher importance while meeting their stringent delay constraints, to maximize the constructed video quality at the receivers' side. More specifically, the packet schedule is determined in such a way that not only the backlog, but also the contributions to the reconstructed videos as well as the stringent delay constraints and time-varying network condition are all considered. This is enabled by our proposed priority model composed of two major components: content priority and scheduling priority. For the content priority, we develop a fast and efficient packet-level transmission distortion model to accurately predict the corresponding distortion for a lost packet at the encoder side. For scheduling priority, we consider delay requirement and dynamic network conditions of subsequent transmission links. Our extensive simulations demonstrate that the proposed transmission distortion model and the CDAS policy significantly improve the performance of multi-session video streaming over wireless mesh networks.

We consider rate-distortion (RD) optimized multi-flow video delivery in unstructured overlay networks. We show that this problem can be studied as a distributed rate allocation. To solve the problem over the participating peers in the overlay, we apply classical decomposition techniques such that the network-wide utility of video distortion is minimized. Media packets are assumed to be piggy-backed with RD preambles that contain information regarding their impact on decoder video distortion and their size. This allows for converting the calculated optimal rate allocation at every node into simple forwarding or dropping actions. Furthermore, the proposed distributed media streaming framework employs a network inference algorithm for minimizing the flow of duplicate packets over the network and utilizing thus more efficiently the available resources. Our simulation results indicate that significant quality benefits can be achieved when the precise RD characteristics of a media presentation are taken into account.

Most consumer digital color cameras capture video using a single chip. Single chip cameras do not capture RGB triples for every pixel, but a subsampled version with only one color component per pixel (e.g. Bayer pattern). Conventionally, a full resolution video is constructed from the Bayer pattern by demosaicing before being converted to YUV domain for compression. In order to lower the encoding complexity, we propose in this work a novel color space conversion in the pre-processing step. Compared to the conventional method, the proposed scheme reduces the encoding complexity almost by half. Moreover, it improves the reconstructed video quality by up to 1.5 dB in CPSNR, when H.264/AVC is used for compression. To further lower the encoding complexity, we additionally use our Wyner-Ziv video coder for compression. Again, we observe in our experiments a similar gain of the proposed scheme over the conventional one.

This paper addresses a fundamental problem in computer vision, curve matching. Curve matching and comparison play a key role in various applications. High-level vision problems usually require comparing curves, and the quality of tackling these problems relies much on the underlying curve matching techniques. Our goal is to define a distance on the space of plane (space) curves. The space of curves is taken as a manifold (topological space), and we consider Riemannian metrics on the manifold. The distance induced by a Riemannian metric is a metric, which, if not trivial, can be used as a similarity metric. This work also deals with the problem of partial curve matching given their starting points are known. Dynamic programming is used to implement partial matching, giving an efficient computational method. Experiments are conducted to test the distance invariant to translation and scaling.

For real-time imaging in surveillance applications, visibility of details is of primary importance to ensure customer confidence. If we display High Dynamic-Range (HDR) scenes whose contrast spans four or more orders of magnitude on a conventional monitor without additional processing, results are unacceptable. Compression of the dynamic range is therefore a compulsory part of any high-end video processing chain because standard monitors are inherently Low- Dynamic Range (LDR) devices with maximally two orders of display dynamic range. In real-time camera processing, many complex scenes are improved with local contrast enhancements, bringing details to the best possible visibility. In this paper, we show how a multi-scale high-frequency enhancement scheme, in which gain is a non-linear function of the detail energy, can be used for the dynamic range compression of HDR real-time video camera signals. We also show the connection of our enhancement scheme to the processing way of the Human Visual System (HVS). Our algorithm simultaneously controls perceived sharpness, ringing ("halo") artifacts (contrast) and noise, resulting in a good balance between visibility of details and non-disturbance of artifacts. The overall quality enhancement, suitable for both HDR and LDR scenes, is based on a careful selection of the filter types for the multi-band decomposition and a detailed analysis of the signal per frequency band.

We first develop a simple method for detecting the local orientation of image contours and then use this detection to design an edge-adaptive image interpolation strategy. The detection is based on total variation: small total variation along a candidate curve implies that the image is approximately constant along that curve, which suggests it is a good approximation to the contours. The proposed strategy is to measure the total variation over a "contour stencil," a set of parallel curves localized over a small patch in the image. This contour stencil detection is used to design an edge-adaptive image interpolation strategy. The interpolation is computationally efficient, operates robustly over a variety of image features, and performs competitively in a comparison against existing methods. The method extends readily to vector-valued data and is demonstrated for color image interpolation. Other applications of contour stencils are also discussed.

This paper presents a new adaptive Boxcar/Wavelet transform for image compression. Boxcar/Wavelet decomposition emphasizes the idea of average-interpolation representation which uses dyadic averages and their interpolation to explain a special case of biorthogonal wavelet transforms (BWT). This perspective for image compression together with lifting scheme offers the ability to train an optimum 2-D filter set for nonlinear prediction (interpolation) that will adapt to the context around the low-pass wavelet coefficients for reducing energy in the high-pass bands. Moreover, the filters obtained after training is observed to posses directional information with some textural clues that can provide better prediction performance. This work addresses a firrst step towards obtaining this new set of training-based fillters in the context of Boxcar/Wavelet transform. Initial experimental results show better subjective quality performance compared to popular 9/7-tap and 5/3-tap BWTs with comparable results in objective quality.

In this paper, we present a spatially adaptive method to reduce compression artifacts observed in block discrete cosine transform (DCT) based image/video compression standards. The method is based on the bilateral filter, which is very effective in denoising images without smoothing edges. When applied to reduce compression artifacts, the parameters of the bilateral filter should be chosen carefully to have a good performance. To avoid over-smoothing texture regions and to effectively eliminate blocking and ringing artifacts, in this paper, texture regions and block boundary discontinuities are first detected; these are then used to control/adapt the spatial and intensity parameters of the bilateral filter. Experiments show that the proposed method improves over the standard non-adaptive bilateral filter visually and quantitatively.

A new stabilization algorithm to counterbalance the shaking motion in a video based on classical Kandade-Lucas- Tomasi (KLT) method is presented in this paper. Feature points are evaluated with law of large numbers and clustering algorithm to reduce the side effect of moving foreground. Analysis on the change of motion direction is also carried out to detect the existence of shaking. For video clips with detected shaking, an affine transformation is performed to warp the current frame to the reference one. In addition, the missing content of a frame during the stabilization is completed with optical flow analysis and mosaicking operation. Experiments on video clips demonstrate the effectiveness of the proposed algorithm.

A novel trained filter based scheme for video de-interlacing is proposed and described in detail. This scheme uses different classifiers, called error functions, on the input, and mixes several sub-de-interlacers depending on them. The approach differs from the earlier works in this area due to focus on more complex classification rather than on complex sub-de-interlacers. The proposed scheme is flexible and allows various combinations of error functions with sub-de-interlacers. In this article we describe a test implementation of this concept with five different sub-de-interlacers and five error functions composing a spatial-temporal de-interlacing method. The description of the test implementation is supported by simulations where we evaluate the contribution of different sub-de-interlacers and error function to output de-interlacing quality.

Occlusion detection is an essential ingredient in high quality picture rate up-conversion and view interpolation applications. Many integrated approaches to occlusion detection and classification have been proposed, particularly in the stereo literature. However, due to their high complexity and 1-dimensional nature (baseline stereo), few of them are suitable for real-time use in picture rate up-conversion. This paper reviews fast, deterministic methods for occlusion detection and proposes a new method, suitable for real-time use in motion compensated picture rate up-conversion.

In this paper we present a noise reduction filter for video processing. It is based on the recently proposed two dimensional steering kernel, extended to three dimensions and further augmented to suit the spatial-temporal domain of video processing. Two alternative fillters are proposed - the time symmetric kernel and the time asymmetric kernel. The first reduces the noise on single sequences, but to handle the problems at scene shift the asymmetric kernel is introduced. The performance of both kernels is tested on simulated data and on a real video sequence together with the original steering kernel. The proposed kernels improves the Rooted Mean Squared Error (RMSE) compared to the original steering kernel method on video material significantly.

H.264 main profile supports two intra modes at macroblock (MB) level: intra_4x4 and intra_16x16. There are nine possible spatial prediction directions per 4x4 block in intra_4x4 mode and four per 16x16 block in intra_16x16 mode. The prediction direction selection per MB in intra_4x4 can be very time-consuming, if all the nine possible prediction directions have to be checked per 4x4 block by a RD-based method. This paper proposes a method that can early detects those 4x4 blocks which select most probable mode (MPM) as the best mode. Once the encoder early detects the MPM as the best mode of one 4x4 block, the encoder can reduce nine RD computations to one and therefore save lots of computation. The saving in selecting a best prediction direction per 4x4 block depends on sequences and quantization parameters, for example, ranging from 16.80% to 60.19% for F1car sequence by using different measures. Simulation results demonstrated that the proposed method significantly reduce the processing time without performance degradation.

An online combined feature evaluation method for visual object tracking is put forward in this paper. Firstly, a feature set is built by combining color histogram (HC) bins with gradient orientation histogram (HOG) bins to emphasize the color representation and contour representation of an object respectively. Then a feature confidence evaluation approach in a Particle Filter framework is proposed to make that features of larger confidence can play more important roles in the instantaneous tracking, ensuring that the tracking can adapt to the appearance changes of either foreground or background. In this way, we extend the traditional filter framework from modeling motion states to modeling feature evaluation. The temporal consistency of particles can also ensure that the evolution of feature confidence is always gentle. Examples are presented to illustrate how the method adapts to changing appearances of both tracked object and background. Experiments and comparisons demonstrate that object tracking with evaluated combined features are highly reliable even when objects go across complex backgrounds.

Many reversible watermarking solutions have been proposed since 1996: spread-spectrum approaches, circular interpretation on histogram approaches, lossless compression approaches, expansion approaches and histogram approaches. In this paper, we propose a solution whose embedding capacity goes beyond all those reversible schemes. For certain images, the reach payload of the proposed method is over 2 bpp. This solution is an improvement of the Coltuc reversible watermarking approach published in ICIP'2007.

In this paper, a 3D auto-regressive (AR) model is proposed for bi-directional prediction. The prediction is composed of two 3D AR models, which are along the forward and backward directions, respectively. Applying the 3D AR model, each pixel in the current frame is predicted as a weighted summation of pixels within a spatial neighborhood along the forward/backward motions within the forward/backward reference frames. Ultimately, the prediction of each pixel is obtained as the combination of predictions generated by the two 3D AR models. To derive accurate AR coefficients, this paper proposes a framework that performs simultaneous coefficient estimation and image interpolation. As opposed to other methods, the predicted pixels generated by one 3D AR model are further used to predict the pixels in adjacent frame along the motion trajectory. Consequently, each pixel in one forward/backward reference frame can be predicted as a nonlinear combination of pixels within an enlarged spatial neighborhood along the motion in one backward/forward reference frame. An iterative algorithm using a nonlinear least squares method is then devised to compute the optimum 3D AR coefficients. Various experiments conducted in this paper have confirmed that the proposed method has superior performance for bi-directional prediction.

Even though substantial progress has been made in super resolution research, many issues regarding robust sub-pixel estimation and fast implementation of feature preserving restoration still exist. To obtain more reliable sub-pixel information, we proposed to correct mis-aligned sub-pixels by motion vector (MV) processing based on hierarchical block partition and weighted vector median filtering (WVMF). Two indices - relative displaced frame difference and motion vector similarity degree - are computed and compared with trained thresholds to classify the motion blocks into reliable and unreliable groups. Then the unreliable blocks are divided into four sub-blocks with their motion vector processed by WVMF based on the reliability information of their neighborhood blocks. To preserve the local features such as edge direction, strength as well as its spread region, anisotropic kernels are learned from local gradient fields to represent edge information. Finally, a kernel constrained projection is established for restoring high resolution frames. The experimental results show that the proposed algorithm preserves important features in the images and outperforms the traditional POCS method.

Due to the recent explosion of multimedia formats and the need to convert between them, more attention is drawn to picture rate conversion. Moreover, growing demands on video motion portrayal without judder or blur requires improved format conversion. The simplest conversion repeats the latest picture until a more recent one becomes available. Advanced methods estimate the motion of moving objects to interpolate their correct position in additional images. Although motion blur and judder have been reduced using motion compensation, artifacts, especially around the moving objects in sequences with fast motion, may be disturbing. Previous work has reduced this so-called 'halo' artifact, but the overall result is still perceived as sub-optimal due to the complexity of the heuristics involved. In this paper, we aim at reducing the heuristics by designing LMS up conversion filters optimized for pre-defined local spatio-temporal image classes. Design and evaluation, and a benchmark with earlier techniques will be discussed. In general, the proposed approach gives better results.

Traditional mechanisms for congestion control in multimedia streaming systems reduce the data transmission rate when congestion is detected. Unfortunately, decreasing the rate of the media stream also decreases the media quality, but it is the only way to combat congestion when it is caused by overwhelming traffic that exceeds the capacity of the network. However, if the bottleneck is a wireless link, congestion is often derived from retransmissions caused by bit errors in the radio link. If this is the case, it might be beneficial not to reduce the transmission rate, but allow delivery of packets containing bit errors up to the application layer first. In this scenario, the quality of media will be impacted by bit errors instead of lower coding rate. In this paper, we propose a system concept allowing bit errors in packets in order to relieve congestion. We have built a simulation to compare the performance of the proposed system against traditional congestion control. The results show that the proposed approach can improve the overall performance both by increasing the throughput over the wireless and improving the perceived video quality in terms of peak signal-to-noise ratio (PSNR).

In the framework of Wyner-Ziv Coding of Video, the coding efficiency depends on the quality of the side information (SI) at the decoder, where the side information is constructed from the key frames available at the decoder. In this paper, we propose a novel frame interpolation method for Wyner-Ziv coding, where the motion compensation is bidirectional and allows pixelwise estimation. The proposed interpolation method allows to obtain better SI quality than the one obtained by state-of-the-art interpolation methods.

In this paper, we propose an incremental-based basic unit (BU) level quantization parameter (QP) determination algorithm for H.264/AVC rate control. Unlike traditional BU level QP computation in existing rate control schemes, the proposed algorithm does not perform target bit allocation and predict coding complexities. Instead, it exploits bit increment to directly determine QP for a BU, aiming at reducing the variations of encoding bits used among BUs within a frame and improve subjective visual quality. To better handle buffer fullness and reduce buffer overflow/ underflow, we explore an enhanced Proportional-Integral-Derivative buffer controller. In addition, the algorithm can also effectively intra-code all frames in a video sequence and has low computational complexity making it suitable for real-time applications. Our experimental results demonstrate that, the proposed algorithm outperforms the rate control algorithm JVT-W042, adopted in the recent H.264/AVC reference model JM13.2, by achieving accurate rate regulation, reducing frame skipping, decreasing quality fluctuation, and improving coding quality up to 1 dB.

In this paper, a new method for license plate detection based on AdaBoost is proposed. In the new method, character frequency feature, which is powerful feature for detecting license plate character, are introduced to feature pool. The frequency features obtained from the FFT of horizontal projection of binary image are selected by AdaBoost. Then, Haar-like features selected by AdaBoost are used to capture subtle structure of license plate. Furthermore, considering the characteristic of Chinese license plate that there are two types of license plate: deeper background-lighter character and lighter background-deeper character license plates, two detectors are designed to extract different license plates respectively. Experimental results show the efficiency of the proposed method.

As online image sharing services are becoming popular, the importance of correctly annotated tags is being emphasized for precise search and retrieval. Tags created by user along with user-generated contents (UGC) are often ambiguous due to the fact that some tags are highly subjective and visually unrelated to the image. They cause unwanted results to users when image search engines rely on tags. In this paper, we propose a method of measuring tag confidence so that one can differentiate confidence tags from noisy tags. The proposed tag confidence is measured from visual semantics of the image. To verify the usefulness of the proposed method, experiments were performed with UGC database from social network sites. Experimental results showed that the image retrieval performance with confidence tags was increased.

A novel template design algorithm is presented for lossless compression of color halftone images. First, we extract the line pattern in the neighbor local region of each bi-level pixel. Then, the representative line pattern is evaluated from these obtained lines, using the least square error minimization. According to the evaluated line pattern and two design constraints, therefore, the context template is shaped. With the designed template, finally, each color channel image is compressed by a context-based binary arithmetic encoder. Based on the adaptiveness of the template to the input image, the proposed templates yield better compression performance than the conventional JBIG templates, which saves 35% of the JBIG bitstream.

Spatial intra prediction has been added recently to the latest video coding standard H.264/AVC. In the intra prediction of H.264/AVC, there are 9, 9 and 4 prediction modes for 4×4, 8×8 and 16×16 blocks, respectively. Prediction signals are generated by using one or several reference pixels. The value of a reference pixel is copied as the prediction value. In some prediction modes, we calculate a weighted mean by averaging several pixels. The same prediction value is copied to several of the pixels lying in the prediction direction. However, if original image has patterns like gradations, the residual energy could increase which would result in low coding efficiency. In this paper, we propose a new intra prediction that generates prediction signals with a spatial gradient to deal with this problem. Simulation results show that it improves the picture quality and reduce the bit-rate by about 0.14 dB and 1.0 % on average for CIF sequences, respectively. It is also confirmed that our method is effective at high bit-rates.

We consider streaming video content over an overlay network of peer nodes. We propose a novel streaming strategy that is built on utility-based packet scheduling and proportional resource sharing in order to flight against free-riders. Each of the peers employs a mesh-pull mechanism to organize the download of media packets from its neighbours. For eficient resource utilization, data units are requested from neighbours based on their utility. The packet utility is driven by both its importance for the video reconstruction quality at the receiving peer and its popularity within the peer neighbourhood. In order to discourage free-riding in the system, requesting peers then share the upload bandwidth of a sending peer in proportion to their transmission rate to that peer . Our simulation results show that the proposed protocols increase the performance of a mesh-pull P2P streaming system. Significant improvements are registered relative to existing solutions in terms of average quality and average decoding rate.

In this paper we present a new method to produce low overhead redundant bits used to detect transmission errors of J2K streams on noisy communication channels. This method takes advantage of algorithms already existing on the JPEG 2000 standard and does not require any intrusive alterations on the coder. We will show that it is possible to improve standard JPEG 2000 error resilience in exchange for null resource footprint. Remark that this capability can be very important, for example in wireless communications that are mainly used by mobile devices with small processing capabilities.

A 4-D wavelet-based transform is used for efficient and scalable compression of multi-view video data. It is composed of a 1-D temporal wavelet transform, namely Motion Compensated Temporal Filtering (MCTF), a 1-D view-directional wavelet transform, namely Disparity Compensated View Filtering (DCVF), and a 2-D spatial transform. The latter is the subject of study in this paper. Whereas usually fixed isotropic wavelet or wavelet packet transforms have been used in the past for the spatial decomposition of the temporal-view-directional highpass bands, we now introduce the usage of adaptive anisotropic wavelet packet transforms as a generalization of wavelet and wavelet packet transforms. An efficient algorithm to adaptively find the rate-distortion optimal joint anisotropic basis for temporal-view-directional multi-view video subbands is derived. It is shown that the adaptive anisotropic transform performs best, compared with conventional wavelet or wavelet packet transforms.

This paper describes complexity reduction in MPEG-2 to H.264 transcoding with resolution reduction. The methods developed are applicable to transcoding any DCT based video such as MPEG-2, MPEG-4, and H.263 to the recently standardized H.264 video at a reduced resolution. H.264 is being adopted by mobile device industry and devices such as iPod use H.264. The mobile devices, however, need the video at a reduced resolution. The proposed transcoder accelerates the H.264 encoding stage by performing motion estimation for only one block size as determined by the trained decision trees. Our solution allows conversion of MPEG-2 video to the H.264 video format at a reduced resolution with substantially less computing complexity. We use machine learning based approaches to significantly reduce the complexity of this transcoding. Experimental results show a reduction in transcoding time of about 67% (a 3x speedup) with a less than -0.5 dB change in PSNR.

Quantitative metrics for successful image inpainting currently do not exist, with researchers instead relying upon qualitative human comparisons to evaluate their methodologies and techniques. In an attempt to rectify this situation, we propose two new metrics to capture the notions of noticeability and visual intent in order to evaluate inpainting results. The proposed metrics use a quantitative measure of visual salience based upon a computational model of human visual attention. We demonstrate how these two metrics repeatably correlate with qualitative opinion in a human observer study, correctly identify the optimum uses for exemplar-based inpainting (as specified in the original publication), and match qualitative opinion in published examples.

Shannon entropy as a measure of image information is extensively used in image processing applications. This measure requires estimating a high-dimensional image probability density function which poses a limitation from a practical standpoint. A number of approaches have been introduced in the literature for estimating image spatial entropy based on the assumption of Markovianity or homogeneity. This paper provides an overview of these existing approaches and their differences with Shannon entropy. These definitions are compared by applying them to synthesized test images. These images are designed in such a way that the spatial arrangements of pixels are changed without altering the histogram, thus allowing the emphasis to be placed on evaluating image spatial entropy. Furthermore, the computational complexity aspect of the definitions are discussed. The comparison results show that although the definition of image spatial entropy based on Aura Matrix provides the most effective outcome among the existing definitions, there are still deficiencies associated with this definition.

When combined with acoustical speech information, visual speech information (lip movement) significantly improves Automatic Speech Recognition (ASR) in acoustically noisy environments. Previous research has demonstrated that visual modality is a viable tool for identifying speech. However, the visual information has yet to become utilized in mainstream ASR systems due to the difficulty in accurately tracking lips in real-world conditions. This paper presents our current progress in tracking face and lips in visually challenging environments. Findings suggest the mean shift algorithm performs poorly for small regions, in this case the lips, but it achieves near 80% accuracy for facial tracking.