Texts represent an important class of information in our daily lives. This paper studies the problem of super-resolution (SR) of texts, namely reconstructing high-resolution texts from low-resolution video captured by handheld cameras. Such type of video is called nonideal due to uncontrolled imaging condition, unknown point spread function and inevitable distortion caused by compression algorithms. Motivated by the different consideration in SR from mosaicing, we investigate the error accumulation in homography-based registration of multi-view images. We advocate the nonuniform interpolation approach towards SR that can achieve resolution scalability at a low computational cost and study the issues of phase consistency and uncertainty that are difficult to be addressed under the conventional framework of treating SR as an inverse problem. We also present a nonlinear diffusion aided blind deconvolution technique for simultaneous suppression of compression artifacts and enhancement of textual information. The performance of the proposed SR-of-texts technique is demonstrated by extensive experiments with challenging real-world sequences.

Super-resolution imaging techniques reconstruct a high resolution image from a set of low resolution images that are taken from almost the same point of view. The problem can be subdivided into two main parts: an image registration part where the different input images are aligned with each other, and a reconstruction part, where the high resolution image is reconstructed from the aligned images. In this paper, we mainly consider the first step: image registration. We present three frequency domain methods to accurately align a set of undersampled images. First, we describe a registration method for images that have an aliasing-free part in their spectrum. The images are then registered using that aliasing-free part. Next, we present two subspace methods to register completely aliased images. Arbitrary undersampling factors are possible with these methods, but they have an increased computational complexity. In all three methods, we only consider planar shifts. We also show the results of these three algorithms in simulations and practical experiments.

Theoretical and practical limitations usually constrain the achievable resolution of any imaging device. Super-Resolution (SR) methods are developed through the years to go beyond this limit by acquiring and fusing several low-resolution (LR) images of the same scene, producing a high-resolution (HR) image. The early works on SR, although occasionally mathematically optimal for particular models of data and noise, produced poor results when applied to real images. In this paper, we discuss two of the main issues related to designing a practical SR system, namely reconstruction accuracy and computational efficiency. Reconstruction accuracy refers to the problem of designing a robust SR method applicable to images from different imaging systems. We study a general framework for optimal reconstruction of images from grayscale, color, or color filtered (CFA) cameras. The performance of our proposed method is boosted by using powerful priors and is robust to both measurement (e.g. CCD read out noise) and system noise (e.g. motion estimation error). Noting that the motion estimation is often considered a bottleneck in terms of SR performance, we introduce the concept of "constrained motions" for enhancing the quality of super-resolved images. We show that using such constraints will enhance the quality of the motion estimation and therefore results in more accurate reconstruction of the HR images. We also justify some practical assumptions that greatly reduce the computational complexity and memory requirements of the proposed methods. We use efficient approximation of the Kalman Filter (KF) and adopt a dynamic point of view to the SR problem. Novel methods for addressing these issues are accompanied by experimental results on real data.

Video cameras must produce images at a reasonable frame-rate and with a reasonable depth of field. These requirements impose fundamental physical limits on the spatial resolution of the image detector. As a result, current cameras produce videos with a very low resolution. The resolution of videos can be computationally enhanced by moving the camera and applying super-resolution reconstruction algorithms. However, a moving camera introduces motion blur, which limits super-resolution quality. We analyze this effect and derive a theoretical result showing that motion blur has a substantial degrading effect on the performance of super resolution. The conclusion is, that in order to achieve the highest resolution, motion blur should be avoided. Motion blur can be minimized by sampling the space-time volume of the video in a specific manner. We have developed a novel camera, called the "jitter camera," that achieves this sampling. By applying an adaptive super-resolution algorithm to the video produced by the jitter camera, we show that resolution can be notably enhanced for stationary or slowly moving objects, while it is improved slightly or left unchanged for objects with fast and complex motions. The end result is a video that has a significantly higher resolution than the captured one.

Performance of current face recognition algorithms reduces significantly when they are applied to low-resolution face images. To handle this problem, super-resolution techniques can be applied either in the pixel domain or in the face subspace. Since face images are high dimensional data which are mostly redundant for the face recognition task, feature extraction methods that reduce the dimension of the data are becoming standard for face analysis. Hence, applying super-resolution in this feature domain, in other words in face subspace, rather than in pixel domain, brings many advantages in computation together with robustness against noise and motion estimation errors. Therefore, we propose new super-resolution algorithms using Bayesian estimation and projection onto convex sets methods in feature domain and present a comparative analysis of the proposed algorithms with those already in the literature.

The term super-resolution is typically used in the literature to describe the process of obtaining a high resolution (HR) image or a sequence of HR images from a set of low resolution (LR) observations. This term has been applied primarily to spatial and temporal resolution enhancement. However, intentional pre-processing and downsampling can be applied during encoding and super-resolution techniques to upsample the image can be applied during decoding when video compression is the main objective. In this paper we consider the following three video compression models. The first one simply compresses the sequence using any of the available standard compression methods, the second one pre-processes (without downsampling) the image sequence before compression, so that post-processing (without upsampling) is applied to the compressed sequence. The third model includes downsampling in the pre-processing stage and the application of a super resolution technique during decoding. In this paper we describe these three models but concentrate on the application of super-resolution techniques as a way to post-process and upsample a compressed video sequences. Experimental results are provided on a wide range of bitrates for two very important applications: format conversion between different platforms and scalable video coding.

In this paper, a new technique for robust super-resolution (SR) from compressed video is presented. The proposed method exploits the differences between low-resolution images at the pixel level, in order to determine the usability of every pixel in the low-resolution images for SR enhancement. Only the pixels, from the lowresolution images, that are determined to be usable, are included in the L₂-norm minimization procedure. Three different usability criterions are proposed, maximum distance from the median - MDM, maximum distance from initial image - MDIM, and maximum distance from the SR estimate - MDSRE. The results obtained with real video sequences demonstrate superior quality of the resulting enhanced image in the presence of outliers and same quality without outliers when compared to existing L₂-norm minimization techniques. At the same time, the proposed scheme produces sharper images as compared to L₁-norm minimization techniques.

This paper proposes an example-based Super-Resolution (SR) algorithm of compressed videos in the Discrete Cosine Transform (DCT) domain. Input to the system is a Low-Resolution (LR) compressed video together with a High-Resolution (HR) still image of similar content. Using a training set of corresponding LR-HR pairs of image patches from the HR still image, high-frequency details are transferred from the HR source to the LR video. The DCT-domain algorithm is much faster than example-based SR in spatial domain⁶ because of a reduction in search dimensionality, which is a direct result of the compact and uncorrelated DCT representation. Fast searching techniques like tree-structure vector quantization¹⁶ and coherence search¹ are also key to the improved efficiency. Preliminary results on MJPEG sequence show promising result of the DCT-domain SR synthesis approach.

We consider the problem of complexity reduction in motion/disparity estimation for multiview video coding. We propose predictive fast search algorithms that, after either the motion field or the disparity field has been estimated, obtain with low complexity a good set of candidate vectors for the other field. The proposed scheme performs predictive motion search from view to view and predictive disparity search from one time instant to another time. We also propose an efficient search pattern that starts with the candidate vectors from the proposed algorithms. Simulation results show a very significant reduction in encoding complexity with slight coding efficiency degradation as compared to the full search in both motion and disparity estimations.

A new complexity-scalable framework for motion estimation is proposed to efficiently reduce the motioncomplexity of encoding process, with focus on long term memory motion-compensated prediction of the H.264 video coding standard in this work. The objective is to provide a complexity scalable scheme for the given motion estimation algorithm such that it reduces the encoding complexity to the desired level with insignificant penalty in rate-distortion performance. In principle, the proposed algorithm adaptively allocates available motion-complexity budget to macroblock based on estimated impact towards overall rate-distortion (RD) performance subject to the given encoding time limit. To estimate macroblock-wise tradeoff between RD coding gain (J) and motion-complexity (C), the correlation of J-C curve between current macroblock and collocated macroblock in previous frame is exploited to predict initial motion-complexity budget of current macroblock. The initial budget is adaptively assigned to each blocksize and block-partition successively and motion-complexity budget is updated at the end of every encoding unit for remaining ones. Based on experiment, proposed J-C slope based allocation is better than uniform motion-complexity allocation scheme in terms of RDC tradeoff. It is demonstrated by experimental results that the proposed algorithm can reduce the H.264 motion estimation complexity to the desired level with little degradation in the rate distortion performance.

Image-based rendering techniques require capturing an object or scene from many viewpoints. Often, depth maps are used in addition to the original images for rendering new iews for novel viewpoints not coinciding with one of the original camera positions. Due to the high amount of data, efficient compression is necessary. When the data is stored or transmitted, it is not only desirable to compress the image data but also the depth information. In this paper, the case of sequences recorded with a single, hand-held camera is investigated. These sequences are used for unstructured lumigraph rendering. In addition, the case of multi-view video sequences is analyzed. For both cases, depth maps are compressed using H.264/AVC, and the achievable data rates are studied. For the case of unstructured lumigraph rendering, the effect of depth map compression on the quality of rendered images is analyzed in a second step.

The use of shape-adaptive transforms is a popular approach for coding arbitrarily shaped objects in image/video coding due to their adaptability at object edges and low complexity. In this respect shape adaptive DCT (SA-DCT) and shape adaptive DWT (SA-DWT) have been proposed in previous literature. The Integer Transform (IT), a derivative of the 4x4 DCT, has been adopted in the latest H.264/AVC standard for coding image blocks in residual data (texture). The associated integer arithmetic guarantees fast and accurate coding/decoding. In this paper, we propose a novel Shape Adaptive Integer Transform (SA-IT) which can be effectively used in future for enabling arbitrary shaped object coding in H.264. Though Integer Transforms are a derivative of 4x4 DCTs, in H.264, to maintain integer arithmetic capability, the post-and pre-scaling factors of transform process are integrated into the forward and inverse quantiser stages respectively for reducing the total number of multiplications and avoiding the loss of accuracy. Thus SA-IT considerably differs from SA-DCT and calls for novel design and implementation considerations based on combining those merits of both SA-DCT and IT. We provide theoretical proofs and support them with experimental justifications.

In this paper, we consider the problem of optimal bit allocation for wireless video transmission over fading channels. We use a newly developed hybrid scalable/multiple-description codec that combines the functionality of both scalable and multiple-description codecs. It produces a base layer and multiple-description enhancement layers. Any of the enhancement layers can be decoded (in a non-hierarchical manner) with the base layer to improve the reconstructed video quality. Two different channel coding schemes (Rate-Compatible Punctured Convolutional (RCPC)/Cyclic Redundancy Check (CRC) coding and, product code Reed Solomon (RS)+RCPC/CRC coding) are used for unequal error protection of the layered bitstream. Optimal allocation of the bitrate between source and channel coding is performed for discrete sets of source coding rates and channel coding rates. Experimental results are presented for a wide range of channel conditions. Also, comparisons with classical scalable coding show the effectiveness of using hybrid scalable/multiple-description coding for wireless transmission.

This paper develops a combined space-time multiple description video coding (ST-MDVC) scheme and corresponding error concealment methods in order to adapt to error prone packet networks, especially for multiple channels. It generates four descriptions, instead of two in most other MDVC schemes, with much lower redundancy than others. The three-loop slice group MDVC approach of [8] is used to achieve spatial description splitting. MDVC is very suitable for multiple channel environments, and especially able to maintain acceptable quality when some of these channels fail completely, i.e. in an on-off MDC environment, without experiencing any drifting problem. Our MDC scheme coupled with the proposed concealment approaches proved to be suitable not only for the on-off MDC environment case (data from one channel fully lost), but also for the case where only some packets are lost from one or more channels. Several steps to achieve high performance concealments are done with the received packets. Results show the promising performance with very low redundancy for both the balanced and unbalanced channel cases. Generating four descriptions make it more adapted to multi-path scenarios.

In this paper, we present a joint source channel decoding technique for CABAC encoded data based on sequential decoding. The proposed method is developed to be compatible with CABAC features, i.e., handling binary arithmetic coding, adaptive probabilities, and context modeling. Redundancy due to the binarization step is exploited and no extra redundancy is compulsory. An hypothesis test allowing to adjust the trade-off between decoding complexity and error resilience performances is proposed. Simulation results show that our method outperforms those using classical sequential algorithms.

Intra prediction in H.264/AVC uses 9 modes in 4×4 luma block, 4 in 16×16 luma and 8×8 chroma blocks. Intra prediction modes utilize similarity of current macroblock with pre-encoded neighboring macroblocks and directionality of image in pixel domain. Therefore its coding efficiency in H.264/AVC is improved compared to the conventional video codecs such as MPEG-2 and MPEG-4. However coding efficiency in intra-coded frame is much lower compared to that of inter slice since the intra prediction modes use only limited neighboring macro blocks and then to obtain good matched reference block image using intra prediction may be more difficult than inter prediction. Therefore for higher coding efficiency, it is very important to get good matched prediction block in a given envirionment. In general video codec which uses general 4:2:0 YCbCr format, overall coding efficiency issue is mainly focused on luma relative to chroma components. In this paper, we propose additional intra luma prediction mode using collocated chroma pixels and weight values. The proposed method utilizes collocated chroma macroblocks as reference image for more efficient intra luma prediction.

SP and SI frames in the H.264 video coding standard can be used for error resilience, bitstream switching or random access. Despite a widespread interest in these new types of frames, no work so far has investigated, in a systematic way, their rate-distortion efficiency. In this paper, we propose a model for the rate-distortion performance of SI and SP frames. A comparison to experimental results, obtained with our implementation of an SP encoder, recently adopted by JVT, confirms its validity. The model predicts how the relative sizes of SP and SI frames can be traded off. We analyze, both theoretically and experimentally, how this can be used to minimize the transmitted bit-rate when SP frames are used for video streaming with packet losses.

The coding efficiency of a Wyner-Ziv video codec relies significantly on the quality of side information extracted at the decoder. The construction of efficient side information is difficult thanks in part to the fact that the original video sequence is not available at the decoder. Conventional motion search methods are widely used in the Wyner-Ziv video decoder to extract side information. This substantially increases the Wyner-Ziv video decoding complexity. In this paper, we propose a new method to construct side estimation based on the idea of universal prediction. This method, referred to as Wyner-Ziv video coding with Universal Prediction(WZUP), does not perform motion search or assume underlying model of original input video sequences at the decoder. Instead, WZUP estimates the side information based on its observations on past reconstructed video data. We show that WZUP can significantly reduce decoding complexity at the decoder and achieve fair side estimation performance, thus make it possible to design both the video encoder and the decoder with low computational complexity.

We present a hybrid scalable video coding algorithm in which two types of scalabilities, i.e., layered scalability and parallel scalability, are combined to achieve higher order of flexibility for heterogeneous networks and mixed client requirements. In space domain, SNR scalable video coding is used to give layered scalability on frame quality; while in temporal domain, a multiple description approach, the Multiple State Video Coding (MSVC), provides parallel scalability to the base layer video. Two coarsely quantized sub-streams are yielded using MSVC and form the base layer. All residual signals of the video sequence are coded into a third sub-stream and form the enhancement layer. The hybrid scalable video protects the base layer using multiple description and has better performance than conventional layered scalable video when a base layer picture is damaged. It gives similar performance as SNR scalable video when the enhancement layer encounters errors. In general packet drop networks, the proposed scalable video achieves higher PSNR than the original MSVC video and single description video in high packet drop rate environments.

Wyner-Ziv video coding has gained considerable interests in the research community. In this paper, we examine the Wyner-Ziv video coding performance and compare it with conventional motion-compensated prediction (MCP) based video coding. Theoretical and simulation results show that although Wyner-Ziv video coding can achieve as much as 6dB gain over conventional video coding without motion search, it still falls 6dB or more behind current best MCP-based INTER-frame video coding. We further investigate the use of sub-pixel and multi-reference motion search methods to improve Wyner-Ziv video coding efficiency.

This paper describes a new approach for improving the coding efficiency of spatially scalable video coding. For spatial scalability implemented within the traditional hybrid block transform motion-compensated video coding framework, it has been surprisingly difficult to achieve coding efficiency significantly better than simulcast. One of the most difficult tasks to do efficiently is the motion compensation in the enhancement layer. The new approach for motion compensation presented here uses techniques borrowed from frequency scalability. The three frequency scalability approaches to motion compensation that can be adapted to spatial scalability are the pyramid scheme, the subband scheme, and the conditional replacement scheme. This paper describes the pyramid scheme and presents experimental results comparing the pyramid approach to non-scalable H.264 and simulcast.

A system for sender-driven video streaming from multiple servers to a single receiver is considered in this paper. The receiver monitors incoming packets on each network path and returns, to the senders, estimates of the available bandwidth on all the network paths. The senders in turn employ this information to compute transmission schedules for packets belonging to the video stream sent to the receiver. An optimization framework is proposed that enables the senders to compute their transmission schedules in a distributed way, and yet to dynamically coordinate them over time such that the resulting video quality at the receiver is maximized. Experimental results demonstrate that the proposed streaming framework provides superior performance over distortion-agnostic transmission schemes that perform proportional packet scheduling based only on the available network bandwidths.

Current state-of-the-art algorithms that process visual information for end use by humans treat images and video as traditional "signals" and employ sophisticated signal processing strategies to achieve their excellent performance. These algorithms also incorporate characteristics of the human visual system (HVS), but typically in a relatively simplistic manner, and achievable performance is reaching an asymptote. However, large gains are still realizable with current techniques by aggressively incorporating HVS characteristics to a much greater extent than is presently done. Achieving these gains requires HVS characterizations which better model natural image perception ranging from sub-threshold perception (where distortions are not visible) to supra-threshold perception (where distortions are clearly visible). This paper reviews classical psychophysical HVS characterizations focused on the visual cortex (V1), pertaining to the contrast sensitivity function, summation, and masking, which have been obtained using unrealistic stimuli such as sinusoids and white noise. The direct applicability of these results to natural images is often not clear. Complementary results are then presented which have been obtained using realistic stimuli derived from or consisting of natural images, along with several applications of these results. Finally, a new structure-based masking model is proposed to model masking in homogeneous natural image patches as a function of the image type: textures, edges, or structures.

Some emerging applications may require flexible playback features for time-based media, such as video, that cannot be directly supported by current compression standards, because for these decoding of frames can only be done in a predetermined order. An example would be a video application where both backward and forward frame-by-frame playback are to be supported. A standard codec could support this by decoding complete GOPs in the desired order, and then playing back one frame at a time. Thus, potentially significant added delay and memory are needed to support backward playback, which can be lowered if small GOP sizes are chosen, at the cost of reduced coding efficiency. Other example applications where flexible playback may be desirable include switching between different views in multiview video coding, and accessing individual spectral bands in hyperspectral imagery. In this work we address flexible playback by showing that it becomes feasible when a particular data unit (e.g., a video frame) can be decoded using information from either one of a number of other data units (e.g., in the video case the next frame or the previous frame). Note that this is different from structures such as bi-directionally predicted frames, which require both predictor frames to be available at the decoder. We cast this problem as one of source coding with uncertainty about decoder side-information and propose a solution based on distributed source coding. In addition, we propose macroblock-based mode switching algorithms in the context of distributed video coding to improve coding efficiency. Our results show that, using forward/backward playback as an example, our proposed solution can achieve good coding efficiency without incurring additional delay and memory overhead.

We propose the use of H.264 redundant slices for Systematic Lossy Error Protection (SLEP) of a video signal transmitted over an error-prone channel. In SLEP, the video signal is transmitted to the decoder without channel coding. Additionally, a Wyner-Ziv encoded version of the video signal is transmitted in order to provide error-resilience. In the event of channel errors, the Wyner-Ziv description is decoded as a substitute for the error-prone portions of the primary video signal. Since the Wyner-Ziv description is typically coarser than the primary video signal, SLEP is a lossy error protection technique which trades-off residual quantization distortion for improved error-resilience properties, such as graceful degradation of decoder picture quality. We describe how H.264 redundant slices can be used to generate the Wyner-Ziv description, and present simulation results to demonstrate the advantages of this method over traditional methods such as FEC.

Correlation estimation plays a critical role in resource allocation and rate control for distributed data compression. A Wyner-Ziv encoder for distributed image compression is often considered as a lossy source encoder followed by a lossless Slepian-Wolf encoder. The source encoder consists of spatial transform, quantization, and bit plane extraction. In this work, we find that Gray code, which has been extensively used in digital modulation, is able to significantly improve the correlation between the source data and its side information. Theoretically, we analyze the behavior of Gray code within the context of distributed image compression. Using this theoretical model, we are able to efficiently allocate the bit budget and determine the code rate of the Slepian-Wolf encoder. Our experimental results demonstrate that the Gray code, coupled with accurate correlation estimation and rate control, significantly improves the picture quality, by up to 4 dB, over the existing methods for distributed image compression.

From the information-theoretic perspective, as stated by the Wyner-Ziv theorem, the distributed source encoder doesn't need any knowledge about its side information in achieving the R-D performance limit. However, from the system design and performance analysis perspective, correlation modeling plays an important role in analysis, control, and optimization of the R-D behavior of the Wyner-Ziv video coding In this work, we observe that videos captured from a wireless video sensor network (WVSN) are uniquely correlated under the multi-view geometry. We propose to utilize this computer vision principal, as well as other existing information, which is already available or can be easily obtained from the encoder, to estimate the source correlation structure. The source correlation determines the R-D behavior of the Wyner-Ziv encoder, and provide useful information for rate control and performance optimization of the Wyner-Ziv encoder.

There are mainly two key points which can affect the efficiency of multi-view video capture and transmission system largely: communication between cameras and computing complexity of encoder. In this paper, we propose a practical framework of distributed multi-view video coding, in which inter-camera communication is avoided and the large computing complexity is moved from encoder to decoder. In this scheme, multi-camera video sources are encoded separately and decoded dependently, and the traditional inter frame is replaced by Wyner-Ziv frame. To reach this goal, Wyner-Ziv theory on source coding with side information is employed as the basic coding principle. A Wyner-Ziv coding method based on wavelet transform and turbo codes is used as the core of the scheme. To further improve the coding performance, we also consider exploiting the large redundancy between adjacent views. A more flexible prediction method that can jointly use temporal and view correlations is proposed to generate the side information at the decoder. The experimental results show that the coding performance of proposed DMVC scheme is very promising compared to the traditional intra coding.

The free viewpoint switching is one of the most important features of multi-view video streaming. The key problem lies in how to achieve the best performance when the camera processing capability and the network bandwidth are limited. In this paper, we propose a novel free viewpoint switching scheme for multi-view video scenario, in which the distributed video coding technique is employed. In this scheme, the multi-camera video sources are encoded separately with the traditional hybrid video coding scheme, and meanwhile an alternative bitstream is produced for every frame based on the Wyner-Ziv coding method for the purpose of error correction when the viewpoint switching occurs. When switching happens, the Wyner-Ziv bits corresponding to the actual reference frame at the switching point is transmitted and used to recover the true reference. Instead of completely removing the mismatch, the proposed switching scheme tries to reduce the mismatch to an acceptable level so as to save the bits for the switching frame. A wavelet transform domain Wyner-Ziv coding method is proposed to produce the Wyner-Ziv bits for the switching frame. Conclusively, with the proposed scheme, the inter-camera communication can be avoided and the drifting error can be controlled efficiently when the viewpoint switching occurs.

In this paper, we present a parallel multiple target tracking framework using multiple cooperative trackers. The multiple target occlusion problem is handled by modeling the interaction among different targets' observations and solving the data association through a recursive estimation. The computational complexity of the proposed approach increases linearly with the number of targets which yields a much faster implementation than existing multiple target tracking algorithms. Experimental results have been demonstrated on real-world videos.

In the visual tracking domain, Particle Filtering (PF) can become quite inefficient when being applied into high dimensional state space. Rao-Blackwellisation ^[1] has been shown to be an effective method to reduce the size of the state space by marginalizing out some of the variables analytically ^[2]. In this paper based on our previous work ^[3] we proposed RBPF tracking algorithm with adaptive system noise model. Experiments using both simulation data and real data show that the proposed RBPF algorithm with adaptive noise variance improves its performance significantly over conventional Particle Filter tracking algorithm. The improvements manifest in three aspects: increased estimation accuracy, reduced variance for estimates and reduced particle numbers are needed to achieve the same level of accuracy.

This paper proposes a purely image-based TV channel logo detection algorithm that can detect logos independently from their motion and transparency features. The proposed algorithm can robustly detect any type of logos, such as transparent and animated, without requiring any temporal constraints whereas known methods have to wait for the occurrence of large motion in the scene and assume stationary logos. The algorithm models logo pixels as outliers from the actual scene content that is represented by multiple 3-D histograms in the YC_BC_R space. We use four scene histograms corresponding to each of the four corners because the content characteristics change from one image corner to another. A further novelty of the proposed algorithm is that we define image corners and the areas where we compute the scene histograms by a cinematic technique called Golden Section Rule that is used by professionals. The robustness of the proposed algorithm is demonstrated over a dataset of representative TV content.

We develop a new approach to finding corners in images that combines foveated edge detection and curvature calculation with saccadic placement of foveal fixations. Each saccade moves the fovea to a location of high curvature combined with high edge gradient. Edges are located using a foveated Canny edge detector with spatial constant that increases with eccentricity. Next, we calculate a measure of local corner strength, based on a product of curvature and gradient. An inhibition factor based on previous visits to a region of the image prevents the system from repeatedly returning to the same locale. A long saccade is move thes fovea to previously unexplored areas of the image. Subsequent short saccades improve the accuracy of the location of the corner approximated by the long saccade. The system is tested on two natural scenes and the results compared against subjects observing the same test images through an eyetracker. Results show that the algorithm is a good locator of corners.

This paper presents a disparity estimation algorithm that combines three different kinds of techniques: Gabor transform, variational refinement and region-based affine parameter estimation for disparity calculation. The Gabor transform is implemented using a set of quadrature-pair filters for estimating the two-dimensional cor- respondences between the two images without the calibration information, and the estimated coarse disparity maps are applied to a variational refinement process which involves solving a set of partial differential equations (PDEs). Then the refined disparity values are used with the image segmentation information so that the parameters of affine transforms for the correspondence of each region can be calculated by singular value decomposition (SVD), and these affine parameters can be applied in turn to bring more refined disparity maps.

Plane-Based calibration algorithms have been widely adopted for the purpose of camera calibration task. These algorithms have the advantage of robustness compared to self-calibration and flexibility compared to traditional algorithms which require a 3D calibration pattern. While the common assumption is that the intrinsic parameters during the calibration process are fixed, limited consideration has been given to the general case when some intrinsic parameters maybe varying, others maybe fixed or known. We first discuss these general cases for camera calibration. Using a counting argument we enumerate all cases where plane-based calibration may be utilized and list the number of images required for each of these cases. Then we extend the plane-based framework to the problem of cameras with zoom variation, which is the most common case. The approach presented and may be extended to incorporate additional varying parameters described in the general framework. The algorithm is tested using both synthetically generated images and actual images captured using a digital camera. The results indicate that the method performs very well and inherits the advantage of robustness and flexibility from plane-based calibration algorithms.

The paper proposes a new type of nonlinear filters, classification-based hybrid filters, which jointly utilize spatial, rank order and structural information in image processing. The proposed hybrid filters use a vector containing the observation samples in both spatial and rank order. The filter coefficients depend on the local structure of the image content, which can be classified based on the luminance pattern in the filter window. The optimal coefficients for each class are obtained by the Least Mean Square optimization. We show that the proposed classification-based hybrid filters exhibit improved performance over linear filters and order statistic filters in several applications, image de-blocking, impulsive noise reduction and image interpolation. Both quantitative and qualitative comparison have also been presented in the paper.

This study presents an alternative method to estimate motion parameters to the gradient-based method, which is known as Lucas-Kanade algorithm. The proposed method is a faster version of a hyperplane-intersection method. The hyperplane-intersection method estimates the motion parameters between images as an intersection position of estimated hyperplanes in a parameter space. The hyperplanes approximate the zero positions of partial derivatives of a continuous similarity measure with respect to each parameter. The method employs a straightforward computation to estimate the parameters, instead of using an iterative framework. The noniterative method is suitable for hardware implementation. The method is the region-based and intensity-based technique that is capable of using any dissimilarity or similarity measure such as sum of squared differences (SSD) or zero-mean normalized cross correlation (ZNCC), which can be selected adequately in consideration of a property of input image sequence and a required computation time. The faster version of the method is realized with pre-computed warped images of the template, which reduce the computational cost for each input frame. This study also compares the computational cost and the accuracy of the estimated parameters of the proposed algorithm with those of the gradient descent method. Experiments using synthesized-motion sequences and real image sequences are performed to confirm the comparisons. The faster version of the hyperplane-intersection method using ZNCC demonstrates robustness to a non-uniform illumination change in the image sequences.

We present a wavelet-based near-lossless coder with L∞-error scalability. The method presented consists of a wavelet-based lossy layer encoder followed by multiple stages of residual refinements for L∞-error scalability. We introduce a successive refinement scheme in L∞-error sense for residual layers reminiscent of popular set-partitioning algorithms for embedded coding. The initial near-lossless error bound is attained by our previously proposed two-stage scheme which determines the bit-rate for the lossy layer 'on-the-fly' without any iteration. The resulting residual which is defined by the difference between the original and the initial near-lossless reconstruction is then further refined in L∞-error sense up to lossless reconstruction. This scheme provides a combination of L²-error embedded lossy reconstruction up to an automatically determined high-fidelity plus optional layers of scalable L∞-error. This may turn out to be particularly useful for scalable archival applications where the fidelity of reconstructions on the high-bit end needs to be strictly controlled.

One interesting feature of image compression is support of region of interest (ROI) access, in which an image sequence can be encoded only once and then the decoder can directly extract a subset of the bitstream to reconstruct a chosen ROI of required quality. In this paper, we apply Three-dimensional Subband Block Hierarchical Partitioning (3-D SBHP), a highly scalable wavelet transform based algorithm, for volumetric medical image compression to support ROI access. The codeblock selection method by which random access decoding can be achieved is outlined and the performance empirically investigated. The experimental results show that there are a number of parameters that affect the effectiveness of ROI access, the most important being the size of the ROI size, code-block size, wavelet composition level, number of filter taps and target bit rate. Finally, one possible way to optimize ROI access performance is addressed.

JPEG2000 technology has been adopted even for encoding motion pictures because of its high coding performance and the scalability of the stream format. Since it includes the additional advantage of employing intra frame coding technology such as JPEG2000, it allows drastic reduction in the latency accompanied by encoding and decoding process can be extremely reduced. On the other hand, a low delay transmission has been required especially for a FPU (Field Pick-up Unit) terminal or a piece of wireless camera equipment which may be utilized for a real-time remote hookup. In many such terminals, compression coding was conducted by typical MPEG-2 video, and the codec latency of more than 300 msec was forced basically from its coding algorithm. In this paper, HDTV is assumed to be a typical video application, and an optimal rate control mechanism for the JPEG2000 encoder is introduced as a key technology to achieve a low delay in transmission while maintaining high coding performance. Our study introduces advanced key technologies as yet unrecognized officially. Coding experiments using those technologies confirmed that significant coding delay elimination was achieved when compared to conventional encoding schemes.

Previously we presented a network-driven Wyner-Ziv video coding method, in which the motion vectors are derived at the decoder and sent back to the encoder through a reliable backward channel. In this paper, we consider the scenario when the backward channel is error resilient. We study the performance of error resilient methods for our codec. A symmetrical Reversible Variable Length Code (RVLC) is used to reduce the bandwidth requirement of the backward channel. A hybrid scheme with selective coding is proposed to improve the coding efficiency when transmission delay occurs. The experimental results show that these error resilient methods can consistently improve the video quality at the decoder.

TCP is one of the most widely used transport protocols for video streaming. However, the rate variability of TCP makes it difficult to provide good video quality. To accommodate the variability, video streaming applications require receiver-side buffering. In current practice, however, there are no systematic guidelines for the provisioning of the receiver buffer, and smooth playout is insured through over-provisioning. In this work, we are interested in memory-constrained applications where it is important to determine the right size of receiver buffer in order to insure a prescribed video quality. To that end, we characterize video streaming over TCP in a systematic and quantitative manner. We first model a video streaming system analytically and derive an expression of receiver buffer requirement based on the model. Our analysis shows that the receiver buffer requirement is determined by the network characteristics and desired video quality. Experimental results validate our model and demonstrate that the receiver buffer requirement achieves desired video quality.

In this work, we present an effective overlay multicast tree constructing algorithm to meet delay constraint for real-time media service with the minimum networking price over Differentiated-Service networks. In addition, a dynamic and scalable tree maintaining algorithm is proposed for seamless service by updating the small parts of the tree when some End-systems join or leave the multicast group. An effective solution is obtained by changing the tree structure among Data-server, Proxy-senders and End-systems and the class vector of Proxy-senders. Finally, experimental results are presented to compare the performance of the proposed algorithm.

In this paper, we propose a novel source rate control algorithm for video streaming over the Internet. With the incorporation of a virtual network buffer management mechanism (VB), the QoS requirements of the application can be translated into the source rate constraints, based on which the source rate control is implemented. The maximum admissible bandwidth (or send rate) constraint, which is imposed by the encoder and decoder buffer sizes of the application, is also derived. To make sure the send rate not exceeding the maximum bandwidth constraint, a rate regulator using the token bucket approach is adopted at the application layer to limit the output rate of the encoder buffer when necessary. Simulation results show that our proposed algorithm can help the application reduce the overflow and underflow of the decoder buffer, and achieve better video quality and quality smoothness than traditional source rate control algorithms.

Media encryption technologies actively play the first line of defense in securing the access of multimedia data. Traditional cryptographic encryption can achieve provable security but is unfortunately sensitive to a single bit error, which will cause an unreliable packet to be dropped to create packet loss. In order to achieve robust media encryption, error resilience in media encryption can be treated to be equivalent to error resilience in media transmission. This study proposes an embedded block hash searching scheme at the decoder side to achieve motion estimation and recover the lost packets, while maintaining format compliance and cryptographic provable security. It is important to note that the proposed framework is a kind of joint error-resilient video transmission/encryption and copyright protection.

This paper introduces an anisotropic decomposition structure of a recently introduced 3-D dual-tree discrete wavelet transform (DDWT), and explores the applications for video denoising and coding. The 3-D DDWT is an attractive video representation because it isolates motion along different directions in separate subbands, and thus leads to sparse video decompositions. Our previous investigation shows that the 3-D DDWT, compared to the standard discrete wavelet transform (DWT), complies better with the statistical models based on sparse presumptions, and gives better visual and numerical results when used for statistical denoising algorithms. Our research on video compression also shows that even with 4:1 redundancy, the 3-D DDWT needs fewer coefficients to achieve the same coding quality (in PSNR) by applying the iterative projection-based noise shaping scheme proposed by Kingsbury. The proposed anisotropic DDWT extends the superiority of isotropic DDWT with more directional subbands without adding to the redundancy. Unlike the original 3-D DDWT which applies dyadic decomposition along all three directions and produces isotropic frequency spacing, it has a non-uniform tiling of the frequency space. By applying this structure, we can improve the denoising results, and the number of significant coefficients can be reduced further, which is beneficial for video coding.

We propose two approaches to scalable lossless coding of motion video. They achieve SNR-scalable bitstream up to lossless reconstruction based upon the subpixel-accurate MCTF-based wavelet video coding. The first approach is based upon a two-stage encoding strategy where a lossy reconstruction layer is augmented by a following residual layer in order to obtain (nearly) lossless reconstruction. The key advantages of our approach include an 'on-the-fly' determination of bit budget distribution between the lossy and the residual layers, freedom to use almost any progressive lossy video coding scheme as the first layer and an added feature of near-lossless compression. The second approach capitalizes on the fact that we can maintain the invertibility of MCTF with an arbitrary sub-pixel accuracy even in the presence of an extra truncation step for lossless reconstruction thanks to the lifting implementation. Experimental results show that the proposed schemes achieve compression ratios not obtainable by intra-frame coders such as Motion JPEG-2000 thanks to their inter-frame coding nature. Also they are shown to outperform the state-of-the-art non-scalable inter-frame coder H.264 (JM) lossless mode, with the added benefit of bitstream embeddedness.

The extracted low resolution video from a motion compensated 3-D subband/wavelet scalable video coder is unnecessarily sharp and sometimes contains significant aliasing, compared to that by the MPEG4 lowpass filter. In this paper, we propose a content adaptive method for aliasing reduction in subband/wavelet scalable video coding. We try to make the low resolution frame (LL subband) visually similar to that of the MPEG4 decimation filter through frequency roll-off. Scaling of the subbands is introduced to make the variances of the subbands comparable in these two cases. Thanks to the embedded properties of the EZBC coder, we can achieve the needed scaling of energies in each subband by sub-bitplane shift in the extractor and coefficient scaling in the decoder. An analysis is presented for the relationship between sub-bitplane shift and scaling, which shows that our selected sub-bitplane shift works well for high to medium bitrates. Two different energy-matching structures, i.e. a dyadic decomposition and non-dyadic decomposition, are proposed. The first dyadic method has low complexity but low accuracy for energy matching, while the second non-dyadic method is more accurate for energy matching but has higher analysis/synthesis complexity. The first scheme introduces no PSNR loss for full resolution video, while the second one introduces a slight PSNR loss for full resolution, due to our omission of interband context modeling in this case. Both methods offer substantial PSNR gain for lower spatial resolution, as well as substantial reduction in visible aliasing.

This paper presents an adaptive in-band motion compensated temporal filtering (MCTF) scheme for 3-D wavelet based scalable video coding. The proposed scheme overcomes the problem of visual artifacts due to motion mismatch when motion information of the spatial lowpass subband is inaccurately applied to the highpass subbands in decoding high spatial resolution video. More specifically, the proposed scheme determines if the mismatch energy is beyond an acceptable threshold in the wavelet domain and then adaptively switches between using the motion information from the spatial lowpass and highpass subbands in the MCTF prediction step. As a result, the amount of motion overhead needs to be transmitted is minimized and a good trade-off between motion information and texture is realized. Experimental results show that the proposed scheme greatly improves the quality of the decoded video at high spatial resolutions while achieving a high coding efficiency. Furthermore, the proposed adaptive scheme requires only modifications when performing MCTF in the highpass subbands, hence the original strength of in-band MCTF for decoding low spatial resolution video is well preserved.

This paper presents a new rate allocation algorithm for MCTF-based video coding with the aim to control quality fluctuation. Distortion analysis is conducted for MCTF using a simplified signal model. Based on the analysis, the aim to control quality fluctuation is posed as a quadratic programming problem and its solution forms the basis for our proposed algorithm. After discussions on some extensions of the proposed rate allocation method, we verify it on MPEG SVC reference software and the experimental results demonstrate that the proposed rate allocation scheme can reduce quality fluctuation significantly.

Previously we have presented a selective image sharpening method based on the coupled nonlinear diffusion process composed of a nonlinear diffusion term, a fidelity term and an isotropic peaking term, and it can sharpen only blurred edges without increasing the noise visibility. Our previously presented prototypal color-image sharpening methods based on the coupled nonlinear-diffusion process have been formulated on the linear color models, namely, the channel-bychannel model and the 3D vectorial model. Our prototypal methods can sharpen blurred color step edges, but they do not necessarily enhance contrasts of signal variations in complex texture image regions so well as in simple step-edge regions. To remedy the drawback, this paper extends our coupled nonlinear-diffusion color-image sharpening method to the nonlinear non-flat color model, namely, the chromaticity-brightness model, which is known to be closely relating to human color perception. We modify our time-evolution PDE's for the non-flat space of the chromaticity vector and present its digital implementations. Through experimental simulations, we compare our new color-image sharpening method based on the chromaticity-brightness model with our prototypal color-image sharpening methods based on the linear color models.

MPEG-4 is the first standard that combines synthetic objects, like 2D/3D graphics objects, with natural rectangular and non-rectangular video objects. The independent access to individual synthetic video objects for further manipulation creates a large space for future applications. This paper addresses the optimization of such complex multimedia algorithms for implementation on multiprocessor platforms. It is shown that when choosing the correct granularity of processing for enhanced parallelism and splitting time-critical tasks, a substantial improvement in processing efficiency can be obtained. In our work, we focus on non-rectangular (also called arbitrary-shaped) video objects decoder. In previous work, we motivated the use of a multiprocessor System-on-Chip(SoC) setup that satisfies the requirements on the overall computation capacity. We propose the optimization of the MPEG-4 algorithm to increase the decoding throughput and a more efficient usage of the multiprocessor architecture. First, we present a modification of the Repetitive Padding to increase the pipelining at block level. We identified the part of the padding algorithm that can be executed in parallel with the DCT-coefficient decoding and modified the original algorithm into two communicating tasks. Second, we introduce a synchronization mechanism that allows the processing for the Extended Padding and postprocessing (Deblocking & Deringing) filters at block level. The first optimization results in about 58% decrease of the original Repetitive-Padding task computational requirements. By introducing the previously proposed data-level parallelism and exploiting the inherent parallelism between the separated color components (Y, Cr, Cb), the computational savings are about 72% on the average. Moreover, the proposed optimizations marginalize the processing latency from frame size to slice order-of-magnitude.

We propose a novel framework for fast mode decision in the simple and main profiles of the H264 video coding standard. Our framework consists of a specific combination of algorithms, each achieving computational savings while retaining Rate Distortion (RD) performance very similar to the standard. In particular, we utilise a set of skip mode conditions for P and B slices, two heuristics that reduce the cardinality of the inter mode set to be examined, inter/intra mode prediction and the monotonicity property of the Rate Distortion cost functions. We achieve content dependent savings in run times between 5.8 and 90.1% as compared to H264. Compared to other work that was used as input to the standard, our scheme is faster by 9-23% for very similar RD performance. The proposed framework can be used wholly or partially for computational speed-ups, it is independent of the motion search method used and is applicable in both the rate controlled and non rate controlled cases.

According to advances in digital image processing techniques, interest in high-quality still images has been increased. This paper proposes an effective discrete wavelet transform (DWT)-based algorithm that efficiently interpolates a highresolution (HR) still image from a low-resolution (LR) one. The decay and persistence properties of DWT coefficients are utilized for HR image reconstruction. DWT coefficients are initially estimated based on the decay of the Lipschitz exponential and the similarity of DWT coefficients across resolution scales is used for coarse estimation. For optimal fine estimation the mean squared error is iteratively minimized. The proposed DWT-based interpolation algorithm yields better performance than the conventional methods in terms of the peak signal to noise ratio and subjective image quality.

We consider a portable video device which captures and compresses video data. The compressed bit stream is either stored in an on-board storage device or transmitted to a remote user through wireless channels. The storage space or the amount of energy and channel bandwidth available for data transmission determines the total number of bits that the video encoder can generate. Besides this bit resource constraint, the portable video device also operates under data processing energy constraint. This is because video compression is computationally intensive and energy-consuming. One of the central challenging issues in portable video communication system design is how to maximize the operational lifetime. In this work, we will investigate the following research problem: given a portable device, which has a certain amount of power supply E and the maximum number of bits it can store or transmit is R, and the required video quality level is D, what is the maximum operational lifetime of this portable device, and how to allocate the bit and energy resources to achieve this maximum lifetime? To address this problem, we design an energy-scalable video encoder and study its power-rate-distortion (P-R-D) behavior. We theoretically study the optimum bit and energy resource allocation and analyze the energy saving performance. Our results show that for typical consumer electronic videos, using the proposed P-R-D video encoding technology, the operational lifetime of the portable device can be doubled or even tripled. This has a significant impact in energy-efficient portable video communication system design.

Wireless video sensor networks have been envisioned for a wide variety of important applications. In this work, we investigate the following research problem: given a mobile video sensor with an initial resource configuration, what is the maximum video quality it can provide, and what is the best video encoding and streaming strategy to achieve this performance? To address this problem, we study the energy consumption of each sensor node in video compression and wireless data transmission. We statistically model the node mobility and study its impact on wireless video transmission. We develop a task-oriented performance metric to measure the video sensing performance of the sensor node. We develop an optimal resource allocation and scheduling scheme for the video sensor to maximize the performance metric. The results provide an important step stone for our future performance study of wireless video sensor networks.

This paper presents a shot boundary detection (SBD) method that finds boundaries between shots using the changes in visual content elements such as objects, actors, and background. Our work presented in this paper is based on the property that the features do not change significantly within a shot whereas they change substantially across a shot boundary. Noticing this characteristic of shot boundaries, we propose a SBD algorithm using the scale- and rotationinvariant local image descriptors. To obtain information of the content elements, we employ the scale invariant feature transform (SIFT) that has been commonly used in object recognition. The number of matched points is large within the same shot whereas zero or the small number of matched points is detected at the shot boundary because all the elements in the previous shot change abruptly in the next shot. Thus we can determine the existence of shot boundaries by the number of matched points. We identify two types of shot boundaries (hard-cut and gradual-transition such as tiling, panning, and fade in/out) with a adjustable frame distance between consecutive frames. Experimental results with four test videos show the effectiveness of the proposed SBD algorithm using scale invariant feature matching.

The paper proposes a novel unsupervised video object segmentation algorithm for image sequences with low depth-offield (DOF), which is a popular photographic technique enabling to represent the intention of photographer by giving a clear focus only on an object-of-interest (OOI). The proposed algorithm largely consists of two modules. The first module automatically extracts OOIs from the first frame by separating sharply focused OOIs from other out-of-focused foreground or background objects. The second module tracks OOIs for the rest of the video sequence, aimed at running the system in real-time, or at least, semi-real-time. The experimental results indicate that the proposed algorithm provides an effective tool, which can be a basis of applications, such as video analysis for virtual reality, immersive video system, photo-realistic video scene generation and video indexing systems.

In block-based video compression technology, blocking artifacts are obvious because of the luminance and chrominance discontinuities which are caused by block-based discrete cosine transform (DCT) and motion compensation. As a kind of solution, an in-loop filter has been successfully used in H.264 adapting to quantization parameter and video content. In this paper, blocking artifacts distribution properties are analyzed carefully to reflect the blocking effect more accurately in the low bit rate applications. Two important parameters, named blocking severity and pixel variation, are defined to describe the boundary strength and the gradient of the samples across the edge respectively. Through series of statistical data retrieval and analysis for these parameters using multiple representative video sequences, a novel blocking artifacts distribution model is concluded. Based on this distribution model, an improved filter is proposed to H.264 with novel strength determination rule and different alpha model. Comparing with H.264 anchor results, the proposed de-blocking filter shows better performance especially in subjective aspect, which could be widely used in low bit rate applications.

Intra prediction coding is one of the many coding-efficiency oriented tools of H.264/AVC, but it requires high computational complexity. Many fast intra coding algorithms have been proposed to reduce the computational complexity of intra prediction, but most of them have been focused on the mode decision methods themselves. In this paper, we propose a fast algorithm in which new intra modes are substituted for certain of the conventional intra modes, so that the number of intra modes can be reduced. The proposed intra modes, namely the weighted mean and median modes, can effectively represent the directions of a block in a frame. The simulation results showed that the proposed method could reduce the encoding time of the overall sequence by about 11 % and that of the I-frames by about 28%, without any noticeable degradation of the coding efficiency

This paper presents how to split touching particles. The algorithm mainly consists of two sub-algorithms; they are splitting based on concavities of boundaries and splitting based on object skeleton histogram. It first traces particle boundaries based on discontinuities, then splits touching parts of particles based on shape information, where no gray information can be used, by using morphological mathematics method (e.g. distance map) combining particle shape information (information is classified by using fuzzy mathematics). After that, the algorithm split the remaining touching parts by using the information of concavities of particle boundaries The algorithm can automatically delineate particles not only on a simple image (a few particles touching) and also on a complex image (many particles touching together). The algorithm has been tested in a laboratory; it works well for various particle images.

In this paper, we propose a new featured-based geo-registration technique for aerial images that helps nearly real-timely calibrate and update the telemetry data and sensor model on flight. Firstly by using the available telemetry information, we bring both images into a common projection space. Second, we use the distribution of local geometric properties to guess the initial rotation and translation parameters. After the coarse alignment, the Harris operator is adopted to produce the initial feature points set, while the corresponding feature points set in the target image can be established through normalized correlation in local neighborhood of the selected points' position. Finally after the matched feature point sets are well constructed, the transform parameters can be solved by least square error estimation. The experiment shows our algorithm can effectively reduce the error in telemetry data caused by various noises.

A new segmentation approach usable for fixed or motion compensated camera is described. Instead of the often used RGB color space we operate with the invariant Gaussian color model proposed by Geusebroek and temporal information which eliminates unsteady regions surrounded by the moving objects. The Gaussian color model has never been used in video segmentation. Comparison with some state of the art methods in which both subjective and objective evaluation are applied proof the good performance of the proposed method.

In this paper, we address the problem of scrambling regions of interest in a video sequence. We target applications such as video surveillance preserving privacy, anonymous video communications, or TV news safeguarding the anonymity of a source. We propose an efficient solution based on transform-domain scrambling. More specifically, the sign of selected transform coefficients is flipped during encoding. We address the two cases of Motion JPEG 2000 and MPEG-4. Simulation results show that it can be successfully applied to conceal information in regions of interest in the scene while providing with a good level of security. Furthermore, the scrambling is flexible and allows adjusting the amount of distortion introduced. Finally, this is achieved with a small impact on coding performance and negligible computational complexity increase.

This paper presents a novel approach for automatic and robust object detection. It utilizes a component-based approach that combines techniques from both statistical and structural pattern recognition domain. While the component detection relies on Haar-like features and an AdaBoost trained classifier cascade, the topology verification is based on graph matching techniques. The system was applied to face detection and the experiments show its outstanding performance in comparison to other face detection approaches. Especially in the presence of partial occlusions, uneven illumination and out-of-plane rotations it yields higher robustness.

A new embedded rate scalable and resolution scalable, lossless image compression technique based on Highly Scalable - Set Partitioning in Hierarchical Trees (HS-SPIHT) is developed. The new method, called Resolution Scalable-SPIHT (RS-SPIHT), encodes the regions in the next higher resolution that are most likely to be significant along with the current resolution. The advantage of doing so is two fold: (1) the resulting data stream is rate scalable - spatially scalable bit stream, that is at any given data rate the next higher image resolution can be achieved which is not possible using SPIHT, and (2) the largest possible recoverable resolution at a particular date rate has a higher quality than could be previously reached using HS-SPIHT.

We present robust global motion estimation by MDPE algorithm for video stabilization. MDPE algorithm is a robust model fitting method, which has high breakdown point. We evaluate the performance of global motion estimation by robust model fitting methods, M-estimator, RANSAC and MDPE. An improvement of the MDPE for video stabilization is also presented. Finally, the video stabilization based on our robust global motion estimation is examined.

Originally, mathematical morphology was a theory of signal transformations which are invariant under Euclidean translations. An interest in the extension of mathematical morphology to spatially-variant (SV) operators has emerged due to the requirements imposed by numerous applications in adaptive signal (image) processing. This paper presents a general theory of spatially-variant mathematical morphology in the Euclidean space. We define the binary and gray-level spatially-variant basic morphological operators (i.e., erosion, dilation, opening and closing) and study their properties. We subsequently derive kernel representations for a large class of binary and gray-level SV operators in terms of the basic SV morphological operators. The theory of SV mathematical morphology is used to extend and analyze two important image processing applications: morphological image restoration and skeleton representation of binary images. For morphological image restoration, we obtain new realizations of adaptive median filters in terms of the basic SV morphological operators. For skeleton representation, we develop an algorithm to construct the optimal structuring elements, in the sense of minimizing the cardinality of the spatially-variant morphological skeleton representation. Experimental results show the power of the proposed theory of spatially-variant mathematical morphology in practical image processing applications.

In this paper, we address the problem of video frame rate up conversion (FRC) in compressed domain. FRC is often recognized as video temporal interpolation. The problem is very challenging when targeted for a video sequence with an inconsistent camera and object motion. A novel compressed domain motion compensation scheme is presented and applied in this paper. The proposed algorithm uses MPEG-2 compressed motion vectors to undergo a cumulative spatiotemporal interpolation over a temporal sliding window of frames. An iterative rejection scheme based on the affine motion model is exploited to detect the global camera motion. Subsequently, the foreground object separation is performed by examining the temporal consistency of the output of iterative rejections. This consistency check process helps to conglomerate the resulting foreground macroblocks and weeds out the unqualified blocks, thus further refines the crude segmentation results. Finally, different strategies for compensating the camera motion and the object motion are applied to interpolate the new frames. Illustrative examples are provided to demonstrate the efficacy of the proposed approach. Experimental results are compared with the popular block based frame interpolation approach.

The JPEG2000 coding scheme is slowly getting into use in imaging applications. The standard offers a set of new features and a better compression efficiency compared to the baseline JPEG. It requires, however, a high computational cost, making its usage in constrained devices, such as mobile phones or PDAs, challenging. We present a precompression rate allocation technique to reduce the encoding complexity and help adoption of JPEG2000 in such contexts.

In this work, we propose a method to control distortion fluctuation in 3D wavelet based video coding. A novel technique of assigning priorities to temporal subbands at different levels are used to smoothen the temporal PSNR. The wavelet filter properties are explored and an optimum criterion is derived to reduce the distortion fluctuation.

In this paper, we propose a novel computationally efficient post-processing algorithm to reduce ringing artifacts in the decoded DCT-coded video without using coding information. While the proposed algorithm is based on edge information as most filtering-based de-ringing algorithms do, this algorithm solely uses one single computationally efficient nonlinear filter, namely sigma filter, for both edge detection and smoothing. Specifically, the sigma filter, which was originally designed for nonlinear filtering, is extended to generate edge proximity information. Different from other adaptive filtering-based methods, whose filters typically use a fixed small window but flexible weights, this sigma filter adaptively switches between small and large windows. The adaptation is designed for removing ringing artifacts only, so the algorithm cannot be used for de-blocking. Overall, the proposed algorithm achieves a good balance among removing ringing artifacts, preserving edges and details, and computational complexity.

In scalable video coding (SVC), video quality can be controlled by bit-stream extraction. The bit-stream extraction is to extract network abstraction layer (NAL) units for the required quality. Since scalable bit-stream is packed in NAL units, quality control of SVC video is highly related to NAL unit extraction policy at a given bit-rate. Therefore, effective extraction policy is required to provide SVC video with optimal quality. In this paper, a SVC bit-stream extraction method based on perceptual quality is proposed. The main goal of this work is to find the optimal extraction policy for SVC bit-stream that contain the spatial, temporal, and SNR scalability at a given bit-rate. Also, to consider perceptual quality relied on video characteristics, the video segments are classified into different classes, namely action, crowd, dialog, scenery, and text&graphic. As a result of a subjective test on the classified video scenes, consistent characteristic of perceptual quality preference is achieved. Based on this, quality information table (QIT) has been determined for each class which guides the bit-stream extraction process. The determined QIT is applied to SVC bit-stream extraction depending on to which class a video segment belongs. In the experiment, the proposed extraction scheme is applied for SVC bit-stream extraction belongs to action class. Extraction policy to maximize perceptual quality of action class is applied based on QIT for action class. The extracted video and multi-dimensional scalability resulted from the proposed scheme are also described.

We propose an optimal framework for active surface extraction from video sequences. An active surface is a collection of active contours in successive frames such that the active contours are constrained by spatial and temporal energy terms. The spatial energy terms impose constraints on the active contour in a given frame. The temporal energy terms relate the active contours in different frames to preserve desired internal and external properties of the active surface. For computational efficiency, we reduce the 3-D active surface ((x,y,t) coordinates) optimization problem to a 2-D model ((φ,t) coordinates) by considering only point indices along normal lines φ of each contour and define the energy terms in a causal way. We develop an n-D dynamic tree programming algorithm to find the optimum of n-D semi-causal functions. We prove that the n-D dynamic tree programming algorithm converges to the global optimum. In particular, the classical 1-D dynamic programming algorithm is a special case of the n-D dynamic tree programming algorithm. The optimal active surface is subsequently obtained by using the 2-D dynamic tree programming algorithm. Simulation results show the efficiency and robustness of the proposed approach in active surface extraction for video tracking and segmentation of the human head in real-world video sequences.

In this paper, we address the problem of robust transmission of packet based H.264/AVC video over direct sequence-code division multiple access (DS-CDMA) channels. H.264 based data partitioning is used to produce video packets of unequal importance with regards to their need in terms of the decoded video quality. In the proposed transmission system, the data partitioned video packets are packetized as per IP/UDP/RTP protocol stack and are sorted into different levels for giving unequal error protection (UEP) using Rate Compatible Punctured Convolutional (RCPC) codes. Constant size framing is done at the link layer and Cyclic Redundancy Check header (CRC) is attached for error detection. Link layer buffering and packet interleaving schemes are proposed to improve the efficiency of the system. A multipath Rayleigh fading channel with Additive White Gaussian Noise (AWGN) and interference from other users is considered at the physical layer. The link layer frames are channel encoded, spread and transmitted over the channel. The received data is despread/demodulated using the Auxiliary Vector (AV) filter or RAKE matched filter (RAKE-MF) receiver and subsequently channel and source decoded. Our experimental results show the effectiveness of using data partitioning for wireless transmissions when compared to the system not using data partitioning. Also the superior interference mitigation capabilities of AV receiver is shown in comparison to the RAKE-MF receiver.

In this paper, a novel VLC method based on 2nd-run-level coding and dynamic truncation is proposed to compress the DCT coefficients efficiently. In the proposed VLC, 2nd-run-level is first employed, following the traditional run-level coding, to further reduce the considerable redundancy existing in the original level and run sequences. In order to achieve a higher degree of context adaptability and coding efficiency, dynamic truncation is introduced and employed in the sequential coding of 2D symbols without large amounts of 2D-VLC tables required, though adding extra coding complexity to some extent. Adaptive EG/GR selection is also presented and recommended since it brings extra improvement in compression efficiency without increasing any computational complexity. Experimental results show that when compared with context-based 2D-VLC, the proposed VLC method gains 0.25 dB ~0.79 dB in PSNR and achieves 5.30% ~ 11.58% improvement in bit rate reduction.

H.264 High Profile, one of the standard video coding schemes, can be utilized for super high definition video since target image resolution is up to 4096×2304. However, in the case of encoding super high definition video such as the 4Kx2K system whose resolution corresponds to 4096×2160, the coding syntax might be redundant since available macroblock size become quite small compared to the image resolution. The problem is caused by the limitation of the macroblock size by which motion compensated prediction is conducted. To overcome the problem within the framework of the H.264 High Profile encoder, extending the limitation of macroblock size is required. From this perspective, this paper describes an efficient coding scheme applicable to super high definition video based on extending the H.264 High Profile. As the key technology of the proposed coding scheme, an approach for extending the limitation of available macroblock size is introduced. From experimental results, it was confirmed that the outstanding coding gain was achieved by the proposed scheme especially for the low bit-rate condition.

In this paper, we suggest a new importance sampling scheme to improve a particle filtering based tracking process. This scheme relies on exploitation of motion segmentation. More precisely, we propagate hypotheses from particle filtering to blobs of similar motion to target. Hence, search is driven toward regions of interest in the state space and prediction is more accurate. We also propose to exploit segmentation to update target model. Once the moving target has been identified, a representative model is learnt from its spatial support. We refer to this model in the correction step of the tracking process. The importance sampling scheme and the strategy to update target model improve the performance of particle filtering in complex situations of occlusions compared to a simple Bootstrap approach as shown by our experiments on real fish tank sequences.

Certain applications require the extraction of patches of color from an image, their size and location. These applications may be: color harmonization algorithms, non-photorealistic rendering, etc. These applications use not too big a palette of colors, and in both cases large areas of homogeneous color are favored along with high detail preserved in the smaller areas with a lot of color activity. The main problem this paper will tackle is to identify the underlying color in an image region, which will be referred to as its underlying color patch, and also try to protect as much as possible the high color activity detail areas. No perfect scene object segmentation is intended in this process, since different objects may be quantized to the same color, the result may be a merged color patch.

Stereo video becomes an important issue with the developments of 3D display technologies. While a stereo system provides the perception of 3D depth, the amount of data for stereo video may be doubled compared to mono video. So an efficient stereo video coding technique is essential. Since the stereo video is taken from the same object in the two different views, most of objects in a stereo pair are translated only to the horizontal direction, while objects in subsequent frames of a mono video can be translated to an arbitrary direction, rotated, and zoomed. Hence, unlike in the mono video, the disparity in a stereo pair can be well estimated by a translational motion or disparity if the object boundary is exactly described. In this paper, we propose an efficient disparity estimation scheme based on a edge model describing object boundary in a block, and apply the estimated disparity to stereo video compression. In addition, a disparity regularization scheme, which is proper for the edge model, is proposed to reduce the bits required for coding the block motion vectors and disparity values. It has been found that the proposed algorithms significantly improve the coding efficiency of stereo video sequences.

In this paper, adaptive Lagrangian multiplier λ estimation in Larangian R-D optimization for video coding is presented that is based on the ρ-domain linear rate model and distortion model. It yields that λ is a function of rate, distortion and coding input statistics and can be written as λ(R, D, σ²) = β(ln(σ²/D) + δ)D/R + k₀, with β, δ and k₀ as coding constants, σ² is variance of prediction error input. λ(R, D, σ²) describes its ubiquitous relationship with coding statistics and coding input in hybrid video coding such as H.263, MPEG-2/4 and H.264/AVC. The lambda evaluation is de-coupled with quantization parameters. The proposed lambda estimation enables a fine encoder design and encoder control.