A new fuzzy filter is presented for the noise reduction of images corrupted with additive Gaussian noise. The filter consists of two stages. The first stage computes a fuzzy gradient for eight different directions around the currently processed pixel. The second stage uses the fuzzy gradient to perform fuzzy smoothing by taking different contributions of neighboring pixel values. The two stages are both based on fuzzy rules which make use of membership functions. The filter can be applied iteratively to effectively reduce heavy noise. The shape of the membership functions is adapted according to the remaining noise level after each iteration, making use of the distribution of the homogeneity in the image. The fuzzy operators are implemented by the classical min/max. Experimental results are obtained to show the feasibility of the proposed approach. These results are also compared to other filters by numerical measures and visual inspection.

High resolution images are often required in applications such as remote sensing, frame freeze in video, military and medical imaging. Digital image sensor arrays, which are used for image acquisition in many imaging systems, are not dense enough to prevent aliasing, so the acquired images will be degraded by aliasing effects. To prevent aliasing without loss of resolution, a dense detector array is required. But it may be very costly or unavailable, thus, many imaging systems are designed to allow some level of aliasing during image acquisition. The purpose of our work is to reconstruct an unaliased high resolution image from the acquired aliased image sequence. In this paper, we propose a spatially adaptive regularized iterative high resolution image reconstruction algorithm for blurred, noisy and down-sampled image sequences. The proposed approach is based on a Constrained Least Squares (CLS) high resolution reconstruction algorithm, with spatially adaptive regularization operators and parameters. These regularization terms are shown to improve the reconstructed image quality by forcing smoothness, while preserving edges in the reconstructed high resolution image. Accurate sub-pixel motion registration is the key of the success of the high resolution image reconstruction algorithm. However, sub-pixel motion registration may have some level of registration error. Therefore, a reconstruction algorithm which is robust against the registration error is required. The registration algorithm uses a gradient based sub-pixel motion estimator which provides shift information for each of the recorded frames. The proposed algorithm is based on a technique of high resolution image reconstruction, and it solves spatially adaptive regularized constrained least square minimization functionals. In this paper, we show that the reconstruction algorithm gives dramatic improvements in the resolution of the reconstructed image and is effective in handling the aliased information. The proposed algorithm is also shown to be robust in the presence of severe registration error. Experimental results are provided to illustrate the performance of the proposed reconstruction algorithm. Comparative analyses with other reconstruction methods are also provided.

This paper proposes a Bayesian method for the restoration of video sequences compressed using the discrete cosine transform (DCT). Two elements, both part of the Bayesian observation model, distinguish the proposed algorithm from the majority of other methods in the literature. The proposed algorithm incorporates temporal information from nearby frames -- past, present and future -- when forming an estimate of the current frame. Furthermore, this work uses a spatially-varying noise model to account for the noise introduced by quantization of the DCT coefficients. These two aspects of the observation model are used in conjunction with a Huber-Markov Random Field (HMRF) model to form a Bayesian estimate of each frame in the compressed video sequence.

This paper introduces a new framework of sequential error concealment techniques for block-based image coding systems. Unlike previous approaches which simultaneously recover the pixels inside the missing block, we propose to recover them in a sequential fashion. The structure of sequential recovery enhances the capability of handling complex texture patterns in the image and serious block loss situations during the transmission. Under the framework of sequential recovery, we present a novel spatially adaptive scheme to interpolate the missing pixels along the edge orientation. We also study the problem of how to fully exploit the information from the available surrounding neighbors with the sequential constraint. Experiment results have shown that novel sequential recovery techniques are superior to most existing parallel recovery techniques in terms of both subjective and objective quality of reconstructed images.

We present a technique that provides progressive transmission and near-lossless compression in one single framework. The proposed technique produces a bitstream that results in progressive reconstruction of the image just like what one can obtain with a reversible wavelet codec. In addition, the proposed scheme provides near-lossless reconstruction with respect to a given bound after each layer of the successively refinable bitstream is decoded. We formulate the image data compression problem as one of asking the optimal questions to determine, respectively, the value or the interval of the pixel, depending on whether one is interested in lossless or near-lossless compression. New prediction methods based on the nature of the data at a given pass are presented and links to the existing methods are explored. The trade-off between non- causal prediction and data precision is discussed within the context of successive refinement. Context selection for prediction in different passes is addressed. Finally, experimental results for both lossless and near-lossless cases are presented, which are competitive with the state-of-the-art compression schemes.

In this work, near-lossless compression, i.e., yielding strictly bounded reconstruction error, is proposed for high- quality data compression. An interframe causal DPCM scheme is presented for interframe compression of remotely sensed optical data, both multispectral and hyperspectral, as well as of volumetric medical data. The proposed encoder relies on a classified linear-regression prediction, followed by context- based arithmetic coding of the outcome prediction errors. It provides outstanding performances, both for reversible and for irreversible, i.e., near-lossless, compression. Coding time are affordable thanks to fast convergence of training. Decoding is always performed in real time.

In this work we present some image coding models based on adaptive approximation techniques. The image coding models presented are based on Matching Pursuit and High Resolution Pursuit, which are the most popular adaptive approximation techniques. These models have a similar computational complexity and structure. The models expands an image along an overcomplete dictionary. The dictionary was selected according to a best basis metric or a training strategy. From such expansion, the model selects the coefficients that correspond to the most important image structures. Selected coefficients are quantized just when they are chosen, in order to minimize error propagation along the process. These coefficients represent an optimal image decomposition, or a reduced image representation. This representation, in some way, corresponds to a coded image with a high compression rate. A simple reconstruction algorithm recovers the original image with a high visual quality.

With desktop imaging devices becoming ubiquitous, effectively managing the images in large collections has become a challenge. The requirements for a modern imaging system now demand not only efficient storage (low bit rate coding), but also easy manipulation, indexing and retrieval of images. In this paper, we introduce a new method for color image coding based on a visual appearance model of local color image patterns. The visual appearance of small image patterns is characterized by their spatial pattern, color direction and local energy strength. To encode the local visual appearance, an approach based on vector quantization (VQ) is introduced. A separate VQ is designed for the spatial pattern and color direction respectively. It is shown that the method not only achieves good image coding results in terms of rate distortion criterion, it also enables content-based retrieval to be performed in the compressed domain easily and conveniently.

The International Standards Organization (ISO) has proposed a family of standards for compression of image and video sequences, including the JPEG, MPEG-1 and MPEG-2. The latest MPEG-4 standard has many new dimensions to coding and manipulation of visual content. A video sequence usually contains a background object and many foreground objects. Portions of this background may not be visible in certain frames due to the occlusion of the foreground objects or camera motion. MPEG-4 introduces the novel concepts of Video Object Planes (VOPs) and Sprites. A VOP is a visual representation of real world objects with shapes that need not be rectangular. Sprite is a large image composed of pixels belonging to a video object visible throughout a video segment. Since a sprite contains all parts of the background that were at least visible once, it can be used for direct reconstruction of the background Video Object Plane (VOP). Sprite reconstruction is dependent on the mode in which it is transmitted. In the Static sprite mode, the entire sprite is decoded as an Intra VOP before decoding the individual VOPs. Since sprites consist of the information needed to display multiple frames of a video sequence, they are typically much larger than a single frame of video. Therefore a static sprite can be considered as a large static image. In this paper, a novel solution to address the problem of spatial scalability has been proposed, where the sprite is encoded in Discrete Wavelet Transform (DWT). A lifting kernel method of DWT implementation has been used for encoding and decoding sprites. Modifying the existing lifting scheme while maintaining it to be shape adaptive results in a reduced complexity. The proposed scheme has the advantages of (1) avoiding the need for any extensions to image or tile border pixels and is hence superior to the DCT based low latency scheme (used in the current MPEG-4 verification model), (2) mapping the in place computed wavelet coefficients into a zero tree structure without actually rearranging them, thereby saving allocation of additional memory. The proposed solutions provide efficient implementation of the sprite coder making possible a VLSI realization with a reduced real estate.

Prediction is an essential operation in many image processing applications, such as object detection and image and video compression. When the image is modeled as Gaussian, the optimal predictor is linear and easy to obtain. However, image texture and clutter are often non-Gaussian and in such cases, optimal predictors are difficult to find. In this paper, we have derived an optimal predictor for an important class of non-Gaussian image models, the block-based multivariate Gaussian mixture model. This predictor has a special non- linear structure: it is a linear combination of the neighboring pixels but the combination coefficients are also functions of the neighboring pixels, not constants. The efficacy of this predictor is demonstrated in object recognition experiments where the prediction error image is used to identify 'hidden' objects. Results indicate that when the background texture is non-linear, i.e., with fast- switching gray-level patches, it performs significantly better than the optimal linear predictor.

In semantic content-based image/video browsing and navigation systems, efficient mechanisms to represent and manage a large collection of digital images/videos are needed. Traditional keyword-based indexing describes the content of multimedia data through annotations such as text or keywords extracted manually by the user from a controlled vocabulary. This textual indexing technique lacks the flexibility of satisfying various kinds of queries requested by database users and also requires huge amount of work for updating the information. Current content-based retrieval systems often extract a set of features such as color, texture, shape motion, speed, and position from the raw multimedia data automatically and store them as content descriptors. This content-based metadata differs from text-based metadata in that it supports wider varieties of queries and can be extracted automatically, thus providing a promising approach for efficient database access and management. When the raw data volume grows very large, explicitly extracting the content-information and storing it as metadata along with the images will improve querying performance since metadata requires much less storage than the raw image data and thus will be easier to manipulate. In this paper we maintain that storing metadata together with images will enable effective information management and efficient remote query. We also show, using a texture classification example, that this side information can be compressed while guaranteeing that the desired query accuracy is satisfied. We argue that the compact representation of the image contents not only reduces significantly the storage and transmission rate requirement, but also facilitates certain types of queries. Algorithms are developed for optimized compression of this texture feature metadata given that the goal is to maximize the classification performance for a given rate budget.

In this paper an algorithm is presented to design encoders that achieve good compression and classification. The goal is to minimize the classification error introduced by quantizing the data using encoders operating on low dimension inputs, which are subsets of the high dimension data used by the classifier for classification. In the encoder design information from the other dimensions of the vector is used to develop efficient encoders which are capable of achieving lower classification error for a given distortion. The design allows a trade-off between distortion and classifications costs providing more flexibility in the overall system design. The algorithm is tested on Gaussian mixture data, which is classified using a classifier which takes as input vectors of quantized values. The proposed technique can trade performance to achieve lower complexity, which is desirable in devices having limited computational resources. For 4 dimensional Gaussian mixture data the misclassification was about 2.2% more than that achieved by using encoders of the same dimension as the classifier, while the encoding complexity was reduced by more than a factor of 2.

In the image interpretation domain, one of very scrupulous stages rests in the partitioning of pixels into principal components that correspond to the part of the image. However, the approaches used for a segmentation process, either have a very strong specificity or require many sensible parameters, which make their use in a general case very difficult. Using them in inappropriate cases leads to very bad performance in terms of result quality and relevance. Broadly speaking, to obtain an acceptable segmentation result, we have to choose the appropriate method which best adapted method and we have to turn it very carefully. However, even though the need for an adaptable methodology has been long recognized, few proposes have been appeared. In this paper we present a cooperative and concurrent framework for image segmentation. Our approach integrate several region based methods and detector based boundary finding using Multi-Agents System (MAS). Thanks to natural MAS properties, the image analysis is done at several semantic levels. The novelty of our approach is that is a multi directional cooperation whereby each separate method improves its own result as well as the global one through mutual cooperation and information sharing.

We present an adaptive unsupervised segmentation technique, in which spectral features are obtained and processed without a priori knowledge of the spectral characteristics. The proposed technique is based on an iterative method, in which segmentation at a given iteration depends closely on the segmentation results at the previous iteration. The hyperspectral images are first coarsely segmented and then the segmentation is successively refined via an iterative spectral dissimilarity measure. The algorithm also provides reduced computational complexity and improved segmentation performance. The algorithm consists of (1) an initial segmentation based on a fixed spectral dissimilarity measure and the k-means algorithm, and (2) subsequent adaptive segmentation based on an iterative spectral dissimilarity measure over a local region whose size is reduced progressively. The iterative use of a local spectral dissimilarity measure provided a set of values that can discriminate among different materials. The proposed unsupervised segmentation technique proved to be superior to other unsupervised algorithms, especially when a large number of different materials are mixed in complex hyperspectral scenes.

Today's standard video coders employ the hybrid coding scheme on a macroblock basis. In these coders blocks of 16 X 16, and 8 X 8 pixel are used for motion compensation of non- interlaced video. The Discrete Cosine Transform (DCT) is then applied to the prediction error on blocks of size 8 X 8. The emerging coding standard H.26L employs a set of seven different block sizes for motion compensation. The size of these blocks varies from 4 X 4 to 16 X 16. The block sizes smaller than 8 X 8 imply that the 8 X 8 DCT cannot be used for transform coding of the prediction error. In the current test model an integer approximation of the 4 X 4 DCT matrix is employed. In this paper the concept of Adaptive Block Transforms is proposed. In this scheme, the transform block size is adapted to the block sizes used for motion compensation. The transform exploits the maximum possible signal length for transform coding without exceeding the compensated block boundaries. The proposed scheme is integrated into the H.26L test model. New integer approximations of the 8 X 8 and 16 X 16 DCT matrices are introduced. Like the TML 4 X 4 transform the coefficient values of these matrices are restricted to a limited range. The results presented here are based on an entropy estimation. They reveal an increased rate/distortion performance of approximately 1.1 dB for high rates on the employed test sequences.

Pre-processing algorithms improve on the performance of a video compression system by removing spurious noise and insignificant features from the original images. This increases compression efficiency and attenuates coding artifacts. Unfortunately, determining the appropriate amount of pre-filtering is a difficult problem, as it depends on both the content of an image as well as the target bit-rate of compression algorithm. In this paper, we explore a pre- processing technique that is loosely coupled to the quantization decisions of a rate control mechanism. This technique results in a pre-processing system that operates directly on the Displaced Frame Difference (DFD) and is applicable to any standard-compatible compression system. Results explore the effect of several standard filters on the DFD. An adaptive technique is then considered.

Lossless video coding is useful in applications where no loss of information or visual quality is tolerable. In embedded to lossless coding an encoded video stream can be decoded into any bit rate up to the lossless bit rate, which is quite useful in numerous applications. In this paper, the research we present is based on lossless video coding, which uses motion compensated prediction to eliminate temporal redundancy. We specifically address the problem of embedded to lossless coding of the motion compensated prediction residuals. Since the statistical properties of the residuals are different from still images, the best wavelet bases for still images do not perform well for residuals. Since residuals contain a higher portion of high frequency information in high motion regions, a fixed transform for the entire frame is not very efficient. We introduce a spatially adaptive wavelet transform method, which takes the local frame statistics into account before choosing the wavelet base. This transform technique provided the best performance for most of the test frames.

Common motion compensated hybrid video coding standards such as H.263, MPEG-1, MPEG-2, MPEG-4 are based on a fractional-pel displacement vector resolution of 1/2-pel. Recent approaches like MPEG-4 ACE and H.26L (TML5) use a 1/4-pel displacement vector resolution. In order to estimate and compensate fractional-pel displacements, the image signal has to be interpolated. Therefore different interpolation filters are used in the standards. In this paper an enhanced motion compensated hybrid video codec is presented, which is based on high-resolution displacement vectors. For this purpose, displacement vector resolutions of 1/8- and 1/16-pel are used in order to improve the motion compensated prediction and the coding efficiency. The coding results for different resolutions are presented and the dependence on different interpolation filters is analyzed. It turned out that the higher the displacement vector resolution is, the higher the influence of the filter on the coding gain is. A gain up to 3.0 dB PSNR is obtained compared to a hybrid video codec, which is based on 1/2-pel resolution and bilinear interpolation like H.263, MPEG-1,2,4. Compared to 1/4-pel displacement vector resolution and a wiener interpolation filter as it is used in MPEG-4 ACE and H.26L (TML5), a gain up to 1.0 dB PSNR is obtained.

Object-based coding approaches, such as the MPEG-4 standard approach, where a video scene is composed by several video objects, require that the rate control is performed by using two levels: the scene rate control and the object rate control. In this context, this paper presents a new scene level and object level rate control algorithm for low delay MPEG-4 video encoding capable of performing bit allocation for the several VOs in the scene, encoded at different VOP rates, and aiming at obtaining a better trade-off among spatial and temporal quality for the overall scene. The proposed approach combines rate-distortion modeling using model adaptation by least squares estimation and adaptive bit allocation to 'shape' the encoded data in order that the overall subjective quality of the encoded scene is maximized.

In this paper, we propose a method for generating arbitrary view image by interpolating images between three cameras using epipolar geometry. Projective geometry has recently been used in the field of computer vision, because projective geometry can be easily determined comparing with Euclidean geometry. In the proposed method, three input camera images are rectified so that the vertical and horizontal directions can be completely aligned to the epipolar planes between the cameras. This rectification provides Projective Voxel Space (PVS), in which the three axes are aligned with the direction of camera's projection. Such alignment simplifies the procedure for projection and back projection between the 3D space and the image planes. First, we apply shape-from-silhouette with taking advantage of PVS. The consistency of color value between the images is evaluated for final determination of the object surface voxel. Therefore, consistent matching in three images is estimated and images can be interpolated from the matching information. Synthesized images are based on 3D shape in PVS, so the occlusion of the object is reproduced in the generated images, however it requires only weak calibration.

In this paper we propse an approach to generate a panorama depth map which is dense enough to be used as a template for cutting out a texture image by the depth, i.e. structuring the scene into layers to construct a shared 3-D image space. For depth estimation, we make use of census transform for robust determination of correspondence of a stereo pair. To interpolate unknown disparities, we introduced a process influenced by the K-means algorithm. For densification of the depth data, we make use of Region Competition. During the stitching, the confidence of the data is improved for the overlapped areas by multiple evaluation of the disparity data.

This paper presents a regularized smoothing algorithm for 3D reconstruction from image sequence. Depth data estimated from a stereo pair or multiple image frames can easily be corrupted by various types of noise such as quantization and imperfect matching. We propose a regularized image restoration algorithm which enhances the surface of depth maps based on spatially adaptive image fusion. We can also enhance the resolution of the surfaces and preserve discontinuities.

In this paper, we report on a convenient and unified framework for generating a photo-realistic interactive virtual environment (piVE) using heterogeneous multiview cameras, while not using bluescreen techniques and special rendering hardware. In spite of the rapid growth of computer hardware, rendering a photo-realistic virtual environment on the fly is still a challenging problem. With the proposed framework, exploiting stereo images/videos, piVE can be rendered in realtime without using expensive high-end computer with rendering hardware. The proposed framework consists of three main parts, i.e. (1) photo-realistic virtual space generation exploiting a camera with stereoscopic adapter, (2) generation of a video avatar (a special object representing the user) by exploiting multiview camera, and (3) graphics object rendering according to the given camera parameters and the user's interaction. We also address z-keying issues among background video, graphics objects and video avatar.

This article deals with video sequence compression for very low bitrate. We assume static scene captured with a monocular moving camera. We propose a method using several 3D models in order to reconstruct the original video sequence or a virtual one. A complete scheme for video sequence compression is presented and compared with classical compression schemes.

Block-based disparity compensation is an efficient prediction scheme for encoding multi-view image data. Available scene geometry can be used to further enhance prediction accuracy. In this paper, three different strategies are compared that combine prediction based on depth maps and 3-D geometry. Three real-world image sets are used to examine prediction performance for different coding scenarios. Depth maps and geometry models are derived from the calibrated image data. Bit-rate reductions up to 10% are observed by suitably augmenting depth map-based with geometry-based prediction.

This paper presents a fully software-based MPEG-2 encoder architecture, which uses scene-change detection to optimize the Group-of-Picture (GOP) structure for the actual video sequence. This feature enables easy, lossless edit cuts at scene-change positions and it also improves overall picture quality by providing good reference frames for motion prediction. Another favorable aspect is the high coding speed obtained, because the encoder is based on a novel concept for parallel MPEG coding on SMP machines. This concept allows the use of advanced frame-based coding algorithms for motion estimation and adaptive quantization, thereby enabling high- quality software encoding in real-time. Our proposal can be combined with the conventional parallel computing approach on slice basis, to further improve parallelization efficiency. The concepts in the current SAMPEG implementation for MPEG-2 are directly applicable to MPEG-4 encoders.

Parallel encoding of video inevitably gives varying frame rate performance due to dynamically changing video content and motion field since the encoding process of each macro-block, especially motion estimation, is data dependent. A multiprocessor schedule optimized for a particular frame with certain macro-block encoding time may not be optimized towards another frame with different encoding time, which causes performance degradation to the parallelization. To tackle this problem, we propose a method based on a batch of near-optimal schedules generated at compile-time and a run-time mechanism to select the schedule giving the shortest predicted critical path length. This method has the advantage of being near- optimal using compile-time schedules while involving only run- time selection rather than re-scheduling. Implementation on the IBM SP2 multiprocessor system using 24 processors gives an average speedup of about 13.5 (frame rate of 38.5 frames per second) for a CIF sequence consisting of segments of 6 different scenes. This is equivalent to an average improvement of about 16.9% over the single schedule scheme with schedule adapted to each of the scenes. Using an open test sequence consisting of 8 video segments, the average improvement achieved is 13.2%, i.e. an average speedup of 13.3 (35.6 frames per second).

This paper addresses the problem of efficiently decoding high- definition (HD) video for display at a reduced resolution. The decoder presented in this paper is intended for applications that are constrained not only in memory size, but also in peak memory bandwidth. This is the case, for example, during decoding of a high-definition television (HDTV) channel for picture-in-picture (PIP) display, if the reduced resolution PIP-channel decoder is sharing memory with the full-resolution main-channel decoder. The most significant source of video quality degradation in a reduced-resolution decoder is prediction drift, which is caused by the mismatch between the full-resolution reference frames used by the encoder and the subsampled reference frames used by the decoder. to mitigate the visually annoying effects of prediction drift, the decoder described in this paper operates at two different resolutions -- a lower resolution for B pictures, which do not contribute to prediction drift and a higher resolution for I and P pictures. This means that the motion-compensation unit (MCU) essentially operates at the higher resolution, but the peak memory bandwidth is the same as that required to decode at the lower resolution. Storage of additional data, representing the higher resolution for I and P pictures, requires a relatively small amount of additional memory as compared to decoding at the lower resolution. Experimental results will demonstrate the improvement in video quality achieved by the addition of the higher-resolution data in forming predictions for P pictures.

The two dimensional discrete cosine transform (2-D DCT) has been chosen as the basis in almost all of the recent international image and video coding standards. This paper first categorized the 2-D DCT and inverse DCT (IDCT) architectures. Then a new VLSI architecture for 2-D DCT/IDCT without transpose memory was proposed. The proposed 2-D DCT/IDCT architecture eliminates special transpose circuits and uses general memory modules to store the intermediate results after row-wise transform. The row-wise and column-wise transforms are performed with the different data flow provided by the configurable computation units and data alignment module. The accuracy testing system is set up to search the optimum word-length parameters. Based on the accuracy testing system, the proposed architecture has achieved the smallest word-length compared with reported 2-D DCT architectures.

Fractal image compression has several useful properties, including resolution independence, high compression ratios, rapid decoding, and subjectively good image quality. Despite this, its adoption has been limited, largely because of the very high computational burden of compression. In this paper, an original ASIC design is described which performs the comparisons required for fractal image compression of grayscale images. The design is based around a pipeline of many individual block comparison units through which the set of domain blocks are propagated. Each unit performs the required comparisons between a single range block and the series of domain blocks, and stores the best possible match. The ASIC is designed using synthesizable VHDL, allowing it to be targeted to virtually any digital technology. The small portions of the design which are specific to a particular application or technology, such as the bus interfaces and memory blocks, have been kept separate from the application- independent portions of the design. Simulations of the design suggests that an actual hardware implementation would be about one thousand times faster than a general purpose microprocessor based on similar IC technology, reducing the time required to optimally compress a 256 X 256 image using 8 X 8 range blocks from a few minutes to a fraction of a second.

In this paper we present the Elliptical Weighted Average filtering algorithm and an optimized implementation of a two- pass algorithm and used in digital image and video warping. Two-pass algorithms are well suited for hardware implementation due to their reduced complexity in using 1-D re-sampling and anti-aliasing filters. But, the primary disadvantage is the need for a large buffer to store the temporary image since warping is performed in two passes. The size of the temporary buffer is equal to or greater than the size of the input image. A dedicated, hardware, implementation for this algorithm implies huge cost in terms of real estate on chip. In our approach, Wolberg-Boult's resampling algorithm is modified to use only two rows of temporary buffer thereby making the algorithm more amenable for hardware implementation. We present the complexity analysis based on number of arithmetic and logic operations (add, shift, compare, multiply, clip and divide) per macroblock. In the case of EWA filters, it is the most cost-effective high- quality filtering method because point inclusion testing can be done with one function evaluation and the filter weights can be stored in lookup tables for reduction in computation. For mapping the quadrilaterals, four equations were needed for the four lines of the quadrilaterals, which was computationally complex, wherein the computational cost was directly proportional to the number of input pixels accessed. Also we present the complexity analysis per macroblock.

This paper describes a method for establishing dense correspondence between two images in a video sequence (motion) or in a stereo pair (disparity) in case of large displacements. In order to deal with large-amplitude motion or disparity fields, multi-resolution techniques such as blocks matching and optical flow have been used in the past. Although quite successful, these techniques cannot easily cope with motion/disparity discontinuities as they do not explicitly exploit image structure. Additionally, their computational complexity is high; block matching requires examination of numerous vector candidates while optical flow-based techniques are iterative. In this paper, we propose a new approach that addresses both issues. The approach combines feature matching with Delaunay triangulation, and thus reliable long-range correspondences result while the computational complexity is not high (sparse representation). In the proposed approach, feature points are found first using a simple intensity corner detector. Then, correspondence pairs between two images are found by maximizing cross-correlation over a small window. Finally, the Delaunay triangulation is applied to the resulting points, and a dense vector field is computed by planar interpolation over Delaunay triangles. The resulting vector field is continuous everywhere, and thus does not reflect motion or depth discontinuities at object boundaries. In order to improve the rendition of such discontinuities, we propose to further divide Delaunay triangles whenever the displacement vectors within a triangle do not allow good intensity match. The approach has been extensively tested on stereoscopic images in the context of intermediate view reconstruction where the quality of estimated disparity fields is critical for final image rendering. The first results are very encouraging as the reconstructed images are of high quality, especially at object boundaries, and the computational complexity is lower than that of multi- resolution block matching.

The quality of the interpolated images in picture rate upconversion is predominantly dependent on the accuracy of the motion vector fields. Block based MEs typically yield incorrect vectors in occlusion areas, which leads to an annoying halo in the upconverted video sequences. In the past we have developed a cost-effective block based motion estimator, the 3D Recursive Search ME, and an improved accuracy version for tackling occlusion, the tritemporal ME. In this article we describe how the vector field from this tritemporal ME is further improved by a retimer, using information from a foreground/background detector. More accurate motion vector fields are of importance to other applications also (e.g. video compression, 3D, scene analysis ...).

In this paper, we propose a fast adaptive diamond search algorithm (FADS) for block matching motion estimation. Many fast motion estimation algorithms reduce the computational complexity by the UESA (Unimodal Error Surface Assumption) where the matching error monotonically increases as the search moves away from the global minimum point. Recently, many fast BMAs (Block Matching Algorithms) make use of the fact that global minimum points in real world video sequences are centered at the position of zero motion. But these BMAs, especially in large motion, are easily trapped into the local minima and result in poor matching accuracy. So, we propose a new motion estimation algorithm using the spatial correlation among the neighboring blocks. We move the search origin according to the motion vectors of the spatially neighboring blocks and their MAEs (Mean Absolute Errors). The computer simulation shows that the proposed algorithm has almost the same computational complexity with DS (Diamond Search), but enhances PSNR. Moreover, the proposed algorithm gives almost the same PSNR as that of FS (Full Search), even for the large motion with half the computational load.

In this paper we address the classical problem of estimating, from a pair of consecutive frames of a video sequence, the motion (or velocity field, or optical flow) produced by planar translations and rotations, in the context of the Gauss- Laguerre Transform (GLT) theory. This contribution extends some previous works of the authors on wavelet based Optimum Scale-Orientation Independent Pattern Recognition. In particular here we make use of an orthogonal system of Laguerre-Gauss wavelets. Each wavelet represents the image by translated, dilated and rotated versions of a complex waveform whereas, for a fixed resolution, this expansion provides a local representation of the image around any point. In addition each waveform is self-steerable, i.e. it rotates by simple multiplication with a complex factor. These properties allow to derive an iterative joint translation and rotation field Maximum Likelihood (ML) estimation procedure based on a bank of CHWs. In this contribution the coarse estimate obtained by the memoryless, point wise, ML estimator is refined by resorting to a compound Markovian model that takes into account the spatial continuity of the motion field associated to a single object, by heavily penalizing abrupt changes in the motion intensity and direction not located in correspondence of intensity discontinuities (i.e. mostly object boundaries).

The purpose of our work is to report a solution for the problem of the Joint Source-Channel Coding (JSCC) of the H.263 bit stream to be transmitted through error prone channels. Our investigations led us to classify the H.263 syntactical elements into several classes of different significance. By identifying the most sensitive elements we developed a data partitioning (DP) technique which exhibits improved error resilience. A good reconstructed video quality is obtained for a constant 64 kbit/s transmission rate using Rate Compatible Punctured Convolutional (RCPC) codes of different rates for forward error protection. By exploiting the different syntactical element sensitivities we presented an Unequal Error Protection (UEP) scheme that surpasses the optimal Equal Error Protection (EEP). The forward error correction adopted has resulted in PSNR improvements over 20 dB for bit error rates higher than of 4 X 10^-3.

Video communication over lossy packet networks such as the Internet is hampered by limited bandwidth and packet loss. This paper presents a system for providing reliable video communication over these networks, where the system is composed of two subsystems: (1) multiple state video encoder/decoder and (2) a path diversity transmission system. Multiple state video coding combats the problem of error propagation at the decoder by coding the video into multiple independently decodable streams, each with its own prediction process and state. If one stream is lost the other streams can still be decoded to produce usable video, and furthermore, the correctly received streams provide bidirectional (previous and future) information that enables improved state recovery for the corrupted stream. This video coder is a form of multiple description coding (MDC), and its novelty lies in its use of information from the multiple streams to perform state recovery at the decoder. The path diversity transmission system explicitly sends different subsets of packets over different paths, as opposed to the default scenarios where the packets proceed along a single path, thereby enabling the end- to-end video application to effectively see an average path behavior. We refer to this as path diversity. Generally, seeing this average path behavior provides better performance than seeing the behavior of any individual random path. For example, the probability that all of the multiple paths are simultaneously congested is much less than the probability that a single path is congested. The resulting path diversity provides the multiple state video decoder with an appropriate virtual channel to assist in recovering from lost packets, and can also simplify system design, e.g. FEC design. We propose two architectures for achieving path diversity, and examine the effectiveness of path diversity in communicating video over a lossy packet network.

In this work, each packet is associated with a varying degree of significance based on the impact of its loss, which is estimated with a corruption model. For a compressed video stream, initial and propagation errors are first described by the macroblock (MB)-based corruption model by taking into account error concealment, temporal dependency, and loop filtering. Next, at the packet-level, a corruption model using the relative priority index (RPI) is constructed by integrating MB-based models. Furthermore, the effect of multiple-packet loss is analyzed and the associated RPI is calculated to improve the model accuracy at a higher packet loss rate. With the RPI-based corruption model for each packet, coordinated delivery of packetized video over QoS networks is investigated. Finally, a per-packet optimization framework with unequal error protection (UEP) is realized to consider both end-to-end video performance and pricing. The performance of the proposed solution is verified with intensive simulations.

In this paper we investigate the use of a fully rate scalable wavelet codec known as SAMCoW (Scalable Adaptive Motion Compensated Wavelet) for use in robust video streaming. We develop a theory based on the notion of additive temporal distortion to predict the performance of the bit stream under error conditions. Due to the regular nature of SAMCoW, a closed-form solution is found and compared experimentally to a SAMCoW stream in a simulated channel.

An image sequence stabilization system that removes translational jitter while preserving intentional camera pan is presented. The video sequence is processed to acquire global camera translations from frame to frame (global interframe motion vectors) by motion estimation. The resulting motion vectors are accumulated to construct an absolute frame position vs. frame number signal. This signal is low-pass filtered to remove high frequency components caused by jitter, and retain low frequency parts representing the intentional camera pan. Correction vectors for image frames are obtained by subtracting the absolute frame position from the low-pass filtered value, and stabilization is achieved by the corresponding translation of image frames.

The complementary nature of sensors in the visible light and in the infrared attracts research interest in multi-sensor image sequence analysis. The information made available in each wavelength can be combined. The combination of multi- sensor data with the temporal information makes target detection more robust against different sensor artifacts. In this paper we analyze the moving object segmentation and tracking with the fusion of multi-sensor data in two levels: in the detection level and in the decision level. In the first level the accuracy of the detection of all moving objects is analyzed while in the decision level the moving objects are classified as target or non-target, with the computed information in different wavelengths. Performance evaluation is done through ROC curves and with synthetic degradation methods. In the detection level approach registration techniques are used to transmit detected moving object coordinates from the visible to the infrared band and in the opposite direction. The results are compared to the detection rate in the destination band without fusion. In the decision level tracks from different sensors are fused and evaluated considering new ROC curves. The first promising results of the algorithm applied to the experimental data and the algorithm evaluation are presented.

Meaningful region is the intermediate level between the original image and the interesting object of image. This level is an effective visual level for the representation of images and the successful extraction of meaningful regions from images helps to perform semantic segmentation. This paper proposes a scheme for roughly extracting meaningful regions in an image. By using multi-dimensional low-level feature analysis the local level of reliability of different features can be determined in order to adaptively weight the contribution of each feature to the segmentation process. Since the large variance of one feature always indicates that this feature would distinguish different objects clearly, a new weighted non-parametric clustering algorithm in the density space is implemented with suitably decided weights for different features. This permits us to utilize all the features efficiently and to extract semantic meaning from images. The above technique is proposed along with a retrieval application of landscape images. In this application, the object recognition plays an important role. The meaningful regions extracted should be merged into objects and more subtly semantic meaning could be obtained. Experiments on extracting meaningful regions both from still images and video clips are carried out with some satisfactory results.

This paper introduces a system for video object extraction useful for general applications where foreground objects move within a slow changing background. Surveillance of indoor and outdoor sequences is a typical example. The originality of the approach resides in two related components. First, the statistical change detection used in the system does not require any sophisticated parametric tuning as it is based on a probabilistic method. Second, the change is detected between a current instance of the scene and a reference that is updated continuously to take into account slow variation of the background. Simulation results show that the proposed scheme performs well in extracting video objects, with stability and good accuracy, while being of relative reduced complexity.

Tracking of moving-objects in image sequences is needed for several video processing applications such as content-based coding, object oriented compression, object recognition and more recently for video object plane extraction in MPEG-4 coding. Tracking is a natural follow-up of motion-based segmentation. It is a fast and efficient method to achieve coherent motion segments along the temporal axis. Segmenting out moving objects for each and every frame in a video sequence is a computationally expensive approach. Thus, for better performance, semi-automatic segmentation is an acceptable compromise as automatic segmentation approaches rely heavily on prior assumptions. In semi-automatic segmentation approaches, motion-segmentation is performed only on the initial frame and the moving object is tracked in subsequent frames using tracking algorithms. In this paper, a new model for object tracking is proposed, where the image features -- edges, intensity pattern, object motion and initial keyed-in contour (by the user) form the prior and likelihood model of a Markov Random Field (MRF) model. Iterated Conditional Mode (ICM) is used for the minimization of the global energy for the MRF model. The motion segment for each frame is initialized using the segment information from the previous frame. For the initial frame, the motion segment is obtained by manually keying in the object contour. The motion-segments obtained using the proposed model are coherent and accurate. Experimental results on tracking using the proposed algorithm for different sequences -- Bream, Alexis and Claire are presented in this paper. The results obtained are accurate and can be used for a variety of applications including MPEG-4 Video Object Plane (VOP) extraction.

With the introduction of the standard in video indexing, research in shot boundary detection has recently become popular. While new solutions for long transition detection have recently emerged, there is not much literature that focuses on what to do with the frames within the transition when they are detected. After the approximate position of the long transition is detected, the raw sequence cannot be used for segmentation and indexing due to the special effects incorporated. Here, a technique aims at extracting extra information from the transition after the existence of a long transition is confirmed. This technique consists of four stages, namely, shot boundary refinement, shot type determination, frame reconstruction for soft transition and pixel classification for hard transition. This paper gives the overview as well as performance of each stage. This technique allows detected transitions to be analyzed without human intervention.

Automatic generation of user profiles, as specified in the MPEG-7 user preference description scheme, for personized broadcast services is investigated in this work. Our research has focused on categorization of user-favored video into different semantically meaningful classes. This knowledge is then used in media filtering guidance and user preferred AV content selection. Several visual and motion features are extracted from source video sequences, such as the number of intra-coded macroblocks, the macroblock motion information, temporal variances and shot activity histograms, for the classification purposes. Moreover, to further improve the accuracy of classification results, a 'fuzzy nearest prototype classifier' is applied in this work. It is shown by experimental results that the proposed classification scheme is efficient and accurate.

In this paper, within the context of the MPEG-4 standard we report on preliminary experiments in three areas -- authoring of MPEG-4 content, a player/browser for MPEG-4 content, and streaming of MPEG-4 content. MPEG-4 is a new standard for coding of audiovisual objects; the core of MPEG-4 standard is complete while amendments are in various stages of completion. MPEG-4 addresses compression of audio and visual objects, their integration by scene description, and interactivity of users with such objects. MPEG-4 scene description is based on VRML like language for 3D scenes, extended to 2D scenes, and supports integration of 2D and 3D scenes. This scene description language is called BIFS. First, we introduce the basic concepts behind BIFS and then show with an example, textual authoring of different components needed to describe an audiovisual scene in BIFS; the textual BIFS is then saved as compressed binary file/s for storage or transmission. Then, we discuss a high level design of an MPEG-4 player/browser that uses the main components from authoring such as encoded BIFS stream, media files it refers to, and multiplexed object descriptor stream to play an MPEG-4 scene. We also discuss our extensions to such a player/browser. Finally, we present our work in streaming of MPEG-4 -- the payload format, modification to client MPEG-4 player/browser, server-side infrastructure and example content used in our MPEG-4 streaming experiments.

In this paper, we describe a system for wandering in a realistic environment over the Internet. The environment is captured by the concentric mosaic, compressed via the reference block coder (RBC), and accessed and delivered over the Internet through the virtual media (Vmedia) access protocol. Capturing the environment through the concentric mosaic is easy. We mount a camera at the end of a level beam, and shoot images as the beam rotates. The huge dataset of the concentric mosaic is then compressed through the RBC, which is specifically designed for both high compression efficiency and just-in-time (JIT) rendering. Through the JIT rendering function, only a portion of the RBC bitstream is accessed, decoded and rendered for each virtual view. A multimedia communication protocol -- the Vmedia protocol, is then proposed to deliver the compressed concentric mosaic data over the Internet. Only the bitstream segments corresponding to the current view are streamed over the Internet. Moreover, the delivered bitstream segments are managed by a local Vmedia cache so that frequently used bitstream segments need not be streamed over the Internet repeatedly, and the Vmedia is able to handle a RBC bitstream larger than its memory capacity. A Vmedia concentric mosaic interactive browser is developed where the user can freely wander in a realistic environment, e.g., rotate around, walk forward/backward and sidestep, even under a tight bandwidth of 33.6 kbps.

The delivery of high quality video program existing in a video server through heterogeneous networks is a fast emerging service and has attracted much attention recently. Transcoding technique that converts a pre-encoded video sequence from a high bitrate to a low bitrate is a key component in such a system. Based on our previously proposed VBR coding for fixed storage application, we propose in this paper a VBR transcoding architecture to accomplish video streaming over heterogeneous networks. First, we assume that the pre-encoded bitstream is generated using the proposed VBR encoding scheme at a relatively high bitrate. In addition, the relationship between the amount of bits for each frame and all possible quantization factors, i.e. rate (R) and quantization (Q) function, is also generated at the video server. Once the users provide their access requirements, the video server will optimally compute the appropriate quantization factors for each frame according to the constraints of desired bitrate and finite buffer size. Then the compressed bitstream will be decoded to DCT domain and requantized with the updated quantization factors. Because the generation of R-Q function is based on the original frame and is not related to the quantization factors, it is possible to reuse the motion information embedded in the compressed bitstream and implement the transcoding in the DCT domain. The computational expense required by this proposed transcoder is much lower than those required by schemes which decode the compressed bitstream to the pixel domain and transcode to the desired bitrate using a complete encoder. Furthermore, this VBR transcoding scheme is able to generate a CBR-like output while retaining all advantages of VBR encoding. This facilitates the delivery of VBR bitstream through various existing CBR channels. Experimental results demonstrate that our proposed VBR transcoding not only is capable of achieving higher mean PSNR and more consistent decoded visual quality, but also requires much less computational expense compared with conventional pixel domain CBR transcoder.

Several embedded video coding schemes have been recently developed for multimedia streaming over IP. In particular, Fine-Granular-Scalability (FGS) video coding has been recently adopted by the MPEG-4 standard as the core video-compression method for streaming applications. From its inception, the FGS scalability structure was designed to be packet-loss resilient especially under Unequal Packet-loss Protection (UPP). In this paper, we introduce the notion of Fine Grained Loss Protection (FGLP), which provides UPP within the FGS enhancement-layer, and we develop an analytical framework for evaluating FGLP bounds. Based on these bounds, we show the impact of applying fine-grained protection to the FGS enhancement-layer for different types of video sequences and over a wide range of bit-rates and packet-loss ratios. Subsequently, the FGLP framework has been implemented using forward error correction codes and its performance for Internet video streaming has been evaluated. As illustrated by our extensive simulation results, fine-grained loss protection for the FGS enhancement- layer can provide significant resilience under moderate-to- high packet-loss ratios (e.g. 5 - 10%).

Recently the problem of transmitting data over heterogeneous networks has received considerable attention. Our specific interest is the communication of still images. In this work we present a Multiple Description system based on Polyphase DownSampling algorithm (PDMD) that splits an image source in an arbitrary number of balanced descriptions. Our goal is to obtain both the best reconstruction quality from the received descriptions (also only one) and a subjective indistinguishable reconstruction from the original, if all the descriptions are available.

This paper presents a new, scalable coding technique that can be used in interactive image/video communications over the Internet. The proposed technique generates a fully embedded bit stream that provides scalability with high quality for the whole image and it can be used to implement region based coding as well. The embedded bit stream is comprised of a basic layer and many enhancement layers. The enhancement layers add refinement to the quality of the image that has been reconstructed using the basic layer. The proposed coding technique uses multiple quantizers with thresholds (QT) for layering and it creates a bit plane for each layer. The bit plane is then partitioned into sets of small areas to be coded independently. Run length and entropy coding are applied to each of the sets to provide scalability for the entire image resulting in high picture quality in the user-specific area of interest (ROI). We tested this technique by applying it to various test images and the results consistently show high level of performance.

A scalable video streaming system that includes video preprocessing, differential JPEG-2000 (DJ2K) video codec, and content-aware rate control is proposed. The proposed streaming solution tackles the problem of maintaining smooth video quality even when there is a significant change in the source content or the transmission bandwidth. The wavelet-based JPEG- 2000 video codec provides temporal, spatial and SNR scalabilities. A rate controller, which is separated from the encoder, can deal with diverse demands of a wide variety of users and various traffic conditions in a heterogeneous network by leveraging offered scalabilities and content- awareness. Simulations are conducted to demonstrate various advantages of the proposed solution.

We proposed an omni-face tracking system for video annotation in this paper, which is designed to find faces from arbitrary views in complex scenes. The face detector first locates potential faces in the input by performing skin-tone detection. The subsequent processing consists of two largely independent components, the frontal face module and the side- view face module, responsible for finding frontal-view and side-view faces, respectively. The frontal face module uses a region-based approach wherein regions of skin-tone pixels are analyzed for gross oval shape and the presence of facial features. In contrast, the side-view face module follows an edge-based approach to look for curves similar to a side-view profile. To extract the trajectories of faces, the temporal continuity between consecutive frames within the video shots is considered to speed up the tracking process. The main contribution of this work is being able to find faces irrespective of their poses, whereas contemporary systems deal with frontal-view faces only. Information regarding to human faces is encoded in XML format for semantic video content representation. The effectiveness of human face for video annotation is demonstrated in a TV program classification system that categories the input video clip into predefined types. It is shown that the classification accuracy is improved saliently by the employment of face information.

An important topic in face recognition as well as in video coding or multi-modal human machine interfaces is the automatic detection of faces or head-and-shoulder regions in visual scenes. The algorithms therefore should be computationally fast enough to allow an online detection and parallel processing of the detected objects. In this paper we describe our ongoing work on face detection using an approach that models the face appearance by edge orientation information. We will show that edge orientation is a powerful local image feature to describe objects like faces or body parts. We will present a sample and efficient method for template matching and object modeling based solely on edge orientation information. We also show how to obtain an optimal face model in the edge orientation domain from a set of training images. Unlike many approaches that model the gray level appearance of the face our approach is computationally very fast. It takes less than 0.1 seconds on a Pentium II 500 MHz for a 190 X 140 image to be processed using a multi- resolution search with three resolution levels. We demonstrate the capability of our detection method on an image database of 7000 images taken from more than 200 different people. The variations in head size, lighting and background are considerable. The obtained detection rate is more than 97% on that database. We have also extended the algorithm for face tracking. The tracking capabilities are shown using results from a real-time implementation of the proposed algorithm.