The representation of video information in terms of its content is at the foundation of many multimedia applications, such as broadcasting, content-based information retrieval, interactive video, remote surveillance and entertainment. In particular, object-based representation consists in decomposing the video content into a collection of meaningful objects. This approach offers a broad range of capabilities in terms of access, manipulation and interaction with the visual content. The basic difference when compared with pixel-based procedures is that instead of processing individual pixels, image objects are used in the representation. To exploit the benefits of object-based representation, multimedia applications need automatic techniques for extracting such objects from video data, a problem that still remains largely unsolved. In this paper, we first review the extraction techniques that enable the separation of foreground objects from the background. Their field of applicability and their limitations are discussed. Next, automatic tools for evaluating their performances are introduced. The major applications that benefit from an object-based approach are then analysed. Finally, we discuss some open research issues in object-based video.

This paper presents a novel method to automatically extract moving objects using motion and color information. Based on pattern recognition and object tracking principles, the proposed method can handle sequences both with moving and stationary backgrounds. It derives for each physical object a two-dimensional binary model, with the model points corresponding to the edges that were detected by the Canny Operator. The resulting binary models combined with the temporal and spatial information will then guide the extraction of the actual VOPs from the video sequence. The performance of the segmentation technique is illustrated by simulations carried out on standard video sequences.

In this paper, a multi-metric evaluation protocol is proposed to evaluate performance of user-assisted video object extraction systems. Evaluation metrics are the essential element in performance evaluation methodology. Recent works on video object segmentation/extraction are mostly restricted to a single objective metric to judge the overall performance of algorithms. Motivated by a novel framework for performance evaluation on image segmentation using Pareto front, we propose a multi-metric evaluation protocol, including metrics for contour-based spatial matching, temporal consistency, user workload and time consumption. Taking the characteristic of a user-assisted video object extraction system into consideration, we formulate the metrics in a way simple but close to the assessment of human visual system. For spatial matching, we define three types of errors: sharp error, smooth error and mass error, which can precisely score an extraction result. The time consistency is introduced to evaluate the stability over time of a system. In addition, as far as a user-assisted system is concerned, the workload of users is also in our metric. Incorporating multi-metric into one 4-D fitness space, we adopt the Pareto front to find the best choice of a system with optimal parameters. The tests of our evaluation method show that the multi-metric protocol is effective.

We propose measures to evaluate the performance of video object segmentation and tracking methods quantitatively without ground-truth segmentation maps. The proposed measures are based on spatial differences of color and motion along the boundary of the estimated video object plane and temporal differences between the color histogram of the current object plane and its neighbors. They can be used to localize (spatially and/or temporally) regions where segmentation results are good or bad; and/or combined to yield a single numerical measure to indicate the goodness of the boundary segmentation and tracking results over a sequence. The validity of the proposed performance measures without ground truth have been demonstrated by canonical correlation analysis of the proposed measures with another set of measures it with ground-truth on a set of sequences (where ground truth information is available). Experimental results are presented to evaluate the segmentation maps obtained from various sequences using different segmentation and tracking algorithms.

The present paper introduces a very specific and pragmatic approach to segmentation. It is driven by a particular application context: in the framework of mixed-reality, Tranfiction (“transportation into fictional spaces”) is designed to mix synthetic and natural images in real time while allowing users to interact in these input/output screens. Segmentation is therefore used to provide both the immersion and interaction capabilities. The former aspect is achieved by composing the image of the user within the projected virtual scenes, while the later is achieved thanks to basic body/gesture analysis on the segmented silhouettes. According to indoor or outdoor usages, two real-time techniques are developed. Results are analyzed with respect to the overall application, not only in terms of absolute quality but also in terms of perception by the users.

In this paper, we describe a new way to create an object oriented video surveillance system that monitors activity in a site. The process is performed in two steps: first, detection of human faces as a guess for objects of interest is done and tracking of these entities through a video stream. The guidelines here are not to perform a very accurate detection and tracking, based on the contours for example, but to provide a global image processing system on a simple Personal Computer taking advantage from co-operation of detection and tracking. So the scheme we propose here provides a simple, fast solution that tracks few specific points of interest on the object boundary and possibly engage a motion based detection in order to recover the object of interest in the scene or to detect new object of interest as well. This tracker also enables learning motion activities, detecting unusual activities, and supplying statistical information about motion in a scene.

The emergence and proliferation of email, mobile communication devices, internet chatrooms, shared virtual environments, advanced tele-conferencing platforms and other telecommunication systems underline the importance of developing measurement methods that are sensitive to the human experience with these systems. In this paper, we discuss the concepts of social presence and connectedness as complementary notions, each relating to a different set of media properties that serve distinct communication needs. We aim to broaden the scope of current presence technologies and applications, illustrating the various factors that play a role in establishing, enhancing, and enriching the experience of human connectedness through communication media. Based on existing literature, we discuss a number of user requirements for home communication and awareness systems. To make these ideas tangible, we finish the paper by briefly discussing the ASTRA project as a case study in designing and evaluating an awareness system for the home.

In this paper, a next generation 3-D video conferencing system is presented that provides immersive tele-presence and natural representation of all participants in a shared virtual meeting space. The system is based on the principle of a shared virtual table environment which guarantees correct eye contact and gesture reproduction and enhances the quality of human-centered communication. The virtual environment is modeled in MPEG-4 which also allows the seamless integration of explicit 3-D head models for a low-bandwidth connection to mobile users. In this case, facial expression and motion information is transmitted instead of video streams resulting in bit-rates of a few kbit/s per participant. Beside low bit-rates, the model-based approach enables new possibilities for image enhancements like digital make-up, digital dressing, or modification of scene lighting.

This contribution describes a studio system that supports the production of dynamic 3D content. The system is based on a multi-camera approach to capture dynamic scenes in a studio equipped with a chroma-keying facility. This paper focusses on the real-time functionality of the system. Three different methods for the computation of the visual hull from silhouette images were implemented and compared. Two methods use a volumetric representation (3D-array and octree representation). The third method uses a new line representation and has been shown to deliver more accurate surface approximations of objects. Further to the capture functionality, the system provides visualisation tools for planning the production and on-set visualisation. The latter includes a pre-view of the composed programme for the director and an immersive feedback for the actor. The immersive feedback is implemented using a view-dependent projection onto a special retro-reflective chroma-keying cloth and is not interfering with the shape capturing sub-system.

One of the most challenging tasks in stereopsis is to find corresponding points in a stereo image pair to obtain depth information. Half occlusion and order-reversal of objects further complicate the problem. In this paper, we propose the use of hue and intensity of images for stereo correspondences. Stereo correspondence is performed row by row. Planar first-order least squares curve-fitting is exploited to extract line-segments with similar hue. A sloping intensity profile of line-segment is used to indicate the presence of a sloping surface with the same hue. A Left-Left Right-Right (LL-RR) constraint is introduced for matching line-segment pair: the possible occluded part should be located both at the leftmost part of the left-line-segment and at the rightmost part of the right-line-segment. This constraint is used to detect half-occlusion in line-segment pair matching. Finally, an object is formed from a block of contiguous line-segments separated from other blocks by gaps. Experiments were performed on stereo image pairs and the estimated disparity maps are used to evaluate the performance of the proposed algorithm.

In this paper, we will describe a real-time stereo vision algorithm that determines the disparity map of a given scene by an evaluation of the object contours, relying on a reference image displaying the scene without objects. Contours are extracted from the full-resolution absolute difference image between current and reference image by binarization with several locally adaptive thresholds. To estimate disparity values, contour segments extending over several epipolar lines are used. This approach leads to very accurate disparity values. The algorithm can be configured such that no image region that differs from the reference image by more than a given minimum statistical significance is overlooked, which makes it especially suitable for safety applications. We successfully apply this contour based stereo vision (CBS)algorithm to the task of video surveillance of hazardous areas in the production environment, regarding several thousands of test images. Under the harsh conditions encountered in this setting, the CBS algorithm achieves to faithfully detect objects entering the scene and determine their three-dimensional structure. What is more, it turns out to cope with small objects and very difficult illumination and contrast settings.

Current binocular stereoscopic displays cause visual discomfort when the objects with large disparities are present in the scene. With this technique, the improvement of visual comfort has been reported by blurring far background and foregrounds in the scene. However, this technique has a drawback of degrading overall image quality. To lesson visual discomfort caused by large disparities while maintaining high-perceived image quality, we use a novel disparity-based asymmetrical filtering technique. Asymmetrical filtering, which refers to the filtering applied to the image of one eye only, has been showen to maintain the sharpness of a stereoscopic image, provided that the amount of filtering is low. Disparity-based asymmetrical filtering usese the disparity information in a stereoscopic image for controlling the severity of blurring. We investigated the effects of this technique on stereoscopic video by measuring visual comfort and apparent sharpness. Our results indicate that disparity-based asymmetrical filtering does not always improve visual comfort but it maintains image quality.

In stereoscopic television, there is a trade-off between visual comfort and 3D impact with respect to the baseline-stretch of 3D camera. It has been reported that an optimal condition can be reached when we set the baseline-stretch at about the distance of human pupils1. However, we cannot get such distance in case that the sizes of the lens and CCD module are big. In order to overcome this limitation, we attempt to control the baseline-stretch of stereoscopic camera by synthesizing virtual views at the desired location of interval between two cameras. Proposed technique is based on the stereo matching and view synthesis techniques. We first obtain a dense disparity map using a hierarchical stereo matching with the edge-adaptive shifted window. And then we synthesize virtual views using the disparity map. Simulation results with various stereoscopic images demonstrate the effectiveness of the proposed technique.

This paper presents a new, highly flexible, scalable image coder based on a Matching Pursuit expansion. The dictionary of atoms is built by translation, rotation and anisotropic refinement of gaussian functions, in order to efficiently capture edges in natural images. In the same time, the dictionary is invariant under isotropic scaling, which interestingly leads to very simple spatial resizing operations. It is shown that the proposed scheme compares to state-of-the-art coders when the compressed image is transcoded to a lower (octave-based) spatial resolution. In contrary to common compression formats, our bit-stream can moreover easily and efficiently be decoded at any spatial resolution, even with irrational re-scaling factors. In the same time, the Matching Pursuit algorithm provides an intrinsically progressive stream. This worthy feature allows for easy rate filtering operations, where the least important atoms are simply discarded to fit restrictive bandwidth constraints. Our scheme is finally shown to favorably compare to state-of-the-art progressive coders for moderate to quite important rate reductions.

Various techniques for SNR scalability in hybrid block-based video coding exist in the literature and in different standards. A new approach based on transform coefficient refinement is proposed in this article. The coefficient difference is computed after quantization, subsequently entropy-encoded, and transmitted for reconstruction of a high-quality layer at the decoder. The approach achieves in most cases only a moderate increase in bit rate as compared to other schemes. The bit rate of our two-layer framework converges towards the rate of a single-layer system as the quality gap between both layers increases. The gains come at the cost of increased computational complexity and memory requirements.

The paper studies scalable video coding based on multiresolution video representations generated by multi-scale subband motion compensated temporal filtering (MCTF) and spatial wavelet transform. Since MCTF is performed subband by subband in the spatial wavelet domain, motion vectors are available for reconstructing video sequences of any possible reduced spatial resolution, restricted by the dyadic decomposition pattern and the maximal spatial decomposition level. The multiresolution representations naturally provide a framework with which both spatial scalability and temporal scalability can be very conveniently and efficiently supported by a video coder that utilizes such multiresolution video representations. Such video coders can be fully scalable by incorporating wavelet-domain bit-plane image coding techniques. This paper examines the performance, including scalability and coding efficiency, of a scalable video coder that utilizes such multi-scale video representations together with the EZBC image coder. A wavelet-domain variable block size motion estimation algorithm is introduced to enhance the performance of the subband MCTF. Experiments show that the proposed coder outperforms the state of the art fully scalable coder MC-EZBC in terms of the spatial scalability.

This paper proposes a robust video transmission system over lossy packet-switched networks with block-based fine granularity scalable (B-FGS) coding including source coding, efficient rate allocation and packetization, which works effectively when packet loss occurs. The compressed B-FGS bit-stream can be truncated in fine grain for its block-based bit-plane coding at enhancement layer. Therefore channel adaptive rate allocation can be implemented and the time-varying bandwidth can be exploited efficiently. Distortion associated rate allocation method is proposed to substitute Rate-Distortion optimal rate allocation in order to reduce the streaming server’s complexity. In addition, packetizing with independent data provides robust video transmission, which restrains the error propagation across packets. Furthermore, in order to mitigate quality flicking of decoded video induced by packets loss, spatial interleaving is applied in packetizing. Thus, in proposed video transmission system, available bandwidth variation and packet loss, two key problems in video streaming applications, are well dealt with. In order to validate the effectiveness of proposed transmission system, MPEG-4 FGS transmission system and that with conditional retransmission are compared in simulations. Proposed transmission system is more robust to packet-loss errors than the other two, especially when packet-loss rate is high.

We investigate multicast/broadcast of digital video over spread-spectrum CDMA cellular networks, a platform targeted at various kinds of multimedia services. In particular, we propose an end-to-end embedded transmission scheme which combines a scalable video source coder, adaptive power allocation, adaptive channel coding and an embedded multiresolution modulation strategy to simultaneously deliver a basic quality-of-service (QoS) to less capable receivers and an enhanced QoS for more capable receivers. We demonstrate the efficacy of this approach using the ITU-T H.263+ video source coder, although the approach is generally applicable to other scalable source coding schemes as well.

FGS is a video coding scheme that can control picture quality with high scalability, but it has lower efficiency than the single layer video coding scheme because of its hierarchical structure. This paper proposes application scheme of three methods for efficiency improvement to the FGS encoder and decoder. The first method is the bit plane coding method using two bit group separation based on the distribution of a significant bit. The second method is the introduction of motion compensation estimation to the enhancement layer. The third method is a complement method to make a high quality reference picture from both the base layer and the enhancement layer. In conclusion, the proposed methods can improve the coding efficiency in the enhancement layer of conventional FGS by 2.2% and the PSNR performance of conventional FGS by 1.1dB, as well as reduce propagation of the drift error caused by low rate transmission and channel congestion. The results show that the proposed method is more efficient and more suitable for video streaming on a heterogeneous network with channel deviation.

This paper describes the development of a layered structure of a multi-resolutional scalable video codec based on the Color Set Partitioning in Hierarchical Trees (CSPIHT) scheme. The new structure is designed in such a way that it supports the network Quality of Service (QoS) by allowing packet marking in a real-time layered multicast system with heterogeneous clients. Also, it provides (spatial) resolution/ frame rate scalability from one embedded bit stream. The codec is self-identifiable since it labels the encoded bit stream according to the resolution. We also introduce asymmetry to the CSPIHT encoder and decoder which makes it possible to decode lossy bit streams at heterogeneous clients.

In this paper, we propose a bitstream switching scheme for progressive fine granularity scalable (PFGS) video coding that aims to improve the streaming performance of the scalable bitstreams. By switching among PFGS video bitstreams with different enhancement layer’s reference rates based on the same base layer, significant performance gain can be achieved, especially when the bandwidth fluctuates broadly. Furthermore, compared with other bitstream switching schemes, the proposed scheme shows its flexibilities and various advantages due to using the common base layer. The experimental results illustrate that the proposed scheme can significantly improve the streaming performance.

Multi-dimensional rate control schemes, which jointly adjust two or three coding parameters, have been recently proposed to achieve a target bit rate while maximizing some objective measures of video quality. The objective measures used in these schemes are the peak signal-to-noise ratio (PSNR) or the sum of absolute errors (SAE) of the decoded video. These objective measures of quality may differ substantially from subjective quality, especially when changes of spatial resolution and frame rate are involved. The proposed schemes are, therefore, not optimal in terms of human visual perception. We have investigated the impact on subjective video quality of the three coding parameters: spatial resolution, frame rate, and quantization parameter (QP). To this end, we have conducted two experiments using the H.263+ codec and five video sequences. In Experiment 1, we evaluated the impact of jointly adjusting QP and frame rate on subjective quality and bit rate. In Experiment 2, we evaluated the impact of jointly adjusting QP and spatial resolution. From these experiments, we suggest several general rules and guidelines that can be useful in the design of an optimal multi-dimensional rate control scheme. The experiments also show that PSNR and SAE do not adequately reflect perceived video quality when changes in spatial resolution and frame rate are involved, and are therefore not adequate for assessing quality in a multi-dimensional rate control scheme. This paper describes the method and results of the investigation.

Video streaming over wireless channel is challenging due to a number of factors such as limited bandwidth and loss sensitivity. In this paper, we develop a novel rate control algorithm for MPEG-4 video coding. Unlike traditional rate control schemes, we jointly consider the encoding complexity variation and buffer variation as well as human visual properties to optimize the rate control efficiency. We also analyze the sensitivity of a macroblock (MB) as a result of bit errors and calculate its error sensitivity metric. This metric is used in unequal error protection (UEP) of the MB. Simulation results show that proposed approach can improve the decoded picture quality in wireless video coding and transmission.

State-of-the-art video coders feature a high level of adaptivity to meet the properties of their input signals. Video signal statistics are apt to sometimes discontinuously vary in all three dimensions: spatially and temporally. Consequently for video compression, different coding tools are employed at different coordinates of the signal to adaptively and maximally reduce redundancy. Since the decision for a certain tool and for the apropriate parameters is highly signal-dependent, a video coder forms a non-linear system and optimization is not trivial to perform. Lagrange Rate/Distortion-Optimization (RDO) has become an important tool for video encoding. It has achieved high gains in coding efficiency when applied to the independent encoding of macroblocks (MBs). For a chosen quantizer, bit rate is allocated to each MB's coefficients and prediction parameters subject to minimization of a Lagrange cost function of rate and distortion. In effect, the cheaper option is chosen to either use a more precise prediction, i.e. motion vectors and block tiling, or to spend more bit rate on the coefficients. Often, the video coding objective is that of constant quality. This is approximately achieved by a constant quantizer for all MBs and RDO within each MB. Due to the recursive structure of hybrid video coders though, MBs are dependent on each other, demanding for dependent RDO. This has early been formulated, but is hardly solvable due to the high dimensionality of this problem. In the following, possible simplified ways to take dependencies into account are explored. The variations in all three dimensions of the signal properties are differentiated and simplified models used for a dependent optimization.

This paper presents a virtual frame rate control algorithm and a bit allocation algorithm at frame level for efficient MPEG-2 video encoding. The proposed frame rate control scheme is composed of three steps. At first step, teh scan format of an input video sequence is converted into progressive scan format before video encoding. At the second step, an average motion activity of the frames within a previous temporal window of a pre-defined size is examined, and a proper frame rate of a current temporal window is adaptively determined based on the computed average motion activity. At the final step, the frames located at particular positions in a current temporal window are virtually skipped according to the determined frame rate. We can skip the selected frames by deliberately fixing the coding types of all the macroblocks in those frames to 'skipped MB.' We also propose a modified version of MPEG-2 TM5 bit allocation algorithm suitable for virtual frame rate control. Experimental results show that the proposed algorithm noticeably improves coding efficiency in comparison with conventional MPEG-2 video encoding.

Future consumer terminals (TV sets, set-top boxes, displays) will increasingly be based on programmable platforms instead of only dedicated hardware. When the available resources are insufficient to deal with the worst-case requirements then resource overloads might lead to system instability and reduced output quality. For robust and cost-effective media processing on programmable platforms, dynamic resource management is required together with resource-quality scalable video algorithms. A way to prevent resource overloads is to stop processing when the assigned resources have been used. This may lead to drops and changes in quality. We propose to use shot-change information prior to the actual processing to predict difficult situations and react on them in a proper way. For a scalable motion estimator we show that, after a shot-change, generating motion vectors that indicate zero or close to zero motion prevents overloads, saves resources and can lead to a higher quality.

There is a growing need for mechanisms to shape the rate of precompressed video for adaptation to network constraints. The motion vector data component in an encoded bitstream might constitute some considerable portion of P and especially B type pictures, yet no technique currently exists to scale it. Motion vectors of neighboring macroblocks tend to be well-correlated in both horizontal and vertical directions. However, the current standards only exploit the horizontal correlations through predictive coding, introducing dependency and thus resulting in a nonscalable representation. In this paper, we propose a new representation and encoding algorithm for the motion vector information enabling the exploitation of two-dimensional (2-D) correlations, and realizing the signal-to-noise ratio (SNR) scaling of an already encoded video object to an alternate (lower) rate - (higher) distortion level. In the proposed approach, the values of the motion vectors are represented in a binary format. Consequently, the individual bit planes of this representation are coded through a novel 2-D hierarchical bit plane encoding scheme. Scaling, when necessary, is achieved by discarding the least significant bit plane(s). Simulations based on MPEG-2 standard demonstrate that up to 20% scaling is achievable without visually noticeable distortion.

A flow control for streaming multimedia data over UDP on IP network is presented. The bitrate adaptation algorithm embedded in the protocol is considered to be an end-to-end mechanism in the application level. The flow control system constantly maintains the streaming buffer at a prescribed capacity even with bursty network losses by adapting the multimedia bitrate B from the streaming codec. A congestion control algorithm which is considered to be located in a lower level than that of the flow control mechanism is presented. It works together with the presented flow control to resolve network congestion problems while maintaining a degree of TCP-friendliness by changing the sending rate R at the server. Simulation results obtained from NS2 have shown better resource allocation can be obtained, and an overall incrase in the average sending rate and hence better quality of streaming media is observed.

This paper presents an approach dedicated to accurately track one or several semantic objects in a video sequence. The accurate tracking of the partition object boundary is obtained by a label prediction. This prediction is performed thanks to motion vectors obtained with two different block-matching uses. In the predicted partition, a local segmentation is necessary only where matching failed and close to the predicted boundaries, in order to get the most accurate boundaries. This local segmentation is then followed by a classification step. During the classification a backward projection is used to assign or not a region to a given object.

In this paper, we propose a method of tracking soccer players using multiple views. As many researchers have done on soccer scene analysis by using trajectories of the playser and the soccer ball, it is desirable to track soccer players robustly. Soccer player tracking enables strategy analysis, scene recovery, making scenes for broadcasting, and automatic system of the camera control. However, soccer is a sport that occlusion occurs in many cases, and tracking often fails by the occlusion of the players. It is difficult to track the players by using a single camera alone. Therefore, we use multiple view images to avoid the occlusion problem, so that we can obtain robustness in player tracking. As a first step, inner-camera operation is performed independently in each camera to track the players. In any case that player can not be tracked in the camera, inter-camera operation is performed as a second step. Tracking information of all cameras are integrated by using the geometrical relationship between cameras called homography. Inter-camera operation makes it possible to get the location of the player who is not detected in the image, who is occluded by the other player, and who is outside the angle of view. Experimental results show that robust player tracking is available by tracking advantage using multiple cameras.

In this paper, an adaptive stereo object tracking system based on a block-based SSD(sum of squared difference) algorithm and camera configuration parameters is proposed. That is, by applying SSD algorithm to the reference image of the previous frame and the input image of the current frame, each location coordinate of a moving target in the right and left images are extracted and the respective shifted distances from the reference target position which is assumed to be at the origin in the initial frame are detected. Using the pan/tilt's moving angle calculated from the target's shifted distance and the configuration parameters of a stereo camera system, the block-mask size of a target object can be adaptively determined. The target image segmented with this block mask is used as a reference image in the next stage of tracking, and it is automatically updated according to the same procedure during the course of target tracking. From some experiments using sequential 48 frames of the dynamic stereo image, it is analyzed that the horizontal and vertical stereo disparities on the target object after stereo tracking are kept to be very low values of 1.5 and 0.4 pixels on average, respectively.

Data association and model selection are important factors for tracking multiple targets in a dense clutter environment without using apriori information about the target dynamic. We propose Interacting Multiple Model-Expectation Maximization (IMM-EM) algorithm, by incorporating different dynamic models for the target and Markov Random Field (MRF) for data association, and hence it is possible to track maneuvering and non-maneuvering targets simultaneously in a single batch mode (sequential). Moreover it can be used for real time application. The proposed method overcomes the problem of data association by incooperating all validated measurements together using EM algorithm and exploiting MRF. It treats the data association problem as incomplete data problem. In the proposed method, all validated measurements are used to update the target state. It uses only measurement association as missing data, which simplifies E-step and M-step of the algorithm. In the proposed approach probability density function (pdf) of an observed data given target state and measurement association, is treated as a mixture pdf. This allows to combine likelihood of a measurement due to each model, and the association process is defined to incorporate IMM and consequently, it is possible to track any arbitrary trajectory. We also consider two different cases for association of measurement to target: Case I:- association of each measurement to target is independent of each other, Case II:- association of a measurement influences an association of its neighbor measurement.

Tracking is a composite job involving the co-operation of autonomous activities which exploit a complex information model and rely on a distributed architecture. Both information and activities must be classified and related in several dimensions: abstraction levels (what is modelled and how information is processed); topology (where the modelled entities are); time (when entities exist); strategy (why something happens); responsibilities (who is in charge of processing the information). A proper Object-Oriented analysis and design approach leads to a modular architecture where information about conceptual entities is modelled at each abstraction level via classes and intra-level associations, whereas inter-level associations between classes model the abstraction process. Both information and computation are partitioned according to level-specific topological models. They are also placed in a temporal framework modelled by suitable abstractions. Domain-specific strategies control the execution of the computations. Computational components perform both intra-level processing and intra-level information conversion. The paper overviews the phases of the analysis and design process, presents major concepts at each abstraction level, and shows how the resulting design turns into a modular, flexible and adaptive architecture. Finally, the paper sketches how the conceptual architecture can be deployed into a concrete distribute architecture by relying on an experimental framework.

We present a crossbreed feature-based head tracking technique in natural and unspecified environment. Kalman filter is a famous estimation technique in many areas to predict the route of moving object. We tested and developed a Kalman filter to track unpredictable and fast moving objects. Depth information could generate robust tracking result that is little affected by background texture and color. However this is also limited by selected conditions like distance, accuracy of stereo camera, and object occlusion at same distance, etc. To overcome these restrictions, we combined multiple features together into single tracking system that does largely depend on depth feature. We consider multi people environment with rapid walking path.

In this paper, we present a technique for tracking non-rigid video objects in a sequence. It assumes that the object in the initial image has been previously defined by an object partition. Most of object tracking methods usually rely on the motion homogeneity of the object to be tracked. They do not assume that the selected objects present either motion or spatial homogeneity. When they assume these, they employed a spatial or temporal criterion separately to achieve the object tracking. Our proposed object tracking approach relies on the concept of backward partition projection using a likelihood joint spatial-temporal model to deal with occlusion, uncover areas and fast motion problems. Several examples in different scenarios are finally presented in order to demonstrate the performance of the proposed method.

For various applications, such as data compression, structure from motion, medical imaging and video enhancement, there is a need for an algorithm that divides video sequences into independently moving objects. Because our focus is on video enhancement and structure from motion for consumer electronics, we strive for a low complexity solution. For still images, several approaches exist based on colour, but these lack in both speed and segmentation quality. For instance, colour-based watershed algorithms produce a so-called oversegmentation with many segments covering each single physical object. Other colour segmentation approaches exist which somehow limit the number of segments to reduce this oversegmentation problem. However, this often results in inaccurate edges or even missed objects. Most likely, colour is an inherently insufficient cue for real world object segmentation, because real world objects can display complex combinations of colours. For video sequences, however, an additional cue is available, namely the motion of objects. When different objects in a scene have different motion, the motion cue alone is often enough to reliably distinguish objects from one another and the background. However, because of the lack of sufficient resolution of efficient motion estimators, like the 3DRS block matcher, the resulting segmentation is not at pixel resolution, but at block resolution. Existing pixel resolution motion estimators are more sensitive to noise, suffer more from aperture problems or have less correspondence to the true motion of objects when compared to block-based approaches or are too computationally expensive. From its tendency to oversegmentation it is apparent that colour segmentation is particularly effective near edges of homogeneously coloured areas. On the other hand, block-based true motion estimation is particularly effective in heterogeneous areas, because heterogeneous areas improve the chance a block is unique and thus decrease the chance of the wrong position producing a good match. Consequently, a number of methods exist which combine motion and colour segmentation. These methods use colour segmentation as a base for the motion segmentation and estimation or perform an independent colour segmentation in parallel which is in some way combined with the motion segmentation. The presented method uses both techniques to complement each other by first segmenting on motion cues and then refining the segmentation with colour. To our knowledge few methods exist which adopt this approach. One example is \cite{meshrefine}. This method uses an irregular mesh, which hinders its efficient implementation in consumer electronics devices. Furthermore, the method produces a foreground/background segmentation, while our applications call for the segmentation of multiple objects. NEW METHOD As mentioned above we start with motion segmentation and refine the edges of this segmentation with a pixel resolution colour segmentation method afterwards. There are several reasons for this approach: + Motion segmentation does not produce the oversegmentation which colour segmentation methods normally produce, because objects are more likely to have colour discontinuities than motion discontinuities. In this way, the colour segmentation only has to be done at the edges of segments, confining the colour segmentation to a smaller part of the image. In such a part, it is more likely that the colour of an object is homogeneous. + This approach restricts the computationally expensive pixel resolution colour segmentation to a subset of the image. Together with the very efficient 3DRS motion estimation algorithm, this helps to reduce the computational complexity. + The motion cue alone is often enough to reliably distinguish objects from one another and the background. To obtain the motion vector fields, a variant of the 3DRS block-based motion estimator which analyses three frames of input was used. The 3DRS motion estimator is known for its ability to estimate motion vectors which closely resemble the true motion. BLOCK-BASED MOTION SEGMENTATION As mentioned above we start with a block-resolution segmentation based on motion vectors. The presented method is inspired by the well-known $K$-means segmentation method \cite{K-means}. Several other methods (e.g. \cite{kmeansc}) adapt $K$-means for connectedness by adding a weighted shape-error. This adds the additional difficulty of finding the correct weights for the shape-parameters. Also, these methods often bias one particular pre-defined shape. The presented method, which we call $K$-regions, encourages connectedness because only blocks at the edges of segments may be assigned to another segment. This constrains the segmentation method to such a degree that it allows the method to use least squares for the robust fitting of affine motion models for each segment. Contrary to \cite{parmkm}, the segmentation step still operates on vectors instead of model parameters. To make sure the segmentation is temporally consistent, the segmentation of the previous frame will be used as initialisation for every new frame. We also present a scheme which makes the algorithm independent of the initially chosen amount of segments. COLOUR-BASED INTRA-BLOCK SEGMENTATION The block resolution motion-based segmentation forms the starting point for the pixel resolution segmentation. The pixel resolution segmentation is obtained from the block resolution segmentation by reclassifying pixels only at the edges of clusters. We assume that an edge between two objects can be found in either one of two neighbouring blocks that belong to different clusters. This assumption allows us to do the pixel resolution segmentation on each pair of such neighbouring blocks separately. Because of the local nature of the segmentation, it largely avoids problems with heterogeneously coloured areas. Because no new segments are introduced in this step, it also does not suffer from oversegmentation problems. The presented method has no problems with bifurcations. For the pixel resolution segmentation itself we reclassify pixels such that we optimize an error norm which favour similarly coloured regions and straight edges. SEGMENTATION MEASURE To assist in the evaluation of the proposed algorithm we developed a quality metric. Because the problem does not have an exact specification, we decided to define a ground truth output which we find desirable for a given input. We define the measure for the segmentation quality as being how different the segmentation is from the ground truth. Our measure enables us to evaluate oversegmentation and undersegmentation seperately. Also, it allows us to evaluate which parts of a frame suffer from oversegmentation or undersegmentation. The proposed algorithm has been tested on several typical sequences. CONCLUSIONS In this abstract we presented a new video segmentation method which performs well in the segmentation of multiple independently moving foreground objects from each other and the background. It combines the strong points of both colour and motion segmentation in the way we expected. One of the weak points is that the segmentation method suffers from undersegmentation when adjacent objects display similar motion. In sequences with detailed backgrounds the segmentation will sometimes display noisy edges. Apart from these results, we think that some of the techniques, and in particular the $K$-regions technique, may be useful for other two-dimensional data segmentation problems.

Compression and content-based video retrieval (CBVR) are essential needs for efficient and intelligent utilizations of vast multimedia databases over the Internet. In video sequences, object based extraction techniques are gaining importance in achieving compression and performing content-based video retrieval. In this paper, a novel technique is developed to extract objects from video sequences based on spatiotemporal independent component analysis (stICA) and multiscale analysis. The stICA is used to extract the preliminary source images containing moving objects in the video sequences. The source image data obtained after stICA analysis are further processed using wavelet based multiscale image segmentation and region detection techniques to improve the accuracy of the extracted object. Preliminary results demonstrate great potential for stICA based object extraction technique in content-based video processing applications.

This paper presents a new algorithm for video-object segmentation, which combines motion-based segmentation, high-level object-model detection, and spatial segmentation into a single framework. This joint approach overcomes the disadvantages of these algorithms when applied independently. These disadvantages include the low semantic accuracy of spatial segmentation and the inexact object boundaries obtained from object-model matching and motion segmentation. The now proposed algorithm alleviates three problems common to all motion-based segmentation algorithms. First, it completes object areas that cannot be clearly distinguished from the background because their color is near the background color. Second, parts of the object that are not considered to belong to the object since they are not moving, are still added to the object mask. Finally, when several objects are moving, of which only one is of interest, it is detected that the remaining regions do not belong to any object-model and these regions are removed from the foreground. This suppresses regions erroneously considered as moving or objects that are moving but that are completely irrelevant to the user.

This paper proposes a new method for the detection of moving cast shadows in natural video sequences. The variations of illumination generate in a shadow area are modelized assuming that the source light is fixed and unique, and that the surface on which the shadow is projected is plane and Lambertian. A local and global matching of this model is then done on the current image in order to obtain a first detection of the moving shadow areas. This matching process is based on a reference image which is assumed to contain no moving shadow. A spatio-temporal follow up of the obtained areas is applied in order to remove false detection. The proposed segmentation method was tested and validated on real video sequences.

A representation using eigenimages that achieves in two stages identify the object and locate its pose is addressed.In this paper we demonstrate how a mixture of two approaches based on eigenspaces with some little modifications resolve the problem of identification and the location of the object. In the first stage we recognize the object by means of PCA* (Principal Component Analysis) method combined with a neural network classifier, and in the second step, the object’s pose is obtained using a modification of typical PCA (we name as PCA2 method). We present the results obtained using a database made with 25 real objects.

In the general multiple video object coder, more interesting objects such as a speaker or a moving object is consistently coded with higher priority. Since the priority of each object may not be fixed in the whole sequence and be variable on a frame by frame basis, it must be adjusted in a frame. In this paper, we analyze the independent rate control algorithm and the global algorithm that the QP value is controlled by static parameters, object importance or priority, the target PSNR and the weighted distortion. The priority among static parameters is analyzed and adjusted into dynamic parameters according to the visual interests or importance obtained by a camera interface. The target PSNR and the weighted distortion are proportional to magnitude, motion, and distortion. We apply these parameters for the weighted distortion control and the priority-based control leading to an efficient bit-rate distribution. As result, we have achieved that fewer bits are allocated for video objects which have less importance and more bits for those which have higher visual importance. The period to reach stability in the visual quality is reduced to less than 15 frames of the coded sequence. With respect to the PSNR, the proposed scheme shows higher quality of over 2dB than the conventional schemes. Thus the coding scheme interfaced to human-eyes proves to be an efficient video coder dealing with the multiple video objects.

This paper considers the statistics of local appearance based measures that are suitable for the visual parsing of sport events. The moments of the colour information are computed, and the shape content in the frames is characterised by the moments of local shape measures. Their generation process is very low cost. The temporal evolution of the features then is modelled with a Hidden Markov Model. The HMM is used to generate higher level information by classifying the shots as close ups, court views, crowd shots and so on. The paper illustrates how those simple features, coupled with the HMM, can be used for parsing snooker and tennis footages.

This paper proposes novel technique for elimination of moving objects self-generated illumination variations with respect to a fixed reference frame (background). In particular, proposed techniques can be used by a video-based surveillance system able to automatically detect potentially dangerous or particular situations happening within a guarded area. Proposed techniques are based on chromatic properties of cast shadows and illumination and achieved results demonstrate that the proposed approach can greatly improve performances of a video-surveillance system in term of the precision of detected objects. Presented processing techniques are necessary in the case of a real system that must be able to work with good performances twenty-four hours a day.

The research on Content-based Image Retrieval (CBIR) has been very active in recent years. The performance of a CBIR system can be significantly improved by selecting a good indexing feature space to represent image characteristics. In this paper, we introduce a statistical-model based technique for analyzing and extracting image features in the wavelet domain. The images are decomposed into a set of wavelet subspaces in the wavelet domain and for each wavelet subspace, a two component Gaussian mixture model is developed to describe the statistical characteristics of the wavelet coefficients. The model parameters, which are a good reflection of image features in the wavelet subspaces, are obtained by an EM (Expectation-Maximization) algorithm and employed to construct the indexing feature space for a CBIR system. We apply the new method on the Brodatz image database to demonstrate its performance. The experimental results indicate that our indexing feature space is very effective in representing image characteristics and provides a high retrieval rate in the CBIR system. When compared with some other conventional feature extraction methods, the new method achieves comparable retrieval performance with less number of features in the feature space, which means it is more computationally efficient.

In this paper, we describe a three-step content-based approach to image retrieval and mining. At a first step, visual features such as color and shape are generated from images by improving a few existing feature extraction techniques. Then, both visual features and descriptive data (i.e., metadata) are considered for a coarse-grain similarity analysis using a conceptual clustering approach called formal concept analysis (concept lattice theory). This approach is designed and implemented so that exploratory mechanisms such as browsing, zooming/shrinking and visualization allow the user to discover and refine the cluster which is the most appropriate to his/her target image. At this second stage, issues such as dimension reduction, cluster construction and association rule generation are handled. The last step consists to conduct a fine-grain similarity analysis on some selected cluster(s) identified at the second stage by using two newly proposed similarity measures.

Content based image retrieval has gained considerable attention in nowadays as a very useful tool in a plethora of applications. Web has become the most important application, because over 70% of it is devoted to images, and looking for a specific image is a really daunting task. The vast majority of these images are JPEG compressed. An extensive study of eighteen similarity measures used for image retrieval has been conducted and the corresponding results are reported in the present communication. The energy histograms of the low frequency DCT coefficients have been used as the feature space for similarity testing. Query-by-image-example was used in all tests.

This paper presents an object detection framework applied to cinematographic post-processing of video sequences. Post-processing is done after production and before editing. At the beginning of each shot of a video, a slate (also called clapperboard) is shown. The slate contains notably an electronic audio timecode that is necessary for audio-visual synchronization. This paper presents an object detection framework to detect slates in video sequences for automatic indexing and post-processing. It is based on five steps. The first two steps aim to reduce drastically the video data to be analyzed. They ensure high recall rate but have low precision. The first step detects images at the beginning of a shot possibly showing up a slate while the second step searches in these images for candidates regions with color distribution similar to slates. The objective is to not miss any slate while eliminating long parts of video without slate appearance. The third and fourth steps are statistical classification and pattern matching to detected and precisely locate slates in candidate regions. These steps ensure high recall rate and high precision. The objective is to detect slates with very little false alarms to minimize interactive corrections. In a last step, electronic timecodes are read from slates to automize audio-visual synchronization. The presented slate detector has a recall rate of 89% and a precision of 97,5%. By temporal integration, much more than 89% of shots in dailies are detected. By timecode coherence analysis, the precision can be raised too. Issues for future work are to accelerate the system to be faster than real-time and to extend the framework for several slate types.

In this paper, we describe an algorithm which can automatically recognize human gesture in a sequence of natural video image by utilizing two dimensional features extracted from bodily region of the images. In the algorithm, we first construct a gesture model space by analyzing the statistical information of sample images with principle component analysis method. And then, input images are compared to the model and individually symbolized to one part of the model space. In the last step, the symbolized images are recognized with HMM as one of model gestures. The feature of our method is to use a combination of partial and global information of two-dimensional abstract bodily motion, consequently it is very convenient to apply to real world situation and the recognition results are very robust.

This paper describes a strategy to visualise a data-set of multi-component feature vectors using multidimensional scaling (MDS). MDS is employed, instead of more commonly applied mapping techniques, as it can utilise arbitrary distance measures and hence can easily incorporate the non-linear distance metrics employed when matching multi-component vectors. To test this mapping approach, we have applied it to a data-set of two hundred and sixty eight images, segmented into multiple components each represented by a shape descriptor. The inter-image distances are measured using a series of simple non-location based image distance metrics. The maps are encouraging, with well clustered areas for duplicate or near duplicate trademarks. This gives a clear indication that MDS can be used for this type of visualisation task. However, the maps themselves also significantly highlight the inadequacies of the segmentation and matching phases. Particularly for the images with an overall impression that doesn’t correspond to the segmented parts, for example figure/ground reversal or macro texture.

The study presented in this paper analyses descriptions extracted with MPEG-7-descriptors from visual content from the statistical point of view. Good descriptors should generate descriptions with high variance, a well-balanced cluster structure and high discriminance to be able to distinguish different media content. Statistical analysis reveals the quality of the used description extraction algorithms. This was not considered in the MPEG-7-design process where optimising the recall was the major goal. For the analysis eight basic visual descriptors were applied on three media collections: the Brodatz dataset (monochrome textures), a selection of the Corel dataset (colour photos) and a set of coats-of-arms images (artificial colour images with few colour gradations). The results were analysed with four statistical methods: mean and variance of descriptor elements, distribution of elements, cluster analysis (hierarchical and topological) and factor analysis. The main results are: The best descriptors for combination are Color Layout, Dominant Color, Edge Histogram and Texture Browsing. The other are highly dependent on these. The colour histograms (Color Structure and Scalable Color) perform badly on monochrome input. Generally, all descriptors are highly redundant and the application of complexity reduction transformations could save up to 80% of storage and transmission capacity.

The unprecedented increase in the level of heterogeneity of emerging wireless networks and the mobile Internet emphasizes the need for scalable and adaptive video solutions both for coding and transmission purposes. However, in general, there is an inherent tradeoff between the level of scalability (e.g., in terms of bitrate range, levels of spatial resolutions, and/or levels of temporal resolutions) and the video-coding penalty incurred by such scalable video schemes as compared to non-scalable coders. In other words, the higher the level of scalability, the lower the overall video quality of the scalable stream that is needed to support the desired scalability level. In [1][2][3], we introduced the notion of TranScaling, which is a generalization of (non-scalable) transcoding. With TranScaling, a scalable video stream covering a given bandwidth range, is mapped into one or more scalable video streams covering different bandwidth ranges. In this paper, we illustrate the benefits of Spatial TranScaling in the context of a recently developed scalable and adaptive inband motion compensated temporal filtering scheme (IBMCTF) [4][5]. We show how using TranScaling, the already high coding efficiency performance of such adaptive IBMCTF schemes can be further improved.

This paper presents a coding mode decision algorithm for MPEG-2 spatial transcoding. The optimization for coding mode and quantization scale are formulated in an operational rate distortion sense and solved by Lagarange multiplier method. The experimental results show that the proposed transcoder with optimized coding mode and quantizer can achieve better quality and lower bit rate than those obtained using cascaded transcoder or MPEG-2 TM5 encoder.

In this text we present a system for streaming video content encoded using the motion-compensated Embedded Zero Block Coder (EZBC). The system incorporates unequal loss protection in the form of multiple description FEC (MD-FEC) coding, which provides adequate protection for the embedded video bitstream when the loss process is not very bursty. The adverse effects of burst losses are reduced using a novel motion-compensated error concealmet method.

Achieving very low bitrates for video applications in a mobile environment is essential. Video codecs standards evolution had a tremendous impact in the development of digital video, making possible today widespread usage of video data. At the same time it had an impact on the direction of research, biasing most of the efforts towards block based video. Alternative approaches may though lead towards different architectures able to provide higher compression efficiency. In this paper we propose a Matching Pursuit (MP) based video coder that adopts a generalized sub-pixel motion compensation and arithmetic coding. We introduce a new dictionary, and an adaptive grid for atom coding combined with an arithmetic coder. Comparison with H.264 show an improvement of up to 20% in compression efficiency for the same PSNR

In today multimedia audio-video communication systems, data compression plays a fundamental role by reducing the bandwidth waste and the costs of the infrastructures and equipments. Among the different compression standards, the MPEG-4 is becoming more and more accepted and widespread. Even if one of the fundamental aspects of this standard is the possibility of separately coding video objects (i.e. to separate moving objects from the background and adapt the coding strategy to the video content), currently implemented codecs work only at the full-frame level. In this way, many advantages of the flexible MPEG-4 syntax are missed. This lack is due both to the difficulties in properly segmenting moving objects in real scenes (featuring an arbitrary motion of the objects and of the acquisition sensor), and to the current use of these codecs, that are mainly oriented towards the market of DVD backups (a full-frame approach is enough for these applications). In this paper we propose a codec for MPEG-4 real-time object streaming, that codes separately the moving objects and the scene background. The proposed codec is capable of adapting its strategy during the transmission, by analysing the video currently transmitted and setting the coder parameters and modalities accordingly. For example, the background can be transmitted as a whole or by dividing it into “slightly-detailed” and “highly detailed” zones that are coded in different ways to reduce the bit-rate while preserving the perceived quality. The coder can automatically switch in real-time, from one modality to the other during the transmission, depending on the current video content. Psychovisual masks and other video-content based measurements have been used as inputs for a Self Learning Intelligent Controller (SLIC) that changes the parameters and the transmission modalities. The current implementation is based on the ISO 14496 standard code that allows Video Objects (VO) transmission (other Open Source Codes like: DivX, Xvid, and Cisco’s Mpeg-4IP, have been analyzed but, as for today, they do not support VO). The original code has been deeply modified to integrate the SLIC and to adapt it for real-time streaming. A personal RTP (Real Time Protocol) has been defined and a Client-Server application has been developed. The viewer can decode and demultiplex the stream in real-time, while adapting to the changing modalities adopted by the Server according to the current video content. The proposed codec works as follows: the image background is separated by means of a segmentation module and it is transmitted by means of a wavelet compression scheme similar to that used in the JPEG2000. The VO are coded separately and multiplexed with the background stream. At the receiver the stream is demultiplexed to obtain the background and the VO that are subsequently pasted together. The final quality depends on many factors, in particular: the quantization parameters, the Group Of Video Object (GOV) length, the GOV structure (i.e. the number of I-P-B VOP), the search area for motion compensation. These factors are strongly related to the following measurement parameters (that have been defined during the development): the Objects Apparent Size (OAS) in the scene, the Video Object Incidence factor (VOI), the temporal correlation (measured through the Normalized Mean SAD, NMSAD). The SLIC module analyzes the currently transmitted video and selects the most appropriate settings by choosing from a predefined set of transmission modalities. For example, in the case of a highly temporal correlated sequence, the number of B-VOP is increased to improve the compression ratio. The strategy for the selection of the number of B-VOP turns out to be very different from those reported in the literature for B-frames (adopted for MPEG-1 and MPEG-2), due to the different behaviour of the temporal correlation when limited only to moving objects. The SLIC module also decides how to transmit the background. In our implementation we adopted the Visual Brain theory i.e. the study of what the “psychic eye” can get from a scene. According to this theory, a Psychomask Image Analysis (PIA) module has been developed to extract the visually homogeneous regions of the background. The PIA module produces two complementary masks one for the visually low variance zones and one for the higly variable zones; these zones are compressed with different strategies and encoded into two multiplexed streams. From practical experiments it turned out that the separate coding is advantageous only if the low variance zones exceed 50% of the whole background area (due to the overhead given by the need of transmitting the zone masks). The SLIC module takes care of deciding the appropriate transmission modality by analyzing the results produced by the PIA module. The main features of this codec are: low bitrate, good image quality and coding speed. The current implementation runs in real-time on standard PC platforms, the major limitation being the fixed position of the acquisition sensor. This limitation is due to the difficulties in separating moving objects from the background when the acquisition sensor moves. Our current real-time segmentation module does not produce suitable results if the acquisition sensor moves (only slight oscillatory movements are tolerated). In any case, the system is particularly suitable for tele surveillance applications at low bit-rates, where the camera is usually fixed or alternates among some predetermined positions (our segmentation module is capable of accurately separate moving objects from the static background when the acquisition sensor stops, even if different scenes are seen as a result of the sensor displacements). Moreover, the proposed architecture is general, in the sense that when real-time, robust segmentation systems (capable of separating objects in real-time from the background while the sensor itself is moving) will be available, they can be easily integrated while leaving the rest of the system unchanged. Experimental results related to real sequences for traffic monitoring and for people tracking and afety control are reported and deeply discussed in the paper. The whole system has been implemented in standard ANSI C code and currently runs on standard PCs under Microsoft Windows operating system (Windows 2000 pro and Windows XP).

In this paper, we propose a layer-based integrated real-time scheduling algorithm in a single scalable stream and an online dynamic resource allocation algorithm among multiple concurrent users for scalable streaming media server over a network with packet loss and variable delay. The layer-based real-time scheduling algorithm efficiently schedules the packets in the buffer of the scalable streaming media server for transmission. The on-line resource allocation algorithm can allocate the server’s resource among all the concurrent streams fairly and improve the playback quality in client. Simulation results show that our proposed algorithms outperform the frame-based scheduling algorithm and the off-line resource allocation algorithm in various situations with different round-trip times, channel errors, etc. The low complexity of the proposed algorithms also enables them to be applied in real-time applications.

This paper addresses the optimal rate control in a communication network consisting of multiple users, transmitting video. We suggest that rate control in the Internet should take the signal properties into account. An optimal distribution of rates should maximise the aggregate quality of the received signals. This paper addresses video sequences specifically, but the results are expected to hold for more generalized signals as well. The optimal transmission rate of a user will vary in time. This is due to its dependence on time-varying parameters such as the traffic routing, network topology and the demands of other users. Moreover, the optimal transmission rate is dependent on the metric selected to describe the video quality. We show that when congestion is controlled `correctly' the optimal distribution of rates is an attractor. Using the proposed rate control mechanisms, the transmission rates will then converge weakly towards this attractor. The algorithm for rate control mechanisms for video transmission is then given. The proposed control mechanisms are directly implementable in the Internet without requiring intelligent feedback from the Internet gateways. Simulations using the proposed congestion control mechanisms have been conducted using different metrics to describe the video quality.

This paper investigates a novel active buffer management scheme, "Jitter Detection" (JD) for gateway-based congestion control to stream multimedia traffics in packet-switched networks. The quality of multimedia presentation can be greatly degraded due to network delay variation or jitter when transported over packet-switched network. Jitter degrades the timing relationship among packets in a single media stream and between packets from different media streams and hence creates multimedia synchronization problems. Moreover, too much jitter will also degrade the performance of the streaming buffer in the client. Packets received by client will render useless if they have accumulated a large enough jitter. The proposed active buffer management scheme will improve the quality of service in multimedia networking by detecting and discarding useless packets that accumulated large enough jitter. Such as to maintain a high bandwidth for packets within the multimedia stream's jitter tolerance. Simulation results have shown that the proposed scheme can effectively lower the average received packet jitter and increase the goodput of the received packets when compared to random early detection (RED), and Droptail used in gateway-based congestion control. Furthermore, simulation results have also revealed that the proposed scheme can maintain the same TCP-friendliness when compared to that of RED and Droptail used for multimedia streams.

International recommendations for subjective video quality assessment (e.g., ITU-R BT.500-11) include specifications for how to perform many different types of subjective tests. Some of these test methods are double stimulus where viewers rate the quality or change in quality between two video streams (reference and impaired). Others are single stimulus where viewers rate the quality of just one video stream (the impaired). Two examples of the former are the double stimulus continuous quality scale (DSCQS) and double stimulus comparison scale (DSCS). An example of the latter is single stimulus continuous quality evaluation (SSCQE). Each subjective test methodology has claimed advantages. For instance, the DSCQS method is claimed to be less sensitive to context (i.e., subjective ratings are less influenced by the severity and ordering of the impairments within the test session). The SSCQE method is claimed to yield more representative quality estimates for quality monitoring applications. This paper considers data from six different subjective video quality experiments, originally performed with SSCQE, DSCQS and DSCS methodologies. A subset of video clips from each of these six experiments were combined and rated in a secondary SSCQE subjective video quality test. We give a method for post-processing the secondary SSCQE data to produce quality scores that are highly correlated to the original DSCQS and DSCS data. We also provide evidence that human memory effects for time-varying quality estimation seem to be limited to about 15 seconds.

International recommendations for subjective video quality assessment (e.g., ITU-R BT.500-11) include specifications for how to perform many different types of subjective tests. In addition to displaying the video sequences in different ways, subjective tests also have different rating scales, different words associated with these scales, and many other test variables that change from one laboratory to another (e.g., viewer expertise and criticality, cultural differences, physical test environments). Thus, it is very difficult to directly compare or combine results from two or more subjective experiments. The ability to compare and combine results from multiple subjective experiments would greatly benefit developers and users of video technology since standardized subjective data bases could be expanded upon to include new source material and past measurement results could be related to newer measurement results. This paper presents a subjective method and an objective method for combining multiple subjective data sets. The subjective method utilizes a large meta-test with selected video clips from each subjective data set. The objective method utilizes the functional relationships between objective video quality metrics (extracted from the video sequences) and corresponding subjective mean opinion scores (MOSs). The objective mapping algorithm, called the iterated nested least-squares algorithm (INLSA), relates two or more independent data sets that are themselves correlated with some common intermediate variables (i.e, the objective video quality metrics). We demonstrate that the objective method can be used as an effective substitute for the expensive and time consuming subjective meta-test.

This paper presents the results of a quality evaluation of video sequences encoded for and transmitted over a wireless channel. We selected content, codecs, bitrates and bit error patterns representative of mobile applications, focusing on the MPEG-4 and Motion JPEG2000 coding standards. We carried out subjective experiments using the Single Stimulus Continuous Quality Evaluation (SSCQE) method on this test material. We analyze the subjective data and use them to compare codec performance as well as the effects of transmission errors on visual quality. Finally, we use the subjective ratings to validate the prediction performance of a real-time non-reference quality metric.

In this paper, we propose a new video quality evaluation method based on multi-feature and radial basis function neural network. Multi-feature is extracted from a degraded image sequence and its reference sequence, including error energy, activity-masking and luminance-masking as well as blockiness and blurring features. Based on these factors we apply a radial basis function neural network as a classifier to give quality assessment scores. After training with the subjective mean opinion scores (MOS) data of VQEG test sequences, the neural network model can be used to evaluate video quality with good correlation performance in terms of accuracy and consistency measurements.

This paper describes a model to estimate human subjective video quality scores based on a framework of double-ended with a full reference system or end-to-end network video systems where sending video and receiving videos are available. The model incorporates multi-dimensional and multi-variables all of which are objectively measurable. Its aim is to estimate subjective score in the difference mean opinion score between a reference video and corresponding degraded videos. The model is developed based on an application of the multivariable analysis and trained by a set of a priori known relationship between physically obtainable parameters and corresponding subjective score. The paper describes how the model incorporates a mathematical algorithm as an executable programme module, and in what degree the model can be applicable in order to account actual examples of video pairs for both training video set used and foreign video sets. Referring to an application of the developed model to a set of foreign videos, it reports the capability of the model to different quality range. In addition to the worked results, some issues for further study are highlighted.

In this paper we present a no-reference objective quality metric (NROQM) that has resulted from extensive research on impairment metrics, image feature metrics, and subjective image quality in several projects in Philips Research, and participation in the ITU Video Quality Experts Group. The NROQM is aimed at requirements including video algorithm development, embedded monitoring and control of image quality, and evaluation of different types of display systems. NROQM is built from metrics for desirable and non-desirable image features (sharpness, contrast, noise, clipping, ringing, and blocking artifacts), and accounts for their individual and combined contributions to perceived image quality. We describe our heuristic, incremental approach to modeling quality and training the NROQM, and its advantages to deal with imperfect data and imperfect metrics. The results of training the NROQM using a large set of video sequences, which include degraded and enhanced video, show high correlation between objective and subjective scores, and the results of the first performance test show good objective-subjective correlations as well. We also discuss issues that require further research such as fully content-independent metrics, measuring over-enhanced video quality, and the role of temporal impairment metrics.

This paper presents a new and general concept, PQSM (Perceptual Quality Significance Map), to be used in measuring the visual distortion. It makes use of the selectivity characteristic of HVS (Human Visual System) that it pays more attention to certain area/regions of visual signal due to one or more of the following factors: salient features in image/video, cues from domain knowledge, and association of other media (e.g., speech or audio). PQSM is an array whose elements represent the relative perceptual-quality significance levels for the corresponding area/regions for images or video. Due to its generality, PQSM can be incorporated into any visual distortion metrics: to improve effectiveness or/and efficiency of perceptual metrics; or even to enhance a PSNR-based metric. A three-stage PQSM estimation method is also proposed in this paper, with an implementation of motion, texture, luminance, skin-color and face mapping. Experimental results show the scheme can improve the performance of current image/video distortion metrics.

A novel method of detecting local impairment of the coded picture caused by transmission failure on digital television transmission using a reduced reference framework is proposed. The method utilizes the SSSWHT method, which is based on spread spectrum and extraction of WHT coefficients, and precisely estimates the MSE in a small region within the frame using reduced references to detect local impairment caused by MPEG bitstream error. Computer simulations show that the proposed method detects the impaired frames completely at a reference path bitrate of 425 kbps. The method also detects slices that have impaired pixel blocks with accuracy of approximately 70% at a reference path bitrate of 425 kbps. These results confirm that the proposed method is effective for television transmission monitoring.

The paper provides an overview of the fundamental issues confronting scalable video compression, together with some of the most promising general approaches to addressing these issues. For motivation, the paper sketches a compelling application in remote browsing of video, whose realization is not possible without efficient highly scalable video compression technology. The paper outlines some of the most important structural alternatives for wavelet-based scalable video compression systems, together with experimental findings in support of particular alternatives. In particular, the paper describes novel spatio-temporal decomposition structures, a novel finely embedded motion coding structure, and a complete compression system which addresses the needs of a wide range of potential applications. Preliminary compression performance results are provided together with information on an implementation which is capable of real-time decoding at CIF resolutions and beyond.

Recently a methodology for representation and adaptation of arbitrary scalable bit-streams in a fully content non-specific manner has been proposed on the basis of a universal model for all scalable bit-streams called Scalable Structured Meta-formats (SSM). According to this model, elementary scalable bit-streams are naturally organized in a symmetric multi-dimensional logical structure. The model parameters for a specific bit-stream along with information guiding decision-making among possible adaptation choices are represented in a binary or XML descriptor to accompany the bit-stream flowing downstream. The capabilities and preferences of receiving terminals flow upstream and are also specified in binary or XML form to represent constraints that guide adaptation. By interpreting the descriptor and the constraint specifications, a universal adaptation engine sitting on a network node can adapt the content appropriately to suit the specified needs and preferences of recipients, without knowledge of the specifics of the content, its encoding and/or encryption. In this framework, different adaptation infrastructures are no longer needed for different types of scalable media. In this work, we show how this framework can be used to adapt fully scalable video bit-streams, specifically ones obtained by the fully scalable MC-EZBC video coding system. MC-EZBC uses a 3-D subband/wavelet transform that exploits correlation by filtering along motion trajectories, to obtain a 3-dimensional scalable bit-stream combining temporal, spatial and SNR scalability in a compact bit-stream. Several adaptation use cases are presented to demonstrate the flexibility and advantages of a fully scalable video bit-stream when used in conjunction with a network adaptation engine for transmission.

In interframe wavelet video coding, wavelet-based motion-compensated temporal filtering (MCTF) is combined with spatial wavelet decomposition, allowing for efficient spatio-temporal decorrelation and temporal, spatial and SNR scalability. Contemporary interframe wavelet video coding concepts employ block-based motion estimation (ME) and compensation (MC) to exploit temporal redundancy between successive frames. Due to occlusion effects and imperfect motion modeling, block-based MCTF may generate temporal high frequency subbands with block-wise varying coefficient statistics, and low frequency subbands with block edges. Both effects may cause declined spatial transform gain and blocking artifacts. As a modification to MCTF, we present spatial highpass transition filtering (SHTF) and spatial lowpass transition filtering (SLTF), introducing smooth transitions between motion blocks in the high and low frequency subbands, respectively. Additionally, we analyze the propagation of quantization noise in MCTF and present an optimized quantization strategy to compensate for variations in synthesis filtering for different block types. Combining these approaches leads to a reduction of blocking artifacts, smoothed temporal PSNR performance, and significantly improved coding efficiency.

We propose a novel approach that uses disparity-compensated lifting for wavelet compression of light fields. Disparity compensation is incorporated into the lifting structure for the transform across the views to solve the irreversibility limitation in previous wavelet coding schemes. With this approach, we obtain the benefits of wavelet coding, such as scalability in all dimensions, as well as superior compression performance. For light fields of an object, shape adaptation is adopted to improve the compression efficiency and visual quality of reconstructed images. In this work we extend the scheme to handle light fields with arbitrary camera arrangements. A view-sequencing algorithm is developed to encode the images. Experimental results show that the proposed scheme outperforms existing light field compression techniques in terms of compression efficiency and visual quality of the reconstructed views.

This paper proposes an advanced motion-threading technique to improve the coding efficiency of the 3D wave-let coding. We extend the original motion-threading technique to the lifting wavelet structure. This extension solves the artificial motion thread truncation problem in long support temporal wavelet filtering, and enables the accuracy of motion alignment to be fractional-pixel with guaranteed perfect reconstruction. Furthermore, the mismatch problem in the motion-threading caused by occlusion or scene-change is considered. In general, the temporal wavelet decomposition consists of multiple layers. Unlike the original motion-threading scheme, in the proposed scheme each layer owns one set of motion vectors so as to achieve both high coding efficiency and temporal scalability. To reduce the motion cost, direct mode is used to exploit the motion vector correlation. An R-D optimized technique is introduced to estimate motion vectors and select proper prediction modes for each macroblock. The proposed advanced motion-threading scheme can outperform the original motion-threading scheme up to 1.5~5.0 dB. The experimental results also demonstrate that the 3D wavelet coding scheme can be competitive with the start-of-the-art JVT video standard on coding efficiency.

Techniques for full scalability with motion-compensated temporal filtering (MCTF) in the wavelet-domain (in-band) are presented in this paper. The application of MCTF in the wavelet domain is performed after the production of the overcomplete discrete wavelet transform from the critically-sampled decomposition, a process that occurs at both the encoder and decoder side. This process, which is a complete-to-overcomplete discrete wavelet transform, is critical for the efficiency of the system with respect to scalability, coding performance and complexity. We analyze these aspects of the system and set the necessary constraints for drift-free video coding with in-band MCTF. As a result, the proposed architecture permits the independent operation of MCTF within different resolution levels or even different subbands of the transform and allows the successive refinement of the video information in resolution, frame-rate and quality.

This paper describes low rate FGS video compression algorithms based on closed-loop temporal prediction combined with motion-compensated spatio-temporal wavelet analysis. After presenting an overall coding architecture, the paper reviews different motion-compensated (MC) temporal filtering solutions and discusses their amenability for compression and fine grain scalability. The coding architecture relies on a rate-constrained hierarchical motion estimation and a 3D-EBCOT algorithm. The analysis of different MC-temporal filtering solutions is substantiated with various quantitative elements on filter length, on motion models and temporal transform reliability. Some of their limitations are then addressed by designing algorithms which combine closed-loop temporal prediction with MC-temporal analysis. The PSNR performances of the approaches are compared with those obtained with MPEG-4 part 2 and H.264 JM2.1.

In this paper we compare the coding performance of the JPEG2000 still image coding standard with the INTRA coding method used in the H.26L project. We discuss the basic techniques of both coding schemes and show the effect of improved I-frame coding to the overall performance of a H.26L-based system. The coding efficiency as well as the runtime behaviour is considered in our comparison.

An extension of the JPEG 2000 standard is presented for non-conventional images resulting from an adaptive subdivision process. Samples, generated through adaptive subdivision, can have different sizes, depending on the amount of subdivision that was locally introduced in each region of the image. The subdivision principle allows each individual sample to be recursively subdivided into sets of four progressively smaller samples. Image datasets generated through adaptive subdivision find application in Computational Physics where simulations of natural processes are often performed over adaptive grids. It is also found that compression gains can be achieved for non-natural imagery, like text or graphics, if they first undergo an adaptive subdivision process. The representation of adaptive subdivision images is performed by first coding the subdivision structure into the JPEG 2000 bitstream, ina lossless manner, followed by the entropy coded and quantized transform coefficients. Due to the irregular distribution of sample sizes across the image, the wavelet transform must be applied on irregular image subsets that are nested across all the resolution levels. Using the conventional JPEG 2000 coding standard, adaptive subdivision images would first have to be upsampled to the smallest sample size in order to attain a uniform resolution. The proposed method for coding adaptive subdivision images is shown to perform better than conventional JPEG 2000 for medium to high bitrates.

In this paper, we propose a new method for removing coding artifacts appeared in JPEG 2000 coded images. The proposed method uses a fuzzy control model to control the weighting function for different image edges according to the gradient of pixels and membership functions. Regularized post-processing approach and recursive line algorithm are described in this paper. Experimental results demonstrate that the proposed algorithm can significantly improve image quality in terms of objective and subjective evaluation.

A new set of boundary-handling algorithms has been developed for discrete wavelet transforms in the ISO/IEC JPEG-2000 Still Image Coding Standard. Two polyphase component extrapolation policies are specfied: a constant extension policy and a symmetric extension policy. Neither policy requires any computations to generate the extrapolation. The constant extension policy is a low-complexity option that buffers just one sample from each end of the input being extrapolated. The symmetric extension policy has slightly higher memory and conditional-logic requirements but is mathematically equivalent to wholesample symmetric pre-extension when used with whole-sample symmetric filter banks. Both policies can be employed with arbitrary lifted filter banks, and both policies preserve resolution scalability and reversibility. These extension policies will appear in Annex H, "Transformation of images using arbitrary wavelet transformations," in Part 2 ("Extensions") of the JPEG-2000 standard.

In this paper we put forward a novel V9/3 wavelet that constructed by human visual system model (HVS). Using this wavelet in JPEG2000 standard, there will be better visual effect and lower computation complexity than that of Daubechies9/7 in JPEG2000. The wavelet base coefficients and its lift scheme discussed in paper. The experiment result display this wavelet have better visual effect.

JPEG2000 is a family of technologies based on the image compression system defined in IS 15444-1. Presently, the ISO/IEC Joint Technical Committee of Photographic Experts (JPEG) is developing an international standard for interactivity with JPEG2000 files, called JPIP; it will become Part 9 of the standard. One of the main goals of JPIP is to exploit the multi-resolution and spatially random access properties of JPEG2000, to permit “smart dissemination” of the data for client-server based applications. The purpose of this paper is to discuss the principles and philosophy of operation underlying the JPIP standard, to outline aspects of the architecture of JPIP systems, and to report on the performance of a prototype implementation.

Surveillance imagery is used for supporting the decision processes in strategic, operational and tactical tasks. The visual tasks associated with the interpretation of such imagery may require the detection of targets or the recognition of image contents. It is important for an image compression scheme to be tuned to maximise the imagery's interpretability (or content recognition) performance. Importance prioritisation coding of an image, where image contents are prioritised in order of importance for interpretability, is hence a very desirable feature for such applications. The paper presents a flexible framework for such prioritisation in JPEG2000 using regions of interests (ROIs) and importance maps. The coding scheme allows multiple and arbitrary shaped ROIs to be defined at different spatial locations, sub-band orientations and scale, and subsequently be prioritised using arbitrary importance scores. No ROI or importance map information need to be signalled to the decoder, and progressive lossy to lossless reconstruction of ROIs and the background is also possible. Furthermore, the prioritisation maintains the JPEG2000 code-stream syntax and can be decoded by any generic JPEG2000 decoder.

With the continual expansion of multimedia and Internet applications, the needs and requirements of advanced technologies, grew and evolved. With the increasing use of multimedia technologies, image compression techniques require higher performance as well as new features. Significant progress has recently been made in image compression techniques using discrete wavelet transforms. The overall performance of these schemes may be further improved by properly designing of efficient entropy coders. In this paper, we describe an efficient architecture for JPEG2000 entropy coder, which is a new standard to address the needs in the specific area of still image encoding. Our proposed architecture consists of two main parts, the coefficient bit modeler (CBM) and the binary arithmetic coder (BAC), which communicate through a FIFO buffer. Optimizations have been made in our proposed architecture to reduce accesses to memories. Our Proposed architecture is fast and modular and is suitable for real-time applications.

This paper presents some ideas which extend the functionality and the application fields of a spatially selective coding within a JPEG2000 framework. At first, the image quality drop between the Regions of Interest (ROI) and the background (BG) is considered. In a conventional approach, the reconstructed image quality steeply drops along the ROI boundary; however, this effect could be considered or perceived objectionable in some use cases. A simple quality decay management is proposed here, which makes use of concentric ROI with different scaling factors. This allows the technique to be perfectly consistent with the JPEG2000 part 2 ROI definition and description. Another considered issue is the extension of the selective ROI coding to a 3D Volume of Interest coding. This extension is currently under consideration for the part 10 of JPEG2000, JP3D. An easy and effective 2D to 3D extension for the VOI definition and description is proposed here: a VOI is defined by a set composition of ROI generated solids, where ROI are defined along one or more volume cutting direction, and is described by the relative set of ROI parameters. Moreover, the quality decay management can be applied to this extension. The proposed techniques could have a significant impact on the selective coding of medical images and volumes. Image quality issues are very important but very critical factors in that field, which also constitutes the dominant market for 3D applications. Therefore, some experiments are presented on medical images and volumes in order evaluate the benefits of the proposed approaches in terms of diagnostic quality improvement with respect to a conventional ROI coding usage.

With the rapid development of wireless communication systems, transmission of digital multimedia has become widely spread. This brings with itself the issue of copyright protection for digital work. Digital watermarking is the process of embedding data inside a host image such that it does not degrade the perceptual quality of the image. In recent years, there have been many approaches to introduce different watermarking algorithms for the purposes of copyright protection, broadcast monitoring and covert communication. In this paper, a new transform based watermarking algorithm for digital images is introduced. This method uses the Singular Value Decomposition to obtain the eigenimages of a given image which are known to be the best orthogonal basis that can express that image in a least squares sense. The watermark is embedded by changing the strength of the singular values. The strength of the watermark is determined based on the entropy of the eigenimages to ensure robustness and imperceptibility at the simultaneously. A corresponding watermark detection and extraction algorithm is proposed. The performance of the algorithm under different types of attacks that a digital image can go through during transmission is illustrated through an example.

In this paper we propose the use of a Bayesian framework to allow characterisation of image tampering from a library of attacks. We use the double watermarking strategy proposed in our previous work to derive sufficient information to drive the classifier. A non-parametric Bayesian classifier, trained on data derived from Monte Carlo simulations is used. In addition to classification, the effects of varying the input parameters are studied. The results obtained show that the non-parametric Bayesian classifier has a very low misclassification rate for this type of problem. Explanations as to the nature of the results, and some of the practical considerations, are given.

The widespread adoption of digital video techniques has generated a requirement for authenticity verification in applications such as criminal evidence, insurance claims and commercial databases. This paper extends our previous work and improves the watermark estimation procedure of a spatial digital video watermarking system designed to detect and characterize time-base attacks. Most watermark extraction processes utilize the noise masking levels of the image. These levels change during transmission, especially when the host signal is compressed at low bit-rates. The blocking artifacts that are introduced by this process modify the noise masking levels and influence the ability to form a good estimate of the embedded watermark. This paper describes a novel filter procedure to eliminate these artifacts from the noise masking levels. The efficiency is compared with the standard MPEG-4 deblocking and deringing filters. Extracting the watermark from only the encoded Macroblocks, excluding the skipped Macroblocks improves the performance significantly without an increase in computational complexity. The functionality of this system within an MPEG-4 implementation is demonstrated with a receiver operating characteristic.

Unlike most previous work, which used a random number of a sequence of bits or an image as watermark directly, this paper proposed a new image vectorisation method for digital image watermarking. A watermark image (image to be embedded) is firstly contourised into a sequence of contour curves by constructing a "vector" for each of the grey level values. In the contourisation process, a topology analysis method is applied for looking for local maxima, minima and saddle points to implement a topology table. It is well known that the volume of "vector" data from real image contourisation may be up to an order of magnitude greater than the raster representation. Therefore, a simplification method is adopted to analysis the great expansion of data, and to determine which contours it is necessary to preserve in a given image and which contours can be discarded in that image. With help of previously obtained the topology table, image is decomposed into a number of adjacent sub regions known as catchments basins, each of which typically surrounds of a local maximum or minimum and is defined by a contour which is referred to as a watershed or watershed boundary. Then the simplified contour points are embedded as watermark onto the cover image by using the well known spread spectrum technique. After the contour points are extracted from watermarked image, the watermark image is reconstructed by constructing the triangle mesh defined by the contour map and rendering it using conventional rendering method.

In this paper, we discuss how we can enhance the performance of the MPEG-4 Visual Texture Coding algorithm (VTC). Runtime analysis reveals the major coding stages and shows a weak point within the vertical filtering stage. A useful cache-access strategy is considered, which lifts this problem almost entirely. Additionally, we perform the DWT and the zerotree coding stage in parallel using OpenMP. The improved sequential version of the vertical filtering improves also the parallel eaciency significantly. We present results from 2 different multiprocessor platforms (SGI Power Challenge: 20 IP25 RISC CPUs, running at 195 Mhz; SGI Origin3800:128 MIPS RISC R12000 CPUs, running at 400MHz).

H.264 is an emerging video coding standard, providing significant improvements with respect to its ancestors, like MPEG-2 and MPEG-4. In this paper, we present an evaluation of the most important H.264 coding tools in terms of visual quality and compression efficiency

The universal scalability, which integrates different types of scalabilities and consequently provides a large scaling range for each parameter, is of high interests to the applications in the current heterogeneous surroundings. In our previous work, an MPEG-4 universal scalable codec basing on a layered path-tree structure [1,2] has been addressed, in which a video layer and the coding order of two consecutive layers are interpreted as a node and the parent-to-child relationship in a path-tree, respectively. Since individual video layers can be coded separately using different coding tools in MPEG-4 simple scalable profile (SSP) [3] and fine-granularity scalable profile (FGS) [4], the proposed scalable video coder may include spatial, temporal and SNR enhancements simultaneously. In this paper, based on some visual observations we first address some encoding strategies for the universal scalable coding, including spatial-temporal quality tradeoff, region sensitivity and frequency weighting. Applying these strategies will take the content characteristics into consideration and can determine better coding parameters. As a result, bit allocation becomes more sensitive to those perceptually important parts of spatial, temporal and SNR enhancements. Next, a batch encoding process is conducted to generate universal scalable streams automatically, in which all the abovementioned encoding strategies are fully integrated together. The preliminary experiments show that better visual quality can be obtained within the full bitrate range.

This paper focuses on the encoder of ITU-T H.264 | ISO/IEC MPEG-4 Part-10 AVC (hereafter “JVT”), especially the Lagrange parameter for the rate distortion optimization (RDO) scheme. Since the Lagrange optimization method is very effective from the viewpoint of coding efficiency optimization, it is specified as an essential experimental condition for performance evaluation in JVT standardization activity. After discussing the background of this mechanism, we verify this method and the currently used parameter theoretically. Computer simulation results show that the Lagrange multiplier might not necessarily be original value; smaller values provide better performance in QCIF, for example.

The unequal error protection (UEP) codes with wavelet-based algorithm for video compression over wide-band code division multiple access (W-CDMA), additive white Gaussian noise (AWGN) and Rayleigh fading channels are analysed. The utilization of Wavelets has come out to be a powerful method for compress video sequence. The wavelet transform compression technique has shown to be more appropriate to high quality video applications, producing better quality output for the compressed frames of video. A spatially scalable video coding framework of MPEG2 in which motion correspondences between successive video frames are exploited in the wavelet transform domain. The basic motivation for our coder is that motion fields are typically smooth that can be efficiently captured through a multiresolutional framework. Wavelet decomposition is applied to video frames and the coefficients at each level are predicted from the coarser level through backward motion compensation. The proposed algorithms of the embedded zero-tree wavelet (EZW) coder and the 2-D wavelet packet transform (2-D WPT) are investigated.

The demand for mobile multimedia broadcasting service is increasing consistently as more people expect seamless outdoor connections and communication capabilities. In this paper, we introduce the digital multimedia broadcasting (DMB) system based on Eureka-147 that has been tentatively adopted in Korea. Since Eureka-147 is originally designed for broadcasting digital audio data, it provides a bit error rate (BER) of about 10^-4 while the transmission of compressed video data, for example, requires the BER of about 10^-9. To deal with this mismatch, the Korean DMB standard is considering the addition of the RS(204,188) coder to Eureka-147. In this paper, we apply the RS(204,188) coder to the Eureka-147 and present the simulation results on the performance of this modified system at various transmission and protection modes. We conclude that the addition of RS(204,188) coder to Eureka-147 in the Korean DMB system results in the satisfactory level of BER for mobile multimedia broadcasting services.

This paper proposes a bit error recovery method for Internet facsimile images compressed with Group 3, Extended 2 Dimensional MMR coding scheme. When an error occurs in an MMR coded bitstream, the bitstream cannot be correctly decoded after the error point. To prevent losing valid information after an error, we developed an error recovery system that detects bit errors and applies bit-inversion algorithms to correct the errors. In case the bit-inversion cannot correct the error, the system applies new algorithms that utilize syntactical structure information of the coded bitstream. Testing results show that around 95% of bit errors are corrected with our bit-inversion algorithms. The error recovery system also recovers nearly all the data for the remaining 5% of the images.

When designing an encoder for a real-time video application over a wireless channel, we must take into consideration the unpredictable fluctuation of the quality of the channel and its impact on the transmitted video data. This uncertainty motivates the development of an adaptive video encoding mechanism that can compensate for the infidelity caused either by data loss and/or by the post-processing (error concealment) at the decoder. In this paper, we first explore the major factors that cause quality degradation. We then propose an adaptive progressive replenishment algorithm for a packet loss rate (PLR) feedback enabled system. Assuming the availability of a feedback channel, we discuss a video quality assessment method, which allows the encoder to be aware of the decoder-side perceptual quality. Finally, we present a novel dual-feedback mechanism that guarantees an acceptable level of quality at the receiver side with modest increase in the complexity of the encoder.

In this paper we address the problems of video transmission in error prone environments. A novel error-resilient method is proposed that uses a data embedding scheme for header parameters in video coding standards, such as MPEG-2 and H.263. In case of requiring taking the loss of data information into account except for header errors, the video decoder hides visual degradation as well as possible, employing an error concealment method using an affine transform. Header information is very important because syntax elements, tables, and decoding processes all depend on the values of the header information. Therefore, transmission errors in header information can result in serious visual degradation of the output video and also cause an abnormal decoding process. In the proposed method, the header parameters are embedded into the least significant bits (LSB) of the quantized DCT coefficients. Then, when errors occur in the header field of the compressed bitstream, the decoder can accurately recover the corrupted header parameters if the embedded information is extracted correctly. The error concealment technique employed in this paper uses motion estimation considering actual motions, such as rotation, magnification, reduction, and parallel motion, in moving pictures. Experimental results show that the proposed error-resilient method can effectively reconstruct the original video sequence without any additional bits or modifications to the video coding standard and the error concealment method can produce a higher PSNR value and better subjective video quality, estimating the motion of lost data more accurately.

In this paper, some applications of a local version of Singular Value Decomposition (SVD) to texture classification and texture segmentation are studied. We introduce two measures, obtained from SVD transform, which capture some of the perceptual and conceptual features in an image. One of the measures classifies the textures by their roughness and structures. Experimental results show that these measures are suitable for texture clustering and image segmentation and they are robust relative to changes in lighting.

Stroke is a major cause of disability and health care expenditure around the world. Existing stroke rehabilitation methods can be effective but are costly and need to be improved. Even modest improvements in the effectiveness of rehabilitation techniques could produce large benefits in terms of quality of life. The work reported here is part of an ongoing effort to integrate virtual reality and machine vision technologies to produce innovative stroke rehabilitation methods. We describe a combined object recognition and event detection system that provides real time feedback to stroke patients performing everyday kitchen tasks necessary for independent living, e.g. making a cup of coffee. The image plane position of each object, including the patient’s hand, is monitored using histogram-based recognition methods. The relative positions of hand and objects are then reported to a task monitor that compares the patient’s actions against a model of the target task. A prototype system has been constructed and is currently undergoing technical and clinical evaluation.

It was previously shown that sparse representations can improve and simplify the estimation of an unknown mixing matrix of a set of images and thereby improve the quality of separation of source images. Here we propose a multiscale approach to the problem of blind separation of images from a set of their mixtures. We take advantage of the properties of multiscale transforms such as wavelet packets and decompose signals and images according to sets of local features. The resulting partial representations on a tree of data structure depict various degrees of sparsity. We show how the separation error is affected by the sparsity of the decomposition coefficients, and by the misfit between the prior, formulated in accordance with the probabilistic model of the coefficients' distribution, and the actual distribution of the coefficients. Our error estimator, based on the Taylor expansion of the quasi Log-Likelihood function, is used in selection of the best subsets of coefficients, utilized in turn for further separation. The performance of the proposed method is assessed by separation of noise-free and noisy data. Experiments with simulated and real signals and images demonstrate significant improvement of separation quality over previously reported results.

In this paper, we propose an original framework for an intuitive tuning of parameters in image and video segmentation algorithms. The proposed framework is very flexible and generic and does not depend on a specific segmentation algorithm, a particular evaluation metric, or a specific optimization approach, which are the three main components of its block diagram. This framework requires a manual segmentation input provided by a human operator as he/she would have performed intuitively. This input allows the framework to search for the optimal set of parameters which will provide results similar to those obtained by manual segmentation. On one hand, this allows researchers and designers to quickly and automatically find the best parameters in the segmentation algorithms they have developed. It helps them to better understand the degree of importance of each parameter's value on the final segmentation result. It also identifies the potential of the segmentation algorithm under study in terms of best possible performance level. On the other hand, users and operators of systems with segmentation components, can efficiently identify the optimal sets of parameters for different classes of images or video sequences. In a large extent, this optimization can be performed without a deep understanding of the underlying algorithm, which would facilitate the exploitations and optimizations in real applications by non-experts in segmentation. A specific implementation of the proposed framework was obtained by adopting a video segmentation algorithm invariant to shadows as segmentation component, a full reference segmentation quality metric based on a perceptually motivated spatial context, as the evaluation component, and a down-hill simplex method, as optimization component. Simulation results on various test sequences, covering a representative set of indoor and ourdoor video, show that optimal set of parameters can be obtained efficiently and largely improve the results obtained when compared to a simple implementation of the same segmentation algorithm with ad-hoc parameter setting strategy.

Much research has been undertaken in the area of streaming video across computer networks in general and the Internet in particular, but relatively little has been undertaken in the field of streaming 3-D wireframe animation. Despite superficial similarities, both being visual media, the two are significantly different. Different data passes across the network so loss affects signal reconstruction differently. Regrettably, the perceptual effects of such loss have been poorly addressed in the context of animation to date and much of the work that there has been in this field has relied on objective measures such as PSNR in lieu of those that take subjective effects into account. In this paper, we bring together concepts from a number of fields to address the problem of how to achieve optimal resilience to errors in terms of the perceptual effect at the receiver. To achieve this, we partition the animation stream into a number of layers and apply Reed-Solomon (RS) forward error correction (FEC) codes to each layer independently and in such a way as to maintain the same overall bitrate whilst minimizing the perceptual effects of error, as measured by a distortion metric derived from related work in the area of static 3-D mesh compression. Experimental results show the efficacy of our proposed scheme under varying network bandwidth and loss conditions for different layer partitionings. The results indicate that with the proposed Unequal Error Protection (UEP) combined with Error Concealment (EC) and efficient packetization scheme, we can achieve graceful degradation of streamed animations at higher packet loss rates than other approaches that do not cater for the visual importance of the layers and use only objective layering metrics. Our experiments also demonstrate how to tune the packetization parameters in order to achieve efficient layering with respect to the subjective metric of surface smoothness.

Multiple State Video Coding (MSVC) is a Multiple Description Coding Scheme where the video is coded into multiple independently decodable streams, each with its own prediction process and state. The system subject to this work is composed of two subsystems: 1- multiple state encoding/decoding, 2- path diversity transmission system. In [1] we discussed how to optimize the rate allocation of such a system while maximizing the average reconstructed frame PSNR at the decoder and minimizing the PSNR variations between the streams given the total bitrate R_T and the balanced (equal loss probabilities) or unbalanced (unequal) loss probabilities p₁ and p₂ over the two paths. In our current work we establish a theoretical framework to estimate the rate-(decoder) distortion (R-Dd) function and have taken into account the MSVC-structure, the rate allocation, channel impairments and reconstruction strategies respectively. The video sequence is modeld as an AR(1) source and the distortion associated with each reconstructed frame in both threads is a lossy transmission environment is estimated recursively depending on the system parameters.

This paper presents a post-processing error concealment technique for handling transmission of DWT image coefficients. Our technique recovers the lost regularity information as well as irregularity information. First, the Harmonic Function is employed for overall smoothness recovery. Next, in order to enhance local edges, the spatial coherence between DWT subbands is explored. The amplitudes of those detected singular points are then modified, according to the regions they belong to. Our proposed method reveals edges from blurred areas due to smooth recovery accurately.

Measures for coping with transmission errors in wireless environments are essential for mobile video communication. This paper proposes a “flat-scalable” scheme that has transmission error tolerance and is applicable regardless of the video compression method. The proposed scheme, which uses two sets of encoders and decoders, is a diversity method in the application layer. This scheme, by differentiating the temporal position of I-pictures between the two encoders, obtains two decoded pictures whose coding noises differ. When there is no error on both streams, it averages the two decoded pictures to reduce noise and improve PSNR. When one of the streams has errors, it outputs the decoded picture without errors. This scheme greatly improves the received picture quality compared to the conventional single stream scheme especially when there are transmission errors.

Recent advances in video coding technology have resulted in rapid growth of application in mobile communication. With this explosive growth, reliable transmission and error resilient technique become increasingly necessary to offer high quality multimedia service. This paper discusses the error resilient performances of the MPEG-4 simple profile under the H.324/M and the H.264 baseline under the IP packet networks. MPEG-4 simple profile has error resilient tools such