In this paper, we present an integrated system for video surveillance developed within the European IST WCAM project, using only standard multimedia and networking tools. The advantages of such a system, while allowing cost reduction and interoperability, is to benefit from the fast technological evolution of the video encoding and distribution tools.

In this paper, we present an integrated system for smart encoding in video surveillance. This system, developed within the European IST WCAM project, aims at defining an optimized JPEG 2000 codestream organization directly based on the semantic content of the video surveillance analysis module. The proposed system produces a fully compliant Motion JPEG 2000 stream that contains regions of interest (typically mobile objects) data in a separate layer than regions of less interest (e.g. static background). First the system performs a real-time unsupervised segmentation of mobiles in each frame of the video. The smart encoding module uses these regions of interest maps in order to construct a Motion JPEG 2000 codestream that allows an optimized rendering of the video surveillance stream in low bandwidth wireless applications, allocating more quality to mobiles than for the background. Our integrated system improves the coding representation of the video content without data overhead. It can also be used in applications requiring selective scrambling of regions of interest as well as for any other application dealing with regions of interest.

The WCAM project aims to provide an integrated system for secure delivery of video surveillance data over a wireless network, while remaining scalable and robust to transmission errors. To achieve these goals, the content is encoded in Motion-JPEG2000 and streamed with a specific RTP protocol encapsulation to prevent the loss of packets containing the most essential data. Protection of the video data is performed at content level using the standardized JPSEC syntax, along with flexible encryption of quality layers or resolution levels. This selective encryption respects the JPEG2000 structure of the stream, not only ensuring end-to-end ciphered delivery, but also enabling dynamic content adaptation within the wireless network (quality of service, adaptation to the user's terminal). Some DRM modules from OPENSDRM platform will be added to manage all authenticated peers on the WLAN (from end-users to cameras), as well as to manage the rights to display conditionally the video data. This whole integrated architecture addresses several security problems such as data encryption, integrity and access control. Using several protection layers, the level of confidentiality can depend on both the content characteristics and the user rights, thus also addressing the critical issue of privacy.

The EU FP6 WCAM (Wireless Cameras and Audio-Visual Seamless Networking) project aims to study, develop and validate a wireless, seamless and secured end-to-end networked audio-visual system for video surveillance and multimedia distribution applications. This paper describes the video transmission aspects of the project, with contributions in the areas of H.264 video delivery over wireless LANs. The planned demonstrations under WCAM include transmission of H.264 coded material over 802.11b/g networks with TCP/IP and UDP/IP being employed as the transport and network layers over unicast and multicast links. UDP based unicast and multicast transmissions pose the problem of packet erasures while TCP based transmission is associated with long delays and the need for a large jitter buffer. This paper presents measurement data that have been collected at the trial site along with analysis of the data, including characterisation of the channel conditions as well as recommendations on the optimal operating parameters for each of the above transmission scenarios (e.g. jitter buffer sizes, packet error rates, etc.). Recommendations for error resilient coding algorithms and packetisation strategies are made in order to moderate the effect of the observed packet erasures on the quality of the transmitted video. Advanced error concealment methods for masking the effects of packet erasures at the receiver/decoder are also described.

In this paper, we present a smart video surveillance system based on standard technologies and wired or wireless IP networking. The camera is performing the following operations. First, a video analysis module detects suspicious events and identifies regions of interest in the scene. The video is then compressed using JPEG 2000. The resulting compressed stream is signed and scrambled for the purpose of data integrity verification and confidentiality. Finally, the stream is protected for robustness to transmission errors based. The system is designed in such a way to protect the privacy of people under surveillance. More specifically, during the analysis stage, people in the scene are detected by change detection or face detection techniques. Scrambling is then applied only to the corresponding regions. Furthermore, the amount of distortion can be controlled by restricting scrambling to some resolution levels or quality layers. In this way, people under surveillance cannot be recognized, but the remaining of the scene is clear. The encryption key is kept under control of judicial authorities, who can grant authorization to unlock the protection and view the whole scene. Per consequent, this system successfully addresses the loss of privacy issue.

We present a systematic framework for arriving at audio classes for detection of crimes in elevators. We use a time series analysis framework to analyze the low-level features extracted from the audio of an elevator surveillance content to perform an inlier/outlier based temporal segmentation. Since suspicious events in elevators are outliers in a background of usual events, such a segmentation help bring out such events without any a priori knowledge. Then, by performing an automatic clustering on the detected outliers, we identify consistent patterns for which we can train supervised detectors. We apply the proposed framework to a collection of elevator surveillance audio data to systematically acquire audio classes such as banging, footsteps, non-neutral speech and normal speech etc. Based on the observation that the banging audio class and non-neutral speech class are indicative of suspicious events in the elevator data set, we are able to detect all of the suspicious activities without any misses.

In this paper, we present an object detection and tracking algorithm for low-frame-rate applications. We extend the standard mean-shift technique such that it is not limited within a single kernel but uses multiple kernels centered around high motion areas obtained by change detection. We also improve the convergence properties of the mean-shift by integrating two additional likelihood terms. Our simulations prove the effectiveness of the proposed method.

Automatic video surveillance systems are used in many different fields of applications like intelligent vehicles, intelligent highways or in security tasks. An automatic and correct interpretation of video sequences is based on the detection, tracking and classification of various objects under highly diverse conditions. This requires high sophisticated algorithms as well as high computational performance to fulfil real-time constraints. Here, the Infineon Technologies AG develops a fully programmable, scalable multi-processor architecture optimized for video processing, which provides a processing performance similar to actual PCs but at much lower costs, lower power consumption and lower physical size. This architecture supports data-parallelism for low-level image processing, parallelism of tasks in the medium- and high-level as well as control oriented processing. A cycle accurate, virtual prototype of the architecture is available. A library of optimized image processing functions supports a comfortable application development and the reuse of existing application software. Beside the implementation of standard low-level operators new efficient approaches for motion estimation, object detection and tracking are developed and tested in applications for intelligent vehicles and intelligent highway scenarios. The integration of these application specific tasks into the image processing library results in a powerful embedded vision platform for video surveillance systems, which supports a comfortable application development.

Good pedestrian classifiers that analyze static images for presence of pedestrians are in existence. However, even a low false positive error rating is sufficient to flood a real system with false warnings. We address the problem of pedestrian motion (gait) modeling and recognition using sequences of images rather than static individual frames, thereby exploiting information in the dynamics. We use two different representations and corresponding distances for gait sequences. In the first a gait is represented as a manifold in a lower dimensional space corresponding to gait images. In the second a gait image sequence is represented as the output of a dynamical system whose underlying driving process is an action like walking or running. We examine distance functions corresponding to these representations. For dynamical systems we formulate distances derived based on parameters of the system taking into account both the structure of the output space and the dynamics within it. Given appearance based models we present results demonstrating the discriminative power of the proposed distances

The presented work describes an innovative architecture for multi-sensor distributed video surveillance applications. The aim of the system is to track moving objects in outdoor environments with a cooperative strategy exploiting two video cameras. The system also exhibits the capacity of focusing its attention on the faces of detected pedestrians collecting snapshot frames of face images, by segmenting and tracking them over time at different resolution. The system is designed to employ two video cameras in a cooperative client/server structure: the first camera monitors the entire area of interest and detects the moving objects using change detection techniques. The detected objects are tracked over time and their position is indicated on a map representing the monitored area. The objects’ coordinates are sent to the server sensor in order to point its zooming optics towards the moving object. The second camera tracks the objects at high resolution. As well as the client camera, this sensor is calibrated and the position of the object detected on the image plane reference system is translated in its coordinates referred to the same area map. In the map common reference system, data fusion techniques are applied to achieve a more precise and robust estimation of the objects’ track and to perform face detection and tracking. The work novelties and strength reside in the cooperative multi-sensor approach, in the high resolution long distance tracking and in the automatic collection of biometric data such as a person face clip for recognition purposes.

We present a software for complete metric calibration of synchronized multicamera setups. The minimum is three cameras, but there is no upper limit. The method is fully automatic and a freely moving bright spot is the only calibration object. No camera pre-calibration is required. The software computes complete set of intrinsic camera parameters including the parameters of non-linear distortion as well as external orientation of the cameras in one common coordinate system. The software is written in Matlab, runs in non-interactive mode and produces both textual and intuitive graphical output. It is very robust and may be run with the same set of parameters in different setups. The software is free. We suggest a trick on how to adapt a very ordinary laser pointer to be a useful calibration object. Projections of such calibration object are localized with sub-pixel accuracy. Average reprojection error around 1/5 pixel was reached in well synchronized multicamera setups even for wide-angle lenses with severe non-linear distortion. We show several virtual reality and telepresence applications demonstrating the usefulness of having multicamera setups being fully calibrated.

This paper presents a new method for remote and interactive browsing of long video surveillance sequences. The solution is based on interactive navigation in JPEG 2000 coded mega-images. We assume that the video 'key-frames' are available through automatic detection of scene changes or abnormal behaviors. These key-frames are concatenated in raster scanning order forming a very large 2D image, which is then compressed with JPEG 2000 to produce a scalable video summary of the sequence. We then exploit a mega image navigation platform, designed in full compliance with JPEG 2000 part 9 "JPIP", to search and visualize desirable content, based on client requests. The flexibility offered by JPEG 2000 allows highlighting key-frames corresponding to the required content within a low quality and low-resolution version of the whole summary. Such a fine grain scalability is a unique feature of our proposed JPEG 2000 video summaries expansion. This possibility to visualize key-frames of interests and playback the corresponding video shots within the context of the whole sequence enables the user to understand the temporal relations between semantically similar events. It is then particularly suited to analyzing complex incidents consisting of many successive events spread over a long period.

Hyperspectral images are acquired incrementally in a “push-broom” fashion by on-board sensors. Since these images are highly voluminous, buffering an entire image before compression requires a large buffer and causes latency. Incremental compression schemes work on small chunks of raw data as soon as they are acquired and help reduce buffer memory requirements. However, incremental processing leads to large variations in quality across the reconstructed image. The solution to this problem lies in using carefully designed rate control algorithms. We propose two such “leaky bucket” rate control algorithms that can be employed when incrementally compressing hyperspectral images using the JPEG2000 compression engine. They are the Multi-Layer Sliding Window Rate Controller (M-SWRC) and the Multi-Layer Extended Sliding Window Rate Controller (M-EWRC). Both schemes perform rate control using the fine granularity afforded by JPEG2000 bitstreams. The proposed algorithms have low memory requirements since they buffer compressed bitstreams rather than raw image data. Our schemes enable SNR scalability through the use of quality layers in the codestream and produce JPEG2000 compliant multi-layer codestreams at a fraction of the memory used by conventional schemes. Experiments show that the proposed schemes provide significant reduction in quality variation with no loss in mean overall PSNR performance.

We propose an integrated, wavelet based, two-stage coding scheme for lossy, near-lossless and lossless compression of medical volumetric data. The method presented determines the bit-rate while encoding for the lossy layer and without any iteration. It is in the spirit of "lossy plus residual coding" and consists of a wavelet-based lossy layer followed by an arithmetic coding of the quantized residual to guarantee a given pixel-wise maximum error bound. We focus on the selection of the optimum bit rate for the lossy coder to achieve the minimum total (lossy plus residual) bit rate in the near-lossless and the lossless cases. We propose a simple and practical method to estimate online the optimal bit rate and provide a theoretical justification for it. Experimental results show that the proposed scheme provides improved, embedded lossy, and lossless performance competitive with the best results published so far in the literature, with an added feature of near-lossless coding.

The curved wavelet transform performs 1-D filtering along curves and exploits orientation features of edges and lines in an image to improve the compactness of the wavelet transform. This paper investigates the issue of efficient data organization and representation of the curved wavelet coefficients. We present an adaptive zero-tree structure that exploits the cross-subband similarity of the curved wavelet transform. The child positions in the adaptive zero-tree structure are not restricted to a square of 2x2 pixels and they vary with the curves along which the WT is performed. Five child patterns have been determined according to different combination of curve orientations. A new image coder, using the curved wavelet transform, is then developed based on this adaptive zero-tree structure and the set partitioning technique. Experimental results using synthetic and natural images show the effectiveness of the proposed adaptive zero-tree structure for encoding of the curved wavelet coefficients. The coding gain of the proposed coder can be as higher as 1.2dB in terms of PSNR compared to the SPIHT coder.

We propose an inter mode decision scheme for P slices in the H264 video coding standard. Our scheme initially exploits neighbourhood information jointly with a set of skip mode conditions for enhanced skip mode decision. It subsequently performs inter mode decision for the remaining macroblocks by using a gentle set of smoothness constraints. For RD performance very close to the standard we achieve 35-58% reduction in run times and 33-55% reduction in CPU cycles for both the rate controlled and the non rate controlled versions of H264. Compared to other work that has been proposed as input to the standard, gains of 9-23% in run times and 7-22% in CPU cycles are also reported.

The amount of data that has to be stored and transmitted for stereo-view video applications can be double of conventional mono-view video applications if a mono-view video coding method is applied directly. Although not designed for stereo-view video coding, the H.264 coding tools can be arranged to take advantage of the correlations between the pair of views of a stereo-view video, and provide very reliable and efficient compression performance as well as stereo/mono-view scalability. This paper introduces methods for coding stereo-view video sequences efficiently using the H.264 coding tools, such as the interlace-coding tools and the sub-sequence coding concept. With the help of the stereo video Supplemental Enhancement Information (SEI) message defined in H.264 Fidelity Range Extensions (FRExt), a decoder can easily synchronize the views, and a streaming server or a decoder can easily detect the scalability of a coded stereo video bitstream. Coding performances are shown in the paper for Main and Baseline profiles using both scalable and non-scalable coding options. The scalable coding options that can be signaled by the H.264 SEI messages could have coding performance similar to or better than that of proprietary system configurations for non-scalable coding.

A multiple global affine motion model is proposed for low bit rate video compression. Block-wise motion segmentation is first performed with the number of motion objects L predefined. The affine motion models for multiple MOs are estimated and coded in the frame header. The scaling parameters a1, a2, a4 and a5 are coded with a 4-dimensional vector-quantizer (VQ), whose 16 most recently used code words are maintained on line and searched for VQ match, and the 300-word long main code book stored offline. The translational parameters a3 and a6 are coded predicatively as a classical motion vector. L new macro-block modes are added to the standard’s list of 7 intra and inter modes. No segmentation information is transmitted, for the mode already indicates that if one of the affine modes is selected by Lagrange rate-distortion optimization. A metric S is defined to measure locality of the motion and will disable use of affine models when a threshold is surpassed. Simulation shows that more than 50% of the MB’s choose one of the affine modes. When 100kbps or lower bandwidths are available, the proposed codec not only saves 1~18% bit rate, but also enhances error-resilience in multiple slice frames and reduces blocking artifacts notably.

The objective of this paper is to develop a robust error-resilient algorithm, called the Synchronous Backward Error Tracking (SBET), to completely terminate the error propagation effects in the error-prone environment for H.264 video coding. The motivation is that if the state of the decoder is available to the encoder, i.e., the state of the encoder can synchronize to the state of the decoder, the effect of error propagation can be entirely terminated because all predictions are based on the same references. Therefore, we assume that a feedback channel is available and the encoder can be aware of the decoder's error concealment by any external means. The pixel-based Precise Backward Error Tracking (PBET) is modified and utilized to track the error locations and reconstruct the state of the decoder in the encoder. The proposed method only involves memory access, simple addition and multiplication operations for the error-contaminated pixels to achieve encoder-decoder synchronization. By observing simulation results, the rate-distortion performance of the proposed algorithm is always better than that of the conventional algorithms. Specifically, SBET outperforms PBET up to 1.21 dB under 3% slice error rate for the QCIF format Foreman sequence. In addition, instead of forced INTRA refreshing, the phenomenon of burst bit rate can be avoided.

Recent advances in digital video coding tools have led to the introduction of the H.264 video coding standard, which promises increased visual quality and reduced bandwidth. In this paper, we analyze and compare MPEG-2 and H.264 video compression methods. Although H.264 is similar to MPEG-2 in that both are hybrid coders that use motion compensation, H.264 also includes advanced features such as improved entropy encoding, in-loop filtering of reference frames, flexible macroblock sizing, and multiple reference frame capability. Many experiments were performed to illustrate the coding gains of each feature in H.264 as well as to compare H.264 to MPEG-2. Quality was measured using two different objective video metrics: peak signal-to-noise ratio and the Structural Similarity Index. A variety of natural video test sequences were used with varying resolutions and data rates. TM5 and JM reference software were used to create MPEG-2 and H.264 compressed bitstreams. Results for all experiments show significant coding gain with H.264 versus MPEG-2 when compressing natural video sequences, especially at low data rates.

Recently it has been theorized that some painters as early as 1420 used concave mirrors (and, later, converging lenses) to project real inverted images onto their supports which they then traced and painted over. We consider a specific painting adduced as evidence for this bold theory, the Lorainnese Baroque master Georges de la Tour’s Christ in the carpenter’s studio (1645). We perform analyses of the reflections and shadows -- “cast” shadows and “form” shadows -- to infer the source(s) of illumination. We find compelling evidence that this source is the candle flame depicted within the painting and held by Christ. We find it implausible that the source is direct solar illumination, which has the intensity demanded by the projection theory, or artificial illumination as hypothesized by theory proponents. Similar analyses of several other paintings by de la Tour uniformly support the conclusion that the illumination is small and artificial within the space of the tableau (i.e., a candle), not extremely powerful illumination from outside the tableau. We created a very simple computer graphics model to test and illustrate part of our conclusions. Our research is the first application of technical shadow analysis to the question whether artists as early as the 15th century used optical projections when painting. Careful reading of the historical record of de la Tour’s working methods supports our technical results and extend the growing image analytic methods and historical sources rebutting the theory.

Handwriting originates from a particular copybook style such as Palmer or Zaner-Bloser that one learns in childhood. Since questioned document examination plays an important investigative and forensic role in many types of crime, it is important to develop a system that helps objectively identify a questioned document’s handwriting style. Here, we propose a computer vision system that can assist a document examiner in the identification of a writer’s handwriting style and therefore the origin or nationality of an unknown writer of a questioned document. We collected 33 Roman alphabet copybook styles from 18 countries. Each character in a questioned document is segmented and matched against all of the 33 handwriting copybook styles. The more characters present in the questioned document, the higher the accuracy observed.

Cartridge cases are important forensic specimen for the identification of weapons. The illumination conditions in the area of the firing pin marks and the breech face marks are very different and have to be treated separately to achieve an appropriate image quality for a visual inspection. Furthermore, not only the comparison but also the detection of the different and independent forensic marks should be automated. Both problems lead to the task of segmenting the different parts of the cartridge case bottom. In this paper, two automated approaches for the segmentation of cartridge cases are investigated and compared. The aim of the segmentation is the detection of the cartridge case border, the primer, the firing pin mark and additionally the letters around the primer. The first approach uses images obtained under systematically varied illumination conditions. After a preprocessing step a circle detection is applied to find the circular structures. The analysis of illumination series combined with a the connected components labeling method detect the letters. In a second approach, the depth-from-focus method is used to obtain 2½ D-data. This data is segmented applying a plane estimation technique. This results directly in the detection of the letters. Afterwards a circle detection algorithm identifies the parameters of the circular structures. With the introduced methods it is possible to optimize the illumination in order to realize a higher contrast of both the striation marks on the cartridge case surface and of the indentation of the firing pin independently. The improved image quality helps the examiner in identifying weapons and will help to improve the automated comparison strategies.

Until recently, the forensic or investigative reconstruction of shredded documents has always been dismissed as an important but unsolvable problem. Manual reassembly of the physical remnants can always be considered, but for large amounts of shreds this problem can quickly become an intangible task that requires vast amounts of time and/or personnel. In this paper we propose and discuss several image processing techniques that can be used to enable the reconstruction of strip-shredded documents stored within a database of digital images. The technical content of this paper mainly revolves around the use of feature based matching and grouping methods for classifying the initial database of shreds, and the subsequent procedure for computing more accurate pairing results for the obtained classes of shreds. Additionally, we discuss the actual reassembly of the different shreds on top of a common image canvas. We illustrate our algorithms with example matching and reconstruction results obtained for a real shred database containing various types of shredded document pages. Finally, we briefly discuss the impact of our findings on secure document management strategies and the possibilities for applying the proposed techniques within the context of forensic investigation.

In this paper, we demonstrate that it is possible to use the sensor’s pattern noise for digital camera identification from images. The pattern noise is extracted from the images using a wavelet-based denoising filter. For each camera under investigation, we first determine its reference noise, which serves as a unique identification fingerprint. This could be done using the process of flat-fielding, if we have the camera in possession, or by averaging the noise obtained from multiple images, which is the option taken in this paper. To identify the camera from a given image, we consider the reference pattern noise as a high-frequency spread spectrum watermark, whose presence in the image is established using a correlation detector. Using this approach, we were able to identify the correct camera out of 9 cameras without a single misclassification for several hundred images. Furthermore, it is possible to perform reliable identification even from images that underwent subsequent JPEG compression and/or resizing. These claims are supported by experiments on 9 different cameras including two cameras of exactly same model (Olympus C765).

In forensic applications of photogrammetric techniques any measurable information is a precious commodity. In this paper, we describe the development of a new geometrical approach to photogrammetric measurements. While classical photogrammetry requires knowledge of point measurements, in this approach we exploit other geometrical constraints that are often available in photo and video forensic evidence. In Part I paper, the first in the series, we introduce line constraints and demonstrate algorithms that combine point and line measurements.

A highly desirable feature in storage video applications is uniform video quality. Variable bit rate (VBR) coding has the potential to produce nearly constant quality throughout an entire movie. This can be defined as a bit allocation problem with a long-term constraint on distortion variation. We consider optimal bit allocation with multiple constraints including disk capacity and distortion bounds on the individual frames. We find the theoretical optimality conditions and propose a practical iterative solution based on Lagrangian methods. While minimizing both average distortion and distortion variation cannot be achieved simultaneously for a given bit budget, the proposed algorithms are able to efficiently trade-off between the two goals. The computational complexity of the exact rate-distortion (R-D) functions for real movies is addressed by a statistical R-D model proposed in this work. The model is formed by a rate-quantization (R-Q) function and the corresponding distortion-quantization (D-Q) function. A novel two-pass MPEG-2 VBR encoder based on the proposed algorithms is developed for coding with long-term nearly constant quality. Experimental results are promising and the encoder effectively achieves the fit-to-disc function and at the same time controls objective quality variation. By incorporating basic subjective coding techniques into the encoder significant visual quality improvement is observed during the preliminary subjective tests.

This paper proposes a game theoretical rate control technique for video compression. Using a cooperative gaming approach, which has been utilized in several branches of natural and social sciences because of its enormous potential for solving constrained optimization problems, we propose a dual-level scheme to optimize the perceptual quality while guaranteeing “fairness” in bit allocation among macroblocks. At the frame level, the algorithm allocates target bits to frames based on their coding complexity. At the macroblock level, the algorithm distributes bits to macroblocks by defining a bargaining game. Macroblocks play cooperatively to compete for shares of resources (bits) to optimize their quantization scales while considering the Human Visual System’s perceptual property. Since the whole frame is an entity perceived by viewers, macroblocks compete cooperatively under a global objective of achieving the best quality with the given bit constraint. The major advantage of the proposed approach is that the cooperative game leads to an optimal and fair bit allocation strategy based on the Nash Bargaining Solution. Another advantage is that it allows multi-objective optimization with multiple decision makers (macroblocks). The simulation results testify the algorithm’s ability to achieve accurate bit rate with good perceptual quality, and to maintain a stable buffer level.

In previous work, a performance bound for multi-hypothesis motion-compensated prediction (MCP) has been derived based on a video signal model with independent Gaussian displacement errors. A simplified form of the result is derived in this work. A performance bound for optimal motion-compensated temporal filtering (MCTF) has also been proposed based on a signal model with correlated Gaussian displacement errors. In this previous work, the optimal MCTF (KLT) was found to perform better than one-hypothesis MCP but not better than infinite-hypothesis MCP. In this work, we derive the performance of multi-hypothesis MCP again based on the signal model with correlated Gaussian displacement errors. Now with the same signal model, we find that optimal MCTF has the same performance as that of infinite-hypothesis MCP.

Leaky prediction layered video coding (LPLC) incorporates a scaled version of the enhancement layer in the motion compensated prediction (MCP) loop, by using a leaky factor between 0 and 1, to balance between coding efficiency and error resilience performance. In this paper, we address the theoretic analysis of LPLC using two different approaches: the one using rate distortion theory and the one using quantization noise modeling. In both approaches, an alternative block diagram of LPLC is first developed, which significantly simplifies the theoretic analysis. We consider two scenarios of LPLC, with and without prediction drift in the enhancement layer, and obtain two sets of rate distortion functions in closed form for both scenarios. We evaluate both closed form expressions, which are shown to conform with the operational results.

Considering inter-picture dependencies when selecting transform coefficient levels in hybrid video coding can be done via formulating the decoding process as a linear signal model and solving a quadratic program. The basic method assumes motion estimation and quantization parameters as being given and then selects the transform coefficient levels. However, when motion vectors are determined in advance, motion estimation must be conducted using uncoded reference pictures which is known to deliver inferior results compared to motion estimation on decoded reference pictures. In this work, we expand the basic method to incorporate the case where the motion estimation is considering decoded reference pictures. We propose an approach that iterates between transform coefficient selection and motion estimation. We find that a simple two-pass iteration works reasonably well. Our simulation results using an H.264/AVC-conforming encoder show coding gains up to 1 dB in comparison to the quantization method specified in the test model of H.264/AVC.

In the scalable video coder MC-EZBC, the scalability for motion vectors was not provided, and this greatly impacts its performance when scaling down to very low bit rates and resolutions. Here we enhance MC-EZBC with scalable motion vector coding using the Context based Adaptive Binary Arithmetic Coder (CABAC). Both a layered structure for motion vector coding and an alphabet general partition (AGP) of the motion vector symbols are employed for SNR and resolution scalability of the motion vector bitstream. With these two new features and the careful arrangement of the motion vector bitstream output from the existing MC-EZBC, we obtain temporal, SNR, and resolution scalability for motion vectors. This significantly improves both visual and objective performance at low bit rates and resolutions with only a slight PSNR loss (about 0.05 dB), but no detectable visual loss, at high bit rates.

Motion-compensated wavelet video coders have been shown to exhibit good coding efficiency over a large range of bit-rates, in addition to providing spatial and temporal scalability. While the rate-distortion performance provided by these coders is well understood, their complexity scalability behavior is not well studied. In this paper, we first analyze the complexity of such wavelet video coders, and determine what the critical components are and how they vary depending on the transmission bit-rates. Subsequently, we construct generic complexity models for the critical components of the scalable wavelet video decoders; such that optimal rate, distortion and complexity bitstreams can be created that fulfill not only various network constraints, but also resource constraints such as memory and power. The generic complexity metrics are independent of the hardware architecture and implementation details of the decoders and capture both the time varying video content characteristics and the corresponding encoding parameters. The generic complexity measures can be converted into real platform specific complexity measures like execution time with limited overhead at runtime. Preliminary results show that the proposed models can predict the complexity of the various components of wavelet video decoders with high accuracy.

Discrete wavelet transforms (DWTs) applied temporally under motion compensation (MC) have recently become a very powerful tool in video compression, especially when implemented through lifting. A recent theoretical analysis has established conditions for perfect reconstruction in the case of transversal MC-DWT, and also for the equivalence of lifted and transversal implementations of MC-DWT. For Haar MC-DWT these conditions state that motion must be invertible, while for higher-order transforms they state that motion composition must be a well-defined operator. Since many popular motion models do not obey these properties, thus inducing errors (prior to compression), it is important to understand what is the impact of motion non-invertibility or quasi-invertibility on the performance of video compression. In this paper, we present new experimental results of a study aiming at a quantitative evaluation of such impact in case of block-based motion. We propose a new metric to measure the degree with which two motion fields are not inverses of each other. Using this metric we investigate several motion inversion schemes, from simple temporal sample-and-hold, through spatial nearest-neighbor, to advanced spline-based inversion, and we compare compression performance of each method to that of independently-estimated forward and backward motion fields. We observe that compression performance monotonically improves with the reduction of the proposed motion inversion error, up to 1-1.5dB for the advanced spline-based inversion. We also generalize the problem of "unconnected" pixels by extending it to both update and prediction steps, as opposed to the update step only used in conventional methods. Initial tests show favorable results compared to previously reported techniques.

Motion compensated temporal lifting is a highly effective means for exploiting motion in wavelet-based video compression algorithms. One way to achieve both spatial and temporal scalability attributes is to apply a conventional spatial DWT to an initial set of temporal subbands. This "t+2D" paradigm may be reversed, performing the spatial transform first and temporally transforming its spatial subbands. In this paper, we show how the two paradigms can be bridged by a family of "vector" motion compensation operators. Different members of this family have different implications for compression efficiency and for artifacts which can appear at reduced resolutions. We show how the vector motion compensation operators can be adaptively selected so as to achieve high compression efficiency, while simultaneously suppressing artifacts which might otherwise occur during scaling. The vector motion compensation paradigm provides an efficient framework for in-band block-based motion modeling, which suppresses the appearance of blocking artifacts. The proposed adaptive motion compensation operators have an added advantage in automatically selecting between different resolution-dependent motion models, so as to maximize energy compaction while avoiding the appearance of artifacts at reduced resolutions. Resolution-dependent motion models extend the useful range of bit-rate scalability over many orders of magnitude.

Scalable video coding is a technique which allows a compressed video stream to be decoded in several different ways. This ability allows a user to adaptively recover a specific version of a video depending on its own requirements. Video sequences have temporal, spatial and quality scalabilities. In this work we introduce a novel fully scalable video codec. It is based on a motion-compensated temporal filtering (MCTF) of the video sequences and it uses some of the basic elements of JPEG 2000. This paper describes several specific proposals for video on demand and video-conferencing applications over non-reliable packet-switching data networks.

In this paper, we investigate an important problem of efficiently utilizing the available resources for video transmission over wireless channels while maintaining a good decoded video quality and resilience to channel impairments. Our system consists of the video codec based on 3-D set partitioning in hierarchical trees (3-D SPIHT) algorithm and employs two different schemes using low-density parity check (LDPC) codes for channel error protection. The first method uses the serial concatenation of the constant-rate LDPC code and rate-compatible punctured convolutional (RCPC) codes. Cyclic redundancy check (CRC) is used to detect transmission errors. In the other scheme, we use the product code structure consisting of a constant rate LDPC/CRC code across the rows of the `blocks' of source data and an erasure-correction systematic Reed-Solomon (RS) code as the column code. In both the schemes introduced here, we use fixed-length source packets protected with unequal forward error correction coding ensuring a strictly decreasing protection across the bitstream. A Rayleigh flat-fading channel with additive white Gaussian noise (AWGN) is modeled for the transmission. The rate-distortion optimization algorithm is developed and carried out for the selection of source coding and channel coding rates using Lagrangian optimization. The experimental results demonstrate the effectiveness of this system under different wireless channel conditions and both the proposed methods (LDPC+RCPC/CRC and RS+LDPC/CRC) outperform the more conventional schemes such as those employing RCPC/CRC.

In this paper, we propose a layered complexity-aware encryption scheme to partially encrypt scalable video bitstreams to achieve reasonable security that the specific application entails and the involved computational complexity allows. The proposed scheme naturally combines wavelet-based scalable video coding technique and the concept of selective encryption. We also study the relationship between rate, distortion, and the involved encryption complexity (R-D-EC). Here, distortion is used to measure the level of security of the encrypted video bitstream, and the percentage of encrypted bitstream is used to denote encryption complexity. Our simulation results indicate that selective encryption using a prioritized bitstream structure provided by scalable video coding can achieve almost bitrate-independent encryption complexity control.

We present a novel detection based particle filtering framework for real-time multi-object tracking (MOT). It integrates object detection and motion information with particle filter detecting and tracking the multiple objects dynamically and simultaneously. To demonstrate the approach, we concentrate on the complex multi-head tracking while the framework is general for any kind of objects. Three novel contributions are made: 1) Distinct with the conventional particle filter which generates particles from the prior density, we propose a novel importance function based on up to date detection and motion observation which is much closer to the desired posterior. 2) By integrating detection, the tracker can do the initialization automatically, handle new object appearance and hard occlusion for MOT. By using motion estimation, it can track fast motion activities. 3) Hybrid observations including color and detection information are used to calculate the likelihood which makes the approach more stable. The proposed method is superior to the available tracking methods for multi-head tracking and can handle not only the changes of scale, lighting, zooming, and orientation, but also fast motion, appearance, and hard occlusion.

In this paper, we examine sensor specific distributions of local image operators (edge and line detectors), which describe the appearance of people in video sequences. The distributions are used to describe a probabilistic articulated motion model to track the gestures of a person in terms of arms and body movement. The distributions are based on work of Sidenbladh where general distributions are examined, collected over images found on the internet. In our work, we focus on the statistics of one sensor, in our case a standard webcam, and examine the influence of image noise and scale. We show that although the general shape of the distributions published by Sidenbladh are found, important anomalies occur which are due to image noise and reduced resolution. Taking into account the effects of noise and blurring on the scale space response of edge and line detectors improves the overall performance of the model. The original distributions introduced a bias towards small sharp boundaries over large blurred boundaries. In the case of arms and legs which often appear blurred in the image, this bias is unwanted. Incorporating our modifications in the distributions removes the bias and makes the tracking more robust.

In this paper we investigate object tracking in video sequences by using the potential of particle filtering to process features from video frames. A particle filter (PF) and a Gaussian sum particle filter (GSPF) are developed based upon multiple information cues, namely colour and texture, which are described with highly nonlinear models. The algorithms rely on likelihood factorisation as a product of the likelihoods of the cues. We demonstrate the advantages of tracking with multiple independent complementary cues compared to tracking with individual cues. The advantages are increased robustness and improved accuracy. The performance of the two filters is investigated and validated over both synthetic and natural video sequences.

Detection and tracking of moving objects is important in the analysis of video data. One approach is to maintain a background model of the scene and subtract it from each frame to detect the moving objects which can then be tracked using Kalman or particle filters. In this paper, we consider simple techniques based on salient points to identify moving objects which are tracked using motion correspondence. We focus on video with a large field of view, such as a traffic intersection with several buildings nearby. Such scenes can contain several salient points, not all of which move between frames. Using public domain video and two types of salient points, we consider how to make these techniques computationally efficient for detection and tracking. Our early results indicate that salient regions obtained using the Lowe keypoints algorithm and the Scale-Saliency algorithm can be used successfully to track vehicles in moderate resolution video.

Vision implies objects, so a vision system front-end needs to produce a feature-based description of image objects. The functional boundaries and specifications for the front-end are derived from analyzing: 1) What feature information can be extracted from context-free video? 2) What feature information will reduce the probability distribution model complexity for statistical object recognition and tracking? 3) How should the feature information be encoded? Segmentation is a flexible tool for extracting features. Recently proposed segmentation algorithms can be adapted to high-performance, low-cost hardware. Inexpensive segmentation will have a multiplying affect on vision system performance/complexity. Two examples are techniques for extending hardware functions into both parallel pixel processes and object tracking.

This paper proposes a real-time 3D user interface using multiple possibly uncalibrated cameras. It tracks the user’s pointer in real-time and solves point correspondences across all the cameras. These correspondences form spatio-temporal “traces” that serve as a medium for sketching in a true 3-D space. Alternatively, they may be interpreted as gestures or control information to elicit some particular action(s). Through view synthesis techniques, the system enables the user to change and seemingly manipulate the viewpoint of the virtual scene even in the absence of camera calibration. It also serves as a flexible, intuitive, and portable mixed-reality display system. The proposed system has numerous implications in interaction and design, especially as a general interface for creating and manipulating various forms of 3-D media.

This paper presents a region-based segmentation method extracting automatically moving objects from video sequences. Non-moving objects can also be segmented by using a graphical user interface. The segmentation scheme is inspired from existing methods based on the watershed algorithm. The over-segmented regions resulting from the watershed are first organized in a binary partition tree according to a similarity criterion. This tree aims to determine the fusion order. Every region is then fused with the most similar neighbour according to a spatio-temporal criterion regarding the region colors and the temporal colors continuity. The fusion can be stopped either by fixing a priori the final number of regions, or by markers given through the graphical user interface. Markers are also used to assign a class to non-moving objects. Classification of moving objects is automatically obtained by computing the Change Detection Mask. To get a better accuracy on the contours of the segmented objects, we perform a simple post-processing filter to refine the edges between different video object planes.

The application of the mean shift algorithm to color image segmentation has been proposed in 1997 by Comaniciu and Meer. We apply the mean shift color segmentation to image sequences, as the first step of a moving object segmentation algorithm. Previous work has shown that it is well suited for this task, because it provides better temporal stability of the segmentation result than other approaches. The drawback is higher computational cost. For speed up of processing on image sequences we exploit the fact that subsequent frames are similar and use the cluster centers of previous frames as initial estimates, which also enhances spatial segmentation continuity. In contrast to other implementations we use the originally proposed CIE LUV color space to ensure high quality segmentation results. We show that moderate quantization of the input data before conversion to CIE LUV has little influence on the segmentation quality but results in significant speed up. We also propose changes in the post-processing step to increase the temporal stability of border pixels. We perform objective evaluation of the segmentation results to compare the original algorithm with our modified version. We show that our optimized algorithm reduces processing time and increases the temporal stability of the segmentation.

Global-motion estimation algorithms as they are employed in the MPEG-4 or H.264 video coding standards describe motion with a set of abstract parameters. These parameters model the camera motion, but they cannot be directly related to a physical meaning like rotation angles or the focal-length. We present a two step algorithm to factorize these abstract parameters into physically meaningful operations. The first step applies a fast linear estimation method. In an optional second step, these parameters can be refined with a non-linear optimization algorithm. The attractivity of our algorithm is its combination with the multi-sprite concept that allows for unrestricted rotational camera motion, including varying of focal-lengths. We present results for several sequences, including the well-known stefan sequence, which can only be processed with the multi-sprite approach.

We present an algorithm to quickly analyse and compress facial images using a 2-dimensional morphable model. It runs in real-time on reasonable resources, and offers considerable opportunities for parallelization. A morphable model associates a "shape vector" and a "texture vector" with each image of a sample set. The model is used to analyze a novel image by estimating the model parameters via an optimization procedure. The novel image is compressed by representing it by the set of best match parameters. For real-time performance, we separate the novel image into shape and texture components by computing correspondences between the novel image and a reference image, and match each component separately using eigenspace projection. This approach can be easily parallelized. We improve the speed of algorithm by exploiting the fact that facial correspondence fields are smooth. By computing correspondences only at a number of "feature points" and using interpolation to approximate the dense fields, we drastically reduce the dimensionality of the vectors in the eigenspace, resulting in much smaller compression times. As an added benefit, this system reduces spurious correspondences, since weak features that may confuse the correspondence algorithm are not tracked.

This paper presents a method for view morphing and interpolation based on triangulation. View morphing here is viewed here as a basic tool for view interpolation. The feature points in each source image are first detected. Based on these feature points, each source image is segmented into a set of triangular regions and local affine transformations are then used for texture mapping of each triangle from the source image to the destination image. This is called triangulation-based texture mapping. However, one of the significant problems associated with this approach is texture discontinuity between adjacent triangles. In order to solve this problem, the triangular patches that might cause these adjacent discontinuities are first detected and the optimal affine transformations for these triangles are then applied. In the subsequent view interpolation step, all source images are transferred to the novel view through view morphing, and the final novel view is the combination of all these candidate novel views. The major improvement over the traditional approach is a feedback-based method proposed to determine the weights for the texture combination from different views. Simulation results show that our method can reduce the discontinuities in triangle-based view morphing and significantly improve the quality of the interpolated views.

Techniques for generating multi-resolution repositories of reusable XML subschemas are investigated in this research. There are three major components to achieve this goal: weighting XML elements, decomposing XML schemas, and clustering the decomposed subschemas. Two approaches are proposed to calculate the weights of XML elements. One is based on the analysis of links and their attributes while the other is based on the information accumulated from all reachable nodes. Elements with higher weights are selected as roots of reusable subschemas, which are further clustered into K groups according to the information amount carried to obtain multi-resolution repositories.

In this paper, an adaptive joint source and channel decoding method is designed to accelerate the convergence of the iterative log-dimain sum-product decoding procedure of LDPC codes as well as to improve the reconstructed image quality. Error resilience modes are used in the JPEG2000 source codec, which makes it possible to provide useful source decoded information to the channel decoder. After each iteration, a tentative decoding is made and the channel decoded bits are then sent to the JPEG2000 decoder. Due to the error resilience modes, some bits are known to be either correct or in error. The positions of these bits are then fed back to the channel decoder. The log-likelihood ratios (LLR) of these bits are then modified by a weighting factor for the next iteration. By observing the statistics of the decoding procedure, the weighting factor is designed as a function of the channel condition. That is, for lower channel SNR, a larger factor is assigned, and vice versa. Results show that the proposed joint decoding methods can greatly reduce the number of iterations, and thereby reduce the decoding delay considerably. At the same time, this method always outperforms the non-source controlled decoding method up to 5dB in terms of PSNR for various reconstructed images.

Broadband multimedia communications will be a key element in the 3rd generation of wireless services. A primary challenge is the support of interactive services requiring synchronous playout of multimedia content with a maximum acceptable delay in a multi-user scenario. Subjective tests show that the subjects prefer a lower quality reproduction than a “hopping” video: the pause needed to re-buffer severely affects the overall quality. To maximize the number of active users that can be served with a predefined Quality of Service (QoS) level by a packet-oriented radio access, we propose the use of a multimedia proxy. This component may transcode the original video stream to match a predefined quality level based on the quality of each user’s channel.

We present a general rate-distortion based scheduling framework that can accommodate cases where multiple encoded versions for the same video are available for transmission. Previous work on video scheduling is mostly focused on those encoding techniques, such as layered coding, which generate only one set of dependent packets. However, it is sometimes preferred to have a codec that produces redundant video data, where multiple different decoding paths are possible. Examples of these scenarios are multiple description layered coding and multiple independently encoded video streams. A new source model called Directed Acyclic HyperGraph (DAHG) is introduced to describe the relationship between different video data units with multiple decoding paths. Based on this model, we propose two low-complexity scheduling algorithms: the greedy algorithm and the M-T algorithm. Experiments are made to compare the performance of these algorithms. It is shown that, in the case of multiple decoding paths, the M-T algorithm outperforms the greedy algorithm by taking into account some of the transmission possibilities available in the near future before making a decision.

In this paper, we proposed a multiple description distributed image coding system for mobile wireless transmission. The innovations of proposed system are two folds: First, when MDC is applied to wavelet subband based image coding; it is possible to introduce correlation between the descriptions in each subband. At the encoder, the correlation information is encoded by systematic Reed Solomon (RS) encoder. Only the parity check bits are sent to channel. At the receiver, when part of descriptions are lost, however, their correlation information are available, the Wyner Ziv decoder can still recover the description by using the correlation information and the partly received description as noisy version and the side information. Secondly, in each description, we use multiple bitstream image coding to achieve error robust transmission. In conventional entropy subband coding, the first bit error may cause the decoder to discard all subsequent bits. A multiple bitstream image encoding is developed based on the decomposition of images in the wavelet domain. We show that such decomposition is able to reduce error propagation in transmission, thus to achieve scalability on PSNR performance over the changes of BER, Experimental result shows that the PSNR could be improved with the same coding rate.

To achieve consumer-level quality, media systems must process continuous streams of audio and video data while maintaining exacting tolerances on sampling and frame rate, jitter, and synchronization. While it is relatively straightforward to design fixed-function hardware implementations to satisfy worst-case conditions, there is a growing trend to utilize programmable multi-tasking solutions for media applications. The flexibility of these systems enables support for multiple current and future media formats, which can reduce design costs and time-to-market. This paper seeks to provide practical engineering solutions to achieve robust media processing on such systems, with specific attention given to power-constrained environments. The techniques covered in this article utilize the fundamental concepts of software optimization, software/hardware partitioning, stream buffering, hierarchical prioritization, and system resource and power management. A novel enhancement to dynamically adjust processor voltage and frequency based on buffer fullness to reduce system power consumption is examined in detail. The application of