Library of Congress Cataloging-in-Publication Data

Andrews, Larry C.

Field guide to probability, random processes, and random data analysis / Larry C. Andrews, Ronald L. Phillips.

p. cm. – (Field guide series)

Includes bibliographical references and index.

ISBN 978-0-8194-8701-8

1. Mathematical analysis. 2. Probabilities. 3. Random data (Statistics) I. Phillips, Ronald L. II. Title.

QA300.A5583 2012

519.2—dc23

2011051386

Published by

SPIE

P.O. Box 10

Bellingham, Washington 98227-0010 USA

Phone: +1.360.676.3290

Fax: +1.360.647.1445

Email: books@spie.org

Web: http://spie.org

The content of this book reflects the work and thoughts of the author. Every effort has been made to publish reliable and accurate information herein, but the publisher is not responsible for the validity of the information or for any outcomes resulting from reliance thereon.

Printed in the United States of America.

First printing

## Introduction to the Series

Welcome to the * SPIE Field Guides*—a series of publications written directly for the practicing engineer or scientist. Many textbooks and professional reference books cover optical principles and techniques in depth. The aim of the

*is to distill this information, providing readers with a handy desk or briefcase reference that provides basic, essential information about optical principles, techniques, or phenomena, including definitions and descriptions, key equations, illustrations, application examples, design considerations, and additional resources. A significant effort will be made to provide a consistent notation and style between volumes in the series.*

**SPIE Field Guides**Each * SPIE Field Guide* addresses a major field of optical science and technology. The concept of these Field Guides is a format-intensive presentation based on figures and equations supplemented by concise explanations. In most cases, this modular approach places a single topic on a page, and provides full coverage of that topic on that page. Highlights, insights, and rules of thumb are displayed in sidebars to the main text. The appendices at the end of each Field Guide provide additional information such as related material outside the main scope of the volume, key mathematical relationships, and alternative methods. While complete in their coverage, the concise presentation may not be appropriate for those new to the field.

The * SPIE Field Guides* are intended to be living documents. The modular page-based presentation format allows them to be easily updated and expanded. We are interested in your suggestions for new Field Guide topics as well as what material should be added to an individual volume to make these Field Guides more useful to you. Please contact us at

**fieldguides@SPIE.org.**

John E. Greivenkamp, **Series Editor**

College of Optical Sciences

The University of Arizona

## The Field Guide Series

Keep information at your fingertips with all of the titles in the Field Guide Series:

**Field Guide to**

* Adaptive Optics*, Robert Tyson & Benjamin Frazier

* Atmospheric Optics*, Larry Andrews

* Binoculars and Scopes*, Paul Yoder, Jr. & Daniel Vukobratovich

* Diffractive Optics*, Yakov G. Soskind

* Geometrical Optics*, John Greivenkamp

* Illumination*, Angelo Arecchi, Tahar Messadi, & John Koshel

* Infrared Systems, Detectors, and FPAs*,

*, Arnold Daniels*

**Second Edition*** Interferometric Optical Testing*, Eric Goodwin & Jim Wyant

* Laser Pulse Generation*, Rüdiger Paschotta

* Lasers*, Rüdiger Paschotta

* Microscopy*, Tomasz Tkaczyk

* Optical Fabrication*, Ray Williamson

* Optical Fiber Technology*, Rüdiger Paschotta

* Optical Lithography*, Chris Mack

* Optical Thin Films*, Ronald Willey

* Polarization*, Edward Collett

* Radiometry*, Barbara Grant

* Special Functions for Engineers*, Larry Andrews

* Spectroscopy*, David Ball

* Visual and Ophthalmic Optics*, Jim Schwiegerling

## Field Guide to Probability, Random Processes, and Random Data Analysis

Developed in basic courses in engineering and science, mathematical theory usually involves deterministic phenomena. Such is the case for solving a differential equation that describes a linear system where both input and output are deterministic quantities. In practice, however, the input to a linear system, such as imaging or radar systems, can contain a “random” quantity that yields uncertainty about the output. Such systems must be treated by probabilistic rather than deterministic methods. For this reason, probability theory and random-process theory have become indispensable tools in the mathematical analysis of these kinds of engineering systems.

Topics included in this * Field Guide* are basic probability theory, random processes, random fields, and random data analysis. The analysis of random data is less well known than the other topics, particularly some of the tests for stationarity, periodicity, and normality.

Much of the material is condensed from the authors’ earlier text * Mathematical Techniques for Engineers and Scientists* (SPIE Press, 2003). As is the case for other volumes in this series, it is assumed that the reader has some basic knowledge of the subject.

Larry C. Andrews

Professor Emeritus

Townes Laser Institute

CREOL College of Optics

University of Central Florida

Ronald L. Phillips

Professor Emeritus

Townes Laser Institute

CREOL College of Optics

University of Central Florida

## Table of Contents

**Glossary of Symbols and Notation x**

**Probability: One Random Variable 1**

Terms and Axioms 2

Random Variables and Cumulative Distribution 3

Probability Density Function 4

Expected Value: Moments 5

Example: Expected Value 6

Expected Value: Characteristic Function 7

Gaussian or Normal Distribution 8

Other Examples of PDFs: Continuous RV 9

Other Examples of PDFs: Discrete RV 12

Chebyshev Inequality 13

Law of Large Numbers 14

Functions of One RV 15

Example: Square-Law Device 16

Example: Half-Wave Rectifier 17

**Conditional Probabilities 18**

Conditional Probability: Independent Events 19

Conditional CDF and PDF 20

Conditional Expected Values 21

Example: Conditional Expected Value 22

**Probability: Two Random Variables 23**

Joint and Marginal Cumulative Distributions 24

Joint and Marginal Density Functions 25

Conditional Distributions and Density Functions 26

Example: Conditional PDF 27

Principle of Maximum Likelihood 28

Independent RVs 29

Expected Value: Moments 30

Example: Expected Value 31

Bivariate Gaussian Distribution 32

Example: Rician Distribution 33

Functions of Two RVs 34

Sum of Two RVs 35

Product and Quotient of Two RVs 36

Conditional Expectations and Mean-Square Estimation 37

**Sums of N Complex Random Variables 38**

Central Limit Theorem 39

Example: Central Limit Theorem 40

Phases Uniformly Distributed on (**−π**, **π**) 41

Phases Not Uniformly Distributed on (**−π**, **π**) 42

Example: Phases Uniformly Distributed on (**−** **α**, **α**) 43

Central Limit Theorem Does Not Apply 45

Example: Non-Gaussian Limit 46

**Random Processes 48**

Random Processes Terminology 49

First- and Second-Order Statistics 50

Stationary Random Processes 51

Autocorrelation and Autocovariance Functions 52

Wide-Sense Stationary Process 53

Example: Correlation and PDF 54

Time Averages and Ergodicity 55

Structure Functions 56

Cross-Correlation and Cross-Covariance Functions 57

Power Spectral Density 58

Example: PSD 59

PSD Estimation 60

Bivariate Gaussian Processes 61

Multivariate Gaussian Processes 62

Examples of Covariance Function and PSD 63

Interpretations of Statistical Averages 64

**Random Fields 65**

Random Fields Terminology 66

Mean and Spatial Covariance Functions 67

1D and 3D Spatial Power Spectrums 68

2D Spatial Power Spectrum 69

Structure Functions 70

Example: PSD 71

**Transformations of Random Processes 72**

Memoryless Nonlinear Transformations 73

Linear Systems 74

Expected Values of a Linear System 75

Example: White Noise 76

Detection Devices 77

Zero-Crossing Problem 78

**Random Data Analysis 79**

Tests for Stationarity, Periodicity, and Normality 80

Nonstationary Data Analysis for Mean 81

Analysis for Single Time Record 82

Runs Test for Stationarity 83

**Equation Summary 85**

**Biography 90**

**Index 91**

#### Glossary of Symbols and Notation

**a**, **x**, **u**, etc.

Random variable, process, or field

**B**_{u}(* R*)

Autocovariance or covariance function of random field

**C**_{x}(τ)

Autocovariance or covariance function of random process

**C**_{xy}(τ)

Cross-covariance function

CDF

Cumulative distribution function

Cov

Covariance

**D**_{x}(τ)

Structure function

* E*[.]

Expectation operator

* E*[

*(*

**g****x**)|

*]*

**A**Conditional expectation operator

**f**_{x}(* x*),

**f**_{x}(

*,*

**x***)*

**t**Probability density function

**f**_{x}(* x*|

*)*

**A**Conditional probability density

**F**_{x}(* x*),

**F**_{x}(

*,*

**x***)*

**t**Cumulative distribution function

**F**_{x}(* x*|

*)*

**A**Conditional cumulative distribution function

$p{F}_{q}$

Generalized hypergeometric function

* h*(

*)*

**t**Impulse response function

* H*(ω)

Transfer function

**I**_{p}(* x*)

Modified Bessel function of the first kind

**J**_{p}(* x*)

Bessel function of the first kind

**K**_{p}(* x*)

Modified Bessel function of the second kind

* m*,

*(*

**m***)*

**t**Mean value

**m**_{k}

* k*’th standard statistical moment

* n*!

Factorial function

Probability density function

Pr

Probability

* Pr*(

*|*

**B***)*

**A**Conditional probability

PSD

Power spectral density

RV

Random variable

**R**_{x}(τ)

Autocorrelation or correlation function

**R**_{xy}(τ)

Cross-correlation function

${\text{\u211c}}_{\mathbf{x}}(\text{\tau})$

Long-time-average correlation function

**S**_{x}(ω), **S**_{u}(κ)

Power spectral density function

* U*(

*−*

**x***)*

**a**Unit step function

Var

Variance

Var[**x**|* A*]

Conditional variance

$\overline{\mathbf{x}(t)}$

Time average

$z*$

Complex conjugate of **z**

γ(* c*,

*)*

**x**Incomplete gamma function

Γ(* x*)

Gamma function

δ(* x*−

*)*

**a**Dirac delta function (impulse function)

μ_{k}

* k*’th central statistical moment

$\widehat{\text{\mu}}(t)$

Estimator of mean value

${\text{\sigma}}^{2},{\text{\sigma}}_{\mathbf{x}}^{2}$

Variance

τ

Time difference **t**_{2}−**t**_{1}

Φ_{x}(* s*)

Characteristic function

||

Absolute value

∈

Belonging to

$\left(\begin{array}{l}a\\ n\end{array}\right)$

Binomial coefficient

〈 〉

Ensemble average

{ }

Event

$\cap $

Intersection