There are clear ecological, economic, and social benefits associated with forests1 which are very important to biological diversity as well as soil and water conservation. However, disturbances are pivotal processes in forest ecosystem dynamics2 which strongly affect the composition, functioning, and structure of forest ecosystems,3 and the temporal and spatial patterns of forested landscapes.4 Forest monitoring and protection remains a significant and urgent task in China.5 Insect damage is an important forest disturbance that disturbs the growth of trees, influencing the production and quality of timber over large areas.6 Foliage discoloration and defoliation are significant variables by which to evaluate forest damage. Although there are multiple agents causing color alteration and loss of foliage, it is assumed that insect pests are the most common cause of foliage discoloration and defoliation.7
Currently, the methods used for monitoring of forest insect defoliation and discoloration mainly include field survey and remote sensing technologies. Field survey can diagnose insect damage more correctly than other approaches, but there is a problem if people cannot work continually, so it is difficult to collect damage information in a timely and comprehensive manner. For the monitoring using satellite remote sensing, researchers have explored numerous studies. For example, De Beurs and Townsend8 used MODIS imagery to estimate the magnitude of defoliation caused by the gypsy moth and concluded that MODIS data can be used to monitor insect defoliation on an annual time scale for large patches. Olsson et al.9 utilized SPOT and MODIS data to detect damage caused by Physokermes inopinatus, but insect attacks can be detected when the resulting defoliation or discoloration is sufficiently severe. Multitemporal Landsat TM images were used to estimate and classify defoliation in boreal coniferous forests,10 but a slight defoliation of coniferous forests was difficult to be detected from the remote sensing data used. In summary, due to low spatial and temporal resolutions of satellite images, it is difficult to identify forest insect damage in small area in a timely fashion. In addition, aerial videography has been proven to be an effective remote sensing tool for the detection of forest insect infestations.11 Video data provide a permanent record at the time of detection and, hence, eliminate the subjectivity of the observers. However, the background is dynamically changing for an airborne based video,12 and aerial videography has a high budget and cannot work continually in practical applications. In this situation, the development of distant video surveillance (DVS) technology provides a new opportunity for the monitoring of forest insect defoliation and discoloration.
DVS is an important new remote sensing method and has been widely used for monitoring. As a major application field, forest monitoring has played an important role in forest resource protection and management. In comparison to the watchtower observation, video monitoring can work continuously. In comparison to the satellite image monitoring, video monitoring has a low cost and is real time. For forest fires, various video-based monitoring systems have been proposed by researchers, and most of these systems can achieve reliable results based on smoke recognition during the day and flame recognition during the night.13,14 Compared with forest fires, the features of insect defoliation and discoloration are subtle, and this also brings some new challenges. For example, the forest insect damage monitoring has a higher requirement for positioning accuracy than forest fire. Therefore, it is difficult to directly apply forest fire video monitoring systems to forest insect damage monitoring.
To effectively reduce the damage of forest pests, the ability to enable command and control centers to have an intuitionistic view of the occurrences in the forest area in real time is very important. Although video surveillance technology is playing an increasingly important role in monitoring systems in various security and military applications,15 most of them do not support geospatial technology integration and analysis from a spatial perspective. Based on the above, the main objective of this study is to design and implement a framework for forest insect defoliation and discoloration monitoring, relying on the use of geospatial technology and video monitoring technology. In this framework, satellite remote sensing is mainly responsible for identifying susceptible areas of forest insect infestations. The susceptibility maps can be used for guiding monitoring strategy in the video surveillance stage. Remote sensing technology has been extensively studied and described by other authors, so we will not elaborate on it. Video monitoring is utilized to visualize and identify forest insect damage, to guarantee real-time monitoring by using a pan-tilt-zoom (PTZ) camera with three degrees of freedom because of its ability to pan, tilt, and zoom. On-site verification is conducted by ground surveys and guided through global positioning system (GPS) integrated in the embedded devices. Furthermore, the proposed approach bridges the gap between aerial monitoring and ground surveys to form a comprehensive monitoring framework for forest insect defoliation and discoloration.
Forest pest damage has obvious visual and dynamical features, and they are usually characterized by defoliation and discoloration. Visual detection of an infested stand is not straightforward in many cases, particularly in large and dispersed forests. Video surveillance technology has been called to address this issue mainly due to two reasons: first, DVS can visualize the forests continually to check the health of woods beyond our own eyes; second, DVS has a wider vision and can monitor extensive forest areas and record these data in real time. In addition, to our knowledge, the application of video surveillance technology for forest insect defoliation and discoloration is rarely reported. This paper focuses on the integrated monitoring framework and explores a new application which will be very helpful for further study.
Compared with previous methods, the video-based method can provide continual and real-time monitoring, and the video data can be stored for a long time. Video surveillance techniques have many attractive attributes, but the most prominent are the real-time monitoring capability and the immediate availability of the electronic signal for both visual interpretation and digital processing.16 Visual images can convey more memorable and meaningful information than written words, numbers, or other types of media.17 Visualization refers to any technique for creating images, diagrams, or animations to communicate a message.18 The visualization of forest scene is essential for forestry departments to detect disturbances and make decisions for pest control. With the development of computer hardware and software, it is possible to visualize complex natural phenomena, such as a three-dimensional (3-D) scene or landscape. Visualization means to represent data in an intuitionistic and perceptible form that can be sensed by people’s vision. The 3-D visualization for forest insect defoliation and discoloration monitoring is supported by a geographic information system (GIS) and is characterized by multiperspective views, depending on the relationships with observers from different directions, distances, or locations.
In the proposed method, satellite remote sensing monitoring is mainly used to identify the susceptible areas of forest insect infestations in order to provide support to make appropriate monitoring strategies that reduce the impact of disturbances.19 The remote sensing sources used to identify the susceptibility areas should be high-resolution images, because forest insect infestation is often studied at the individual tree scale.20 DVS is used to capture video images to integrate with GIS, to provide intuitionistic display for the command and control center, and to obtain the geographic coordinates of insect damage for field survey. Then ground survey by human walkthrough is mainly responsible for conducting field verification and to simultaneously provide complementary details that video images lack. In addition, for those forest areas difficult to access, satellite imagery and video images are often the only means of gathering information. The main framework diagram is shown in Fig. 1. A detailed description of the above-mentioned procedures will be given in the following part of the article.
Video Positioning of Forest Insect Damage
For the system implementation, we used a high-definition CCD camera, digital PTZ, zoom lens, protective slab, decoder, watchtower, and power supply facilities. In the process of video surveillance, PTZ can control the camera to rotate at the horizontal (pan) and vertical planes (tilt) and to change the level of magnification (zoom). The camera, fixed on the PTZ, is responsible for capturing video images. The PTZ can control the camera to rotate smoothly in the horizontal direction and in the vertical direction, to guarantee complete coverage. Briefly, the front-end equipment and the monitoring center are connected through a wireless network.
The geographical position of forest insect damage is traditionally determined by comparing the video scene with a corresponding map. There are some unavoidable subjective factors that affect the accuracy. To improve the accuracy, we designed an automatic positioning algorithm for forest insect damage based on digital elevation model (DEM) and returned parameters from the PTZ camera. The principle of the disaster positioning is to build a ray emitting from the monitoring camera according to the orientation parameters returned from the PTZ. From the perspective of geometry, the process of the disaster positioning is to get the intersection point of the DEM and the ray. The DEM cannot be directly expressed by a mathematical function. Thus, in a discrete space, disaster positioning can be seen as a process of finding the optimal solution regarding the constraint satisfaction problem. To improve the efficiency of the algorithm, the viewshed analysis was used to generate the viewshed layer, which could reduce the search space of the optimal solution. To generate the ray, the Bresenham algorithm21 was used which adopts incremental computation and is a classical algorithm to generate a line in computer graphics. Thus, for each column, just check the sign of an error item, and then the pixel of this column is selected. This algorithm does not include the operation of real numbers, so it can generate a line quickly.
Figure 2 illustrates the spatial model for disaster positioning, where represents the monitoring point and is the disaster point. and are the azimuth angle and pitch angle, respectively. is a vertical angle in the direction of . is a ray from the monitoring point. Based on this model, the flow chart of the positioning algorithm is shown in Fig. 3, where represents the height from the camera to the ground. (, , ) are the projective coordinates of the monitoring point.
Integration of GIS and Video Surveillance
The hardware architecture depicting the system context is shown in Fig. 4. For real-world image capture, several PTZ cameras are utilized. A PTZ camera has three degrees of freedom because of its ability to rotate in the horizontal (pan) and vertical planes (tilt) and to change the level of magnification (zoom).
Three servers are deployed at the server-side and are connected to the Internet with a speed of . The video server uses the D-Link DNS-726-4 to provide the load balancing service and is utilized to distribute front-end video streams and to provide load balancing service for an increasing number of users. The data server can provide basic spatial data service and pest information storage service for the application server. The DVS system is deployed on the application server using a middleware of Internet Information Services 7.0. As shown in Fig. 5, the software structure of the DVS system consists of two-dimensional (2-D) GIS, 3-D GIS, video control, disaster positioning, and communication control. In comparison to previous research, our method integrates 2-D GIS and 3-D GIS synergistically. This can make the monitoring process more intuitionistic.
Functionalities of the 2-D GIS component are related to 2-D visualization, layer management, geospatial data access, feature querying, and identification. Furthermore, the 2-D GIS can navigate monitoring points and aid disaster positioning. In addition, the 2-D GIS can serve as an eagle-eye map of the 3-D GIS component. The 3-D GIS component is responsible for the visualization of the DEM and 2-D geospatial data of the forest area. The 2-D geospatial data are displayed as a texture that is laid over the virtual terrain. Moreover, the disaster points can be added and displayed in the 3-D GIS context.
The video control component is responsible for controlling the camera to pan, tilt, and zoom as well as for setting the window size and the bitrate of the camera video. The communication control component is responsible for the acquisition of the camera’s orientation parameters as well as for communications between different components. The component for disaster positioning is responsible for calculating the geographic coordinates of the forest pest disasters.
In order to evaluate the method, ground survey by human is used to collect field data for on-site verification. The field survey is used for both forward validation and reversed validation (Fig. 6). Forward validation is used to assess the accuracy of damage detection using video surveillance. Reversed validation is used to evaluate the completeness of damage detection. Due to the limited memory and screen size of the embedded equipment, the massive spatial data need to be preprocessed, such as establishing the image pyramid and segmenting the vector data.22 In the system, the ShapeFile format is used for storing all the vector data because it uses less storage space. The pyramid images are managed in FGI format, which consists of a master file (.fgi), a secondary file (.aux), and a coordinate file (.txt). After downloading the processed data into the embedded GIS, field surveys can be conducted to collect the spatial data, attribute information of the pest disasters, and upload the results to the server in time.
Field verification is aided by smartphones or personal digital assistant (PDA) with GPS embedded (Fig. 7). After the geographical coordinates of the forest pest disasters are obtained, the thematic map of disasters can be output and imported into PDA, and afterward the foresters are able to conduct on-site verification following the optimal path planned in advance. In addition, to guarantee the reality and authenticity of the field data, the proposed approach can provide supervision and inspection to enable managers to avoid fake data to some extent.
Experiments and Analysis
The DVS system is implemented in Microsoft Visual Studio 2010, as a Visual C# project. Browser/Server structure is utilized in this system. The main user interface is composed of three components: 2-D view, 3-D view, and video view (Fig. 10). Besides these views, the application interface contains the disaster information table (bottom-left) to record the disaster information including the monitoring point names, the latitude and longitude of disasters, damage degree, and date, which is simultaneously stored in the database. The first component, 2-D view, provides the overall perspective of the forest area based on the forestry data and other geographic spatial data. The component is implemented purely by the web pattern. Basic GIS functions related to the view are map zooming, map panning, object identification, and layer management. Users can choose to display the selected layers of geospatial data according to actual needs.
The second component, 3-D view, is responsible for the generation and visualization of the virtual terrain. The component is implemented based on OpenSceneGraph, which is an open source 3-D graphic engine. At the data layer, Geospatial Data Abstraction Library 1.8.0 is used to analyze the spatial data. At the network layer, the library of libcurl 7.21.3 is used to transmit the Web Map Service terrain/image pyramid models. At the function layer, the terrain rendering library of libMini 10.2 is utilized to implement 3-D terrain visualization. Functions for the 3-D view include roaming, zooming, and visibility analysis. Visibility analysis is used to determine whether a given object located on a terrain is visible from a viewpoint and how much of the object is visible.23 In addition, when the coordinates of disaster points are obtained, they can be simultaneously added to the 3-D scene.
Finally, the third component, video view, displays the on-site video of a selected surveillance camera. With this view, it is feasible and flexible to manipulate the camera by changing the parameters of pan, tilt, and zoom, or utilizing the mouse as a virtual joystick (Fig. 8).
During the process of scanning, when finding dubious phenomena, the PTZ parameters can be adjusted to clearly observe and analyze the video images to detect forest pest disasters. Then, combined with DEM and the angles of azimuth and pitch of the PTZ camera, the geographical coordinates of the disasters can be obtained through the positioning algorithm. Then the disaster information is stored into the disaster database. Users can query disaster by single field or multifield, and the query results can be simultaneously visualized in the 2-D view. In addition, the thematic map can be conveniently output and printed, to aid forestry surveyors to conduct field verification. Furthermore, the 2-D view, 3-D view, and video view can interact with each other. For example, when the user selects a monitoring point, the 3-D view and the video view will transform accordingly. Thus, the user can monitor the on-site situation in real time to detect forest pest disasters timely and accurately. In addition, the thematic map can be generated and printed, providing great convenience for the foresters to conduct on-site verification.
For on-site verification, the system for ground surveys (Fig. 9) is used, which includes the mobile terminal and the server terminal. The disaster information is transmitted to the system for ground surveys by extensible markup language (XML). The mobile terminal is developed by VC++. NET, and MSXML3.0 is used to parse the XML schema documents. The server terminal is developed by VC++6.0, using data access object to operate the database and MSXML6.0 to establish the dynamic form. Web services are used in the server terminal to receive the field survey data from the mobile terminal. The embedded system is connected with the server through a general packet radio service and the survey results are uploaded using these services. The implementation of the system for ground surveys relies on the previous work summarized in Ref. 22.
Experiment Results and Validation
To evaluate the availability of the positioning algorithm, accuracy verification was conducted. First, differential GPS was used to measure the geographic coordinates of simulated disaster points. Combined with local three parameters, the geographic coordinates were transformed into projective coordinates. Next, the simulated disaster points were positioned through the proposed video positioning method. Finally, these two measurement results were compared. The projective coordinates and elevation of the simulated monitoring point are (547,366, 4,466,675, 1420).
From Table 1, we can see that the maximum error for is 49 m and for is 97 m. The minimum discrepancy for is 21 m and for is 45 m. These discrepancies are acceptable for forest pest damage monitoring.24
Video monitoring positioning, GPS measuring, and their discrepancies.
|ID||Video monitoring positioning (m)||GPS measuring (m)||Discrepancies (m)|
Field verification is often considered the preferred method for evaluating insect damage detection derived from video remote sensing. In order to better assess the potential of the proposed approach, two forest areas are selected for field validation. Jinghai County is located in southwestern Tianjin (Fig. 10), with an area of . Longitude ranges from 116°42′E to 117°12′E, and latitude ranges from 38°35′N to 39°04′N. The vegetation types of this area are mainly broadleaved deciduous forest. The Hyphantria cunea damage in this area is representative of this city and was, therefore, selected as the test site for this study. Qianshan County is located in southwestern Anhui (Fig. 10), with an area of . Longitude ranges from 116°14′E to 116°41′E, and latitude ranges from 30°27′N to 31°04′N. The vegetation types of this region are mainly pinewood and mixed deciduous forest. Planted forests account for 88.4% of the whole county, resulting in single forest form and poor biodiversity, which facilitates the occurrence and spread of insect pests. The Dendrolimus punctatus walker damage in this region is representative of the province, so it was selected as one experimental area. Both Hyphantria cunea and Dendrolimus punctatus walker occur with three generations in a year.
To quantitatively evaluate the proposed method, we employ the forward detection ratio (fDR) and reversed detection ratio (rDR) as the assessment principles, which are defined as
According to the different stages of insects, we selected sampling points (Fig. 11) according to the visual features of discoloration and defoliation (Fig. 12). Then, field surveys were carried out to validate the detection results. The data are obtained from the local forest bureau. To further demonstrate the method, we also carried out a reversed field validation task. We selected 20 sampling points by field surveys, respectively, in the two areas and validated them by video monitoring. Tables 2 and 3 show the field validation results.
Accuracy validation of site 1.
|Test site||Objective||Date||Video detection||Field survey||fDR|
|Jinghai||Forward validation||May 15, 2013||19||16||0.84|
|July 15, 2013||30||25||0.83|
|15 September, 2013||20||17||0.85|
|Objective||Date||Field survey||Video detection||rDR|
|Reversed validation||May 15, 2013||20||18||0.90|
|15 July, 2013||20||19||0.95|
|15 September 2013||20||18||0.90|
Accuracy validation of site 2.
|Test site||Objective||Date||Video detection||Field survey||fDR|
|Qianshan||Forward validation||23 April, 2013||17||15||0.88|
|23 June, 2013||30||26||0.87|
|23 August, 2013||20||17||0.85|
|Objective||Date||Field survey||Video detection||rDR|
|Reversed validation||23 April 2013||20||16||0.80|
|23 June, 2013||20||17||0.85|
|23 August, 2013||20||16||0.80|
Here, an assessment of accuracy was conducted for the detection and positioning of forest insect damage. Currently, the identification of forest insect damage by video cameras is usually conducted by a forest fire video monitoring system. Due to low definition of the camera, it is difficult to identify forest insect damage. In our method, we selected a high-definition camera, so the identification rate is improved, as shown in Table 4. The fDR and rDR of the current method are 0.79 and 0.85 in site 1, respectively. The fDR and rDR of the proposed method are 0.84 and 0.92 in site 1. The fDR and rDR of the current method are 0.81 and 0.76 in site 2. The fDR and rDR of the current method are 0.87 and 0.82 in site 2. Experiments show that the accuracy of the proposed method is higher than the current method.
Comparison of the forest insect damage detection method by current method and proposed method.
|Test site||Current method||Proposed method|
Traditionally, the geographical position of forest insect damage is determined by comparing the video scene with the corresponding map. There are some unavoidable subjective factors that affect the accuracy. We utilized the traditional method to determine the position of the simulated disaster points, and the comparison results are shown in Table 5. The overall accuracy is calculated as . Experiments show that the proposed positioning method has better overall accuracy than the traditional method. In addition, the proposed method is automatic and avoids the subjectivity of observers to some extent.
Comparison of the positioning method by traditional method and proposed method.
|ID||Traditional method||Proposed method|
|ΔX (m)||ΔY (m)||ΔZ (m)||Overall (m)||ΔX (m)||ΔY (m)||ΔZ (m)||Overall (m)|
Three are some other types of GIS-based video monitoring systems. Zhang et al.25 presented a prototype system VSGIS integrating 2-D GIS and video surveillance, and it was applied in city safety management. Zong et al.26 combined GIS and video monitoring technology to design a GIS-based video surveillance system by constructing an electronic map with video linkage to display control points. In comparison to them, the proposed method integrates 2-D GIS and 3-D GIS simultaneously, and the three modules are geographically linked and displayed in the same window. This is the basis of implementing the fusion of video images and 3-D GIS in future studies.
The study clearly highlights the potential for video surveillance and geospatial technologies to monitor forest pest disturbances, and the ability for positioning and mapping damage in a timely manner. This monitoring technique tends to be proactive rather than passive aiming to complement the existing methods. An operational surveillance system may detect forest pest disasters as early as possible to reduce losses. Although the approach is designed for monitoring forest insect defoliation and discoloration, it could be modified for monitoring forest resources, ecological environment, and other disturbances. Forest pest damage surveillance is established as part of forest health management to ensure the early detection of pest threats to tree health and to respond in time before there is significant damage. From the perspective of ecology, in the long run, this can facilitate sustainable forestry development.
The study focuses on the monitoring and detecting of the damage characterized by defoliation and discoloration in the sense of human–computer interaction. Little effort, however, was spent on researching on the damage detection and recognition from the video images automatically. Nonetheless, according to the literature retrieval, there is hardly any research on this issue. This issue is a complicated problem that refers to various subjects and fields, such as computer vision, intelligent video analysis, image processing, and data mining. Furthermore, forest pest damage detection and recognition also involves a substantial amount of heuristic knowledge, requiring much effort for domestic and international researchers to carry on a great deal of research.
This study aimed to present an integrated application to monitor forest insect damage. It was not, nor could it be our intention, to evaluate the cost benefit of the system. This procedure should be performed by the competent authorities.
In this study, an integrated framework for video-based monitoring of forest insect defoliation and discoloration was presented. By using video monitoring, we could extend the spatial coverage and number of samples far beyond what was obtained with field surveys. Video images make the monitoring procedure more intuitive and perceptual. The foresters can remotely control the video cameras at monitoring spots and observe the video images displayed on the screens in the monitoring center. Moreover, video images are obtained in real time, thus the forest insect disturbances can be detected as early as possible. Compared with traditional human ground observation, using video cameras the human observer is capable of monitoring a wider area. Cameras equipped with power zoom facilitate the observer to inspect suspected areas. Video data can be stored and this is quite useful for postdisaster analysis. In addition, there are some other issues worthy of consideration. For example, what are the spatial distribution and density of the cameras needed for the DVS system to adequately monitor the forest area depending on landscape traits? This will be our future studies.
The authors would like to thank the reviewers whose comments were very helpful in improving this paper. The work is supported by National Science and Technology Major Project of the Ministry of Science and Technology of China (21-Y30B05-9001-13/15), the National High Technology Research and Development Program of China (863 Program No. 2008AA12Z203), and the National Science and Technology Support Project (No. 2006BAD23B04).
Feifei Zhao is currently a PhD candidate in the Institute of Remote Sensing and Digital Earth, Chinese Academy of Sciences. Her current research interests include remote sensing for forestry, video monitoring, and geographic information system.
Yafei Wang is currently a PhD candidate in the Institute of Geographic Sciences and Natural Resources Research, Chinese Academy of Sciences. His major research interests include remote sensing data mining and geographic information system (GIS) application.