Conventional fire detection systems were designed to detect smoke, heat, and radiant energy from a fire using infrared, optical, and ion sensors.12.–3 However, these methods have problems, namely, it is impossible to tell whether a fire is occurring until smoke or flame spreads to the detection range of sensors and detection takes a long time because a fire alarm is only issued after the flame’s influence exceeds a reference temperature or a set value. In addition, because the detection range is restricted to the local position where sensors are installed, the need for numerous distributed sensors and installation increases the cost. On the other hand, a fire detection system using image processing does not require any additional costs, as it uses surveillance cameras already installed in public places, roads, and tunnels. Thus, fires can be quickly detected without having to wait until flames spread to the sensors, and can be monitored over a broader range.
Early fire detection algorithms used rule-based color models.45.–6 They classify the image pixels as fire or not fire by using the RGB, hue, saturation and intensity, and YCbCr color models. However, these methods have many false classifications. For example, red clothes, moving cars, and lights will be recognized as fires because only color information was used. Han7 used color and motion information to present real-time fire detection. Töreyin et al.8 used temporal and spatial wavelet analysis to separate the fire regions from sequential images. Although they presented good experimental results, this approach is impractical in various environments due to many heuristic thresholds. Z. Teng9 modeled the sequentially changing image pixel values by a hidden Markov model framework. Ko et al.10 applied a radial basis function kernel to two-class support vector machines. This research that uses machine learning algorithms could have an over-fitting problem and are not suitable for various environments. In most of the previous studies, it is necessary to determine whether an individual pixel is a fire or nonfire pixel. However, red objects that have a hue as fire are often mistaken for fire when determining each image pixel. In addition, moving lights, such as automobile headlamps, can be mistaken for fires when using only motion information.
Therefore, we defined the color conditions of fire features in both hue, saturation and value (HSV) and RGB color space. The local features and motion information of fires were expressed using the mean, variance, and image difference. Moreover, temporally blinking properties were modeled by using crossing patches so that it was possible to reduce the false detections described above. We defined 22 image features to classify patches into fire or nonfire areas by using a random-forest-supervised learning algorithm. The additive Markov chain was introduced to reduce false alarms. Also, we designed an embedded surveillance device made from acrylonitrile butadiene styrene (ABS) for stable fire detection in outdoor environments.
All modules, including image features extraction and fire classification, are realized to test fire images of different conditions and verified to be robust in complex environments.
Embedded Camera System
In general, to protect people from fires, fire surveillance systems require supervisors in monitoring rooms to observe videos transferred from CCTVs. Building a surveillance system requires considerable effort to install surveillance cameras, as many repeaters and wiring inside facilities are needed to transfer videos to monitoring rooms, causing increased installation costs. In addition, all images are concentrated on servers in monitoring rooms and so the structure of server computers becomes complex. Therefore, we designed an embedded system with a built-in pan-tilt-zoom (PTZ) camera to observe fire areas more effectively. The proposed system, which has a built-in embedded processor inside the camera, is capable of real-time image processing and consists of a main camera that can monitor fires over a wide range and a subcamera that can quickly zoom into fire areas when a fire occurs.
The early-designed device fixed each module by installing a fixed bracket inside a universal housing, as shown in Fig. 1(a) and a subcamera below the housing. Due to heat from the camera and embedded board, a prolonged operating time rapidly raised the internal temperature of the housing. In addition, the commercial housing suffered from the disadvantages of heavy weight and expensive cost. In the second system design, the cooling performance was improved by projecting the cooling section of the embedded board outside, as shown in Fig. 1(b). This system reduced weight by using an ABS resin housing and a digital output main camera. The ABS housing has the advantage of reducing production costs because it can be produced in large quantities by injection molding. Protection from dust and moisture was accomplished by using a rubber seal and waterproof connector, as various foreign bodies in the form of particulate could otherwise enter the system.
The whole operation of the fire detection system is depicted in Fig. 2. The main camera is a fixed type and acquires raw images to detect fire while monitoring the wide surveillance area. In addition, the installed industrial digital camera is able to remove the noise generated in the process of converting images from analog to digital. This can simplify the system and permit control directly by the program. A PTZ camera with maximum 10-magnification was selected for the subcamera so that it is possible to immediately magnify and transmit images of the area where a fire has occurred. The images acquired from the main camera are processed in the built-in embedded board, and thus fire areas are detected. When the fire areas are detected, magnified images of fires are acquired from the subcamera. The acquired images from the subcamera are converted into digital images by a frame grabber. Information, such as a gazing direction and scale, is transmitted by RS-422 communication to the main processor, for calculating PTZ motion data from the fire area detection of the main camera. Finally, the system transfers the fire location and magnified images of the fire area to the monitoring room. Figure 3 is a graphical user interface screenshot of the fire monitoring system and an image that detects fires and magnifies the detected areas.
Figure 4 shows CPU temperature change in both the customized and commercial housing during 24 h. The measurements were conducted after a detection algorithm operated for 1 h in each housing. It indicates that average improvement of cooling performance is 22.017°C.
Fire Detection Algorithm
In previous studies,67.8.9.–10 information from each pixel in the image was used to determine whether fires exist in the images. Alternatively, our paper proposes a method to model the local characteristics of fire, represented in sequential images based on patches of an input image, and to detect a fire area by using a supervised learning algorithm. The random forest showed a higher recognition rate than other machine learning algorithms.11 It can process substitution branching for lost data processing and handle both categorical and numerical data. In addition, it makes it unnecessary to normalize image features and easy to understand the importance of variables.
First, in order to model the local features of fires, the input image is divided into small patches (P), as shown in Fig. 5. We generate the color conditions, motion, and blink information of fires as features in the patch.
Modeling Fire Features
In general, fires have specific colors in images. In this section, fire features were analyzed in RGB and HSV color space. For color features of fire in the RGB color space, it was observed that the R (red) color is larger than the threshold value and the R color value is the largest in the RGB channel. Moreover, in the HSV color space, it is shown that the H (hue) value of fire is 0 to 70 or 290 to 360. For example, the feature means the saturation will decrease with the increasing red component, and thus the expression indicates that saturation will decrease downward to zero when the red component increases toward the upmost value 255 (Refs. 12, 13). Therefore, the patch is likely to indicate a fire area when there are many pixels satisfying these fire conditions. Features () 1 to 4 show how many pixels satisfy each condition in the patch (Table 1).
Pseudo-code for the feature calculation of f1∼f4.
|For all pixels in a patch|
|if(290≤P(r,c)H≤360 or 0≤P(r,c)H≤70),f4++;|
The pixels in the fire areas are similar to specific colors such as red, yellow and white, and the deviation of colors is relatively uniform. Therefore, Features () 5 to 10 are defined by calculating the color mean and the variance for each RGB channel.
Additionally, it is observed that fires appear to blink in sequential images. Using this, the number of blinks in a certain area accumulated over 10 frames is defined as a feature.
We defined three cross extractors to calculate the brightness difference between the previous and subsequent patches as shown in Fig. 6. If the difference in brightness exceeds a threshold value, then it is considered to be a crossing count. in Eq. (3) is a brightness value at the pixel in the image. In this paper, a total of nine features were generated by using three different thresholds deciding the cross state in order to know the level of blinking.
Finally, this study used the feature that many motions could be observed between previous and present frames in fire areas. Automobile headlamps are known to generate strong noise, due to the dispersion of light when using the common image difference between previous and subsequent frames. Therefore, motion pixels are defined as features after the removal of the noise by using the average brightness of the three previous and subsequent frames.
Figure 7 is the result that depicts the features defined above in an image. The image shown in the upper right indicates the area meeting the color conditions of Features 1 to 4. In addition, the image shown in the lower left represents the motion of overlapping frames corresponding to features 20 to 22. An image shown in the lower right represents the crossing blocks between the previous and present frames.
Learning RF and Classifying Fire Features
In order to classify the features defined in the previous section into fire and nonfire areas, this study used the random-forest supervised-learning method.14
The input feature () reaches the leaf node after following the binary branch of decision tree (). Each node () in the ensembles selects a feature minimizing the Gini impurity () in order to determine the feature and threshold value that can classify the feature into a suitable class most effectively. Finally, probability  of each class is calculated by the summation of each leaf node probability, as shown in Fig. 8.
The assembly of each ensemble classifier generated in this manner is called a random forest. When given ensemble classifiers, feature vector and class , the feature vector is classified into fire as follows:
The whole process of learning and classification is described in Fig. 9. In order to learn fire, the input image is divided into the unit area. The 22 features defined in the previous section are extracted from the divided images. When learners input the sample groups into a learning machine, including classes and features generated from the fire and nonfire images as shown in Fig. 10, the ensemble classifier and decision rule are generated by the random forest. When images are captured in real-time, the random forest decides the fire and nonfire area by the decision rule.
Decision of a Fire
When using the fire detection algorithm proposed in the previous section, it is possible to distinguish between fire and nonfire in various environments. However, false alarms occurred in a couple of frames when an automotive headlamp and taillight continuously intersected at night. In order to solve this problem, the presence or absence of fires was finally determined by recording recent fire detection results during frames and stochastically analyzing the state changes with the use of the additive Markov chain.15
An additive Markov chain is a theory that is affected by previous sequential random variables () and the sum of all the values becomes the final probability.
In this case, a memory function () is defined by the results of fire detection () during frames as a binary number. It represents the frequency of the fire detection and continuity during frames. In addition, the function is generated to give a higher weight to the variable close to the present state. is a constant to determine whether to give a higher weight to the fire’s appearance frequency or continuity, and 0.7 is used in this experiment. is a normalizing constant.
In this study, number of cases is generated. The executing speed can be increased by the pre-calculated values with the Lookup Table.
The detection results are determined during the accumulated five frames, and the following rule is used for the final decision:
To learn fire features, 10,000 and 40,000 samples of a fire and nonfire, respectively, were extracted from video images, such as in a tunnel, downtown, and mountain area. Numerical type feature data were used for the training. The classifier was generated by random-forest supervised-learning.
When using the learning algorithm, the selection of not only features but also samples used for learning can have a large impact on the classification results. For example, if most fire samples are red areas, the red areas will be detected well. However, it becomes difficult to detect the yellow and white areas. When learning only areas with active motion, it becomes difficult to detect the areas with the small motion of a flame. Therefore, the fire samples used for learning should be selected so that the numbers are evenly distributed according to their colors and motions. In selecting the samples of a nonfire area, learning was performed by dividing into several areas as follows: areas similar in color to fires without motion, areas different in color from fires with active motion, and areas completely different in color from fires without motion. In such cases, it is desirable to select a relatively large number of nonfire samples, because many errors mistaking nonfire for fire occur when the number of fire samples is relatively large.
In the first experiment as shown in Fig. 11, Movies 1 to 5 were used to compare the fire detection algorithm proposed by Töreyin, Ko, and the method proposed in this paper. Additionally, Movies 6 to 11 were used to perform experiments in more varied environments. The description of the images can be found in Table 2. True positive means that a fire image is correctly determined as a fire, true negative means that a nonfire image is correctly determined as a nonfire, false positive means that a fire image is misjudged as a nonfire and false negative means a nonfire image is misjudged as a fire.
Descriptions of experimental movies.
|Movie 1||People set fire to trees using equipment.|
|Movie 2||There are trees burning at a close distance.|
|Movie 3||There is a fire in a narrow outside area. This movie is used for testing a small fire and a similar color in a scene.|
|Movie 4||An entire forest is burning with smoke.|
|Movie 5||There are automobile accidents occurring in the tunnel. This movie is used for confirming whether pixels with the similar fire color and motion are accurately classified in the scenes. There is no fire.|
|Movie 6||People create a fire and the equipment extinguishes the fire inside a building.|
|Movie 7||Firefighters set fire to a train in a tunnel.|
|Movie 8||There is fire occurring in an automobile and a house in a city.|
|Movie 9||People set fire in a bucket and there are red objects such as a hat, backpack, and a fire extinguisher.|
|Movie 10||In the movie there is an ambulance blinking a siren. This movie is used for confirming whether pixels with the similar fire color and motion are accurately classified in scenes. There is no fire.|
|Movie 11||There are a lot of automobiles reducing speed at night with headlamps and tail lights on. This movie is used for confirming whether pixels with the similar fire color and motion are accurately classified in scenes. There is no fire.|
NOTE: The movies 1∼5 are available at http://signal.ee.bilkent.edu.tr/VisiFire/. The movies 9 and 11 were recorded by our embedded system. The other movies were downloaded on YouTube.
Overall results of the first experiment are shown in Figs. 12 and 13. The method proposed in this study shows a higher detection rate than the other two methods. In this case, the detection rate is the sum of the values of true positive and true negative. In Movie 1, it was not possible to consider the motion information properly because the frame rate of the video was lower than other images. In Movie 3, the detection rate was lower than other images because the size of the fire area was smaller than other patches used in the experiment. Furthermore, in Movie 5, the false alarm shown in the method by Ko did not occur in the experiment with the proposed method.
As depicted in Fig. 14, the second experiment was performed by using fires in tunnels, inside buildings and automobiles, with a blinking ambulance light, continuously passing automobile headlamps, and tail lights likely to be mistaken for fires. As shown in Fig. 15, the detection rate was an average of 96.45% and we could reduce the average of false negative rate from 2.1% to 1.31% when applying the additive Markov chain. However, as shown in Fig. 16(a) and 16(b), false positives were shown because the fire area was small at the moment when the fire was extinguished in Movie 6 and as the fire started in Movie 9. Depicted in Fig. 16(c) and 16(d), the false negatives occurred because the color, motion and crossing feature were similar to fires. Movie 9 demonstrated whether the classifier works well when exposing a bag, hat, and fire extinguisher similar in color to flames in the image. The results showed that it worked well even when moving the hat and bag. Although there were many areas that had values similar to the fire features defined above in Movies 10 and 11, the classifier could work well because each of the feature values were relatively small.
It is possible to recognize variable importance in the learning process when using random-forest supervised-learning. Figure 17 shows that the importance of 22 features is evenly distributed. In this case, Features 5 and 7 are relatively high because the classifier frequently used the mean value of red and blue color channels during the training. The rate of features () related to the fire color are 51.2%. It means that the color features are more effective than the motion and crossing features (). However, the motion and crossing features perform the role of the fire features very well. It works to prevent miss-classifications when similar colors, such as automobile headlamps and red hats, appear in scenes. In the color features, a contribution of the RGB channel values () is higher than the color conditions (). In the crossing features (), crossing features for small threshold () have more influences on detecting fires.
The algorithm could process approximately 30 frames of image in Core 2 Duo 2.00 Ghz at our embedded system.
In this paper, a fire detection system with the built-in embedded processor, the main camera, and the PTZ camera was designed to perform fire surveillance in various environments. In addition, fire features were modeled by color, motion, and crossing features. This paper proposed an algorithm that could detect fires precisely by using the random forest classifier, making it possible to reduce the false alarms that occurred in previous studies resulting from the complex background. We used an additive Markov chain method for a robust fire detection algorithm. The performances of this method were verified by comparative experiments with other methods in various environments. The proposed fire detection system can detect fires with high reliability in public places, such as buildings, roads and tunnels. Also, it minimizes casualties and property damages by early fire detection. In addition, the automatic surveillance system can reduce facility maintenance costs by reducing the number of monitoring personnel and the fatigue level of the supervisor. Therefore, our system contributes to increased safety in public places because potential fires can be monitored 24 h a day.
This work was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (Nos 2011-0027213 and 2011-0017228) and this work was partly supported by the Human Resources Development program (No. 20114010203080) of the Korea Institute of Energy Technology Evaluation and Planning (KETEP) grant funded by the Korea government Ministry of Trade, Industry and Energy.
Onecue Kim received a BS in computer science from Tongmyong University, Korea, in 2008. He is now a postgraduate and doctoral student in mechanical engineering at Pusan National University, Korea. His current research interests are visual surveillance, machine vision, robots, and pattern recognition.
Dong-Joong Kang received a BS in precision engineering from Pusan National University in 1988 and a PhD in automation and design engineering at KAIST (Korea Advanced Institute of Science and Technology) in 1998. From 1997 to 1999, he was a research engineer at Samsung Advanced Institute of Technology (SAIT) and from 2000 to 2005 he was an assistant professor in the mechatronics engineering department at Tongmyong University. He has been an associate professor at the School of Mechanical Engineering at Pusan National University since 2006 and associate editor of the International Journal of Control, Automation, and Systems since 2007. His current research interests include visual surveillance, intelligent vehicles/robotics, and machine vision.