Two ultrahigh-speed CCD image sensors with different characteristics were fabricated for applications to advanced scientific measurement apparatuses. The sensors are BSI MCG (Backside-illuminated Multi-Collection-Gate) image sensors with multiple collection gates around the center of the front side of each pixel, placed like petals of a flower. One has five collection gates and one drain gate at the center, which can capture consecutive five frames at 100 Mfps with the pixel count of about 600 kpixels (512 x 576 x 2 pixels). In-pixel signal accumulation is possible for repetitive image capture of reproducible events. The target application is FLIM. The other is equipped with four collection gates each connected to an in-situ CCD memory with 305 elements, which enables capture of 1,220 (4 x 305) consecutive images at 50 Mfps. The CCD memory is folded and looped with the first element connected to the last element, which also makes possible the in-pixel signal accumulation. The sensor is a small test sensor with 32 x 32 pixels. The target applications are imaging TOF MS, pulse neutron tomography and dynamic PSP. The paper also briefly explains an expression of the temporal resolution of silicon image sensors theoretically derived by the authors in 2017. It is shown that the image sensor designed based on the theoretical analysis achieves imaging of consecutive frames at the frame interval of 50 ps.
Because escape from a net cage and mortality are constant problems in fish farming, health control and management of facilities are important in aquaculture. In particular, the development of an accurate fish counting system has been strongly desired for the Pacific Bluefin tuna farming industry owing to the high market value of these fish. The current fish counting method, which involves human counting, results in poor accuracy; moreover, the method is cumbersome because the aquaculture net cage is so large that fish can only be counted when they move to another net cage. Therefore, we have developed an automated fish counting system by applying particle tracking velocimetry (PTV) analysis to a shoal of swimming fish inside a net cage. In essence, we treated the swimming fish as tracer particles and estimated the number of fish by analyzing the corresponding motion vectors. The proposed fish counting system comprises two main components: image processing and motion analysis, where the image-processing component abstracts the foreground and the motion analysis component traces the individual’s motion. In this study, we developed a Region Extraction and Centroid Computation (RECC) method and a Kalman filter and Chi-square (KC) test for the two main components. To evaluate the efficiency of our method, we constructed a closed system, placed an underwater video camera with a spherical curved lens at the bottom of the tank, and recorded a 360° view of a swimming school of Japanese rice fish (Oryzias latipes). Our study showed that almost all fish could be abstracted by the RECC method and the motion vectors could be calculated by the KC test. The recognition rate was approximately 90% when more than 180 individuals were observed within the frame of the video camera. These results suggest that the presented method has potential application as a fish counting system for industrial aquaculture.
Particle Tracking Velocimetry, PTV, is one of the most powerful tools those can measure the deformation of fast flowing fluid, such as vorticity, shear rate, and so on, with a high-speed video camera. We have developed a new method to estimate vorticity from randomly located velocity vectors obtained by a PTV. The proposed method employs the Moving Least Square method that is developed for the meshless numerical simulation. The optimal size of fitting area of the proposed method is derived theoretically and is confirmed by the Monte Carlo simulation. The comparison of accuracy of the proposed method is carried out. The result shows that the proposed method is more accurate than the commonly used method in any case.
Although we have aspired to observe dynamic changes in fluorescent images at the cellular level for a long time, the commercially available video cameras are not at all suitable for this purpose because of their low frame rates and photosensitivity. The present work tackles this issue and describes our attempt to find a solution by using our high-speed video camera and an ultrabright illumination system. We used light sources with considerably higher energy because conventional mercury lamps cannot produce sufficient brightness for our video cameras working a rate of more than 4,500 fps to obtain fluorescent images of cells. We observed that the flagellar movement of mice sperms ceased and multiple kinks developed in their tails when exposed to 2.7W of laser illumination for 1 s. In contrast, no significant alterations could be detected when the sperms were subjected to the same amount of energy by intermittent illumination. Since we found that cells can survive short-duration exposure to high-energy light, we attempted to construct an ultrabright Xenon-strobe illumination system. Our fluorescence studies are currently being extended to other types of animal cells, e.g., observation of the conduction of action potentials in the peripheral nerves of frog.
This paper presents preliminary evaluation results of a test sensor of the backside-illuminated ISIS, an ultra-high
sensitivity and ultra-high speed CCD image sensor. To achieve ultra-high sensitivity, the CCD image sensor employs the
following three technologies: backside illumination, cooling and Charge Carrier Multiplication (CCM). The test sensor
has been designed, fabricated and evaluated. At room temperature without cooling, the video camera has about ten-time
higher sensitivity than the previous one, which was supported by a conventional front side illumination technology.
Furthermore, the video camera can detect images at very low signal level, less than 5 e-, by using CCM at -40 degree C.
We are developing an ultra-high-sensitivity and ultra-high-speed imaging system for bioscience, mainly for imaging of microbes with visible light and cells with fluorescence emission. Scarcity of photons is the most serious problem in applications of high-speed imaging to the scientific field. To overcome the problem, the system integrates new technologies consisting of (1) an ultra-high-speed video camera with sub-ten-photon sensitivity with the frame rate of more than 1 mega frames per second, (2) a microscope with highly efficient use of light applicable to various unstained and fluorescence cell observations, and (3) very powerful long-pulse-strobe Xenon lights and lasers for microscopes. Various auxiliary technologies to support utilization of the system are also being developed. One example of them is an efficient video trigger system, which detects a weak signal of a sudden change in a frame under ultra-high-speed imaging by canceling high-frequency fluctuation of illumination light. This paper outlines the system with its preliminary evaluation results.
The visual study of unsteady shock wave dynamics has in the past predominantly been done using single-shot images.
The advent of ultra-fast, good-resolution high-speed digital cameras has changed this state of affairs and allows the true
development of the flow to be studied. It enables the detection of weaker features which are easily overlooked in singleshot
visualizations by virtue of the fact that human vision is very sensitive to detecting the motion of an object, even if it
generates only a faint optical signal. Recent application of these devices to the study of the focusing of a shock wave in a
cylindrical cavity has identified a number of previously unknown features, while other features that previously had been
inadequately reported could be clearly identified and explained The observation of deliberately generated weak
disturbances allows the quantification of which part of the flow is influenced by which part of the boundaries
encompassing it. Whilst the imaging itself is very useful it is also highly desirable to use techniques from which
quantitative data can be obtained. Color, such as in direction- and magnitude-indicating color schlieren, and polychrome
shearing interferometry, adds an additional dimension to such investigations.
A feasibility study is presented for an image sensor capable of image capturing at 100 Mega-frames per second (Mfps). The basic structure of the sensor is the backside-illuminated ISIS, the in-situ storage image sensor, with slanted linear CCD memories, which has already achieved 1 Mfps with very high sensitivity. There are many potential technical barriers to further increase the frame rate up to 100 Mfps, such as traveling time of electrons within a pixel, Resistive-Capacitive (RC) delay in driving voltage transfer, heat generation, heavy electro-magnetic noises, etc. For each of the barriers, a countermeasure is newly proposed and the technical and practical possibility is examined mainly by simulations. The new technical proposals include a special wafer with n and p double epitaxial layers with smoothly changing doping profiles, a design method with curves, the thunderbolt bus lines, and digitalnoiseless image capturing by the ISIS with solely sinusoidal driving voltages. It is confirmed that the integration of these technologies is very promising to realize a practical image sensor with the ultra-high frame rate.
The authors applied an ultra-high-speed video camera to visualize crack propagation in brittle bodies, such as mortar
specimens, under the impact splitting test. Strain of the brittle bodies in impact splitting tests was analyzed by means of
PIV (Particle Image Velocimtery), which is usually used for measurements of flow fields with tracer particles.
The results show that, when the applied impulse on the mortar specimens is increased, the crack propagation velocity
reaches an upper bound. The upper bound of the crack propagation velocity was 2.6 km/sec. The horizontal tensile
strain around the crack tip was estimated to be 370 μ by PIV measurement with the ultra-high-speed camera, and 270 -
375 μ by strain gages, respectively. Those results showed a good agreement with each other.
This paper outlines a special microscope under development, named "Ultra-high-speed bionanoscope" for ultra-highspeed
imaging in biological applications, and preliminary design of the image sensor, which is the key component in the
system. The ultra-high-speed bionanoscope consists of two major subsystems: a video camera operating at more than 10
Mfps with ultra-high-sensitivity and the special microscope to minimize loss of light for seriously reduced illumination
light energy due to the ultra-high-speed imaging. The ultra-high-frame rate is achieved by introducing a special structure
of a CCD imager, the ISIS, In-situ Storage Image Sensor, invented by Etoh and Mutoh. The ISIS has an array of pixels
each of which equips with a slanted linear CCD storage area for more than 100 image signals for reproduction of
smoothly moving images. The ultra-high-sensitivity of the sensor of less than 10 photons is achieved by introducing
three existing technologies, backside-illumination, cooling, and the CCM, Charge Carrier Multiplication invented by
In 2001, an ultra-high-speed video camera of 1,000,000 frames per second was developed in Hydraulics Laboratory of Kinki University. The image sensor of the camera was the ISIS-V2, the In-situ Storage Image Sensor-Version 2. The camera has been applied to visualization of high-speed phenomena in various fields of science and engineering. We observed entrapment phenomena of bubbles resulting from thermal spraying of metals. Thermal spraying is used to improve solid surfaces by spraying melted metal or ceramic particles to the surfaces. One of the problems relating to the thermal spraying is entrapment of air bubbles under the metal or ceramic layers covering the solid surfaces. The bubbles decrease bonding strength of the layers made by the thermal spraying. The entrapment processes were successfully visualized by application of the ultra-high-speed video camera.
A 1 M fps video camera was applied to the observation of drop impacts onto water surfaces. It captured the detailed mechanism of sheet ejection from the contact region between the drop and the water surface, the small droplets separated from the ejected sheet and bubble entrapment by a drop impacting onto a water surface. The 103 consecutive images are enough to form a short movie which is suitable for dynamic recognition.
A high-speed video camera captures bursting phenomena of a bubble at water surface under various surface tension and kinematic viscosity conditions. Surface tension and viscosity of water are changed by adding ethanol which dissolves into water and changes the surface tension, density and kinematic viscosity of water. A technique is proposed in order to separately evaluate effects of viscosity and surface tension on the water particle generation from bubble eruptions by utilizing the peculiar characteristics of the solution.
Presented in this paper is an outline of the R and D activities on high-speed video cameras, which have been done in Kinki University since more than ten years ago, and are currently proceeded as an international cooperative project with University of Applied Sciences Osnabruck and other organizations. Extensive marketing researches have been done, (1) on user's requirements on high-speed multi-framing and video cameras by questionnaires and hearings, and (2) on current availability of the cameras of this sort by search of journals and websites. Both of them support necessity of development of a high-speed video camera of more than 1 million fps. A video camera of 4,500 fps with parallel readout was developed in 1991. A video camera with triple sensors was developed in 1996. The sensor is the same one as developed for the previous camera. The frame rate is 50 million fps for triple-framing and 4,500 fps for triple-light-wave framing, including color image capturing. Idea on a video camera of 1 million fps with an ISIS, In-situ Storage Image Sensor, was proposed in 1993 at first, and has been continuously improved. A test sensor was developed in early 2000, and successfully captured images at 62,500 fps. Currently, design of a prototype ISIS is going on, and, hopefully, will be fabricated in near future. Epoch-making cameras in history of development of high-speed video cameras by other persons are also briefly reviewed.
An improved design is presented for an ISIS, In-situ Storage Image Sensor, previously proposed by the authors for a high frame rate video camera of 1,000,000 pps. CCD channels of the sensor play dual roles for signal storage in an image capturing phase and for signal transfer in a read-out phase, which minimizes unutilized spaces on the light receptive area. The transfer direction is only vertical, which simplifies the structure of the sensor and provides better quality in reproduced images. An overwriting mechanism is built in, which facilitates synchronization of cease of the image capturing phase to the occurrence of a target event. The design is improved by coupling adjacent two CCD channels and two photodiodes, which provides wider spaces to place metal wires to increase rate of charge drive.
The ISIS, In-situ Storage Image Sensor, may achieve the frame rate higher than 1,000,000 pps. Technical targets in development of the ISIS are listed up. A layout of the ISIS is presented, which covers the major targets, by employing slanted CCD storage and amplified CMOS readout. The layout has two different sets of orthogonal axis systems: one is mechanical and the other functional. Photodiodes, CCD registers and all the gates are designed parallel to the mechanical axis systems. The squares on which pixels are placed form the functional axis system. The axis systems are inclined to each other. To reproduce a moving image, at least fifty consecutive images are necessary for ten-second replay at 5 pps. The inclined design inlays the straight CCD storage registers for more than fifty images in the photo- receptive area of the sensor. The amplified CMOS readout circuits built in all the pixels eliminate line defects in reproduced images, which are inherent to CCD image sensors. FPN (Fixed Pattern Noise) introduced by the individual amplification is easily suppressed by digital post image processing, which is commonly employed in scientific and engineering applications. The yield rate is significantly improved by the elimination of the line defects.
It costs a few million dollars to develop an innovative solid- state image sensor. Various image sensors with special useful functions have been proposed for individual scientific and engineering research purposes. However, the enormous cost hampers their development. Presented in this paper is an attempt to classify wide-range of requirements for high-speed image sensors for scientific and engineering use, and to combine them into a limited number of specific designs. This will serve to provide a rational overall development strategy and to reduce overlapping effort and cost for their development, and hopefully contribute to realization of some of the designs. This is a state-of-the-art paper on development of high-speed video cameras, including: (1) classification of advanced image capturing technologies required for scientific and engineering researches; (2) performance criteria for high-speed image capturing; (3) classification and comparison of currently available high- speed image capturing technologies; (4) user's requirements for electronic high-speed imagers, obtained by questionnaires and hearings; (5) comparison of PRIS (parallel readout image sensors) and ISIS (in-situ storage image sensors); (6) current status of fabrication technology of both CCD- and CMOS-image sensors, and (7) recommendation of prototype high-speed image sensors for urgent development.
In 1991, the authors developed a high-speed video camera with a frame rate of 4,500 pps, which was the world's fastest at that time and is currently marketed by a third party. An MCP-type image intensifier is attached directly to the image sensor to realize both very high light sensitivity and frame rate. Three intensified image sensors are combined with a cubic three-way beam splitter prism and three optical filter holders to form this high-speed video/ultra-high- speed triple-framing camera. The camera can capture fast- moving images under the following conditions: (1) 4,500-pps continuous color imaging with R, G & B filters and perfect synchronization of the three images for a full 256 X 256 X 3-pixel frame, or faster color imaging of up to 40,500 pps at a reduced resolution of 64 X 64 X 3 pixels. (2) Triple-speed continuous monochrome imaging without filters and delayed synchronization, i.e. 13,500 pps for a full 256 X 256-pixel frame or up to 121,500 pps for a reduced 64 X 64-pixel frame. (3) Ultra-fast three-frame capture at a speed of 1/50,000,000 s by delayed gating of the image intensifiers. The cubic prism is also convertible, enabling simultaneous image capture by three monochromatic light sources of different wavelengths with no loss of incident light energy. This is in contrast with the large energy loss caused by a cubic prism using metalized half-mirrors, which reduced light transmission to one quarter, coupled with the losses associated with the three optical filters, reducing the light energy incident on each individual sensor still further.
An in-situ image sensor (ISIS) capable of recording moving images at a rate of 1,000,000 pps is proposed, with the following characteristics. (1) The CCD strips for image signal storage are elongated vertically and run below several pixels, unlike existing ISIS implementations. This unidirectional charge transfer reduces the number of layers of wiring required to drive the charges. (2) The CCD strips perform the dual roles of image signal storage and readout. In the image capture phase, the electric potential of the boundary element of each CCD storage strip is kept at a constant high value, and the charge transferred to the element is drained from the image sensor. This defines the number of elements available for storage. In the readout phase, the potential of the boundary element is controlled so as to allow charge transfer downward through the remainder of the CCD strips as in usual CCD operation. The unidirectional transfer and the dual-role CCD make the proposed sensor the simplest and, therefore, the best approach to ISIS, contributing to a higher frame rate, a greater number of sequentially-stored frames, and better quality of reproduced images. In a practical design, space is required for the gate connecting each photosensor to its CCD strip. To produce an interline sensor, a photodiode can be fabricated in the knife-shaped slit between meandering CCD strips. A micro-cylindrical lens could increase the area ratio of the photodiode to around 20%.
A questionnaire was distributed to scientists and engineers all over Japan to compile information on conventional and potential application of high-speed videography and special conditions associated with using them in practice. The distribution of the required frame rate shows that existing high-speed videocameras of 103 to 104 pps cover only 30-40% of the potential usages, while existing image-converter multiframing cameras of 106 to 107 pps cover more than 90%. It is, however, clear from the authors' experiences that dynamic recognition, which is supported by videocameras but not by multiframing cameras, is an essential and very powerful tool in scientific and engineering research. New concepts to produce high-speed videocameras of 105 to 107 pps are therefore presented.
As a project of the Kinki University Joint Research Center on 1991, the authors developed a high-speed videocamera with an extremely sensitive internal MCP-type image intensifier and a 4500 pictures/sec (pps) recording speed. Here, we report on its use over the past two and a half years from the following three perspectives. (1) Past applications of the videocamera. (2) Mechanics developed after experience of the applications. (3) Further improvements in the future.
Basic designs of an ultra high-speed multiframing camera and an ultra high-speed video camera are presented. The proposed video camera has a frame rate of 3 X 107 pps, while the recordable number of frames and the linear resolution are 96 frames and 128 pixels, respectively. The 96 successive frames can be replayed at 5 pps over about 20 seconds. Although motion at 5 pps looks somewhat intermittent, the frame rate is high enough for recognition of continuous movement.