The precise stitching of microscopic images of large-scale biological sequence slices is of great significance for the study of biological structure and function, but the slight scale changes of microscopic images and the blank areas in the images seriously affect the accuracy of mosaic. In this paper, we propose a electron microscope sequence image stitching based on belief propagation algorithm, which basically solves this problem. Firstly, the relative scale of adjacent images is calculated by extracting the sift feature points of the images. Then the global optimization model is used to obtain the absolute scale of each image, and the image is scaled to obtain the microscopic image with consistent scale. Secondly, obtain the relative displacement relationship of adjacent images by template matching method, and then the global positions of all images are optimized by Belief Propagation (BP) algorithm to eliminate the influence of blank regions and repetitive structures on the stitching results. In the case study, the proposed method demonstrates high quality.
Registration of electron microscopy (EM) images is one of the most important steps in reconstructing neurons. Image registration algorithm based on SIFT have been widely used in the EM image registration. But SIFT matching procedure both costs a lot of time and introduce massive false matches. In this paper, we propose an improved EM image registration method using the scale information of SIFT keypoints. In the feature matching procedure, our method saves up to 45.8% of the computation time compared to SIFT. We also added a preprocessing procedure for RANSAC to eliminate false matches in small-scale matches sets. Experimental results show that the method improves the accuracy of results on every test EM image set while highly reducing the registration time.
Imaging serial sections in electron microcopy (EM) is an important volume EM approach for neuronal circuit reconstruction, which has advantages of larger imaging volume and non-destructive for tissue sections. However, the continuity between sections is destroyed when the tissue block is cut into sections physically, and sections suffer stretching, folding and distorting individually during section preparation and imaging. As a result, image registration is a challenging task to recover the continuity of the neurite. The traditional methods use the SIFT or block matching method to extract landmarks between the adjacent sections, which is doubtful when the neurite direction is not perpendicular to the section plane. To get round the difficulty of reliable landmark extraction, we propose a skeleton-based image registration method for serial EM sections of the nerve tissue. The virtual skeletons are traced across the sections after an initial approximate rigid alignment. Then we make assumption that the skeleton shape is smooth adequately in z direction. In company with the constraints that the displacements of the skeleton points in the same section are smooth and small, an energy function is proposed to calculate the new positions of the skeleton points for all of the sections. Finally, the sections are warped according to the adjusted positions of skeleton points. The proposed method is highly automatic and could recover the 3D continuity of the neurite. We demonstrate that our method outperforms the state-of-the-art methods on serial EM sections including a synthetic test case.
Electron microscope image stitching is highly desired to acquire microscopic resolution images of large target scenes in neuroscience. However, the result of multiple Mosaicked electron microscope images may exist severe gray scale inhomogeneity due to the instability of the electron microscope system and registration errors, which degrade the visual effect of the mosaicked EM images and aggravate the difficulty of follow-up treatment, such as automatic object recognition. Consequently, the grayscale correction method for multiple mosaicked electron microscope images is indispensable in these areas. Different from most previous grayscale correction methods, this paper designs a grayscale correction process for multiple EM images which tackles the difficulty of the multiple images monochrome correction and achieves the consistency of grayscale in the overlap regions. We adjust overall grayscale of the mosaicked images with the location and grayscale information of manual selected seed images, and then fuse local overlap regions between adjacent images using Poisson image editing. Experimental result demonstrates the effectiveness of our proposed method.
At present, deep learning is widely used and has achieved excellent results in many fields except in the field of image registration, the reasons are two-fold: Firstly all the steps of deep learning should be derivable; nevertheless, the nonlinear deformation which is usually used in registration algorithms is hard to be depicted by explicit function. Secondly, success of deep learning is based on a large amount of labeled data, this is problematic for the application in real scenes. To address these concerns, we propose an unsupervised network for image registration. In order to integrate registration process into deep learning, image deformation is achieved by resampling, which can make deformation step derivable. The network optimizes its parameters directly by minimizing the loss between registered image and reference image without ground truth. To further improve algorithm's accuracy and speed, we incorporate coarse-to-fine multi-scale iterative scheme. We apply our method to register microscopic section images of neuron tissue. Compared with highly fine-tuning method sift flow, our method achieves similar accuracy with much less time.
Proc. SPIE. 10574, Medical Imaging 2018: Image Processing
KEYWORDS: Image processing algorithms and systems, Image fusion, Cancer, Convolutional neural networks, Data modeling, Image segmentation, Electron microscopes, Scanning electron microscopy, Network architectures, Fusion energy
Recent studies have empowered that the relation between mitochondrial function and degenerative disorder- s is related to aging diseases. Due to the rapid development of electron microscope (EM), stacks delivered by EM can be used to investigate the link between mitochondrial function and physical structure. Whereas, one of the main challenges in mitochondria research is developing suitable segmentation algorithms to obtain the shapes of mitochondria. Nowadays, Deep Neural Network (DNN) has been widely applied in solving the neuron membrane segmentation problems in virtue of its exceptional performance. For this reason, its appli- cation to mitochondria segmentation holds great promise. In this paper, we propose an effective deep learning approach to realize mitochondria segmentation in Automated Tape-Collecting Ultra Microtome Scanning Elec- tron Microscopy (ATUM-SEM) stacks. The proposed algorithm contains three parts: (1) utilizing histogram equalization algorithm as image preprocessing to keep the consistency of dataset; (2) putting forward a fusion fully convolution network (FCN), which is motivated by the principle the deeper, the better, to build a much deeper network architecture for more accurate mitochondria segmentation; and (3) employing fully connected conditional random field (CRF) to optimize segmentation results. Evaluation was performed on a dataset of a stack of 31 slices from ATUM-SEM, with 20 images used for training and 11 images for testing. For comparison, U-Net approach was evaluated through the same dataset. Jaccard index between the automated segmentation and expert manual segmentations indicates that our method (90.1%) outperforms U-Net (87.9%) and has a preferable performance on mitochondria segmentation with different shapes and sizes.
Recently, due to the rapid development of electron microscope (EM) with its high resolution, stacks delivered by EM can be used to analyze a variety of components that are critical to understand brain function. Since synaptic study is essential in neurobiology and can be analyzed by EM stacks, the automated routines for reconstruction of synapses based on EM Images can become a very useful tool for analyzing large volumes of brain tissue and providing the ability to understand the mechanism of brain. In this article, we propose a novel automated method to realize 3D reconstruction of synapses for Automated Tapecollecting Ultra Microtome Scanning Electron Microscopy (ATUM-SEM) with deep learning. Being different from other reconstruction algorithms, which employ classifier to segment synaptic clefts directly. We utilize deep learning method and segmentation algorithm to obtain synaptic clefts as well as promote the accuracy of reconstruction. The proposed method contains five parts: (1) using modified Moving Least Square (MLS) deformation algorithm and Scale Invariant Feature Transform (SIFT) features to register adjacent sections, (2) adopting Faster Region Convolutional Neural Networks (Faster R-CNN) algorithm to detect synapses, (3) utilizing screening method which takes context cues of synapses into consideration to reduce the false positive rate, (4) combining a practical morphology algorithm with a suitable fitting function to segment synaptic clefts and optimize the shape of them, (5) applying the plugin in FIJI to show the final 3D visualization of synapses. Experimental results on ATUM-SEM images demonstrate the effectiveness of our proposed method.
Microscopic image registration is a key component of the neural structure reconstruction with serial sections of neural
tissue. The goal of microscopic neural image registration is to recover the 3D continuity and geometrical properties of
specimen. During image registration, various distortions need to be corrected, including image rotation, translation,
tissue deformation et.al, which come from the procedure of sample cutting, staining and imaging. Furthermore, there is
only certain similarity between adjacent sections, and the degree of similarity depends on local structure of the tissue and
the thickness of the sections. These factors make the microscopic neural image registration a challenging problem.
To tackle the difficulty of corresponding landmarks extraction, we introduce a novel image registration method for
Scanning Electron Microscopy (SEM) images of serial neural tissue sections based on the structure of mitochondria. The
ellipsoidal shape of mitochondria ensures that the same mitochondria has similar shape between adjacent sections, and its
characteristic of broad distribution in the neural tissue guarantees that landmarks based on the mitochondria distributed
widely in the image. The proposed image registration method contains three parts: landmarks extraction between
adjacent sections, corresponding landmarks matching and image deformation based on the correspondences. We
demonstrate the performance of our method with SEM images of drosophila brain.
Extracting the structure of single neurons is critical for understanding how they function within the neural circuits.
Recent developments in microscopy techniques, and the widely recognized need for openness and standardization
provide a community resource for automated reconstruction of dendritic and axonal morphology of single neurons. In
order to look into the fine structure of neurons, we use the Automated Tape-collecting Ultra Microtome Scanning
Electron Microscopy (ATUM-SEM) to get images sequence of serial sections of animal brain tissue that densely
packed with neurons. Different from other neuron reconstruction method, we propose a method that enhances the SEM
images by detecting the neuronal membranes with deep convolutional neural network (DCNN) and segments single
neurons by active contour with group shape similarity. We joint the segmentation and tracing together and they interact
with each other by alternate iteration that tracing aids the selection of candidate region patch for active contour
segmentation while the segmentation provides the neuron geometrical features which improve the robustness of tracing.
The tracing model mainly relies on the neuron geometrical features and is updated after neuron being segmented on
the every next section. Our method enables the reconstruction of neurons of the drosophila mushroom body which is
cut to serial sections and imaged under SEM. Our method provides an elementary step for the whole reconstruction of
Activity-dependent changes in the synaptic connections of the brain are tightly related to learning and memory. Previous studies have shown that essentially all new synaptic contacts were made by adding new partners to existing synaptic elements. To further explore synaptic dynamics in specific pathways, concurrent imaging of pre and postsynaptic structures in identified connections is required. Consequently, considerable attention has been paid for the automated detection of axonal boutons. Different from most previous methods proposed in vitro data, this paper considers a more practical case in vivo neuron images which can provide real time information and direct observation of the dynamics of a disease process in mouse. Additionally, we present an automated approach for detecting axonal boutons by starting with deconvolving the original images, then thresholding the enhanced images, and reserving the regions fulfilling a series of criteria. Experimental result in vivo two-photon imaging of mouse demonstrates the effectiveness of our proposed method.
In this paper we propose an image super-resolution algorithm based on Gaussian Mixture Model (GMM) and a new
adaptive image decomposition algorithm. The new image decomposition algorithm uses local extreme of image to
extract the cartoon and oscillating part of image. In this paper, we first decompose an image into oscillating and
piecewise smooth (cartoon) parts, then enlarge the cartoon part with interpolation. Because GMM accurately
characterizes the oscillating part, we specify it as the prior distribution and then formulate the image super-resolution
problem as a constrained optimization problem to acquire the enlarged texture part and finally we obtain a fine result.
This paper presents a wavelet-domain Hidden Markov Tree(HMT)-based color image superresolution algorithm using multi-channel data fusion. Because there exists correlations among the three channels of a RGB color image, a channel by channel superresolution method almost certain leads to color distortion. In order to solve this problem, first the low-resolution color image is converted into a gray-scale image using the spatially-adaptive approach presented in this paper and the resulting gray-scale image must reflect the human perception of edges in the color image; then by superresolving this gray-scale image, a high-resolution image is obtained; finally, wavelet-domain HMT-based image superresolutions are performed for the three channels of the low-resolution color image using the same posterior state probabilities, which reflect the hidden states of the wavelet coefficients of the high-resolution gray-scale image obtained before, and thus the resulting high-resolution color image is what we desired. Becasue the correlations among the three channels of a RGB color image are considered, there are no color distortions in the reconstructed high-resolution image. Experimental results show that the reconstructed color images have high PSNR and are of high visual quality.