Traditional techniques of microbial density estimation involve analysis of microbes from still images of water samples. However, microorganisms are non-rigid objects that swim in polluted water and are surrounded by static clutter and debris. As a result, the detection and classification of microorganisms based on their shape, size, and orientation becomes a very difficult task. In view of above, a microbial population density estimation technique based on analysis of video images of moving microbes is of considerable interest. Many of these microorganisms have unique motion characteristics and this paper proposes a technique for micro organisms classification based on the motion characteristics. Motion detection is carried out on preprocessed image sequences obtained from a microscope mounted CCD camera. A block matching technique in the frequency domain using the phase shift property of the Discrete Hartley Transform (DHT), is used to estimate the position of the microorganism in the current frame. Tracking of the various microorganisms is performed using the Interacting Motion Models (IMM). The position estimates are used as an input to the IMM tracker. The motion characteristics of the various types of micro organisms are modeled by three models namely: Zero velocity motion model is used to identify slow moving organisms, the constant velocity model defines organisms moving in near straight line trajectories, the third model is the coordinated turn model, a high maneuvering model which is used to characterize spiral and zigzag motion. Multiple microorganisms can be tracked using this technique, and is more reliable in recognition of the track characteristics than using a bank of 'non-interacting' single model-based filters. The track characteristic and the mode probabilities are unique property of a particular organism. The track data obtained from the IMM and the mixing probabilities of the various models used in the track, are used as a feature set. Principal Component Analysis (PCA) is used for feature extraction. Classification is to be carried out using the Hidden Markov Models (HMM). The performance of the proposed technique is evaluated using simulated image sequences and actual images obtained from the field.