In order to fully describe the texture informations of the image, and to distinguish the sub-regions which contain different texture informations, this paper proposes a method of facial expression recognition based on adaptively weighted improved Local Binary Pattern (LBP). Firstly, the whole face region and expression sub-regions of eyebrows, eyes, nose and mouth are isolated by preprocessing. Secondly, the features of the sub-regions are extracted by improved LBP, the Fisher Linear Discriminant (FLD) is applied to calculated the weights of sub-regions, and then the weighted histograms of expression sub-regions are fused as the histogram of facial expression feature. Finally, the fused features are classified by Support Vector Machine (SVM). The experiments are performed on JAFFE and Extended Cohn-Kanada database(CK+), and the experimental results demonstrate that the proposed method has better recognition performance.