Phased Array Ultrasonic Testing (PA-UT) has been proved to be the most feasible way to inspect defects in electrofusion (EF) joints of polyethylene (PE) pipes. The recognition of defects in PA-UT results relies on the experience of operators, and results in inconsistent defective detection rate and low recognition speed. In this paper, an automatic defect recognition model for PA-UT inspection images was proposed based on convolutional neural network (CNN), realizing recognition of four typical defects in EF joint, i.e. wire dislocation, holes, lack of fusion and cold welding. The proposed recognition model consisted of anomaly detection model and defect detection model. The anomaly detection model recognized anomalies of PA -UT inspection images. The Defect detection model classified and located the defects of abnormal PA -UT inspection images. The anomaly detection model was assigned with a large anchor scale, to realize fast recognition, so as to meet the requirement of real-time recognition practical inspection. And defect detection model was designed to achieve high accuracy of defects recognition. By optimizing parameters of learning rate and dataset augmentation, the anomaly detection model and the defect detection reached a good combination of robustness and accuracy. Experiments showed that the proposed recognition model can improve the recognition speed and accuracy compared to single defect detection model.