This paper investigates a new robust optimization framework based on density-convex reliability model and applies it to the dimensional optimization of magnetic resonance (MR) compatible surgical robot. As a justified tool for assessing reliability, the density-convex model is proposed on account of the reality that available data information is always insufficient. Based on the density-convex model, reliability functions of structure are constructed and taken as constraint conditions. The Euclidean norm of the sensitivity Jacobian matrix is selected as robust index and stated as the ultimate objective function. By using finite element method and artificial neural network (FEM–ANN) method, the explicit functions of mechanical response are achieved effectively. The optimization is solved by a gradient-based optimization algorithm in the framework. As an application of the above optimization framework, a prototype robot is designed and manufactured. Finally, a test experiment verifies the high reliability of the robot and further proves the validity and effectiveness of this proposed method.