Annular parts are widely utilized in high-end manufacturing, which are often manufactured by ring forging to meet the requirement on mechanical properties. However, residual stress (RS) is inevitably introduced in the forging process, which affects the working performance and reduces the service life of the part. Therefore, to counter the determent of RS, it is first necessary to accurately obtain it. Different from sheet or structural parts, the RS distribution in an annular part is more complex. Existing methods based on RS measurement and finite element prediction did not incorporate the RS prior, so the accuracy is influenced by the simplified assumption on RS distribution. This paper proposes a new RS inferencing method for annular parts based on the use of deformation force and physical prior. Firstly, the RS distribution prior of the annular blank after heat treatment is obtained by finite element simulation and the clustering algorithm. Then, the prior is employed to calculate the coefficient matrix to reflect the relationship between the deformation force and the RS. Finally, the Tikhonov regularization method is used to solve the deformation force-RS inverse problem. The effectiveness and feasibility of the proposed method are verified in the simulation and experiment, respectively, which shows that the integration of RS prior distribution is able to reduce the ill-condition of the solving process to obtain an accurate solution of RS distribution in an annular part.