Automatic CNN-based 3D/2D non-rigid registration platform for fast 3D femur reconstruction and clinical 3D measurements from Bi-planar radiographs

Purpose: This paper presents an automatic 3D/2D non-rigid registration method for fast 3D reconstruction and clinical measurements of the femur. Approach: The proposed CNN cascade-based 3D/2D registration platform comprises three major steps to fit a generic 3D femur model into 2D bi-planar EOS® radiographs: 1) Pose estimation (CNN_Pose)- a combination of Principal Component Analysis (PCA) and CNN-based 3D/2D similarity registration; 2) 3D shape deformation (CNN_Shape)- a CNN-based 3D displacement estimation of handles followed by Moving Least Square (MLS) shape deformation to extend an as-rigid-as-possible deformation to the entire bone, 3) 3D scale deformation (CNN_Scale)- a CNN-based 3D scale ratio estimation of handles followed by MLS-based model rescaling. Results: The accuracy of the method is evaluated in comparison to, first, a clinically proved semi-automatic method on 15 patients, and second, Computerized Tomography CT scans of five new patients. In the first validation, the mean ± standard deviation (STD) of the Root Mean Square of point-to-surface distance (RMS-P2S) error is 0.88± 0.29 mm. For the second validation, the mean± STD of RMS-P2S error is 2.70± 0.39 mm. Four clinical measurements of the reconstructed 3D femurs are computed and compared with the first validation set. For each clinical measurement, the Mean Absolute Errors (MAE) is below 1 mm or 1°. Conclusions: The presented automatic CNN cascade-based framework efficiently registers the generic 3D femur models into bi-planar radiographs. The CNN-based 3D handles displacement and scale estimation eliminates manual-annotations and user-interventions for MLS deformation while maintaining accuracy and speed. This system is applicable for other bones such as the tibia.