OR-PSGAN-CNN: An Occlusion-Robust Deep Learning Framework for Early Detection of Cerebral Palsy in Infants Using RGB-D Videos

Rajalekshmy K.D ¹*, E.J. Thomson Fredrik ¹

Improves early cerebral palsy detection by resolving pose occlusions through enhanced skeleton generation and feature extraction from infant movement videos.

Abstract

Background: Early detection of Cerebral Palsy (CP) is critical for timely intervention and improved developmental outcomes. While pose estimation techniques such as OpenPose have been applied to CP detection in infants, they often struggle with occlusions and recognition errors—particularly in the upper limbs—leading to incomplete or inaccurate feature data during classification. Methods: To address these challenges, this study proposes an Occlusion-Robust Pose Sequence-aware Generative Adversarial Network with Convolutional Neural Network (OR-PSGAN-CNN) model for CP detection using infant video sequences. Initially, OpenPose is applied to RGB-D videos to estimate infant skeletal joint positions. These skeletons are then augmented using a PS-GAN to generate high-quality, occlusion-resistant representations. Feature Matrices (FMs) are constructed by encoding joint coordinates, incorporating joint motion complexity and motion correlation to capture nuanced movement patterns. The resulting FMs are input into a CNN followed by a softmax classifier for CP classification. Missing values due to occlusion are substituted with zeros to preserve structural integrity in the feature space. Results: The proposed OR-PSGAN-CNN model was evaluated on three benchmark datasets—MINI-RGBD, babyPose, and MIA. It achieved classification accuracies of 93.7%, 93.3%, and 93.2% respectively, outperforming existing CP detection approaches. Conclusion: The OR-PSGAN-CNN model effectively mitigates occlusion issues in infant pose estimation and enhances CP detection accuracy. This approach holds significant potential for developing automated and reliable early diagnostic tools for motor disorders in infants, especially when full-body visibility cannot be guaranteed.

Keywords: Cerebral palsy, PS-GAN, Occlusion, Matrix encoding, Joint motion complexity, Joint motion correlation

References

Adama, D. A., Lotfi, A., & Ranson, R. (2021). A survey of vision-based transfer learning in human activity recognition. Electronics, 10(19), 2412.https://doi.org/10.3390/electronics10192412

Cao, Z., Simon, T., Wei, S. E., & Sheikh, Y. (2017). Real-time multi-person 2D pose estimation using part affinity fields. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7291-7299.https://doi.org/10.48550/arXiv.1611.08050

Devarajan, R. K., & Khader, S. S. (2023). Pose sequence-aware generative adversarial network for augmenting skeleton sequences to improve cerebral palsy detection by deep learner. International Journal of Intelligent Engineering & Systems, 16(5), 512-522.DOI:10.14569/IJACSA.2021.0120383

Farahanipad, F., Rezaei, M., Nasr, M. S., Kamangar, F., & Athitsos, V. (2022). A survey on GAN-based data augmentation for hand pose estimation problem. Technologies, 10(2), 1-13.doi.org/10.3390/technologies10020043

Gao, Q., Yao, S., Tian, Y., Zhang, C., Zhao, T., Wu, D., ... & Lu, H. (2023). Automating General Movements Assessment with quantitative deep learning to facilitate early screening of cerebral palsy. Nature Communications, 14(1), 8294.DOI:10.1038/s41467-023-44141-x

Glass, H. C., Li, Y., Gardner, M., Barkovich, A. J., Novak, I., McCulloch, C. E., & Rogers, E. E. (2021). Early identification of cerebral palsy using neonatal MRI and general movements assessment in a cohort of high-risk term neonates. Pediatric Neurology, 118, 20-25.DOI: 10.1016/j.pediatrneurol.2021.02.003

Hesse, N., Pujades, S., Black, M. J., Arens, M., Hofmann, U. G., & Schroeder, A. S. (2019). Learning and tracking the 3D body shape of freely moving infants from RGB-D sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(10), 2540-2551.DOI: 10.1109/TPAMI.2019.2917908

https://vrai.dii.univpm. it/mia-dataset

Leo, M., Bernava, G. M., Carcagnì, P., & Distante, C. (2022). Video-based automatic baby motion analysis for early neurological disorder diagnosis: state of the art and future directions. Sensors, 22(3), 866.; https://doi.org/10.3390/s22030866

Li, D., Qu, J., Tian, Z., Mou, Z., Zhang, L., & Zhang, X. (2022). Knowledge-based recurrent neural network for TCM cerebral palsy diagnosis. Evidence-Based Complementary and Alternative Medicine, 2022, 1-10. https://doi.org/10.1155/2022/7708376

Lin, Y. (2023). A deep learning algorithm-based visual strategy intervention study for children with autism spectrum disorders–extraction and detection of children’s behavioral features. European Review for Medical & Pharmacological Sciences, 27(11), 4914-4928.DOI:10.26355/eurrev_202306_32608

Migliorelli, L., Moccia, S., Pietrini, R., Carnielli, V. P., & Frontoni, E. (2020). The babyPose dataset. Data in Brief, 33, 106329.

Mushta, S. M., Khandaker, G., Power, R., & Badawi, N. (2021). Cerebral palsy in the Middle East: epidemiology, management, and quality of life. In Handbook of Healthcare in the Arab World, Cham: Springer International Publishing, pp. 2539-2572.DOI:10.1007/978-3-319-74365-3_111-1

Mushta, S. M., King, C., Goldsmith, S., Smithers-Sheedy, H., Badahdah, A. M., Rashid, H., ... & McIntyre, S. (2022). Epidemiology of cerebral palsy among children and adolescents in Arabic-speaking countries: a systematic review and meta-analysis. Brain Sciences, 12(7), 859.DOI: 10.3390/brainsci12070859

Nguyen-Thai, B., Le, V., Morgan, C., Badawi, N., Tran, T., & Venkatesh, S. (2021). A spatio-temporal attention-based model for infant movement assessment from videos. IEEE Journal of Biomedical and Health Informatics, 25(10), 3911-3920.DOI: 10.1109/JBHI.2021.3077957

Paul, S., Nahar, A., Bhagawati, M., & Kunwar, A. J. (2022). A review on recent advances of cerebral palsy. Oxidative Medicine and Cellular Longevity, 2022, 1-20.DOI: 10.1155/2022/2622310

Redd, C. B., Karunanithi, M., Boyd, R. N., & Barber, L. A. (2021). Technology-assisted quantification of movement to predict infants at high risk of motor disability: A systematic review. Research in Developmental Disabilities, 118, 104071.DOI: 10.1016/j.ridd.2021.104071

Sakkos, D., Mccay, K. D., Marcroft, C., Embleton, N. D., Chattopadhyay, S., & Ho, E. S. (2021). Identification of abnormal movements in infants: A deep neural network for body part-based prediction of cerebral palsy. IEEE Access, 9, 94281-94292.DOI:10.1109/ACCESS.2021.30934

Silva, N., Zhang, D., Kulvicius, T., Gail, A., Barreiros, C., Lindstaedt, S., ... & Marschik, P. B. (2021). The future of general movement assessment: The role of computer vision and machine learning–a scoping review. Research in Developmental Disabilities, 110, 103854.https://doi.org/10.1016/j.ridd.2021.103854

Wu, Q., Qin, P., Kuang, J., Wei, F., Li, Z., Bian, R., ... & Xu, G. (2023). A training-free infant spontaneous movement assessment method for cerebral palsy prediction based on videos. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 31, 1670-1679.DOI: 10.1109/TNSRE.2023.3255639

Wu, Q., Xu, G., Wei, F., Chen, L., & Zhang, S. (2021). RGB-D videos-based early prediction of infant cerebral palsy via general movements complexity. IEEE Access, 9, 42314-42324.DOI: 10.1109/ACCESS.2021.3066148

Wu, Q., Xu, G., Wei, F., Kuang, J., Qin, P., Li, Z., & Zhang, S. (2022). Supine infant pose estimation via single depth image. IEEE Transactions on Instrumentation and Measurement, 71, 1-11.DOI: 10.1109/TIM.2022.3178693

Zhang, H., Ho, E. S., & Shum, H. P. (2022). CP-AGCN: Pytorch-based attention informed graph convolutional network for identifying infants at risk of cerebral palsy. Software Impacts, 14, 100419.https://doi.org/10.1016/j.simpa.2022.100419

Zhang, H., Shum, H. P., & Ho, E. S. (2022). Cerebral palsy prediction with frequency attention informed graph convolutional networks. In 44th IEEE Annual International Conference of the IEEE Engineering in Medicine & Biology Society, pp. 1619-1625.DOI: 10.1109/EMBC48229.2022.9871230