Constructing tongue coating recognition model using deep transfer learning to assist syndrome diagnosis and its potential in noninvasive ethnopharmacological evaluation

Tongue coating has been used as an effective signature of health in traditional Chinese medicine (TCM). The level of greasy coating closely relates to the strength of dampness or pathogenic qi in TCM theory. Previous empirical studies and our systematic review have shown the relation between greasy coating and various diseases, including gastroenteropathy, coronary heart disease, and coronavirus disease 2019 (COVID-19). However, the objective and intelligent greasy coating and related diseases recognition methods are still lacking. The construction of the artificial intelligent tongue recognition models may provide important syndrome diagnosis and efficacy evaluation methods, and contribute to the understanding of ethnopharmacological mechanisms based on TCM theory.The present study aimed to develop an artificial intelligent model for greasy tongue coating recognition and explore its application in COVID-19.Herein, we developed greasy tongue coating recognition networks (GreasyCoatNet) using convolutional neural network technique and a relatively large (N = 1486) set of tongue images from standard devices. Tests were performed using both cross-validation procedures and a new dataset (N = 50) captured by common cameras. Besides, the accuracy and time efficiency comparisons between the GreasyCoatNet and doctors were also conducted. Finally, the model was transferred to recognize the greasy coating level of COVID-19.The overall accuracy in 3-level greasy coating classification with cross-validation was 88.8% and accuracy on new dataset was 82.0%, indicating that GreasyCoatNet can obtain robust greasy coating estimates from diverse datasets. In addition, we conducted user study to confirm that our GreasyCoatNet outperforms TCM practitioners, yet only consuming roughly 1% of doctors’ examination time. Critically, we demonstrated that GreasyCoatNet, along with transfer learning, can construct more proper classifier of COVID-19, compared to directly training classifier on patient versus control datasets. We, therefore, derived a disease-specific deep learning network by finetuning the generic GreasyCoatNet.Our framework may provide an important research paradigm for differentiating tongue characteristics, diagnosing TCM syndrome, tracking disease progression, and evaluating intervention efficacy, exhibiting its unique potential in clinical applications.