Developing a Machine Learning–Based System for Nutrient Profiling of Pakistani Cuisine

Background:
Chronic diseases such as diabetes and heart disease are rising in South Asia, with poor diet a major contributor. Most dietary apps are designed for Western foods and struggle with culturally diverse dishes such as Pakistani cuisine.

Objective:
This study aimed to build a machine learning-based system that can recognise Pakistani dishes, estimate portion sizes, and calculate their nutritional value.

Methods:
We developed a multi-stage automated dietary assessment pipeline for Pakistani cuisine using a multi-source dataset covering 83 dish classes across 10 categories. Following data cleaning, class balancing, and targeted augmentation, deep learning architectures were evaluated for food classification and segmentation. ResNet-50, MobileNetV2, and YOLOv8 were used for food recognition, while YOLOv8-Seg and Mask R-CNN enabled pixel-level segmentation for portion and nutrient estimation. All models were optimized using transfer learning and evaluated using standard classification and mask-quality metrics, followed by mobile application usability testing.

Results:
A multi-source dataset of ten categories and 83 Pakistani dishes was used to train and compare deep learning models for food classification and segmentation. Portion size was estimated using a segmentation- and depth-based approach without a reference object, while nutritional values were derived from standardised recipes and established food databases. The models were integrated into a mobile application for usability testing. YOLO-based models achieved the best balance of accuracy and speed for real-time classification and segmentation. Portion estimates were reasonably close to measured values, and usability testing showed the app was easy to use, with minor improvements suggested.

Conclusion:
Overall, this work demonstrates the feasibility of an AI-based dietary assessment system adapted to Pakistani cuisine, with future work focused on reducing processing demands and improving depth-based portion estimation.