COMPUTER SCIENCE PROJECT

Skin Care Assistant

Lightweight multimodal AI system that integrates DenseNet for visual feature extraction and a fine-tuned LLaMA 3.2 3B language model to provide image-aware dermatological guidance in a structured conversational manner.

Try it out →

PROJECT DESCRIPTION

The Skin Care Assistant combines image analysis and conversational reasoning into a single multimodal pipeline. Instead of predicting a single class label, we use a fine-tuned DenseNet-121 to extract visual features that are projected into the embedding space of the language model via a light MLP connector. This enables the system to make joint reasoning about morphological cues - texture, scaling, erythema - along with patient reported symptoms.

The assistant is now focusing on three dermatological categories: acne, eczema and psoriasis. All responses are formulated as a provisional decision support with explicit disclaimers and advice to consult a dermatologist.

KEY FEATURES

Multimodal joint diagnosis
DenseNet-121 visual features into the LLaMA embedding space, thus enabling a single reasoning pass over the image and the collected symptoms, as compared to treating the two modalities as independent signals.

Task-specific LoRA adapters
A common base model with a LoRA adapter on top for each task: symptom collection, joint diagnosis, grounded Q&A. This decouples behaviours that interfere with each other in single-adapter fine-tuning at little overhead. The orchestrator dynamically switches the adapters at runtime.

Structured symptom collection
A fine-tuned anamnesis protocol enforces a one-question-at-a-time policy and emits an explicit termination signal when collection is complete, giving the orchestrator a reliable handoff to the visual pipeline.

Retrieval-augmented follow-up
After diagnosis delivery, follow-up questions are answered through a RAG pipeline that combines dense retrieval (ChromaDB), lexical retrieval (BM25), reciprocal rank fusion, and reranking — grounding responses in a curated knowledge base rather than parametric memory.

THE TEAM

Tugsbayar Bat-ErdeneMaster of Computing (Computer Science), Curtin University

Enkhjargal TogooMaster of Computing (Artificial Intelligence), Curtin University

Rai SandaMaster of Computing (Artificial Intelligence), Curtin University

Romina LopezMaster of Computing (Artificial Intelligence), Curtin University

Santiago BoxigaMaster of Computing (Artificial Intelligence), Curtin University

Tianhao GengMaster of Computing (Computer Science), Curtin University