Models for Hair Parameter Prediction

by Juli Huang (models) and Nick Rui (datasets)

Overview

In video games and virtual 3D environments, replicating realistic hair—especially curly hair—remains a significant challenge. Traditional rendering techniques often fall short, leading to inaccuracies that can misrepresent certain ethnic minority groups. This project leverages Convolutional Neural Networks (CNNs) to predict detailed hair parameters, enabling more accurate rendering of diverse hair types and curl patterns in virtual settings.

Motivation

• Realism in Virtual Environments:
  Realistic hair rendering is key to immersive gameplay and believable digital avatars. Current methods struggle with the complexity of curly hair, often resulting in generic or stereotyped representations.

• Inclusivity and Representation:
  By improving hair parameter prediction, we aim to ensure that virtual representations honor the diversity of hair types, particularly those of ethnic minority groups. This contributes to fairer and more inclusive digital content.

• Technical Innovation:
  This project explores advanced CNN architectures and techniques to address a nuanced problem in computer graphics and AI, bridging the gap between deep learning and realistic hair simulation.

Problem Statement

Rendering hair, especially curly hair, in 3D environments poses unique challenges due to its complex structure and variability. Traditional methods may fail to capture subtle details like curl tightness, frizz, and flow, leading to:

• Inaccurate representations that detract from user immersion.
• Misrepresentation of hair characteristics in ethnic minority groups.
• Increased workload for artists trying to manually correct or enhance generated hair models.

Approach

Our solution is to employ CNN models to predict hair parameters from input images or 3D scans. Key elements of our approach include:

Data Collection & Preprocessing

• Gather diverse datasets covering a wide range of hair types and curl patterns.
• Annotate the data with key hair parameters (e.g., curl type, density, direction, frizz).
• Preprocess images to enhance features relevant for prediction.

CNN Architecture

• Design and train convolutional neural networks tailored for hair parameter extraction.
• Experiment with various architectures (e.g., ResNet, U-Net variations) to optimize performance.
• Incorporate data augmentation techniques to improve generalization across different hair types.

SAM MODEL

SAM segmentation model but I tuned/improved it: -resizing, GPU, etc, changed the device assignment to use GPU if available. Ensured the image resizing function is efficient and does not introduce unnecessary overhead Optimized the mask writing process to reduce I/O overhead. Reran on our curly hair kaggle datasets

Training & Evaluation

• Use loss functions that balance parameter accuracy and visual quality.
• Validate models using both quantitative metrics and qualitative assessments.
• Iteratively refine the model based on performance feedback.

Results

While work is in progress, early experiments have shown promising results in capturing key hair attributes. Our models are able to:

• Differentiate between various curl types with a high degree of accuracy.
• Generalize well across diverse datasets, reducing bias in hair representation.
• Provide a robust foundation for integrating into rendering pipelines for real-time applications.

Getting Started

Prerequisites

• Python 3.8 or higher
• Deep Learning Framework: TensorFlow or PyTorch
• Common libraries: NumPy, OpenCV, Matplotlib, scikit-learn

Installation

1. Clone the repository:

   git clone https://github.com/yourusername/cnn-hair-parameter-prediction.git
   cd cnn-hair-parameter-prediction
   

Create and Activate a Virtual Environment, then install requirements

python -m venv venv
source venv/bin/activate  # On Windows, use `venv\Scripts\activate`

pip install -r requirements.txt