ETBHD‐HMF: A Hierarchical Multimodal Fusion Architecture for Enhanced Text‐Based Hair Design

Simple item page
dc.contributor.authorHe, Rongen_US
dc.contributor.authorJiao, Geen_US
dc.contributor.authorLi, Chenen_US
dc.contributor.editorAlliez, Pierreen_US
dc.contributor.editorWimmer, Michaelen_US
dc.date.accessioned2024-12-19T11:15:22Z
dc.date.available2024-12-19T11:15:22Z
dc.date.issued2024
dc.description.abstractText‐based hair design (TBHD) represents an innovative approach that utilizes text instructions for crafting hairstyle and colour, renowned for its flexibility and scalability. However, enhancing TBHD algorithms to improve generation quality and editing accuracy remains a current research difficulty. One important reason is that existing models fall short in alignment and fusion designs. Therefore, we propose a new layered multimodal fusion network called ETBHD‐HMF, which decouples the input image and hair text information into layered hair colour and hairstyle representations. Within this network, the channel enhancement separation (CES) module is proposed to enhance important signals and suppress noise for text representation obtained from CLIP, thus improving generation quality. Based on this, we develop the weighted mapping fusion (WMF) sub‐networks for hair colour and hairstyle. This sub‐network applies the mapper operations to input image and text representations, acquiring joint information. The WMF then selectively merges image representation and joint information from various style layers using weighted operations, ultimately achieving fine‐grained hairstyle designs. Additionally, to enhance editing accuracy and quality, we design a modality alignment loss to refine and optimize the information transmission and integration of the network. The experimental results of applying the network to the CelebA‐HQ dataset demonstrate that our proposed model exhibits superior overall performance in terms of generation quality, visual realism, and editing accuracy. ETBHD‐HMF (27.8 PSNR, 0.864 IDS) outperformed HairCLIP (26.9 PSNR, 0.828 IDS), with a 3% higher PSNR and a 4% higher IDS.en_US
dc.description.number6
dc.description.sectionheadersORIGINAL ARTICLES
dc.description.seriesinformationComputer Graphics Forum
dc.description.volume43
dc.identifier.doi10.1111/cgf.15194
dc.identifier.pages15 pages
dc.identifier.urihttps://doi.org/10.1111/cgf.15194
dc.identifier.urihttps://diglib.eg.org/handle/10.1111/cgf15194
dc.publisher© 2024 Eurographics ‐ The European Association for Computer Graphics and John Wiley & Sons Ltd.en_US
dc.subjectImage processing
dc.subjectHair editing
dc.subjectMultimodal interaction
dc.subjectEditing capabilities
dc.titleETBHD‐HMF: A Hierarchical Multimodal Fusion Architecture for Enhanced Text‐Based Hair Designen_US
Files
Original bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
10_cgf15194.pdf
Size:
15.01 MB
Format:
Adobe Portable Document Format
Download
Collections
43-Issue 6