NeurIPS 2026
Image-Conditioned UV Texture Generation with Surface-Aware Positional Encoding
The idea
Below, a mesh is unwrapped into four UV islands. Pick a texel next to a seam: its neighbours on the surface end up on a different island, at the other end of the UV map. Whether attention can still relate them depends on how position is measured.
Standard DiT
surface neighbours it can attend to
UV map position = grid index
On the mesh
Ours
surface neighbours it can attend to
UV map position = 3D surface coordinate
On the mesh
Move over any view to place the texel yourself. Schematic illustration on a torus, not model output: neighbours are the tokens within a fixed radius, measured in each encoding's own coordinates.
A five-minute narrated walkthrough of the problem, the method and the results.
Generating high-quality UV textures for 3D meshes remains challenging. Multi-view projection pipelines suffer from occlusion and view inconsistency, and recent methods that generate textures directly in UV space still rely on auxiliary modules to supply 3D information, leaving the attention mechanism tied to UV-grid positions rather than to the underlying surface geometry. This mismatch limits coherence across seams and disconnected UV islands.
We propose DirectUV, an image-conditioned UV texture diffusion framework that operates in the latent UV space of a pretrained image VAE, in which a Diffusion Transformer denoises the UV latent given a single input image and a coarse UV map. At its core, Surface-Aware Positional Encoding (SAPE) replaces the standard 2D-grid positional encoding with encodings derived from per-token 3D surface coordinates obtained via UV-to-surface correspondence. As positional encodings define the distance metric that attention operates on, SAPE enables tokens to attend to each other based on true surface proximity rather than UV-grid distance, restoring coherence across seams and disconnected islands. A multi-level extension further assigns different attention heads to progressively finer subdivisions of the same latent UV patch, allowing the model to reason about surface structure at multiple granularities.
Experiments show that DirectUV produces sharper and more globally consistent textures than other baselines, with the largest improvements in occluded and view-unseen regions where projection-based methods leave gaps or stretched textures.
Positional encodings define the distance metric that attention is compelled to operate on. So instead of handing the model 3D geometry as one more condition it may or may not follow, DirectUV writes the surface into the positional encoding itself.
The UV map is the sole generation target, denoised in the latent space of a frozen Flux VAE. The reference image and the coarse UV map are conditioning signals only. Nothing is projected and then refined, so view-dependent artifacts have no path into the output.
SAPE reads each token's 3D surface coordinate from the CCM and uses it as the RoPE position. The query–key dot product then depends only on the offset between two points on the surface, so texels split across a seam attend to each other directly.
Multi-Level SAPE subdivides the same latent patch into 1×1, 2×2 and 4×4 cells and gives each attention head exactly one cell. Coarse correspondence and sub-patch detail coexist on disjoint head slices, with no added parameters or modules.
All methods receive the same mesh and a single front-facing reference image. Renderings use identical lighting.
The full GSO set is disjoint from the training data, so it serves as an out-of-distribution test. Each textured mesh is rendered under albedo shading from viewpoints with full directional coverage, including top-down and bottom-up, and compared against ground truth.
Bold best · underline second best
| Method | PSNR ↑ | SSIM ↑ | FID ↓ | KID ↓ |
|---|---|---|---|---|
| TEXGen | 24.45 | 0.9417 | 44.09 | 34.32 |
| FlexPainter | 23.21 | 0.9463 | 47.04 | 39.25 |
| Hunyuan3D-2.1 | 24.74 | 0.9450 | 39.872 | 33.34 |
| UniTEX | 24.23 | 0.9504 | 42.11 | 39.21 |
| DirectUV | 25.13 | 0.9527 | 39.17 | 30.03 |
What each part of the positional encoding buys, with the rest of the model held fixed.
| Variant | PSNR ↑ | SSIM ↑ | FID ↓ | KID ↓ |
|---|---|---|---|---|
| w/o SAPE | 23.82 | 0.9376 | 58.42 | 72.36 |
| SAPE, single level | 24.36 | 0.9462 | 40.86 | 36.74 |
| Multi-Level SAPE | 25.13 | 0.9527 | 39.17 | 30.03 |
Behaviour beyond the benchmark: imperfect conditioning, missing views, and meshes that come out of shape generators.
Here the coarse UV map paints the underside of the keyboard white. The reference image and the surrounding surface point to a continuous dark metal casing, and the final texture follows them rather than copying the coarse values.
During training the number of views behind the coarse UV map is sampled from {0, 1, 2, 4}. At inference the model stays plausible across all of them, including a completely blank coarse UV map, where it relies on the reference image and the surface-aware denoising alone.
Paired with shape generators such as Tripo and Rodin, DirectUV completes an image-to-textured-mesh pipeline: the generator produces the mesh, and DirectUV textures it from the same image, across the varied topologies and UV layouts these models output.
Additional results across object categories and mesh complexities.
DirectUV depends on the quality of the UV parameterization. When a layout is shattered into many small slivers and corner patches, each token's surface coordinate averages over a large region per fragment, the CCM signal turns coarse, and the multi-level subdivision cannot fully recover local detail. The result is blur and colour leakage in those regions. UV-aware preprocessing or a tokenization learned for UV-space data are natural next steps.
If DirectUV is useful in your work, please cite the paper.
@article{lin2026directuv,
title = {DirectUV: Image-Conditioned UV Texture Generation
with Surface-Aware Positional Encoding},
author = {Lin, Jiantao and Xu, Yingjie and Sheng, Mingzhi and
Wei, Yangkai and Chen, Hao and Chen, Ying-Cong},
journal = {arXiv preprint arXiv:2609.34651},
year = {2026}
}