Unsupported FBX Format: ByPolygonVertex/Direct (and possible fix)

[olichose123]

Alternative title: Insomnia-fueled exploration of hxd.fmt.fbx

The problem

I came upon a bug with a specific FBX setup. Blockbench (a low-poly 3D modelling software) outputs an FBX model where LayerElementUV has MappingInformationType as ByPolygonVertex and ReferenceInformationType as Direct, which means there are no indices. Compare this to Blender's FBX format which outputs as ByPolygonVertex/IndexToDirect, with indices mapping polygons to UV coordinates.

The function hxd.fmt.fbx.Geometry.hx:getUVs considers the absence of indices as a Maya quirk and generates a list of indices with the exact same values as the geometry's PolygonVertexIndex, while it should, in our case, generate a list of continuous integers.

The current method:

public function getUVs() {
	var uvs = [];
	for( v in root.getAll("LayerElementUV") ) {
		var index = v.get("UVIndex", true);
		var values = v.get("UV").getFloats();
		var index = if( index == null ) {
			// ByVertice/Direct (Maya sometimes...)
			[for( i in getPolygons() ) if( i < 0 ) -i - 1 else i];
		} else index.getInts();
		uvs.push({ values : values, index : index });
	}
	return uvs;
}

My FBX's Geometry:

Vertices: *24 {
	a: 400,800,400,400,800,-400,400,0,400,400,0,-400,-400,800,400,-400,800,-400,-400,0,400,-400,0,-400
}
PolygonVertexIndex: *24 {
	a: 1,0,2,-4,6,4,5,-8,4,0,1,-6,3,2,6,-8,2,0,4,-7,5,1,3,-8
}
LayerElementUV: 0 {
	Version: 101
	Name: ""
	MappingInformationType: "ByPolygonVertex"
	ReferenceInformationType: "Direct"
	UV: *48 {
		a: 0.25,1,0,1,0,0.75,0.25,0.75,0.25,0.46875,0.25,0.71875,0,0.71875,0,0.46875,0.28125,0.75,0.53125,0.75,0.53125,1,0.28125,1,0.53125,0.46875,0.53125,0.71875,0.28125,0.71875,0.28125,0.46875,0.25,0.1875,0.25,0.4375,0,0.4375,0,0.1875,0.8125,1,0.5625,1,0.5625,0.75,0.8125,0.75
			}
		}

in a nutshell

• 8 vertices (24 values)
• 6 quads (24 values)
• 24 UV coordinates (48 values)

Expected output of "index" in getUVs' return output:
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23]: length 24
Where 6 quads are represented in 24 UV coordinates.
(each four UV indices represent a quad)

Current output:
[for( i in getPolygons() ) if( i < 0 ) -i - 1 else i];
or
[1,0,2,3,6,4,5,7,4,0,1,5,3,2,6,7,2,0,4,6,5,1,3,-8] length 24
Which is the same value as PolygonVertexIndex, 1,0,2,-4,6,4,5,-8,4,0,1,-6,3,2,6,-8,2,0,4,-7,5,1,3,-8.

A possible solution

Here's an edited getUVs that would still support Maya and also support ByPolygonVertex/Direct. A cleaner approach would be to use enums, but it's important to note that for the FBX format both ByVertex and ByVertice are valid keywords.

public function getUVs() {
	var uvs = [];
	var mapping_type = root.get("LayerElementUV.MappingInformationType").props[0].toString();
	for( v in root.getAll("LayerElementUV") ) {
		var index = v.get("UVIndex", true);
		var values = v.get("UV").getFloats();
		var indices = null;
		// Ref info type is Direct instead of IndexToDirect
		if (index == null)
		{
			if (mapping_type == "ByVertex" || mapping_type == "ByVertice")
			{
				// Original Maya approach
				indices = [for( i in getPolygons() ) if( i < 0 ) -i - 1 else i];
			}
			else if (mapping_type == "ByPolygonVertex")
			{
				// Blockbench approach
				indices = [for(i in 0...getPolygons().length) i];
			}
		}
		else {
			indices = index.getInts();
		}
		uvs.push({ values : values, index : indices });
	}
	return uvs;
}

Note that I could not test a Maya FBX, but the code for it should not have changed.

[olichose123]

I believe normals have a similar but not quite the same problem. Blockbench maps normals ByPolygon, but heaps support ByVertex/ByVertice (Maya?) and ByPolygonVertex (Blender?) only. The math is a little different there as there doesn't seem to be a separate index array.

See hxd.fmt.fbx.Geometry.hx:processVectors

[olichose123]

Ok, I can confirm there was something similar happening with normals.

Heaps expect normals to be in a ByPolygonVertex/Direct format without indices. Blockbench outputs normals as ByPolygon/Direct. As, unlike UVs, heaps does not generate missing indices for normals, we need to convert ByPolygon-based normals into ByPolygonVertex-based ones.

We must first establish if we are dealing with triangles or quads. Then, we simply multiply our normal count by 3 or 4 for each polygon.

Here is an add-on to hxd.fmt.fbx.Geometry.hx:processVectors to showcase my approach.

else if( root.get(layer+".MappingInformationType").props[0].toString() == "ByPolygon" ) {
	var nout = [];
	var vertice_indices = getPolygons(); // get vertex indices
	var polygon_size = 0;
	for (i in 0...vertice_indices.length) {
		if (vertice_indices[i] < 0) {
			polygon_size = i+1; // determine if we're dealing with triangles or quads
			break;
		}
	}

	for (i in 0...vertice_indices.length) // for each vertex
	{
		for (j in 0...polygon_size) // set our normal coordinates X times (tri or quad)
		{
			nout.push(nrm[i * 3]);
			nout.push(nrm[i * 3 + 1]);
			nout.push(nrm[i * 3 + 2]);
		}

	}
	nrm = nout;
}

Here's a "before and after".

Some problems:

• Right now this solution edits "processVectors", which is used by other parts of the FBX parser.
• my variable names are not in line with the file's standards
• I need someone to validate the maths and test other format standards

Those are the main reasons why I haven't made a PR out of this yet.