Where AI 3D generation stops working

Ironclads, sludge barges, gyrocopters, steam tanks, battering rams and trade carts all became 3D models in a single pass, generated from their sprites and shipped with nothing touching the geometry in between. Seventy four of them in June, in one run.

The foot soldiers did not go that way, and the most useful thing the batch generator does is refuse to try. Point it at a brawler and it stops with a message saying that anything with legs needs a skeleton and animations and belongs on a different route, and that this one handles siege engines, ships, aircraft and carts only.

Three words in that sentence are doing all the work, so here they are plainly.

Image-to-3D is a service that takes one picture of an object and returns a solid 3D model of it: a mesh of triangles with a texture wrapped around it, in a file a game engine can load. You send an image, wait a few minutes, download a model. This project uses Meshy, which has both an HTTP API for batches and a web interface for driving one job at a time by hand. Both routes appear below, and which one a unit takes is the whole subject of this post.

A skeleton, or rig, is the invisible hierarchy of joints inside a character: a hip that a thigh hangs off, a thigh that a shin hangs off, and so on down. Nothing about it is visible in a render. It exists so that the mesh can be told to bend, because a model without one is a statue.

Rigging is the act of placing those joints inside a particular mesh and deciding how much each one pulls on the surface around it. It is the step that turns a statue into something that can walk, and it is the wall this post is named after.

That refusal is the honest shape of image-to-3D for a game with 229 unit types. For one class of asset the machine does the whole job. For the other it does about two thirds and hands the rest back to Adam, who asked for 3D units in the same breath as the buildings, in plain language, with no idea that the request contained two completely different jobs. Finding the line between them, and then writing it down where a script would read it rather than where it had to be remembered, was most of the work.

Sort your assets before you spend anything on generation#

One question predicts the outcome better than subject, complexity or polygon budget: does this thing need a skeleton and named movements, or is it one rigid body?

A rigid asset has no wrong answer available to it. There is no resting pose to get wrong, no soft weighting in the wrong place, and no movement that can be called a walk while actually being an attack swing. What you get is a mesh and a texture, and a mesh either has the silhouette you asked for or it does not, which you can see in one look.

Sort the list by that one question before a single credit is spent. The two answers are not two speeds of the same process, they are different processes with different staffing.

Here that split is written into the generator rather than remembered. Anything that flies or floats is static. So is anything whose name ends in ram, cannon, wagon, wheel, tank, engine, chariot or walker, and so is every trade cart and merchant ship. A gyrocopter's rotor spins in a shader and its hull is one rigid piece, so the mesh is finished the moment it arrives. Ships and vehicles ask for a higher triangle budget than a building does, at 30,000, because they get drawn large on the faction pages and read badly at less.

Rigging a character is still done by hand#

Nothing about the humanoid route is a batch. It is done in Meshy's web interface, one character at a time, by hand, and it looks nothing like the building run.

Two things about the input change. The first is that one picture is not enough. A rigid object can be reconstructed from a single three-quarter view because there is nothing behind the arm that matters. A character has to bend, and the parts a skeleton cares about are precisely the parts a game sprite hides: the inside of an elbow, the back of a knee, where the shoulder actually sits under a pauldron. So a humanoid goes in with several views of the same character instead, generated as a matched set, front and back and a true side profile alongside the isometric one the game uses.

The second is the pose. Game sprites are posed for reading at a distance, which means a bomb lobber is winding up to throw and a shaman is mid-incantation. Neither is any use to a rig, because a joint you cannot see is a joint nobody can place. The reference set is regenerated in an A-pose: standing straight, arms held out and down at roughly forty-five degrees, so the body forms a letter A. It is the industry-standard rigging pose for exactly one reason, which is that no limb is hiding behind another one. Hands are empty and open with the fingers visible, and any thrown prop deliberately removed from the picture: no bomb, no dynamite, nothing in the hands. Held equipment that a unit never puts down, like a caster's staff, stays.

Then there are the retries, and there are a lot of them. The service returns something odd more often on a character than on a hull, and the failure is usually not subtle: a limb fused to a torso, a face that has gone somewhere strange, a cloak that became part of the ground. There is no scoring function for that. You look at it, you decide it is not good enough, and you go again. Budget several attempts per character rather than one, and budget them as attention rather than as compute, because Adam has to be sitting there for each one.

The last part is the one with no automation at all anywhere in sight. There is currently no reliable way to wire a skeleton into a humanoid without placing the joints by hand. Automatic rigging exists and it will happily give you an answer. It will also put an elbow in a forearm, and you will not find out until the unit swings an axe. So the joints get placed, per character, by Adam, and that is the wall the title of this post is about. Everything upstream of it got cheaper by two orders of magnitude. That step did not move at all.

Why generated animations come back with the wrong names#

Even with the rig placed, the movements arrive labelled by guesswork, and the game picks which movement to play by matching text in the label.

The export shape that causes the most trouble returns one file per movement, each carrying a full copy of the character, with names like these:

Armature|clip0|baselayer         the resting pose, not a motion at all
Armature|Basic_Jump|baselayer    a hop the unit never performs in game
Armature|Walking|baselayer       the actual walk
Armature|Unreal Take|baselayer   whatever the exporter had open

Every one of those ends in the same word, and that word happens to be an alias for walking. Left alone, all four collide on the walk slot, and which one wins comes down to sorting order. The resting pose sorted first and became the walk, which froze every four-legged unit in mid-stride. Once that was fixed the hop sorted next and won instead, so a run of units walked by jumping. Cavalry turned up with a walk and no resting pose at all, and had to be given the walk as their base. Each of the three was found by watching a model move wrongly, and each was fixed by a rule about which names to drop before the fallback applies.

None of that is a failure of the model that made the mesh. It is that "a movement called Attack" and "the movement that plays when this unit attacks" are different claims, and only one of them can be checked by a machine. So Adam plays each movement on an inspector screen, watches what the model actually does, and labels it. The renaming tool then rewrites the labels inside the file without ever touching the motion data, and keeps a backup of the first run only, so a second run cannot overwrite a known-good file with a broken one. A companion pass fixes which way the model faces, which is the other thing that comes back wrong and also cannot be judged except by looking.

Simplify the compressed file, not the raw one#

Distant units do not need the detail of a unit under the cursor, so every model ships in several versions of decreasing complexity. Making those cheaper versions means throwing triangles away deliberately, and the single most consequential decision in the whole step is which file you throw them away from.

Textures dominate. A raw export carries around 15.5 MB of full-size images, with several redundant copies per unit. The compressed version of the same model carries about 2 MB in a format the graphics card reads directly, and it is byte-for-byte the same texture the game has already downloaded for the close-up version. Simplify the compressed file and the textures pass straight through untouched, so only the geometry shrinks.

Working from the raw export was not merely bigger, it was backwards. A distant version that costs more bandwidth than the close one is worse than not having it. The same reasoning drives the ladder that replaced the old flat billboards, and the same instinct is behind baking a unit's animation into a texture.

One thing about that step is not obvious and cost a batch of models. The simplifier here is meshoptimizer, an open-source C library, compiled to WebAssembly, that is the de facto standard for this in web 3D: you hand it a mesh and a target triangle count and it decides which triangles to collapse. It works by merging nearby points, and the obvious first move is to weld duplicate points so it has fewer to consider. Generated models deliberately split points apart wherever the texture wraps around a seam. Welding first erases the only evidence that the seam is there, the simplifier merges straight across it, and the results came out visibly swirled and striped. Skip the weld entirely and tell the simplifier that open borders may not collapse.

Two ways a correct model becomes invisible#

Both of these produce a model that loads without an error, throws nothing, and cannot be seen. Both bite the moment you take a compressed model and bake a transform into its geometry, which is what you do to draw many copies of something cheaply or to freeze a pose. Neither is specific to any one service.

The first is the storage format. Compressed models keep each point as a small integer covering a fixed range, which is what makes them small. Baking a transform reads each point out, multiplies it by a matrix, and writes the answer back into that same small integer. A world coordinate does not fit, so the geometry is destroyed silently and the model draws as a speck. The fix is to copy the points out into full floating point numbers first, and then bake. The check, if you would rather find out than fix blindly, is one line in a console: print the type of a model's position data. If it says anything other than a 32-bit float array, every transform you apply is writing into storage that cannot hold the answer.

The second is skinning, and it took two wrong diagnoses to find. A generated skeleton parks real scale on the bones themselves, so reading a character's points straight out of the file gives you the resting pose at the wrong size, between 1.5 and 2.4 times what the player sees. Only the skinning-aware reading, which applies the bones before handing back a point, matches the body on screen. In this game that is what turns a living unit into the corpse it leaves behind when it dies. What made it expensive was the test: the first fix was checked against a synthetic model with no skeleton, so the test passed while real corpses got worse, the ram rider going to 2.98 times its proper size and the death knight to 4.88. Only a measurement against a real generated model catches this, which is the general rule for anything touching a generated skeleton.

What you can and cannot automate for a new faction#

Half of this is now a job that can be started and walked away from, and half of it is a diary appointment. It is worth separating them honestly, because the two halves are staffed differently.

The rigid classes are the buildings again with a bigger triangle budget. A faction's ships, siege engines, flyers and carts is a dozen or so models, they all go up in one wave, and a wave costs one wait rather than a dozen. That is under an hour, and none of it needs anybody in the room.

Anything with legs is per character, with Adam present for all of it: the matched set of reference views, the retries until one comes back clean, the joints, and then watching each movement and naming it. Of the 269 unit models in this game, 147 carry an animation file, and every one of those 147 went through his hands.

So the answer to "could an agent build a new faction end to end" is: the buildings yes, the vehicles yes, the soldiers no, and the gap is a single step in the middle of the second list. Every one of the 229 units and 32 heroes in the game now has a model behind it, which was not true for most of this project's life, and the sprites that fed all of it were generated the same way. The next thing worth building is something that scores a movement by watching it rather than by reading its label, because that is the one place on the hand-built route where Adam is doing work a machine could check.