Generating and rigging game-ready 3D units for a web RTS

Building an RTS with nine asymmetric factions, neutral creep camps guarding contested gold mines, wandering wildlife, and flying capital ships quickly hits the 3D asset wall.

A traditional studio solves this with technical artists or outsourced contractors. A low-poly RTS unit with four animation clips (idle, walk, attack, death) can cost between $300 and $1,500 from a freelance 3D artist and takes two to three weeks of reviews. Multiply that by hundreds of units, creeps, siege weapons, and heroes, and a solo web RTS becomes impossible.

When we began rebuilding the game in 3D, we developed techniques for generating 3D buildings from 2D sprites. Over the past few days, we extended that foundation into a pipeline that turns a text brief into a game-ready, fully animated 3D unit.

It has already produced our new creep families, including walking reef crabs, broodspiders, giant scorpions and fire drakelings, as well as animated goblin peons and aerial capital ships like the Dark Elf Nightbarge.

Today, we are releasing the complete pipeline as an open-source GitHub repository: github.com/AdamJSturrock/shards-asset-toolkit.

Here is the exact toolchain, how our AI agent handles rigging for humanoids and multi-legged beasts, how advanced reference-to-video diffusion pushes motion further, and how you can inspect the live 3D models right in this post.

The pipeline at a glance#

The toolchain connects five stages, passing data forward without manual DCC intervention:

1. Concept generation (Gemini): Prompting for a clean three-quarter isometric orthographic view on a pure neutral background with zero atmospheric clutter.
2. 3D mesh synthesis (Meshy): Converting the cleaned 2D concept into a 30,000-polygon textured GLB.
3. Rigging and animation (Blender + Claude Opus 5 / Astra): Headless Python scripts driven by an AI agent that trace landmarks, segment limbs and props into discrete vertex component masks, fit anatomical armatures, and retarget motion.
4. Motion library bridge (Mixamo & Procedural Gaits): Retargeting humanoid and creature motion clips onto custom skeletal proportions with automated weight repair and stance preservation.
5. Web engine optimisation (glTF-Transform): Decimating to runtime budgets, and transcoding textures to GPU-native KTX2 Basis Universal formats. We also built a baker that stores animation in a texture, but it is experimental; the game currently draws units with standard GPU skinning.

To see how every piece works, let us walk through a real subject from our latest creep update: the Shellback Cutthroat (creep_shellback_cutthroat), an asymmetric crab-folk pirate wielding a notched cutlass in one hand, a massive crushing claw on the other, and a battered tricorn hat perched on its shell.

Stage 1: Concept art and clean prompt engineering#

Everything begins with concept art. We use Google Gemini (Nano Banana Pro / Gemini 3 Pro Image) through our asset generation pipeline.

The prompt specifies a clean full-body three-quarter front view, neutral studio lighting, and a flat keyable background. For our crab pirate, the direction was explicit: an upright crab-folk buccaneer, barnacled red-orange shell, salvaged sailcloth sash, a pirate tricorn wedged on the shell ridge, a heavy right crushing claw, and a notched boarding cutlass in the left claw.

3D generators have no concept of transparency or ambient lighting. If concept art contains smoke trails, embers, or ground shadows, the generator attempts to sculpt them into solid polygonal geometry, resulting in solid curtains of rock between legs or weapons fused to jagged lumps of stone.

Our pipeline enforces three prompt rules:

1. Isolated backdrop: Clean, neutral, single-tone studio background.
2. Zero ground contact clutter: Ground timbers and shadows are excluded so the unit arrives with zero base geometry to stand cleanly on terrain.
3. Clear silhouettes: Extremities and weapons are positioned with clear negative space so limbs do not fuse into the torso.

Stage 2: 3D mesh generation with Meshy#

Once the concept image is generated, we send it to Meshy via its API with a polycount target of 30,000 triangles. Each mesh costs 30 credits.

Meshy excels at surface form, silhouette fidelity, and baking plausible normal and roughness maps from a single viewpoint. The segmented carapace, barnacles, cutlass blade, and leather sash translated into crisp geometry of about 31,000 triangles.

The raw output is a single unified organic sculpt: the cutlass is sculpted into the left grip, the right claw is a continuous carapace sculpt without separate mechanical pincers, and the tricorn hat sits fused to the shell ridge.

Standard auto-riggers assume a biped with symmetrical human limbs. Faced with an asymmetric fantasy creature carrying an oversized claw on one side and a rigid sword on the other, standard humanoid auto-riggers distort the geometry or bend the weapon into rubber. Meshy's own auto-rig refused the Cutthroat outright ("pose estimation failed"), which is why this unit is rigged in Blender.

One lesson from armed units: Meshy's image-to-3D pose_mode: t-pose option re-poses a character into its own T-pose with open hands and silently drops whatever it is holding. For armed characters we feed T-pose concept art and leave pose mode off. The auto-rigger still accepts the result, and we then weight the weapon 100% to the hand bone, because Meshy smears it onto the forearm.

The division of labour is strict: Meshy generates the raw mesh and textures. The rigging, skeletal articulation, and motion authoring belong to our AI-driven Blender toolchain.

Stage 3: AI-driven Blender rigging and component isolation#

To rig our crab pirate, we delegate the work to an AI coding agent (Claude Opus 5 and Astra) running headless Python scripts inside Blender.

The AI inspects the downloaded GLB, locates anatomical landmarks, and fits a custom 24-joint armature compatible with standard humanoid naming conventions:

Armature Joint Hierarchy (24 joints):
Hips
 ├── Spine02 -> Spine01 -> Spine -> neck -> Head (with headfront & head_end)
 ├── LeftUpLeg -> LeftLeg -> LeftFoot -> LeftToeBase -> LeftToeTip
 ├── RightUpLeg -> RightLeg -> RightFoot -> RightToeBase -> RightToeTip
 ├── LeftShoulder -> LeftArm -> LeftForeArm -> LeftHand -> LeftHandTip
 └── RightShoulder -> RightArm -> RightForeArm -> RightHand -> RightHandTip

Topological component isolation#

Without component isolation, automated heat diffusion skinning causes severe defects: turning the head stretches the tricorn, and swinging the sword bends steel like rubber.

The AI agent applies topological component isolation in Blender:

1. Cutlass rigid locking: The cutlass (1,507 vertices) is flood-filled from blade tip to hilt and locked 100% to LeftHand with rigid weight 1.0. Zero smoothing is permitted, keeping the blade stiff.
2. Carapace and hat protection: The tricorn hat and shell ridge (8,923 vertices) are assigned rigidly to Head, moving as a solid unit without squashing into the carapace.
3. Heavy claw weighting: Because the right claw is an organic sculpt without separate mechanical joints, the bulk (1,223 vertices) is weighted to RightForeArm and the tip (1,301 vertices) to RightHand, letting it swing as a crushing bludgeon.
4. Automated validation: The script asserts weights sum to 1.0, caps influences at 4 bones per vertex, and verifies zero unweighted vertices.
5. Never delete geometry to free a part: Our first cutthroat rig cut away Meshy's fused skin between the shell rim and the face and arms so the claws could move freely. That left see-through gaps along the rim: the mesh's open edges went from 31 to 347. The rig now keeps the mesh intact, holds the whole carapace rigid to the head, and blends a short weight gradient (about 10 cm) across the fused skin, the way a shoulder is weighted. The export fails if the rigged mesh has more open edges than the source, and the cutthroat is back to 31.

Interactive 3D Model Inspection#

You can test and inspect the actual 3D assets below directly in your browser. Use the tabs to toggle between the Raw Meshy Mesh (the unrigged 30k polygon sculpt), the AI-Rigged & Animated 3D Model, the Shoreline In-Engine Video, and the Gemini Concept Art.

When the rigged model is active, click any animation button to test the live skeletal motion. You can drag to rotate the camera to any angle, scroll to zoom in close on the barnacles and cutlass, or right-click to pan:

Stage 4: Retargeting Mixamo motion clips and custom animations#

With an articulated skeleton in place, we retarget motion-capture clips from Adobe's Mixamo library with the toolkit's retarget-mixamo command (scripts/blender/retarget_mixamo.py):

# Retarget Mixamo world rotations as deltas relative to rest pose
delta_rot = source_bone.matrix.to_quaternion() @ source_rest.inverted()
 
# Apply stance retention ratio
if bone_name in LEG_ALIGN:
    target_bone.rotation_quaternion = slerp(
        character_rest_rot,
        delta_rot @ character_rest_rot,
        leg_align_factor
    )

Three adjustments make the pirate animations read cleanly:

1. Stance retention (--leg-align 0.3): Standard walk clips pull knees inward. Blending the rest stance preserves the wide, bowed crab posture.
2. Bone-keep damping on spine and neck: Damping spine rotation (boneKeep: { Spine02: 0.6, Spine01: 0.6 }) keeps the heavy shell steady while limbs move freely.
3. Root motion cancellation: Movement is governed by engine pathfinding. Our tool subtracts horizontal displacement across each loop, locking walk cycles firmly at the origin.

One known issue we are still fixing: T-pose concept art with the weapon drawn upright gives the model a thumb-up fist, while Mixamo's rest pose is a palm-down fist with the blade pointing forward. Retargeted clips then carry a roll of about 90 degrees on the held weapon, so a sword can look gripped wrong. The fix, normalising the grip at rest before retargeting, is in progress.

The animation suite#

The Shellback Cutthroat ships with five distinct motion cycles:

Here is the full motion preview contact sheet showing how every clip reads from game isometric angles, front view, and 64-pixel RTS minimap scale:

Beyond humanoids: multi-legged creatures and complex anatomy#

An RTS needs giant arachnids, crawling decapods, and flying dragons. Standard tools like Mixamo cannot rig an eight-legged spider: they have no bone definitions or motion data for non-humanoid skeletons.

Our AI toolchain treats skeletal construction as a programmatic problem:

• Procedural kinematic chains: For arachnids like the Broodspider, the AI agent analyses mesh symmetry, detects cephalothorax and abdomen, and builds an 8-leg kinematic hierarchy.
• Segmented weight distribution: Heat diffusion skinning is constrained strictly per leg segment, preventing neighbouring legs from dragging each other.
• Procedural gaits: The AI generates rhythmic tripod and tetrapod crawling gaits directly in Python keyframe loops.

Here is a condensed side-by-side progression showing three distinct creature architectures: the 8-legged Broodspider, the Reef Crab (six walking legs and two claws), and the asymmetric Shellback Cutthroat:

Broodspider· 8-Legged Arachnid (Complex non-humanoid)

Humanoid auto-riggers completely collapse on eight legs. The AI generates a procedural arachnid armature with an 8-leg kinematic chain, distributes bone weights across all leg segments, and crafts multi-phase crawling gait cycles.

1. 2D Concept🔍

Gemini PromptView ↗

2. 3D Static Mesh🔍

Meshy 30k TrisView ↗

3. Bones Applied🔍

AI Armature FitView ↗

4. Animated Model🔍

Procedural MotionView ↗

5. In-Game Result🔍

▶

10s Video

Live RTS EngineView ↗

Reef Crab· Decapod Critter (six walking legs, two claws)

Low-profile carapace with eyestalks and dual pincer claws. Every leg and claw is traced from its tip and weighted only to its own bones, so neighbouring legs never drag each other, and the walk moves forward with the claws facing the target.

1. 2D Concept🔍

Gemini PromptView ↗

2. 3D Static Mesh🔍

Meshy 30k TrisView ↗

3. Bones Applied🔍

AI Armature FitView ↗

4. Animated Model🔍

Procedural MotionView ↗

5. In-Game Result🔍

▶

10s Video

Live RTS EngineView ↗

Shellback Cutthroat· Asymmetric Crab Buccaneer

Asymmetric anatomy pairing an oversized crushing claw with a handheld boarding cutlass and tricorn hat. The AI isolates rigid prop vertices and retargets Mixamo mocap while locking bowed stances.

1. 2D Concept🔍

Gemini PromptView ↗

2. 3D Static Mesh🔍

Meshy 30k TrisView ↗

3. Bones Applied🔍

AI Armature FitView ↗

4. Animated Model🔍

Procedural MotionView ↗

5. In-Game Result🔍

▶

10s Video

Live RTS EngineView ↗

Advanced workflow: Seedance for complex motion and visual effects#

Procedural gaits and mocap retargeting handle regular walks and basic strikes reliably. However, complex creature behaviours expose the limits of stock libraries:

• Asymmetric combat inertia, such as a heavy claw pulling a unit off balance during a strike.
• Secondary organic physics, including twitching antennae, minor pincers, and trailing sashes.
• Synchronised special effects (VFX): water splashes in shallow surf, ground fracture shockwaves, and magical runes.

To explore motion beyond stock libraries, we are testing an advanced workflow built on ByteDance's Seedance video diffusion model.

Reference-to-Video (R2V) in Blender#

Generating motion purely from text prompts suffers from drifting limbs and erratic cameras. The Seedance workflow uses Blender as spatial ground truth:

1. 3D proxy blocking: In Blender, the rigged model or proxy geometry is animated with crude timing to establish camera framing, step cadence, and impact windows.
2. Playblast export: Blender renders a lightweight viewport preview video.
3. Reference-to-Video diffusion: Seedance takes the playblast as a motion control signal. The diffusion model adheres to the 3D camera and timing while generating fluid weight transfer and dynamic secondary effects.

Extracting dynamics back into the engine#

Because generation is tethered to a 3D reference, the resulting dynamics feed back into the engine:

• Pose tracking to skeletal keyframes: Optical tracking converts fluid weight shifts from video frames back into standard Blender bone rotations.
• VFX sprite extraction: Energy flares, impact shockwaves, and dust wakes are extracted as alpha-masked sprite sequences or Three.js particle coordinates.

This workflow is an advanced frontier on our roadmap rather than part of the base CLI command. However, using 3D blocking to anchor diffusion models like Seedance opens the door to intricate creature motion that stock libraries cannot match.

Final result in the game engine#

Here is the final result running in the game engine with ultra graphics enabled: water reflections and specular glints turned on, the Shellback Cutthroat in the shallow surf with its Shell Grotto creep spawner on the beach, an iron-bound treasure chest, and evergreen forest trees framing the background ridge:

Stage 5: Squeezing models for a browser RTS#

A raw Meshy export carries 2048px textures and around 30,000 triangles, and the browser decodes those textures into uncompressed GPU memory. If eight players meet with twenty units each, that memory adds up fast. As shown in our work on reducing game download size, uncontrolled VRAM footprint kills browser tabs before polycount does.

Our optimisation pipeline applies three passes to make assets web-ready:

1. Texture transcoding to KTX2 / Basis Universal: Resizing textures to 1024px and transcoding to KTX2 (UASTC for normals, ETC1S for colours) keeps them compressed in GPU memory.
2. Geometry compression with meshopt: Compressing vertex streams and keyframe tracks with EXT_meshopt_compression shrinks the download without the decoding hitches of Draco.
3. Vertex Animation Textures (VAT), experimental: Skeletal animation can be baked into 16-bit vertex textures so the vertex shader morphs geometry on the GPU. We have a working baker, but the game does not use this path yet; units currently render with standard GPU skinning.

What it costs: the production ledger#

We have not measured a reliable per-unit total yet: retries, rejected meshes and agent time vary too much from one unit to the next. These are the prices we can stand behind:

Even with a few retries, that is a different order of cost from a contractor quote of hundreds of dollars and weeks of reviews per unit, and it changes the scope of what an independent developer can attempt.

Open source release: Shards Asset Toolkit#

We believe the future of RTS games lies in empowering players, map makers, and modders to build custom maps, campaigns, and entire factions.

Today, we are releasing the complete pipeline as a standalone open-source toolkit on GitHub:

GitHub Repository: https://github.com/AdamJSturrock/shards-asset-toolkit

Quick start installation#

# Clone the open source repository
git clone https://github.com/AdamJSturrock/shards-asset-toolkit.git
cd shards-asset-toolkit
 
# Install dependencies
npm install
 
python3 -m pip install numpy scipy pillow
 
# Configure your API keys in .env
cp .env.example .env   # then fill in your keys

You need GEMINI_API_KEY and MESHY_API_KEY, plus Blender 4.2 or later. ANTHROPIC_API_KEY is optional: only if you drive the Blender steps with a coding agent.

Running the pipeline#

You can try the rigging on the bundled crab pirate without any API keys:

# Rig the bundled raw mesh in Blender from its spec (checks the boundary-edge assertion)
node bin/shards-asset.mjs rig-humanoid \
  --spec examples/anim-authoring/shellback_cutthroat.json \
  --out ./output/anim-authoring/shellback_cutthroat/rig-v3/character.glb
 
# Build, rig and animate a procedural spider stand-in (no mesh, no API keys)
node bin/shards-asset.mjs rig-critter standin --kind spider --out output/standins/spider.glb
node bin/shards-asset.mjs rig-critter prep spider_standin
node bin/shards-asset.mjs rig-critter fit spider_standin
node bin/shards-asset.mjs rig-critter all spider_standin v1

With keys, each stage is its own command:

node bin/shards-asset.mjs concept \
  --prompt "An upright crab-folk buccaneer pirate, barnacled shell, tricorn hat, huge right claw, cutlass in left claw" \
  --output ./output/concepts/shellback_cutthroat.png
node bin/shards-asset.mjs mesh \
  --input ./output/concepts/shellback_cutthroat.png \
  --output ./output/meshes/shellback_cutthroat.glb \
  --polycount 30000
node bin/shards-asset.mjs meshy-rig --task-json ./output/meshes/unit.meshy.json --output ./input/unit/rigged.glb --height 1.7
node bin/shards-asset.mjs dye-mask apply --in animated.glb --out animated.dyed.glb --review dye_review.png
node bin/shards-asset.mjs optimize --input animated.dyed.glb --output character.glb --texture-max 1024

Every command prints its options with --help, including retarget-mixamo and polish-clips for the clip stages.

The toolkit bundles the CLI, the headless Blender rigging scripts for humanoids, mounts and multi-legged critters, the Mixamo retargeting and clip polish tools, a player-colour dye mask tool, and the KTX2 optimiser. The VAT baker is included too, marked experimental.

You write a brief, run the stages, look at the review renders, and get a .glb ready to drop into a custom map package (.sosmod). Full documentation is live on GitHub at github.com/AdamJSturrock/shards-asset-toolkit and in our modding developer guides.