Modular LoRA Subspace Orthogonality vs TIES Merge Dilution (392 Tensors Verified)
Measured over 6 persisted 97.3 MB safetensors checkpoints (24.3M trainable params each, disabled_vs_base max_abs = 0.0)
Cosine Alignment With Active Role Specialist
Modular Routed (set_adapter)
1.0000
PEFT TIES Merge (4 roles)
0.3827 avg
Cross-Role Specialist Pair
|cos| <= 0.00054 (orthogonal)
Why routing beats weight-averaging:
Decomposer, Retriever, Builder & Verifier occupy orthogonal
rank-16 subspaces; TIES attenuates 61.7% of each update,
while set_adapter(role) switches lossless weights at 0 base VRAM.
Held-Out Decomposer Probe NLL (Lower = Better)
Unchanged Base (3B)
1.3098
lora.task-decomposer
0.3681 (-71.9%)
lora.builder-toolcalls
1.6365
lora.verifier-dpo
1.7651
lora.shim-retriever
1.8855
Peak 2xT4 Train VRAM: 11,275 MiB (GPU0) / 11,025 MiB (GPU1)