Adapter-DiLoCo: how it works and what we measured#
This page explains the method Modelen uses and the small-scale results we have measured so far. Everything is research-grade; there is no released model and no "better model" claim.
The problem: bandwidth#
Sending full model weights every step over home connections is not feasible. We combine two ideas:
- LoRA: the base model stays frozen and only a small adapter is trained (r=16, ~10.1 million parameters, ~40 MB for Qwen3-0.6B-Base). An 8 GB volunteer card is enough.
- DiLoCo: workers synchronise with a coordinator every H local steps instead of every step.
Only the adapter crosses the network; the base model stays on the volunteer's machine. Protocol details: Protocol v1.
How a round works#
- The coordinator serves the current adapter and a data shard.
- The volunteer downloads the adapter, verifies its SHA-256 and trains H local steps (20 by default).
- The trained adapter is uploaded.
- The coordinator validates the results, drops outliers, merges them with an outer step (Nesterov momentum) and publishes the next round.
What we measured#
All single seed, small scale; raw data and scripts are in the repository.
| Question | Result | Limit |
|---|---|---|
| Does a 20-minute LoRA run improve Turkish? | Turkish bits per byte 1.297 → 1.241 (−4.3%) | One run; evaluation text from the same domain as training (Wikipedia) |
| Is there English forgetting? | Bits per byte 0.9468 → 0.9597 (+1.4%, slightly worse) | One run |
| Knowledge/reasoning benchmark (TurkishMMLU, 5-shot) | 35.6% → 34.7%: no change (within noise) | Needs far more and more varied data |
| Cost of infrequent synchronisation? | After 8 rounds only 0.0011 bits/byte difference (−0.09%); ~98% of the baseline's total gain reached | One seed, 200 steps, 60k-token evaluation |
| How much less communication? | Calculation: ≈32 GB → ≈1.3 GB, about 25× less | A calculation, not a measurement |
| Does the protocol work end to end? | 2 workers × 3 rounds, no result rejected | Single machine |
Honest limits#
- We do fine-tuning; there is no from-scratch training and no full-weight fine-tuning.
- Language-modelling loss goes down but knowledge benchmarks do not move. Moving them needs much more data and probably a larger base.
- With real volunteers, data shards come from different distributions; network latency, worker loss and malicious contributions were not simulated.
- One study reports that DiLoCo-trained weights may later be incompatible with standard training (arXiv 2511.13761); whether that holds at adapter scale is untested.
- Poisoning protection is basic (Security).
Try the method yourself: Quick start and Volunteer setup.