How do you calculate oversubscription on a leaf switch? Walk me through an example.
Show the answer
Say this first
Divide the server-facing bandwidth by the uplink bandwidth. A leaf with 48 x 25G down has 1,200G of downlink. With 6 x 100G up it has 600G of uplink. 1,200 divided by 600 is 2:1. General compute is often built at 3:1 or better, storage and HPC closer to 1:1, and GPU training fabrics at 1:1.
Go deeper
The math
| Downlinks | Uplinks | Ratio |
|---|---|---|
| 48 x 25G = 1,200G | 6 x 100G = 600G | 2:1 |
| 48 x 10G = 480G | 4 x 40G = 160G | 3:1 |
| 40 x 25G = 1,000G | 4 x 100G = 400G | 2.5:1 |
| 32 x 100G = 3,200G | 8 x 400G = 3,200G | 1:1 |
What the ratio really means
- It is the worst case: every server sending at line rate toward other racks at the same moment. Real traffic is burstier and rarely all leaves the rack.
- In a two-stage fabric the spines are not oversubscribed, so the leaf ratio is the fabric ratio. In a five-stage fabric the pod spine to super-spine ratio multiplies with it.
- Plan for failure. With 6 uplinks and one spine down, 1,200G over 500G is 2.4:1.
- Count only the ports you will really use. A rack with 30 servers on a 48-port leaf has a better effective ratio than the box suggests.
They will ask next
How would you get that 48 x 25G leaf to 1:1?
Use only 24 server ports (600G down against 600G up), choose a leaf with more or faster uplinks such as 400G, or accept the ratio for general compute and put storage on a separate non-blocking design.
Why do GPU training fabrics insist on 1:1?
Collective operations (all-reduce) make every GPU send at line rate at the same time. Any oversubscription shows up directly as longer job time, so the back-end fabric is built non-blocking.
Traps
- Comparing port counts instead of bandwidth.
- Ignoring the peer link and border ports when you budget the uplinks.