- I accidentally 4x P40s and need something to plug them into.
- Original plan fell apart: the Dell R730 outsmarted me. It seems to force any GPUs past the first two to power limit themselves and refuse to draw any external power.
- EPYCs don't seem to make sense in this secenario, the host mobo+cpu alone would cost more then the GPUs.
- Prefer to avoid dual-socket Xeon systems both because of the additional idle power and NUMA.
- With 96GB VRAM, I need at least 128GB system RAM to not have problems.
- I have a large stash of ECC DDR4 2133 modules.
-
CPU:
- Single socket
- Xeon v4
- High single core clock rate: E5-1680 or E5-1660
-
RAM: 8x16gb ECC DDR4 2133mhz
-
Storage: 8x SATA6.0
-
Power:
- 600W primary for motherboard
- 1100W secondary for GPU
-
4x Tesla P40 24GB
-
Host Link: PCIe 3.0 x8 or x16
-
Power: Target 150W-200W per GPU
-
Noise level: Below 35 dbA
- System needs to have some kind of VGA available for BIOS
- Don't require any risers or bifurcators, or at least as few as possible.
- Finding a motherboard that has:
- single-socket Xeon v4
- 4x PCIe 3.0 slots running at x8+
- spaced 2 slots apart
- Booting and using the BIOS requires a display GPU, so we actually need to connect 5 GPUs in total
- Adequate cooling for 200W/GPU with all 4 GPUs running at once with quiet idle and acceptable fully-loaded noise levels
N/A
- This board appears to waste 8 of the Xeon lanes.
- There are a total of 24 lanes coming from the GPU PCIe switches to the host, thus it is not possible to connect 4 GPU at x8 and this board fails our Compute GPU Requirements.
https://www.tweaktown.com/reviews/7887/asus-x99-deluxe-ii-motherboard-review/index.html
https://www.gigabyte.com/Motherboard/GA-X99-UD3-rev-10/sp#sp
- Only has 4 DIMM slots so doesn't meet the 128GB RAM Requirement
https://www.gigabyte.com/Motherboard/GA-X99-UD4-rev-10#ov
- Good slot layout, possible to connect x16+x8+x8+x8 dual width GPU
- Display GPU
- PCIe x1 are obstructed, a 180 degree x8 riser could make enough room for the USB x1 rear-facing
- Alternatively the M.2 x2 connector appears to be positioned nicely for an Oculink 4i M.2 adapter.
- Good slot layout, x8+x8+x8+x8
- Two obstructed x1, but M.2 connector looks good for oclunk
- Expensive? Almost double the price of the UD4.
Manual: https://download.asrock.com/Manual/X99%20WS.pdf
- Only supports 3 GPUs, doesn't meet the 4 PCIe x8 Requirement
Manual: https://download.asrock.com/Manual/QIG/X99%20Extreme11_multiQIG.pdf
- Has 2x PEX bridges but no diagram to show how they're connected to CPU.
- Good slot layout, possible to connect x8+x8+x8+x8 dual width GPU
- Has Molex PCIE_PWR extra connector
- Display GPU
- M.2 x4 connector appears to be positioned nicely for an Oculink 4i M.2 adapter.
- No backup plan here there's no x1 slots to the chipset just 6 total PCIe slots and when in x8+x8+x8+x8 mode 2 are disabled.
https://download.asrock.com/Manual/X99%20WS-E.pdf
- Has 2x PEX bridges but no diagram to show how they're connected to CPU.
https://discourse.mcneel.com/t/my-first-quad-gpu-dual-xeons-kitchen-sink-home-built-pc/41193?page=2
Chipset: Z170
- Z170 chipset does not support Xeon CPUs
Server board.
https://www.asrockrack.com/general/productdetail.asp?Model=EPC612D8A#Specifications
https://download.asrock.com/Manual/EPC612D8A.pdf
- 6 slots with 8x+8x+8x+8x+4x+4x which is incredible.
- Appears to be unavailable for purchase.