Multi-GPU (seven) RTX 3090 workstation, possible? - build critique request

I'm looking to build a new multi-gpu 3090 workstation for deep learning. My original idea was to go with Threadripper 3960x and 4x Titan RTX, but 1) NVidia released RTX 3090, and 2) I stumbled upon this ASRock motherboard with 7 PCIe 4.0 x16 slots. I'm reasonably comfortable building PCs and DIY, but server stuff is a bit new and I'm worried I'm missing something obvious, hence this post.

The requirements (driven by the nature of compute I'm doing):

• NVidia GPUs with largest possible memory - but not Tesla/Quadro

• approx 4 cores / 8 threads and 32GB mem per GPU

• fit as many GPUs into the rig as possible - compute across PCs is more tricky

The build:

• ASRock ROMED8-2T Motherboard

  • do you think this motherboard will actually support 7x GPUs?

  • why: this is the only mobo I could find with 7x PCIe slots for modern affordable CPU with a lot of PCIE lans. Based on this review.

• Supermicro Motherboard

  • why: much more reputable brand in server space

• AMD EPYC ROME 7402P Processor

  • mobo explicitly lists it as supported, so should work ok, no?

  • why: based on this wiki page, I looked at Rome faily (current one), selected the cheapest 24 core one (3.4 cores per GPU)

• Cooler Master ML360 TR4

  • is it going to mount on EPYC socket?

  • why: there seem to be only two AIOs specifically for TR4 socket, second one is Enermax with super bad reputation. Need AIO because tower cooler will obstruct GPUs in my planned case.

• Dynatron A26 Cooler

  • why: dedicated for EPYC, low profile, not water

• Samsung 32GB 3200MHz ECC DDR4 RDIMM Memory

  • I don't know anything about server memory, please help?

  • why: mobo spec says it supports RDIMM, LRDIMM and NV DIMM. This article says up to 32GB modules you should use RDIMM. 8x32GB should be sufficient for all GPUS. NV DIMM seems not applicable here.

• RTX 3090 EVGA GPUs

  • why: I like EVGA due to good customer support, air because easy to swap between rigs, exact model TBD after release

  • plan: build will start with 4x 2080 ti (already have), then add 1x 3090 after initial reviews, confirm no unexpected issues, then keep adding up

• LINKUP PCIe 4.0 X16 Riser Cable

  • zero experience with risers, so any info would be good, is 20cm ok? 50cm too far?

  • why: this is the only PCIE 4.0 riser cable I found

• Samsung 980 (when they release)

  • why: good experience with 970

• EVGA SuperNOVA 1600 G2 PSU

  • zero experience with dual PSUs, any advice or good resources would be super appreciated!

  • plan: 1st PSU: Mobo/CPU/3x3090, 2nd PSU: 4x3090

• Two options for the case:

  • Minestation 1

    • why: mobo directly below GPUs - so can use short risers (good?), but have to use AIO cooler (f** me if it leaks!)

  • Hydra II Rev. B or similar

    • why: mobo in rear compartment - so needs longer risers (50cm, bad?), but can fit air tower cooler (they don't leak!)

  • both are 6u rack, which is 47cm wide, so it should fit 7 triple slot cards (7 * 6cm = 42cm)

  • heat will be a problem..., no? :X

• Noctua NF-F12 iPPC-3000

  • 3x intake + 3x outtake. Tell me heat won't be a problem...

Sorry for the super long post. It would be super helpful is sb more experienced double checked the build. I kind of expect one liner reply: "won't work because of X you silly bumer", which would be super appreciated.

All the best!

EDIT #1:

In case someone is researching this as well, 7 GPU is not a new thing. Found more resources, more motherboards and more info. Most similar build is this one (forum thread). More builds here, here, here, here and of course Linus series "7 Gamers. 1 CPU" part 1, 2, 3 (8 GPU) and his 6 GPU rendering station. Now I really have to do some work.

EDIT #2:

I did more research and it seems this build is a scratch, or at least not suitable for a reliable production system.

The main reason is power supply. Largest decent quality PSU I could find are 2000W [1, 2], this is sufficient for "only" 4x 3090. Based on my (rather minimal) electrical eng. experience you should not bridge PSUs unless they were specifically designed to do so. Person on homelab subreddit raised same concern, I spoke to two electronics engineers and both said the same "might work but don't do it". Miner use dual PSUs but their PCIE x1 extenders might be more isolated (?).

In the end decided to purchase 2x 2nd gen Threadripper 2950x, price slightly higher than single 3960x or EPYC listed above, but seems like much safer way to get 8x GPUs. The CPU power, iter-GPU transfers (now over network) and PCI 4.0 are non factor for me.

Server builds are more than capable to support 8-10GPUs and should work with 2080ti / 3090 as far as I know, server PSUs are specifically designed to operate in parallel / redundant mode. The downside is servers are extremely loud, too loud even for my lab backroom.

Next time!