Rack based on Lenovo Tinys for Red Hat OKD full HA deployment

[sylvaintremblay666]

This is a rack built to host a full-HA OKD setup. For those who don't know, OKD is the community edition of OpenShift, an enterprise-grade Kubernetes distribution packaged by Red Hat.

I'll start this thread by sharing the hardware specs of the rack's content, then tell the story behind this build for those who enjoy a good tale :)

Hardware setup, from top to bottom

• 120mm Noctua top fan
• 2x HUNSN micro firewalls 4x 2.5Gb ports on each (they're a little too wide but I've been imaginative)
• Yuanley 13-port POE 1GB switch
  • 8xPOE (for the vPro KVM consoles)
  • 2x uplink Eth (one for each firewall)
  • 1x uplink SFP (to connect to the SFP+ switch)
• RPi4 with DeskPi HDMI/USBC extension (gives access to the KVM consoles of the hosts)
• Yuanley 8 ports SFP+ switch (10Gb network between the hosts)
• Lenovo Tinys:
  • 3x m920q (i5 8600T, 16GB, nvme root disk, 10Gb PCIe NIC) for the control planes
  • 3x p330 (i7, 8700T, 32GB for now, 2x NVMe storage disks, mSATA root disk, 10Gb PCIe NIC) for workers + distributed storage

Lots of custom 3D-printed stuff, all designed by me, most being available in this collection on MakerWorld. If you see a part you need that I did not publish yet, send me a message and I'll publish it.

The Story behind it

To begin the story, let's first say a few words about me : I'm the (white-bearded geek) co-founder of a company where one of our offerings is on-premise solutions to help those who want to take control of their infrastructure using cloud-native open-source technologies on bare metal. We're building a mesh of micro data centers that are recycling the heat produced by the servers to various uses like heating houses, greenhouses, and much more.

Explaining the concepts of an HA setup, demoing them, and making the customer understand the value and believe what you're explaining isn't easy. We all know that when one wants to explain something to someone else, there's nothing like a "real object" that you can see and feel.

Somewhere before Christmas, I saw the DeskPi RackMate 10-inch rack on Amazon, and the geek in me fell in love instantly, wanting to do something with this cute little thing. Then came the idea : build a nano version of our micro data center ! This could be used for demos, and even to handle real development environment needs for small teams (fully contained HA dev environment cluster with all modern enterprise services).

At first, I was thinking of Raspberry Pi, but I quickly forgot this idea due to memory limitations. Then, I thought about our collaborator JS, who's always talking about his love for Tinys. I should be able to get all the horsepower I need for my use case using this kind of machine !

After some more research, those Tinys were definitely THE answer :

• Half-size PCIe expansion to put a 10Gb network card (for fast distributed storage)
• SODIMM memory upgradable to at least 64GB, theorically up to 96GB
• Possibility to put one small form factor SATA root disk + two NVMe disks on some models (to create storage nodes)
• Core i5 / i7 / i9 available, in T-versions (low power)
• Full vPro support (for KVM and out-of-band control)

Time to start digging through eBay to find some used hardware to build this idea 8-D

Let's begin shopping

I found some Lenovo m920q units that were perfect for control plane use : i5-8600T with 16GB RAM. I ordered them, ordered 128GB NVMe disks for root disks, and ordered the required PCIe risers to put the 10Gb cards in the hosts. As for network cards, the original plan was to use old Mellanox ConnectX-3 dual-port cards we had on hand but later on, I replaced these cards with smaller single-port ones to save space and reduce heat.

For networking, I wanted two switches:

• 1Gb switch to connect the 1Gb interfaces for vPro out-of-band management
• 10Gb switch for the networking between all nodes in the K8s cluster

After hours of research, I found the two Yuanley switches you can see in my build, which are almost exactly what I needed. One of them came with ears for 10-inch rack (!!), and I designed ears for the other one. The PoE wasn't really necessary, but well, it had all the connections I needed, including an SFP uplink to connect both switches together.

I also ordered a batch of slim Ethernet cables and SFP+ cables in different lengths and colors to be able to make a clean setup.

In the pictures, you can see a Tiny in a custom 1U support I designed. At first, I thought I would use one as a firewall, but these plans changed later in the build process. I ended up with the HUNSN micro firewalls, which were perfect for my use case, apart from being a tad too large (more on that later). I also designed a 0.5U wire pass which changed a lot through the development, and isn't quite good enough yet - I need to continue working on it.

Fitting the Tinys

I spent a lot of time just thinking and looking at the rack with the Tinys sitting on the RackMate's tablet on the lowest position. Something wasn't feeling right; the Tinys were a tad too high for a clean fit with some venting.

I finally got my plan : I would remove the bottom plate of the rack and create lower front and rear brackets with new feet. The rear bracket would have holes to screw the Tinys in place, and I'd add a top rear bracket so nothing could move.

Adding a top fan

I then wanted to add a top fan to help ventilation. When you remove the original vented top panel, there are nice slots in the chassis, which were perfect to design a slide-in, two-halves top panel with room for a 120 mm fan. I also made the grille magnetic.

The firewalls

For this rack to be taken seriously regarding its HA claims, it needed two firewalls. The HUNSN firewalls were awesome spec-wise, but a few mm too wide at 120 mm to fit two of them within the usable width of the rack. The space available between the ears is only ~224 mm, so there's no way to fit 240 mm... But it's wider inside the rack - there's a tad over 260 mm available there, which is enough to fit those firewalls... hmmm...

Another thing to keep in mind when looking for a solution : these little firewalls are fanless and use their case as a big heatsink. If I just get them out of their case and screw them on a 3D-printed support, cooling will probably become an issue.

I then got sick for a few days, which gave me time to just think, and I found my solution:

• A rear-held 3D printed bracket with built-in standoffs to hold both boards side by side
• The front of the boards will be a little recessed inside the rack, but I can design a front plate with a recess and some kind of I/O plate to deal with that
• Screw holes to connect the metallic bottom plate I removed from the rack (to fit the Tinys bracket) to my 3D-printed bracket, with the right spacing to fit the heat transfer block between the CPU and the metal plate so that it becomes part of the heatsink.
• Sit this module on top of a full-metal switch, which also adds to the heatsink effect.
• These boards will also be right under the 120 mm top fan, cooling should be OK.

Adding an RPi4

I was looking at my 0.5U wire pass and felt I was wasting some precious room. Could I fit something else in there? After some measuring, I figured out I should be able to fit an RPi in that 0.5U space! Not a lot of room to spare, but it should be doable, so I did it. I designed a new bracket to fit an RPi4 with the DeskPi HDMI/USB-C extension on the side of the wire pass. It will be used to easily connect to the KVMs of the hosts.

Starts to look good, now need workers

This little thing is starting to look good! I am now ready to order the workers. I identified my desired machine to be a Lenovo ThinkStation p330. I found three ones on eBay with interesting specs : i7-8700T, 32GB RAM. I could definitely do with more RAM, but as a starting point, it's not bad (and its SODIMM so easily upgradable). I also ordered mSATA drives with SATA adapters to be able to fit a SATA drive along with the network card and keep both NVMe slots free for dedicated CEPH storage drives. I designed a little bracket to fit the mSATA adapter nicely inside the p330.

Finally, everything in hand, assembly time! :-D

With all the hardware in hand, its finally the long-awaited time to put it all together. What a great Saturday morning I spent assembling this little thing and taking care of making it look as nice and clean as I can :-D

Tadam!

After about two months, here it is! I must say that I'm quite happy with the result.

For those who are still with me, I hope you enjoyed following me in this geeky journey, have a nice day!

[geerlingguy]

Bonus: you even paid the cat tax! Nice rack design.

[sylvaintremblay666]

Bonus: you even paid the cat tax! Nice rack design.

If it's on the net, it needs a cat! haha. Thanks! :-)

[MarcAndreWyss]

@sylvaintremblay666 I was 3 month an ad interim Product Owner for a similar project at my company with Openshift, Redhat DevSpaces and Developer Hub. One of the main issues was getting the hardware because of bureaucracy and regulatory. Therefore I'm personally very interested in you concept / project. Allthough I cannot use it at my company. Too bad. Sounds very interesting...

[SetSplin]

Hello @sylvaintremblay666. Awesome build! I have t1 and the same Yuanley switch. Could you please share a model of ears you used to mount it?

[MeTheKaptain]

What a great write-up and build journey! That's quite the custom rack 😄

[sylvaintremblay666]

@sylvaintremblay666 I was 3 month an ad interim Product Owner for a similar project at my company with Openshift, Redhat DevSpaces and Developer Hub. One of the main issues was getting the hardware because of bureaucracy and regulatory. Therefore I'm personally very interested in you concept / project. Allthough I cannot use it at my company. Too bad. Sounds very interesting...

@MarcAndreWyss We're also using Developer Hub and DevSpaces :-) Was your project publicly available? I'd love to read on it :)

Thanks for the good comments!

[sylvaintremblay666]

Hello @sylvaintremblay666. Awesome build! I have t1 and the same Yuanley switch. Could you please share a model of ears you used to mount it?

Hello @SetSplin ,

The ears are currently custom, putting the switch at the far top and giving more space for the Tinys venting. I will modify them to make them generic with the switch on the middle, upload that and let you know when its available.

[sylvaintremblay666]

Hello @sylvaintremblay666. Awesome build! I have t1 and the same Yuanley switch. Could you please share a model of ears you used to mount it?

Hello @SetSplin ,

The ears are currently custom, putting the switch at the far top and giving more space for the Tinys venting. I will modify them to make them generic with the switch on the middle, upload that and let you know when its available.

Hi @SetSplin ,

Finally got some free time, I ended up uploading both versions : one with the switch in the middle, and the one from the pics in this thread with the switch up top. Here's the link !

[SetSplin]

Thank you @sylvaintremblay666 !

[MarcAndreWyss]

@MarcAndreWyss We're also using Developer Hub and DevSpaces :-) Was your project publicly available? I'd love to read on it :)

Unfortunately no. Hope it will be publicly available in the future. We had (and still have) RedHat employees on site. Even they are not allowed to talk about it in their events at the moment. But the technology is very impressive. Therefore I really like your project.

[Chillinwdylan]

How did you power the Lenovos? I don't see the power bricks? I'm trying to do a similar thing but am trying to avoid having to manage all the cables for 4 power bricks in my mini rack

[sylvaintremblay666]

How did you power the Lenovos? I don't see the power bricks? I'm trying to do a similar thing but am trying to avoid having to manage all the cables for 4 power bricks in my mini rack

for now, power bricks in a cardboard box, not clean at all. We're thinking about a custom setup with a single connection to a psu to provide the 6x 20v power for the Tinys... An idea right now is to use another 4U 10 inch rack to manage power (UPS, PSUs, ...) Not our highest priority now tho, but not that far in the list either.