Skip to content

Instantly share code, notes, and snippets.

@scyto
Last active September 14, 2024 09:12
Show Gist options
  • Save scyto/cd1a8cdd9904f43e319e438551d7f92c to your computer and use it in GitHub Desktop.
Save scyto/cd1a8cdd9904f43e319e438551d7f92c to your computer and use it in GitHub Desktop.
Hardware Setup / Proxmox Base Install

Hardware Setup / Proxmox Base Install

this gist is part of this series

Hardware Setup (perform on all 3 nodes)

  • Connect all 3 NUC to power etc
  • Disable secure boot in BIOS on all nodes
  • Connect all 3 TB4 cables in a rinng topology:
using the numbers printed on the case of the intel13 nucs connect cables as follows (this is important):
- node 1 port 1 > node 2 port 2
- node 2 port 1 > node 3 port 2
- node 3 port 1 > node 1 port 2
  • Prepare bootable usb key by downloading proxmox iso and tools such as 'rufus'
  • Boot from USB

Proxmox Base Install (repeat on each node)

  • choose GUI (why not!)
  • accept terms
  • choose the SSD NOT the NVME for the install
  • set country, keyboard, etc
  • choose a password (this will be for the user - root) and set email
  • choose enp86s0 as the management interface (on proxmox 8 you will see the thundebolt netwrok too if cables are connected)
  • set a name for each node - i used pveX.mydomain.com where X = the node number i wanted
  • set fixed IPv4 address for the management network
    • 192.168.1.81/24 for node 1
    • 192.168.1.82/24 for node 2
    • 192.168.1.83/24 for node 3
  • set external DNS server and gateway
  • let the software partition disk as it sees fit
  • review summary page
  • removed the USB key and reboot when prompted

You should now have 3 nodes. Each should be accessible over the network with the IP addresses you specified. e.g 192.168.1.81:8006, 192.168.1.82:8006, 192.168.1.83:8006

The default username is root and you use the password you defined earlier. Consider making constrained users and doing security properly. I won't be doing that, i will be using root as this a PoC.

Set Package Repositories (only needed if you don't use a subscription)

Note: i will not be using enterprise packages as this is a PoC for evaluation / homeLab purposes only

In node-name > updates > repositories perform the following:

  • add No-Subscription
  • add Ceph Quincy No-Subscription
  • disable pve-enterprise
  • disable ceph-quincy enterprise too

Open the shell and perform apt update && apt upgrade to update all components and then reboot.

@ajax1337
Copy link

after installation my nvme ssd is not showing in boot devices as uefi is enabled , what to do ? i am unable to boot

@travisw3
Copy link

Curious with Proxmox 8.1 supporting Secure boot have you tested this with Secure boot at all now?

@cswaas
Copy link

cswaas commented Jun 10, 2024

Hi, why do you recommend: „choose the SSD NOT the NVME for the install“?
I have three NUCs as well and had Talos OS installed on the NVME to form a cluster and kept the 4TB SSD disk for cluster storage. Now I would like to use your setup.

@anixon604
Copy link

Hi, why do you recommend: „choose the SSD NOT the NVME for the install“? I have three NUCs as well and had Talos OS installed on the NVME to form a cluster and kept the 4TB SSD disk for cluster storage. Now I would like to use your setup.

I think this is likely because proxmox as a base, is not heavy on the I/O and disk performance, another popular alternative Unraid, actually runs off a USB key. Thus, you have more "options" if you save your full NVME drive for after install, then it's easier to have that drive up for Ceph, or as partition it and wipe it as needed, and your actual containers and VMs can go on the NVME.

@edwardtls
Copy link

what are the disk recommendations for example in a situation where I have two 1TB NVME's per node... just one for Proxmox and the other for ceph ?
at the moment I've created a 1TB mirror and assigned 130gb to Proxmox and the remainder to Ceph.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment