Skip to content

Instantly share code, notes, and snippets.

@rohityadavcloud

rohityadavcloud/Linstor

Created October 4, 2023 09:00
Show Gist options
  • Save rohityadavcloud/7844bfa8cbc9586408ce3596c71caa50 to your computer and use it in GitHub Desktop.
Save rohityadavcloud/7844bfa8cbc9586408ce3596c71caa50 to your computer and use it in GitHub Desktop.
Download ZIP
Linstor-setup-notes
Raw
Linstor
Linstor setup on ubuntu 22.04
https://packages.linbit.com/
https://launchpad.net/~linbit/+archive/ubuntu/linbit-drbd9-stack
https://www.youtube.com/watch?v=hI_kTlsbNeU install video
sudo add-apt-repository ppa:linbit/linbit-drbd9-stack
sudo apt-get install drbd-utils drbd-dkms
sudo modprobe drbd
cat /proc/drbd
version: 9.2.5 (api:2/proto:86-122)
GIT-hash: b44520271e63d4b6f359a6642eb4d475b7cc04e0 build by root@kvm3, 2023-09-01 18:27:18
Transports (api:18):
dkms status
drbd/9.2.5-1ppa1~jammy1, 6.2.0-31-generic, x86_64: installed
#wget -O- https://packages.linbit.com/package-signing-#pubkey.asc | apt-key add -
# apt-get install lvm2 nvme-cli
# apt-get install linstor-controller linstor-satellite linstor-client python-linstor
systemctl enable --now linstor-controller
systemctl status linstor-controller
linstor node list
Add in /etc/hosts:
192.168.1.20 kvm1
192.168.1.21 kvm2
192.168.1.22 kvm3
Adding nodes:
On kvm1:
# linstor node create kvm1 192.168.1.20 --node-type combined
# linstor node create kvm2 192.168.1.21
# linstor node create kvm3 192.168.1.22
On other nodes:
linstor --controllers=kvm1 node list
Or, put this on other/all nodes:
# cat /etc/linstor/linstor-client.conf
[global]
controllers=kvm1
The linstor-satellite requires TCP ports 3366 and 3367. The linstor-controller requires TCP port 3370. Verify that you have this port allowed on your firewall.
systemctl restart multipathd
systemctl enable --now linstor-satellite
Disk setup:
# pvcreate /dev/nvme0n1
Physical volume "/dev/nvme0n1" successfully created.
# vgcreate vg_nvme /dev/nvme0n1
Volume group "vg_nvme" successfully created
root@kvm3:/etc/apt/sources.list.d# pvs
PV VG Fmt Attr PSize PFree
/dev/nvme0n1 vg_nvme lvm2 a-- <1.82t <1.82t
root@kvm3:/etc/apt/sources.list.d# vgs
VG #PV #LV #SN Attr VSize VFree
vg_nvme 1 0 0 wz--n- <1.82t <1.82t
Or:
root@kvm3:/etc/apt/sources.list.d# vgcreate vg_nvme /dev/nvme0n1
Create storage pool:
# linstor storage-pool create lvm kvm3 nvme_pool vg_nvme
SUCCESS:
Successfully set property key(s): StorDriver/StorPoolName
SUCCESS:
Description:
New storage pool 'nvme_pool' on node 'kvm3' registered.
Details:
Storage pool 'nvme_pool' on node 'kvm3' UUID is: 4df9d5f5-b71a-439e-a0ed-50b91f8af1ff
SUCCESS:
(kvm3) Changes applied to storage pool 'nvme_pool'
Repeat on:
linstor storage-pool create lvm kvm1 nvme_pool vg_nvme
linstor storage-pool create lvm kvm2 nvme_pool vg_nvme
OR
Just ruN:
linstor physical-storage create-device-pool --pool-name nvme LVMTHIN kvm3 /dev/nvme0n1 --storage-pool nvme_pool
run: pvs; vgs; lvs
root@kvm3:/etc/apt/sources.list.d# pvs
PV VG Fmt Attr PSize PFree
/dev/nvme0n1 linstor_vg_nvme lvm2 a-- <1.82t 0
root@kvm3:/etc/apt/sources.list.d# vgs
VG #PV #LV #SN Attr VSize VFree
linstor_vg_nvme 1 1 0 wz--n- <1.82t 0
root@kvm3:/etc/apt/sources.list.d# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
vg_nvme linstor_vg_nvme twi-a-tz-- <1.82t 0.00 10.41
# Create cloudstack pool in Minster:
root@kvm3:/etc/apt/sources.list.d# linstor resource-group list
╭──────────────────────────────────────────────────────╮
┊ ResourceGroup ┊ SelectFilter ┊ VlmNrs ┊ Description ┊
╞══════════════════════════════════════════════════════╡
┊ DfltRscGrp ┊ PlaceCount: 2 ┊ ┊ ┊
╰──────────────────────────────────────────────────────╯
root@kvm3:/etc/apt/sources.list.d# linstor resource-group create cloudstack --place-count 2 --storage-pool nvme_pool
SUCCESS:
Description:
New resource group 'cloudstack' created.
Details:
Resource group 'cloudstack' UUID is: a7cb8c62-2e45-4753-b6c1-2c897f6081a8
linstor volume-group create cloudstack
# linstor volume-group list cloudstack
╭──────────────────╮
┊ VolumeNr ┊ Flags ┊
╞══════════════════╡
┊ 0 ┊ ┊
Testing volume in linstor:
linstor resource-group spawn cloudstack testres 1Gib
# linstor resource list
╭────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊
╞════════════════════════════════════════════════════════════════════════════════╡
┊ testres ┊ kvm1 ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2023-09-02 01:50:36 ┊
┊ testres ┊ kvm2 ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2023-09-02 01:50:36 ┊
┊ testres ┊ kvm3 ┊ 7000 ┊ Unused ┊ Ok ┊ TieBreaker ┊ 2023-09-02 01:50:33 ┊
╰────────────────────────────────────────────────────────────────────────────────╯
# drbdadm status
testres role:Secondary
disk:Diskless
kvm1 role:Secondary
peer-disk:UpToDate
kvm2 role:Secondary
peer-disk:UpToDate
Set different Nic:
linstor node interface create kvm1 nic_10g 10.1.1.20
linstor node interface create kvm2 nic_10g 10.1.1.21
linstor node interface create kvm3 nic_10g 10.1.1.22
linstor node set-property kvm1 PrefNic nic_10g
linstor node set-property kvm2 PrefNic nic_10g
linstor node set-property kvm3 PrefNic nic_10g
10G optimisation: https://kvaps.medium.com/deploying-linstor-with-proxmox-91c746b4035d
linstor controller drbd-options \
--max-buffers=36864 \
--rcvbuf-size=2097152 \
--sndbuf-size=1048576
linstor controller drbd-options \
--c-fill-target=10240 \
--c-max-rate=737280 \
--c-min-rate=20480 \
--c-plan-ahead=10
# ZFS
# apt-get install zfsutils-linux
# zpool create storage </dev/storage path>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment