Proxmox

• Bootstrapping
• Installing Proxmox
• Preparing Proxmox for TrueNAS
• Installing TrueNAS
• Networking TrueNAS
• Backups
• Proxmox backups
• Design
• Handleidingen

Bootstrapping

Installing Proxmox

This page walks through setting up Proxmox, preparing it for TrueNAS and recovering all VMs.

Install Proxmox

Prepare the installation media

1. Install Rufus: https://rufus.ie/en/
2. Download the latest version of Proxmox (google "proxmox iso")
   
3. Grab a USB drive of about 32GB or larger.
4. Use Rufus to flash the Proxmox ISO to the USB drive.

Install Proxmox

Generally, just follow the questions on screen. When asked what drive to install to, select the smallest one. The system usually has multiple harddrives and multiple SSDs, but the boot SSD should be the odd one out.

Configuring Proxmox

Networking

What we will be setting up is two bridge networks. One is vmbr0, 

Preparing Proxmox for TrueNAS

Introduction

This page walks through setting up networking in preparation for TrueNAS.

Prerequisites

Your setup before is expected to be:

• The machine is physically connected to some form of upstream internet;
• The only network bridge that currently exists is the default vmbr0.

Steps

Creating the inter-VM network bridge

To allow virtual machines to communicate safely, we will be creating a new bridge vmbr1. VMs that require direct communication with each other with no additional routing in between them will use this bridge to talk to each other.

1. Select your Proxmox node.
2. Go to System > Network
3. Click Create > Linux Bridge:
4. A popup should appear. Fill it out like so:

   
   The IPv4 we set here is the address at which the Proxmox node itself will be reachable on this bridge. If left empty, it will not be reachable by VMs.
   

5. Click Create.

6. Apply the changes:
   

The inter-VM network bridge is now created, and Proxmox itself is reachable on the bridge via the address 192.168.255.10.

You are now ready to install TrueNAS.

Installing TrueNAS

Installing TrueNAS

Lots of writing :(

Networking

Assuming you have created the inter-VM bridge network vmbr1, we can now connect it to TrueNAS.

Connect TrueNAS to the newly created bridge

This step requires that you have installed TrueNAS Scale in a VM.

1. Select your TrueNAS VM
2. Go to Hardware
3. Click Add
4. Select Network Device:
   
5. Select the newly created vmbr1 and click Add:
   

   

Viewing TrueNAS network configuration

1. Open the TrueNAS web UI using whatever IP address TrueNAS is reachable at.
2. Navigate to Networking. It should look roughly like this:

Primarily, you will now see two interfaces. Most likely, only one of them will have an IP address assigned already. This is the WAN, or the "where TrueNAS gets internet" interface.

The second interface however will probably not have any IP address. This is the vmbr1 inter-VM bridge network we created.

Configuring the bridge interface

Make absolutely sure you do not modify the primary network interface, as doing so may make the machine unreachable via network.

Configure the inter-VM bridge to not use DHCP, and instead manually assign the IP 192.168.255.20:

Save the settings, click [Test Configuration] and then [Save Configuration].

Summary

• We installed TrueNAS Scale in a VM;
• We connected TrueNAS to our dedicated network bridge vmbr1 that will let Proxmox access TrueNAS directly.

Good work!

Networking TrueNAS

Assuming you have created the inter-VM bridge network vmbr1, we can now connect it to TrueNAS.

Connect TrueNAS to the newly created bridge

This step requires that you have installed TrueNAS Scale in a VM.

1. Select your TrueNAS VM
2. Go to Hardware
3. Click Add
4. Select Network Device:
   
5. Select the newly created vmbr1 and click Add:
   

   

Viewing TrueNAS network configuration

1. Open the TrueNAS web UI using whatever IP address TrueNAS is reachable at.
2. Navigate to Networking. It should look roughly like this:

Primarily, you will now see two interfaces. Most likely, only one of them will have an IP address assigned already. This is the WAN, or the "where TrueNAS gets internet" interface.

The second interface however will probably not have any IP address. This is the vmbr1 inter-VM bridge network we created.

Configuring the bridge interface

Make absolutely sure you do not modify the primary network interface, as doing so may make the machine unreachable via network.

Configure the inter-VM bridge to not use DHCP, and instead manually assign the IP 192.168.255.20:

Save the settings, click [Test Configuration] and then [Save Configuration].

Summary

• We installed TrueNAS Scale in a VM;
• We connected TrueNAS to our dedicated network bridge vmbr1 that will let Proxmox access TrueNAS directly.

Good work!

Backups

Proxmox backups

Design

• From Proxmox, we will be backing up VMs and containers to the TrueNAS that is running in Proxmox.
• In Proxmox, we mount the alpha/proxmox/snapshots dataset via NFS as alpha-snapshots.
• Backups from Proxmox are done hourly on the hour (**:00) and copied to alpha-snapshots.
• In TrueNAS, backups are made from alpha/proxmox/snapshots to Backblaze B2.

Configuration

Mount the target volume

Mount the alpha/proxmox/snapshots dataset via NFS, call it alpha-snapshots.

Make sure to use the dedicated virtual network bridge vmbr1. 

Create the backup job

Select Datacenter > Backups, create a backup job like so:

Mind the Notification system and Mode settings.

The above may change depending on needs, but it's what it's set to at the time of writing.

The only changed setting here is the bandwidth limit. The only reason it's set is to prevent completely hogging the disk I/O for several minutes. Yes, it takes longer as a result, but at least other services using the pool won't be I/O starved.

Configure vzdump

By default, vzdump creates temporary files in the target location (where the backup will go). As a result, it will try to symlink unprivileged container filesystems onto the NFS volume. This causes permission problems, even when NFS is set to mapall user/group.

To fix this problem, modify /etc/vzdump.conf:

The only line changed is the tmpdir: one. Set it to /tmp to prevent doing the compression and such on the target volume.

Design

Storage

Networking

Handleidingen

 

Disks (in Debian installer)

• Gebruik 100% voor partitioning, met LVM en encryption
• Verwijder alle standaard Logical Volumes
• Maak een nieuwe LV: vg-root met size 64G
• Maak een nieuwe LV: vg-swap met size 16G
• Initialiseer vg-root als EXT4 Journaling Filesystem op mount point /
• Initialiseer vg-swap als swap area

Na installatie, gebruik 100% van de resterende LVM ruimte voor een thin pool lv-data:

$ lvcreate --type thin-pool -l 100%FREE -n lv-data naam-van-jouw-vg-hier

Fix Ceph OSDs (na node reinstall)

Edit /etc/pve/ceph.conf en:

• Verwijder alle verwijzingen naar de node onder [global] mon_host = ...
• Verwijder alle verwijzingen naar de mon onder [mds.<node>] en  [mon.<node>]

Dan:

$ ceph-volume lvm activate --all

Nu kan je de OSDs starten in de command line.

Fix IOMMU

Edit /etc/default/grub:

...
GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt pcie_acs_override=downstream,multifunction"
...

En dan:

$ update-grub