VM on Kevin Heruer

Monitoring is important

Fri, 21 Jan 2022 21:00:00 +0100

I’ve been experimenting with monitoring systems because I was not satisfied with Proxmox’s limited reporting of system stats. But that was not the only reason I set up a monitoring system.

I stumbled upon SNMP (Simple Network Management Protocol) and found it it’s implemented in anything that has a network connection, it’s even implemented in printers and routers. This protocol lets you fetch (and set) a limited set of data, but this is enough for monitoring. At first I set up Observium, it’s a valid system but I quickly ran in the problem that it doesn’t play nice with mDNS for some reason. I also have not set up an enterprice grade network so I do not have a local DNS server. I do not want to hack into hosts files or use IP addresses even though all non-user systems have static IP addresses.

Next was Zabbix, it’s open source and very mature software. Setting it up (correctly) was quite a lot of work, and the learning curve is a bit steep if you never really used complete monitoring systems before. However, once Zabbix was set up and using the built-in templates it’s quite easy to start monitoring a server.

Currently I have 11 servers running in VM on Proxmox, adding a server to Zabbix is as easy as apt install zabbix-agent and adding an IP range into the config. The most work I had to do for a server was getting the source and building it myself which was unnecessary because I had to restore the backup anyway (long story short: I upgraded the OS, it got borked).

Once I started adding my hosts I noticed that I was getting long disk write wait’s, this could mean a lot of things. Including failing drives. I ignored it mostly and continued adding all the servers, once that was done I saw more servers throwing this error. I started investigating and found out there were 2 servers running MySQL/MariaDB database servers for the applications running on the servers, turns out that database servers love to read and write (duh). They were slamming the cheap HDD’s I’m running in the server which they didn’t like, I optimized the configs for the databases and now most is well.

I did notice an increase in server/VM responsiveness but the warning(s) still remain, however looking at the graph it does show a great improvement and it currently just barely triggers the warning each time. I’m thinking of increasing the check for now and see if I can upgrade the HDD’s in the future.

Another thing I noticed was that running a DB from an SSD really makes a difference:

It’s obvious SSDs are fast, but seeing a direct impact is really cool! I suggest you start setting up a monitoring system if you have any interest, the things you can do with the data is amazing. I might even connect Home Assistant to Zabbix so I have a single dashboard for all the data.

Increasing VM disk size: Final edition

Fri, 23 Jul 2021 21:47:00 +0100

This post merges both A little follow up on this post and Increasing Disk Size on Existing VM and fixes a couple of mistakes.

First off, go to your Proxmox host. Run the resize command:

bash


qm resize <vmid> <disk> <size>

where vmid is the ID in your VM ID, disk is the identifier of the disk inside the VM, it’s most likely to be scsi0. size is the size you want to add or remove, something like +100G or +1T is both valid.

Now go into your VM, run the following command to check if the system has notices a disk resize:

bash


dmesg | grep sda

Where sda is the disk, it could be vda or sdb.

Run the following command to get your disk and display the partitions, remember to change sda if required.

bash


fdisk -l /dev/sda | grep ^/dev

Remember the number of the partition you want to increase, it’s most likely the biggest one.

Run the following command to open parted with the drive:

bash


parted /dev/sda

Run a list to check if everything is recognized correctly and run the following command:

bash


resizepart <partition number, integer only> 100%

You could change the 100% to add only a percentage of the free space and add the remaining space to another partition, but to keep it simple we’ll give it all to this partition.

If it asks to fix the unused space type F to fix it, if it asks for a partition number and size; just put in the values like above.

Ctrl-C out of Parted and run df -h, you’ll notice that you still do not have your extra space! Run pvresize /dev/<block> to start resizing, remember to subsitute <block> with your partition like sda3.

Now run lvextend -l +100%FREE /dev/mapper/ubuntu--vg-ubuntu--lv and resize2fs /dev/mapper/ubuntu--vg-ubuntu--lv to actually resize your LVM partition. Remember to replace /dev/mapper/ubuntu--vg-ubuntu--lv with your LVM partition, you can find it by running df -h and looking for the one mounted on /.

That’s it! The drive has been expanded and readily available, no need to reboot!

Sources:
https://pve.proxmox.com/wiki/Resize_disks

Follow up on increasing existing VM disk size

Sun, 21 Mar 2021 13:47:57 +0100

A little follow up on this post: Increasing Disk Size on Existing VM Disk

When extending an LVM system you should also run the following commands:

pvresize /dev/sda3 (replace with your partition)

lvextend -l +100%FREE /dev/mapper/ubuntu--vg-ubuntu--lv (replace with your lvm partition)

resize2fs /dev/mapper/dev/ubuntu--vg-ubuntu--lv (again replace with your lvm partition)

Increasing Disk Size on Existing VM Disk

Thu, 21 Jan 2021 17:05:03 +0100

When you’re using your Proxmox server you might run in the problem of not having enough HDD space defined for some of your VMs, using ZFS (pools) it’s really easy to fix this.

Since all my VM disks are simple files you can easily increase their size by running the following command:

bash


qm resize <vmid> <disk> <size>

Let’s say you have a VM with a disk named vm-100-disk-0 and you wan to increase the disk size by 100GB, you just run

bash


qm resize 100 vm-100-disk-0 +100G

and you’re halfway there!

Now you need to tell the VM to use the newly added space, first you need to check if the system sees the new space by running

bash


dmesg | grep sda

Now we need the right partition to add the space to, run the following to check which partitions there are

bash


fdisk -l /dev/sda | grep ^/dev

Remember the number /dev/sda3 for example would be 3.
(Just make sure you’re grepping the right device, vda is also a possibility.)

Now run pared with the right device

bash


parted /dev/sda

It will ask you to fix the unused space, enter F for fix, then run

bash


resizepart 3 100%

This will resize partition (sda)3 to use 100% of the newly added space.

That’s it, no need to reboot!

You can find a more detailed description at the source: https://pve.proxmox.com/wiki/Resize_disks

My Proxmox Setup

Mon, 11 Jan 2021 11:39:26 +0100

Proxmox VE is an interface for managing virtual machines and containers, it’s a lightweight solution that could run from ram/usb stick. Proxmox has more solutions like backup and email etc. but I only use the VE software. It’s free and open-source, is production ready, and is being worked on actively.

With Proxmox VE you can setup VMs on machines and also cluster multiple machines together, it’s easy to use (if you know how to use it) and fast. Currenly I am running 6 VMs on my server, most of them still need some work done but all of them are running Ubuntu Server 20.04.

Proxmox VE also supports hardware passthrough for when a VM needs it, in my case I use it for Plex to use the GPU for transcoding.

Storage management is also easy to do in Proxmox, however when building a server always calculate how much storage you thing you need and then double that. I am currently running into the issue that I’m running out of storage, mainly because I am running in ZFS mirror (RAID1) and cuts my storage in half. Thankfully storage is not terribly expensive, unless you need multiple tens of TB.

There are alternatives to Proxmox VE and it all comes down to your preference, the reason I’m using Proxmox is because it made sense the first time I saw it. It does not have a flashy UI but it works and everything is where it’s supposed to be. But I might change my mind in the future and start using unRAID, who knows.