server on Kevin Heruer

Using the same dataset mounted on an app and share it via SMB

Fri, 28 Jul 2023 01:00:00 +0100

Discovering the intricacies of dataset management can sometimes seem like an overwhelming puzzle. But, after a lot of learning and tinkering, I’ve finally mastered a method to concurrently mount a dataset and share it via SMB.

Why is This Important?

You might wonder, why go through all this trouble? Here’s my use case: I frequently upload videos to my Plex dataset, which is integrated with my Plex TrueNAS app. The conventional approach was to nest a dataset and mount the child dataset while concurrently sharing the parent dataset via SMB. This method, however, became nonviable following an update from TrueNAS.

Fortunately, it opened up two alternatives. The first is to disable host path validation, while the second merges the strengths of NFS and SMB shares using TrueCharts.

Why I Chose the Second Option

Given that I was already utilizing TrueCharts, the second option was more appealing. The process is simple: instead of the traditional host path, I employed the NFS option in the volume mount of the app. Prior to this change, it’s essential to establish the NFS share.

This decision ensured that I could still manually upload videos to my Plex dataset without any disruption or the need for workarounds. By integrating NFS and SMB shares via TrueCharts, I created a more robust and efficient solution.

Seagate Skyhawk Drives suck! Part 2

Sat, 15 Oct 2022 01:00:00 +0100

Last post was about the Seagate Skyhawk HDDs and how they seemed to die like flies. I replaced them with some Toshibas (which are noisier btw) but I wanted to be 100% sure if they were actually dying, so I put them in an external HDD caddy and read out the S.M.A.R.T values. Turns out there is nothing wrong with them??? I also used CrystalDiskInfo and CrystalDiskMark and they seem to be fine.

So the next thing on the list is the RAM, it’s possible the RAM is faulty and makes the system do unexpected things. They are non ECC because I’m running consumer grade hardware but that shouldn’t be the issue, so now I loaded half of the sticks into my desktop and I’m running MemTest86 to see if I can find anything.

So now I have 3x 4TB drives extra, I can’t put them back into the system because they’re 5700RPM instead of the 7200 the others are running on. An alternative for them would be to use them as backup drives, I still have an old 2-bay Synology NAS and backing up the critical stuff shoulnd’t be even close to 4TB making it perfect to run them in mirror.

Seagate Skyhawk Drives suck!

Sat, 08 Oct 2022 20:00:00 +0100

The Seagate Skyhawk drives are advertised as “surveillance” drives, this basically means that its made for lots of write actions.

But let me give you some backstory, I currently run a TrueNAS Scale server with 8TB in total in one pool split among 3 mirrored vdevs. 2 of these vdevs are 2 mirrored 2TB Skyhawks, the other and most recent one is a 4TB mirrored Toshiba vdev.

In a timespan of 2 years this server has run (1.5 years on Proxmox), I had to replace 3 Seagate drives. 1 2TB and both the 4TB ones. I know these drives arent made for read/write but still, they tend do die pretty quick with my setup. And the temperatures arent extremely high either.

I’ll be testing the Toshiba 4TB drives to see if they keep up better since these are advertised as NAS drives.

So at the bottom line, I do NOT recommend Seagate Skyhawk drives as a cheap option for your server, get something else that’s a bit more expensive instead. It’ll be cheaper in the end ;)

P.S. The drives were advertised as 7200 RPM at tweakers.net but in reality they’re 5400 RPM. The drives in queston are the ST4000VX013 models. Turns out I had to do some more research before buying, but they are the cheapest (at time of writing).

Increasing VM disk size: Final edition

Fri, 23 Jul 2021 21:47:00 +0100

This post merges both A little follow up on this post and Increasing Disk Size on Existing VM and fixes a couple of mistakes.

First off, go to your Proxmox host. Run the resize command:

bash


qm resize <vmid> <disk> <size>

where vmid is the ID in your VM ID, disk is the identifier of the disk inside the VM, it’s most likely to be scsi0. size is the size you want to add or remove, something like +100G or +1T is both valid.

Now go into your VM, run the following command to check if the system has notices a disk resize:

bash


dmesg | grep sda

Where sda is the disk, it could be vda or sdb.

Run the following command to get your disk and display the partitions, remember to change sda if required.

bash


fdisk -l /dev/sda | grep ^/dev

Remember the number of the partition you want to increase, it’s most likely the biggest one.

Run the following command to open parted with the drive:

bash


parted /dev/sda

Run a list to check if everything is recognized correctly and run the following command:

bash


resizepart <partition number, integer only> 100%

You could change the 100% to add only a percentage of the free space and add the remaining space to another partition, but to keep it simple we’ll give it all to this partition.

If it asks to fix the unused space type F to fix it, if it asks for a partition number and size; just put in the values like above.

Ctrl-C out of Parted and run df -h, you’ll notice that you still do not have your extra space! Run pvresize /dev/<block> to start resizing, remember to subsitute <block> with your partition like sda3.

Now run lvextend -l +100%FREE /dev/mapper/ubuntu--vg-ubuntu--lv and resize2fs /dev/mapper/ubuntu--vg-ubuntu--lv to actually resize your LVM partition. Remember to replace /dev/mapper/ubuntu--vg-ubuntu--lv with your LVM partition, you can find it by running df -h and looking for the one mounted on /.

That’s it! The drive has been expanded and readily available, no need to reboot!

Sources:
https://pve.proxmox.com/wiki/Resize_disks

Follow up on increasing existing VM disk size

Sun, 21 Mar 2021 13:47:57 +0100

A little follow up on this post: Increasing Disk Size on Existing VM Disk

When extending an LVM system you should also run the following commands:

pvresize /dev/sda3 (replace with your partition)

lvextend -l +100%FREE /dev/mapper/ubuntu--vg-ubuntu--lv (replace with your lvm partition)

resize2fs /dev/mapper/dev/ubuntu--vg-ubuntu--lv (again replace with your lvm partition)

Increasing Disk Size on Existing VM Disk

Thu, 21 Jan 2021 17:05:03 +0100

When you’re using your Proxmox server you might run in the problem of not having enough HDD space defined for some of your VMs, using ZFS (pools) it’s really easy to fix this.

Since all my VM disks are simple files you can easily increase their size by running the following command:

bash


qm resize <vmid> <disk> <size>

Let’s say you have a VM with a disk named vm-100-disk-0 and you wan to increase the disk size by 100GB, you just run

bash


qm resize 100 vm-100-disk-0 +100G

and you’re halfway there!

Now you need to tell the VM to use the newly added space, first you need to check if the system sees the new space by running

bash


dmesg | grep sda

Now we need the right partition to add the space to, run the following to check which partitions there are

bash


fdisk -l /dev/sda | grep ^/dev

Remember the number /dev/sda3 for example would be 3.
(Just make sure you’re grepping the right device, vda is also a possibility.)

Now run pared with the right device

bash


parted /dev/sda

It will ask you to fix the unused space, enter F for fix, then run

bash


resizepart 3 100%

This will resize partition (sda)3 to use 100% of the newly added space.

That’s it, no need to reboot!

You can find a more detailed description at the source: https://pve.proxmox.com/wiki/Resize_disks

Nextcloud

Wed, 20 Jan 2021 10:23:09 +0100

What is Nextcloud

Nextcloud is a complete software package to replace most cloud services like Google, iCloud, Dropbox etc. Nextcloud is branched off from the other solution ownCloud because (from what I’ve heard) a couple of devs did not like the direction it was going so they started their own product.

Nextcloud is not just a storage solution, it’s a complete package, it’s not blazingly fast and it’s written in PHP. But it does offer great integration with other services and has tons of plugins, there are plugins for monitoring your Nextcloud instance itself or third party services. It also supports contacts, calendars, Kanban boards, and even complete chat integrations with video.

Why Nextcloud

I mainly use Nextcloud as a Google Drive/Dropbox replacement, but the expandability is what really got me. I’m pretty sure I want more functionality in the future so using a different service like Seafile is not what I’m looking for in the long run, plus Nextcloud being completely free to host yourself without limitations is great.

A simple Samba/WebDAV was not going to cut it either, having a nice web interface is important since having to install some app to use it sucks and sometimes I work on public systems and having to connect to SMB/WebDAV is just not doable while opening a webpage is easy.

It also has apps, like for iOS, I recently moved all my photos from iCloud to my Nextcloud instance and with the auto upload feature I won’t have to worry about missing any photo’s or videos missing from my Nextcloud environment.

Just owning your own data is great, I’m 99% sure Apple won’t do any crazy things with my data, but you never know. Plus I don’t need to pay a montly fee to get more storage, a HDD is really cheap these days and already having a server certainly did help.

How to set it up

You can set up Nextcloud in multiple ways, I went with the manual one because I want to squeeze/optimize every little thing to save resources on my hardware.

I basically followed the documentation for installation on Linux here: https://docs.nextcloud.com/server/20/admin_manual/installation/source_installation.html I installed every required PHP module (and PHP itself) together with MariaDB, Redis, and SMB.

It’s recommended to install imagick, apcu, memcached, and redis module for PHP so you don’t run into any performance issues down the road. You could also always install these at a later moment when you DO run into these performance issues ofcourse.

Instead of following the Apache configuration I followed the Nginx one: https://docs.nextcloud.com/server/20/admin_manual/installation/nginx.html just because I’m more familiar with Nginx and I personally think it’s less resource hungry and faster. I changed some default things like client_max_body_size to 10G and ofcourse the server_name.

Instead of using SQLite I use MariaDB, it’s much faster and stable for long term use, Nextcloud also only recommends SQLite for development environments.

Optimization

Make sure to read https://docs.nextcloud.com/server/20/admin_manual/installation/server_tuning.html a lot of points here lead to a smooth server, I still need to optimize some things like PHP-FPM but at least make sure you’re using MariaDB/MySQL and Redis caching. These two will speed up things by a lot.

Next check out https://docs.nextcloud.com/server/20/admin_manual/configuration_server/caching_configuration.html it will explain/suggest the best caching settings for your Nextcloud instance. Remember PHP runs best when properly cached/optimized. Just running PHP and PHP-FPM with default settings is just OK, and that’s not good enough for most in production.

At last, if you’re seriously thinking about using Nextcloud as your main cloud storage/Dropbox replacement make sure you check out https://docs.nextcloud.com/server/20/admin_manual/installation/harden_server.html and be sure to run over HTTPS.

Oh and run in RAID1 or ZFS Mirror, I advise using ZFS even though it uses more RAM it’s worth it. Being able to have a pool you can modify however you want and having an abstract layer around your hardware is great and (in theory) I won’t lose any data if any (1) HDD fails in a mirror. Just don’t run striped RAIDs even when the 80% usable capacity is so tempting.

PS. redundancy isn’t a backup, make sure you have one. :)

PPS. this runs in a VM with 3GB of RAM and an intel i5 10500.

Proxmox Tip

Mon, 18 Jan 2021 17:51:06 +0100

When you’re using Proxmox for free without any subscription you may see

bash


TASK ERROR: command 'apt-get update' failed: exit code 100

The issue here is that by default Proxmox is using subscription only repos, you can remove this “error” by either getting a subscription (duh), or change the repo to pve-no-subscription. This repo is not as stable as the subscription one they claim, but for a home server it should be enough. If you really need the stability you’re probably running a critial environment which probably is worth paying for the subscription.

To change the repo edit your /etc/apt/sources.list and set the repo to pve-no-subscription.

bash


deb http://ftp.debian.org/debian buster main contrib
deb http://ftp.debian.org/debian buster-updates main contrib
# PVE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
deb http://download.proxmox.com/debian/pve buster pve-no-subscription
# security updates
deb http://security.debian.org/debian-security buster/updates main contrib

Source: https://pve.proxmox.com/wiki/Package_Repositories#sysadmin_no_subscription_repo

My Proxmox Setup

Mon, 11 Jan 2021 11:39:26 +0100

Proxmox VE is an interface for managing virtual machines and containers, it’s a lightweight solution that could run from ram/usb stick. Proxmox has more solutions like backup and email etc. but I only use the VE software. It’s free and open-source, is production ready, and is being worked on actively.

With Proxmox VE you can setup VMs on machines and also cluster multiple machines together, it’s easy to use (if you know how to use it) and fast. Currenly I am running 6 VMs on my server, most of them still need some work done but all of them are running Ubuntu Server 20.04.

Proxmox VE also supports hardware passthrough for when a VM needs it, in my case I use it for Plex to use the GPU for transcoding.

Storage management is also easy to do in Proxmox, however when building a server always calculate how much storage you thing you need and then double that. I am currently running into the issue that I’m running out of storage, mainly because I am running in ZFS mirror (RAID1) and cuts my storage in half. Thankfully storage is not terribly expensive, unless you need multiple tens of TB.

There are alternatives to Proxmox VE and it all comes down to your preference, the reason I’m using Proxmox is because it made sense the first time I saw it. It does not have a flashy UI but it works and everything is where it’s supposed to be. But I might change my mind in the future and start using unRAID, who knows.

server on Kevin Heruer

Using the same dataset mounted on an app and share it via SMB

The Efficacy of Mounting a Dataset and Sharing It via SMB Simultaneously

Why is This Important?

Why I Chose the Second Option

Further Reading

Seagate Skyhawk Drives suck! Part 2

Seagate Skyhawk Drives suck!

Increasing VM disk size: Final edition

Follow up on increasing existing VM disk size

Increasing Disk Size on Existing VM Disk

Nextcloud

What is Nextcloud

Why Nextcloud

How to set it up

Optimization

Proxmox Tip

My Proxmox Setup