Add Persistent Shared Storage to a Docker Swarm Using GlusterFS

I was interested in setting up persistent shared storage for some docker containers on my little Pi cluster. My research led me to GlusterFS using replicas. I used these sources:

GlusterFS Logo

View Post

Gluster’s docs describes their product as “a scalable, distributed file system that aggregates disk storage resources from multiple servers into a single global namespace.” What this means for me is a shared persistent storage space to save data and config files for my docker containers. Any changes to the mounted glusterfs drive is replicated to all the peer bricks. In this way it doesn’t matter which cluster node a container uses, all will have access to the same data.

I added three Samsung Plus 32GB USB 3.1 Flash Drives to FrankenPi for some additional high speed storage to host the gluster data. My host names are: picluster1, picluster2, picluster3. The operating system is Raspberry Pi OS. I’m using the Buster version released 2020-08-20

  1. Update your software with the apt command:
sudo apt update && sudo apt upgrade -y
  1. Install and configure GlusterFS on each server within the swarm.
sudo apt install software-properties-common glusterfs-server -y
  1. Start and enable GlusterFS with the commands:
sudo systemctl start glusterd
sudo systemctl enable glusterd
  1. If you haven’t already done so, you should generate an SSH key for all nodes and copy them to the other nodes.

    See: How to Setup Raspberry Pi SSH Keys for Authentication
    Use these sections:
    Generating SSH Keys on Linux based systems
    and
    Copying the Public Key using SSH Tools

    Verify the results by trying to connect with the other nodes, you should be able to login without a password after keys are installed. It makes life simpler when moving between nodes:
ssh picluster2
ssh picluster3
  1. Probe the nodes from the master node only (picluster1). This will add the other nodes as Gluster peers. The next command verifies the peers are listed as part of the storage pool. Then we exit root:
sudo -s
gluster peer probe picluster2;gluster peer probe picluster3;

gluster pool list
UUID Hostname State
02c0656a-0637-4889-a185-264b56d6b24f picluster3 Connected
2a5c475a-23de-455b-a6a9-45f08e855b3c picluster2 Connected
bcb63335-b76f-459e-8195-352f1c0e719d localhost Connected

exit
  1. Format the USB drives on each cluster node.

    Start by checking the device names so that you format the correct device with the lsblk command.
lsblk 
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 1 29.9G 0 disk
mmcblk0 179:0 0 29.7G 0 disk
├─mmcblk0p1 179:1 0 256M 0 part /boot
└─mmcblk0p2 179:2 0 29.5G 0 part /

The mmcblk0 device is the micro SD card with the OS partitions, so sda is my target device. If you have other storage devices connected to one of the cluster nodes like I do the results will be different (sdb, sdc, etc.). You should repeat the same command on each device to prevent a horrible mistake.

Format the CORRECT device:

sudo mkfs.xfs -i size=512 /dev/sda -f
  1. Create a mount point for the USB drives on each node. Increment the number at the end so that you can tell the devices apart. The storage devices are not interchangeable, unless you re-format and start over:
sudo mkdir /gluster/bricks/1 -p
  1. Add the device parameters to /etc/fstab to make it permanent, then mount it. Make sure you increment the number appropriately on each node:
sudo su echo '/dev/sda /gluster/bricks/1 xfs defaults 0 0' >> /etc/fstab 
mount -a exit
  1. Create the volume across the cluster. This is only run from one node, picluster1, in my use case:
sudo gluster volume create rep_swarm_vol replica 3 picluster1:/gluster/bricks/1 picluster2:/gluster/bricks/2 picluster3:/gluster/bricks/3 force

Let me break down the command:
sudo – Assume root privileges
gluster volume create – Initialize a new gluster volume
rep_swarm_vol – Arbitrary volume name
replica 3 – The type of pool and number of peers in the storage pool.
picluster1:/gluster/bricks/1 – The host and storage path for each peer.
force – It’s required for the version of Gluster support by the Raspberry Pi.

You can vary the number of replicas if you a different number of nodes. The host and path must reflect your setup if you’ve deviated from what was given above.

  1. Start the volume. This is ran on one node:
sudo gluster volume start rep_swarm_vol
  1. Create a mount point for the replicated gluster volume on each node, add it to fstab to make it permanent, and mount the volume:
sudo -s
mkdir -p /mnt/glusterfs
echo 'localhost:/rep_swarm_vol /mnt/glusterfs glusterfs defaults,_netdev,backupvolfile-server=localhost 0 0' >> /etc/fstab
mount.glusterfs localhost:/rep_swarm_vol /mnt/glusterfs
  1. Set permissions and exit sudo. I use the docker group, but root:root will probably work for most uses:
chown -R root:docker /mnt/glusterfs
exit
  1. Verify all is well. The df -h command should show you backend storage brick, as well as the replicated storage volume, similar to what I’ve highlighted below:
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 29G 4.5G 24G 17% /
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 17M 1.9G 1% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/mmcblk0p1 253M 54M 199M 22% /boot
/dev/sda 30G 408M 30G 2% /gluster/bricks/1
tmpfs 385M 0 385M 0% /run/user/1000
localhost:/rep_swarm_vol 30G 717M 30G 3% /mnt/glusterfs

You should be able to create new files in the /mnt/glusterfs directory and they’ll show up in the same directory on each node.

For example, you want persistent storage for a MySQL database. In your docker YAML files, you could add a section like this:

volumes: 
- type: bind
source: /mnt/glusterfs/staging_mysql
target: /opt/mysql/data

Raspberry Pi4 4GB and Ubuntu 20.04.1 LTS ARM64

I was trying to see if there was better OS for Pi4. I’ve been using RaspberryOS, they have a beta 64 bit version with a pretty long bug list. So I thought I’d give Ubuntu a try and see if it would be good for day to day use.

My initial impression is it’s soooo sloooowwww. The apt update and apt upgrade process took much longer than I’ve seen before on a Pi, including the Pi2b I started with in 2014. Installing cups and hplip to run a print server was a test of patience. Docker was much slower than on RaspberryOS.

I was rather hoping this would work out because there looked to be lower learning curve coming from RaspberryOS. I’m going to look at Apline next. They also support ARM64. And it has smaller footprint.

FrankenPi Lives!

I was using a couple of Raspberry Pi4 4GBs and one Pi3+ single board computers for various tasks within our home network:

It occured to me that I wasn’t fully utilizing the hardware. So I’ve decided to build a Pi4 cluster. I wanted learn more about docker and cluster computing.

Hardware

The Pi4 4GB has a later generation, faster 64 bit, 4 core CPU, and 4 times the memory of the Pi3b+ that’s currently running my Pihole DNS. It also moved Ethernet off of the USB bus removing a major network performance bottleneck compared to the Pi3b+.These improvements make the Pi4B a much better fit for this use case. Plus, I had two on hand already. Frugality is always a plus.

I haven’t decided if I want to make use of the Pi3b+ in this project because of it’s performance constraints. I will eventually add it to the case and use it to run foldforcovid.io, but probably will not add it to the docker swarm. That’s a great project if you have a spare computer and want to use it to help find a cure for COVID-19.

The GeeekPi case was inexpensive when compared to more advanced cases like the Cloudlet case which sells for $69. You do need to disassemble the case to add/remove boards with the cheaper case, but I didn’t foresee a need to do that very often. The GeeekPi case did come with a full complement of Pi3 and Pi4 heatsinks for 4 boards.

The D-Link Switch was supports gigabit and uses a 5V DC power supply. It can also run off the USB charging station with a bit of creative cable modification. Most USB cables use red for +5V. Check the power adapter specs to verify the where the +5V is. Mine had a stripe on the +5V wire from the AC adapter to make it simple.

Edit: I decided to go with the $7 USB A to barrel connector cable after the switch lost power a couple of times. I didn’t have the right size butt connectors to join the power supply wires to the much smaller USB wire, they kept slipping out. The pre-made cable makes it look better as well.

Another valid choice would be using a PoE switch instead. That would require obtaining 3 PoE hats, as well as a compatible PoE Ethernet switch. The PoE switches sell for $50 – $60 on Amazon. The Pi PoE hats are ~$20 each. That would eliminate the USB cables and the charging station and leave just the ethernet cables.

As a point of comparison, the Picocluster 3H with three Pi4s and 3 x 32gb SD cards is $444 if you want to assemble it yourself. It’s $499 if you want it already assembled. Add $20 for US shipping.

The storage for my FrakenPi is a Mediasonic ProBox HF2-SU3S2 4 Bay 3.5” SATA HDD USB 3.0 Enclosure with a pair of 2TB drives, plus a 3TB drive. The pair of 2TB drives are setup has a RAID-1 array and the 3TB is used as an Apple formatted Time Machine backup. This is used to host video files, photos, backup and, a Bittorrent service using Transmission. It takes advantage of the Pi4 USB 3.0. It also supports connection via external SATA. I may pull the old 2TB drives and replace them with 4TB Seagate Ironwolf HDDs. I did pull the existing drives out of an old NAS and an old Western Digital external HDD I had. Frugality is my friend.

Software

There are dozens of tutorials on how to do this. I rather like the approach taken by HackViking using Docker Swarm and Swarmpit. I’m going to start there. That will be my first pass learning experience. I may decide redo the whole thing with k3s lightweight Kubernetes instead. My goal is to learn about cluster computing. There will be no mistakes. Because every mistake is a learning experience.

This is what it looks like after the build:

FrankenPi

YouTube TV

YouTube tv logo

I finally decided on a replacement for Frontier Cable TV. I’ve been using YouTube TV for about 2 1/2 months. It’s pretty responsive compared to Hulu TV. It’s fairly flexible in recording content. It supports multiple profiles with separate recording of programs. It’s still slow compared to cable TV in starting up, changing channels, or starting a recorded program.

You lose all the cable boxes in favor of an Apple TV, Android TV, Roku or Samsung Smart TV. I already had a 4th generation Apple TV, a Roku Tv and a Sony Bravia Android TV. I needed no new devices.

It lack access to Scripps Networks and Discovery Communications programming like HGTV, Travel Channel, Discovery, TLC and Food Network. Hopefully this can be corrected by YouTube coming to and agreement with the content owners. Right now it leaves something of a hole in their programming. We work around it by using the HGTV and Food Network apps to catch at least some of their shows.

Sports and News offerings are more rounded than the lifestyle offerings. Plenty of ESPN and cable news networks are available.

All in all it wasn’t as clean of a transition as I wanted, but we do get most of what we want and have saved money compared to the the TV service from Frontier or Comcast. We do still use Frontier FIOS to support it.

Let’s hope the FCC’s recent decision to gut network neutrality rules doesn’t mean my internet carrier will slow my YouTube TV service just because they can’t extort additional payments from consumers to make up for their horrible business model collapsing under its own weight. It does mean they can and the question becomes when will they?

Grilled Tuna Salad

A bed of arugula, garden fresh peppers and tomatoes. The tuna was about a 6 once portion and an inch thick. It’s seasoned with a bit of sesame oil, soy sauce, grated ginger, garlic powder and onion powder. I cooked in a hot non-stick pan for just 2-3 minutes per side. I use a little balsamic glaze to dress it. From “what can I have for lunch” to completion only took about 10 minutes. Simple, fresh, delicious…

Pinakbet

This is great, I’ve had twice this week. The calabasa is rather like a small green pumpkin and resembles an acorn squash, but with somewhat softer flesh that cooks more quickly. Magic Sarap is a flavor enhancer that contains MSG, feel free to omit is you’re sensitive to it. Long beans resemble 18″ long green beans. We sometimes substitute regular green beans, if they can’t be found at our usual exotic grocery, MD Oriental in Tampa.

Serves 4

  • 1 1/2 cup cubed calabasa (Asian squash)
  • 6 – 8 long beans cut into bite-size pieces
  • 1/2 cup Onion, chopped
  • 2 Cloves of garlic
  • 1/2 cup water
  • 1 medium tomato chopped
  • 1 teaspoon or less of sugar, to taste
  • 1/4 teaspoon Magic Sarap seasoning
  • Shrimp paste (bagoong)
  • Chicharrones
  1. Sauté the onion and garlic for 4 or 5 minutes.
  2. Add the beans, tomato and 1/2 cup of water to help them steam. Cook for 5-6 minutes.
  3. Add the calabasa and Magic Sarap. Simmer until the calabasa is tender. It cooks faster than pumpkin or acorn squash, give it 10-15 minutes.
  4. Top with some crushed chicharron pieces and serve.

Served with smoked bangus and rice

Today’s Lunch: Ensalada Ng Talong (Eggplant Salad)

We did this Filipino style vegetable dish. It’s simple, low calorie and tasty! I ate it before I thought of taking a picture. Maybe next time. I swear I know how to cook other things, I’m just on this Filipino kick lately. It’s under appreciated.

Serves 2

  • 2 Chinese eggplants
  • 1 medium tomato, seeded and chopped
  • Onion, 1/4 cup chopped
  • 1 tablespoon or to taste of Filipino shrimp paste (bagoong), I use spicy from Barrio Fiesta. That’s the best we can find near by.
  1. Pierce the eggplants a few times on each side with a fork and grill or cook under a broiler until the skin just begins to split, about 4 minutes per side.
  2. While that’s cooking seed and chop the tomato and onion and mix with the bagoong. Salt to taste if needed, the bagoong is pretty salty by itself.
  3. Slice the eggplant lengthwise and use a spoon to scrape the meat from the skin. Chop it up and top with the tomato mixture.

The Expanse is back on Syfy Channel!

This is the best science fiction show on TV right now. I just saw it popped back up on my DVR. It’s got great production values, political intrigue and a multilayered story line. I’m catching up with the first two episodes right now. It’s available on their on demand service thru your cable provider, the Syfy app on iOS, TVOS, Roku and AndroidTV.

8

Website Powered by WordPress.com.

Up ↑