Add Persistent Shared Storage to a Docker Swarm Using GlusterFS

I was interested in setting up persistent shared storage for some docker containers on my little Pi cluster. My research led me to GlusterFS using replicas. I used these sources:

GlusterFS Logo

View Post

Gluster’s docs describes their product as “a scalable, distributed file system that aggregates disk storage resources from multiple servers into a single global namespace.” What this means for me is a shared persistent storage space to save data and config files for my docker containers. Any changes to the mounted glusterfs drive is replicated to all the peer bricks. In this way it doesn’t matter which cluster node a container uses, all will have access to the same data.

I added three Samsung Plus 32GB USB 3.1 Flash Drives to FrankenPi for some additional high speed storage to host the gluster data. My host names are: picluster1, picluster2, picluster3. The operating system is Raspberry Pi OS. I’m using the Buster version released 2020-08-20

  1. Update your software with the apt command:
sudo apt update && sudo apt upgrade -y
  1. Install and configure GlusterFS on each server within the swarm.
sudo apt install software-properties-common glusterfs-server -y
  1. Start and enable GlusterFS with the commands:
sudo systemctl start glusterd
sudo systemctl enable glusterd
  1. If you haven’t already done so, you should generate an SSH key for all nodes and copy them to the other nodes.

    See: How to Setup Raspberry Pi SSH Keys for Authentication
    Use these sections:
    Generating SSH Keys on Linux based systems
    and
    Copying the Public Key using SSH Tools

    Verify the results by trying to connect with the other nodes, you should be able to login without a password after keys are installed. It makes life simpler when moving between nodes:
ssh picluster2
ssh picluster3
  1. Probe the nodes from the master node only (picluster1). This will add the other nodes as Gluster peers. The next command verifies the peers are listed as part of the storage pool. Then we exit root:
sudo -s
gluster peer probe picluster2;gluster peer probe picluster3;

gluster pool list
UUID Hostname State
02c0656a-0637-4889-a185-264b56d6b24f picluster3 Connected
2a5c475a-23de-455b-a6a9-45f08e855b3c picluster2 Connected
bcb63335-b76f-459e-8195-352f1c0e719d localhost Connected

exit
  1. Format the USB drives on each cluster node.

    Start by checking the device names so that you format the correct device with the lsblk command.
lsblk 
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 1 29.9G 0 disk
mmcblk0 179:0 0 29.7G 0 disk
├─mmcblk0p1 179:1 0 256M 0 part /boot
└─mmcblk0p2 179:2 0 29.5G 0 part /

The mmcblk0 device is the micro SD card with the OS partitions, so sda is my target device. If you have other storage devices connected to one of the cluster nodes like I do the results will be different (sdb, sdc, etc.). You should repeat the same command on each device to prevent a horrible mistake.

Format the CORRECT device:

sudo mkfs.xfs -i size=512 /dev/sda -f
  1. Create a mount point for the USB drives on each node. Increment the number at the end so that you can tell the devices apart. The storage devices are not interchangeable, unless you re-format and start over:
sudo mkdir /gluster/bricks/1 -p
  1. Add the device parameters to /etc/fstab to make it permanent, then mount it. Make sure you increment the number appropriately on each node:
sudo su echo '/dev/sda /gluster/bricks/1 xfs defaults 0 0' >> /etc/fstab 
mount -a exit
  1. Create the volume across the cluster. This is only run from one node, picluster1, in my use case:
sudo gluster volume create rep_swarm_vol replica 3 picluster1:/gluster/bricks/1 picluster2:/gluster/bricks/2 picluster3:/gluster/bricks/3 force

Let me break down the command:
sudo – Assume root privileges
gluster volume create – Initialize a new gluster volume
rep_swarm_vol – Arbitrary volume name
replica 3 – The type of pool and number of peers in the storage pool.
picluster1:/gluster/bricks/1 – The host and storage path for each peer.
force – It’s required for the version of Gluster support by the Raspberry Pi.

You can vary the number of replicas if you a different number of nodes. The host and path must reflect your setup if you’ve deviated from what was given above.

  1. Start the volume. This is ran on one node:
sudo gluster volume start rep_swarm_vol
  1. Create a mount point for the replicated gluster volume on each node, add it to fstab to make it permanent, and mount the volume:
sudo -s
mkdir -p /mnt/glusterfs
echo 'localhost:/rep_swarm_vol /mnt/glusterfs glusterfs defaults,_netdev,backupvolfile-server=localhost 0 0' >> /etc/fstab
mount.glusterfs localhost:/rep_swarm_vol /mnt/glusterfs
  1. Set permissions and exit sudo. I use the docker group, but root:root will probably work for most uses:
chown -R root:docker /mnt/glusterfs
exit
  1. Verify all is well. The df -h command should show you backend storage brick, as well as the replicated storage volume, similar to what I’ve highlighted below:
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 29G 4.5G 24G 17% /
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 17M 1.9G 1% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/mmcblk0p1 253M 54M 199M 22% /boot
/dev/sda 30G 408M 30G 2% /gluster/bricks/1
tmpfs 385M 0 385M 0% /run/user/1000
localhost:/rep_swarm_vol 30G 717M 30G 3% /mnt/glusterfs

You should be able to create new files in the /mnt/glusterfs directory and they’ll show up in the same directory on each node.

For example, you want persistent storage for a MySQL database. In your docker YAML files, you could add a section like this:

volumes: 
- type: bind
source: /mnt/glusterfs/staging_mysql
target: /opt/mysql/data

Raspberry Pi4 4GB and Ubuntu 20.04.1 LTS ARM64

I was trying to see if there was better OS for Pi4. I’ve been using RaspberryOS, they have a beta 64 bit version with a pretty long bug list. So I thought I’d give Ubuntu a try and see if it would be good for day to day use.

My initial impression is it’s soooo sloooowwww. The apt update and apt upgrade process took much longer than I’ve seen before on a Pi, including the Pi2b I started with in 2014. Installing cups and hplip to run a print server was a test of patience. Docker was much slower than on RaspberryOS.

I was rather hoping this would work out because there looked to be lower learning curve coming from RaspberryOS. I’m going to look at Apline next. They also support ARM64. And it has smaller footprint.

FrankenPi Lives!

I was using a couple of Raspberry Pi4 4GBs and one Pi3+ single board computers for various tasks within our home network:

It occured to me that I wasn’t fully utilizing the hardware. So I’ve decided to build a Pi4 cluster. I wanted learn more about docker and cluster computing.

Hardware

The Pi4 4GB has a later generation, faster 64 bit, 4 core CPU, and 4 times the memory of the Pi3b+ that’s currently running my Pihole DNS. It also moved Ethernet off of the USB bus removing a major network performance bottleneck compared to the Pi3b+.These improvements make the Pi4B a much better fit for this use case. Plus, I had two on hand already. Frugality is always a plus.

I haven’t decided if I want to make use of the Pi3b+ in this project because of it’s performance constraints. I will eventually add it to the case and use it to run foldforcovid.io, but probably will not add it to the docker swarm. That’s a great project if you have a spare computer and want to use it to help find a cure for COVID-19.

The GeeekPi case was inexpensive when compared to more advanced cases like the Cloudlet case which sells for $69. You do need to disassemble the case to add/remove boards with the cheaper case, but I didn’t foresee a need to do that very often. The GeeekPi case did come with a full complement of Pi3 and Pi4 heatsinks for 4 boards.

The D-Link Switch was supports gigabit and uses a 5V DC power supply. It can also run off the USB charging station with a bit of creative cable modification. Most USB cables use red for +5V. Check the power adapter specs to verify the where the +5V is. Mine had a stripe on the +5V wire from the AC adapter to make it simple.

Edit: I decided to go with the $7 USB A to barrel connector cable after the switch lost power a couple of times. I didn’t have the right size butt connectors to join the power supply wires to the much smaller USB wire, they kept slipping out. The pre-made cable makes it look better as well.

Another valid choice would be using a PoE switch instead. That would require obtaining 3 PoE hats, as well as a compatible PoE Ethernet switch. The PoE switches sell for $50 – $60 on Amazon. The Pi PoE hats are ~$20 each. That would eliminate the USB cables and the charging station and leave just the ethernet cables.

As a point of comparison, the Picocluster 3H with three Pi4s and 3 x 32gb SD cards is $444 if you want to assemble it yourself. It’s $499 if you want it already assembled. Add $20 for US shipping.

The storage for my FrakenPi is a Mediasonic ProBox HF2-SU3S2 4 Bay 3.5” SATA HDD USB 3.0 Enclosure with a pair of 2TB drives, plus a 3TB drive. The pair of 2TB drives are setup has a RAID-1 array and the 3TB is used as an Apple formatted Time Machine backup. This is used to host video files, photos, backup and, a Bittorrent service using Transmission. It takes advantage of the Pi4 USB 3.0. It also supports connection via external SATA. I may pull the old 2TB drives and replace them with 4TB Seagate Ironwolf HDDs. I did pull the existing drives out of an old NAS and an old Western Digital external HDD I had. Frugality is my friend.

Software

There are dozens of tutorials on how to do this. I rather like the approach taken by HackViking using Docker Swarm and Swarmpit. I’m going to start there. That will be my first pass learning experience. I may decide redo the whole thing with k3s lightweight Kubernetes instead. My goal is to learn about cluster computing. There will be no mistakes. Because every mistake is a learning experience.

This is what it looks like after the build:

FrankenPi

YouTube TV

YouTube tv logo

I finally decided on a replacement for Frontier Cable TV. I’ve been using YouTube TV for about 2 1/2 months. It’s pretty responsive compared to Hulu TV. It’s fairly flexible in recording content. It supports multiple profiles with separate recording of programs. It’s still slow compared to cable TV in starting up, changing channels, or starting a recorded program.

You lose all the cable boxes in favor of an Apple TV, Android TV, Roku or Samsung Smart TV. I already had a 4th generation Apple TV, a Roku Tv and a Sony Bravia Android TV. I needed no new devices.

It lack access to Scripps Networks and Discovery Communications programming like HGTV, Travel Channel, Discovery, TLC and Food Network. Hopefully this can be corrected by YouTube coming to and agreement with the content owners. Right now it leaves something of a hole in their programming. We work around it by using the HGTV and Food Network apps to catch at least some of their shows.

Sports and News offerings are more rounded than the lifestyle offerings. Plenty of ESPN and cable news networks are available.

All in all it wasn’t as clean of a transition as I wanted, but we do get most of what we want and have saved money compared to the the TV service from Frontier or Comcast. We do still use Frontier FIOS to support it.

Let’s hope the FCC’s recent decision to gut network neutrality rules doesn’t mean my internet carrier will slow my YouTube TV service just because they can’t extort additional payments from consumers to make up for their horrible business model collapsing under its own weight. It does mean they can and the question becomes when will they?

My Take on the Instant Pot

I am absolutely in love with the Instant Pot! Great soups and stews using one of the pressure cooking settings, Perfect rice automatically. Slow cooker settings with a delayed start, timed cook, switches to warm then auto shut off in an intelligent way.

I’ll prepare the ingredients for a stew or soup in the pot, leave it in the refrigerator until work is over, then start it up and have a tender delicious stew that tastes like it was cooked all day in under an hour. Very strong recommendation!

Note that the Instant Pot is far from instant. It takes 10 or 15 minutes to heat up before the cycle starts. In the end, you need to vent either manually or by simply letting it sit for 10 or 15 minutes. Adjust your expectations of what instant means.

I did a beef and cherry tomatoes stew with garlic, onion, 1 lb of lean beef, beef bullion, a pint of cherry tomatoes, salt and pepper in about 45 minutes from beginning to end. It’s a dead simple recipe but very delicious when quality ingredients are used. After a 20 minute pressure cycle with all ingredients, I manually vented, then switched to saute for 10 minutes to cook off the excess water. This left a delicious sauce that’s wonderful over rice.

I also did a version Golden Lentil Soup in about 30 minutes with some turkey sausage added. It was quick and delicious, but as I said, nowhere near instant.

Hulu Live TV

I’ve been looking for an alternative to cable TV for some time. I want to be a cord cutter. I have no desire to rent cable boxes and pay for their overpriced DVR service indefinitely. I don’t want to pay for 300 channels of crap I never watch anyway. I want access to local TV, news networks, AMC, TBS, TNT, Comedy Central, Fox Sports, Food Network and few others. Continue reading “Hulu Live TV”

Blow your face off Chimichurri

  • 1 bell pepper
  • 1/2 a medium onion
  • 1 clove garlic
  • 2 Thai chilis, or until you can no longer feel your face
  • 1 1/2 tsp oil
  • 1 tbsp chimichurri sauce (hot)

3 Weight Watcher points.

Marinate the chicken in the Chimichurri sauce for a few hours or overnight. Stir-fry the garlic and hot peppers with the oil for 1 to 2 minutes in a non-stick pan over medium-high heat. Add the chicken with the sauce, turn after 2 minutes., Add pepper and onion. Stir-fry until onion is tender. Blow, your face off. Damn!

Enjoy with Rage Against the Machine or Linkin Park. Better to not let it touch your lips, OMG, Ouch!!!

If pain is not the goal, maybe moderate the chilies….

Review: Aeropress Coffee Maker

Left to right, Aeropress gear, my cheap grinder and electric kettle.

 

I’ve gone thru a bunch of coffee making methods. I’ve killed 2 espresso makers and tossed out 3 more drip coffee makers. I got disgusted with spending a ton of money on equipment and went with a simple cold brew maker, coupled with a French press for times when demand exceeded supply.

Then, while I was checking out yet another espresso maker I saw the Aeropress. I got one from Amazon a few weeks ago. It resembles a large syringe and is made of mostly BPA free plastic, with a rubber plunger. I’ve been using it exclusively for our morning coffee since then. I love this thing. It marries the speed advantage of French press with the smoothness of cold brew. It produces a rich cup with less bitterness and acidity. If coffee gives you stomach problems, this may bring coffee back into your life.

Good coffee involves several variables.

  • Bean quality and roast
  • Grind
  • Water quality
  • Temperature
  • Contact time
  • Pressure

Good quality beans and filtered water is a must. A grinder makes a world of difference. The Aeropress allows control over the temperature, brew time, and to some small extent the pressure with an accessory. See the update at the bottom.

My process for morning coffee:

  1. I use the electric kettle to heat water to 175 F. Hotter water will give greater extraction and is better for medium and lighters roasted bean, to prevent a washed-out weak taste.
  2. I use the inverted method. Flip the Aeropress so the screen is up and remove the screen. Make sure the plunger is inserted straight, at least 3/4 inch into the tube. This position prevents the coffee from leaking thru the filter while it steeps. The instructions recommend doing it the other way, but I found this easier
  3. It seems to work best for me with medium fine freshly ground coffee, not an espresso grind. That seems to clog the filter and makes using the press a test of strength. Add the coffee to the press, one measure of the provided scoop per cup.
  4. Add water, I use 2 scoops of coffee and fill the press nearly to the top. Stir for 10 to 20 seconds using the included stirrer.
  5. Add a paper filter and replace the screen. Many recipes recommend rinsing the filter with hot water to avoid the paper flavoring the coffee, I can’t tell the difference. I let it sit for 60 more seconds of brew time.
  6. Place it filter side down on a cup and press the plunger slowly down, taking about 30 seconds to fully press it. The goal is about 1:30 of brew time, including the time to press it.

I’m going to compare to the French Press I’ve used for a long while.

The result is flavorful without the bitter kick and fine grit generated by our Bodum French press. It’s a smooth cup of very strong espresso like coffee. I mix it with hot water to make an Americano. You can add hot milk to make a latte. Or use a dark roast and milk and make a Cuban coffee. Or make it with less water, following the scale on the side of the press, to make something like an espresso. To be clear, it’s not an espresso maker. It doesn’t generate sufficient pressure. You get no crema (the tan foam a good espresso maker generates) to speak of. But it’s close in flavor.

You can use a coarser grind and more brew time. Using more water during the brewing process does enhance the bitterness a bit if you prefer that. There are dozens of video recipes on YouTube that change all of the variables. Experiment with them, you can definitely find something that suits you. I started with a scale and timer the first few times. After some time you can leave the training wheels behind and rely on your personal taste and experience.

It’s far easier to clean than a French Press. Hold it over the trash can, remove the screen and eject the spent puck of coffee. Give it a quick rinse and store it with the plunger fully pressed to the bottom. It’s is top rack dishwasher safe. Filters can be rinsed and reused if you’re the frugal type. Or you can get a permanent filter on Amazon.

If you like good coffee, spend the money for a decent conical burr grinder. The one in the picture (Cuisinart DBM-8) doesn’t qualify. It was cheap, actually free, as it was purchased at Kohl’s using Kohl’s cash generated by other purchases. It can be had for around $40 on Amazon. It’s a disc grinder and better than weeks old coffee at the grocery store but doesn’t really do a good consistent grind. Baratza and Capresso make entry-level grinders suitable for use with the Aeropress. Or you can go with a good quality hand grinder and go camping with it. In my opinion, it doesn’t require a high-end grinder suitable for a true espresso maker. Any grinder is better than pre-ground stale coffee.

Pros:

  • A really good cup of coffee
  • Much faster than cold brew
  • It’s quick
  • It’s relatively cheap
  • Easy to clean
  • Very portable
  • The brewing process is very flexible and fully under your control
  • The small paper filters are inexpensive
  • It’s top rack dishwasher safe

Cons:

  • It uses more coffee per cup than other methods
  • It requires filters
  • Since it’s almost all plastic, I think it should be cheaper than ~$30
  • It’s hard to make coffee for more than 2
  • It’s not an espresso maker despite the description by some Amazon sellers

Update: There is an attachment that’s supposed to make espresso. The Fellow Prismo is available from aeropress.com or Amazon for ~$25. It adds a pressure valve to the screen to make a more espresso-like brew. It also eliminates the need for the flip method, so it’s more stable. I haven’t tried it. I just noticed it on Aeropress website.

Grilled Tuna Salad

A bed of arugula, garden fresh peppers and tomatoes. The tuna was about a 6 once portion and an inch thick. It’s seasoned with a bit of sesame oil, soy sauce, grated ginger, garlic powder and onion powder. I cooked in a hot non-stick pan for just 2-3 minutes per side. I use a little balsamic glaze to dress it. From “what can I have for lunch” to completion only took about 10 minutes. Simple, fresh, delicious…

Bangus Sisig

This is Filipino pulutan, food to be eaten while drinking beer or liquor. The traditional version uses pork cheeks, ears, chicken livers, egg, chill peppers, garlic and onions. The meats require a whole lot more prep and have lots of fat. This is lightened up with bangus (also called milkfish), but gets the same sizzling pan prep like fajitas. The fish is convection oven cooked, instead of fried. You can get the fish at an Asian grocery.It’s topped with crushed chicharron and served with a squeeze of lime. It comes in at 3 weight watcher points a serving. Have it with a San Miguel Light beer, cheers!

Bangus Sisig

Serves 4

  • Boneless Bangus, unmarinated
  • 1/2 cup diced onion
  • 2 cloves of garlic
  • Salt & pepper to taste
  • 1 Serrano chili, minced
  • 1 tablespoon of Mayonnaise
  • 1 egg
  • 1/2 oz of chicharron, crushed
  • Lime wedges
  1. Preheat and oven to 425 F, use convection setting if possible. Coat a fajita pan with oil and put into the oven.
  2. Sprinkle the Bangus with salt and pepper and cook 15 minutes
  3. While the fish is cooking, sauté the onion, garlic and chili until onions are tender.
  4. Scrape the fish from the skin and chop. Add to the onion mixture in the sauté pan. Add the mayo.
  5. Put the mix on the preheated fajita pan, it should sizzle.
  6. Quickly mix in the egg or leave the yolk intact if you like. I like some yolky goodness in mine to mix in while eating it. The heated pan should help cook it.
  7. Squeeze with a wedge of lime, top with crushed chicharron and serve immediately.

Powered by WordPress.com.

Up ↑