The Docker stack - the journey starts
The Docker stack I’ll been creating a set of templates over the past few years. Designed to be easier for you and me to spin up our apps. They are free for sure.
Our foundation for the building blocks will now be built
The journey begins here
I’ll been creating a set of templates over the past few years. Designed to be easier for you and me to spin up our apps. They are free of charge as this information and scripts for anyone to use.
The purpose is to automate setting-up vm's.
Using Docker-ce, Docker-Compose, Portainer-ce or Portainer-Agent, Watchtower, NetData, NginX Proxy Manager, Authelia, Heimdall. To be extended with user apps as: NextCloud, Fail2ban, GitLab, MeshCentral, Unbound, Ntop, PiHole, Organizer...
Things to setup
💻 This stack need some skill to set up but I will try to guide you. These scripts will set up the base system on the VM/CT (Docker, Docker-compose, Portainer, Fail2ban, Dozzer, Heimdall, Dashy...) a more advanced set like this one for web interface (NPM, Authelia, Lychee, NextCloud...).
Later the web module we be available.
Also WP and Ghost setups and some other stuff of my interest are planed.
You need to set up your ssh-key
Maybe you should guide me what to script next?
Pre setup
Create a Ubuntu server, always use the LTS versions, current is 20.04. Download-Link
Depending of how much you are going to load onto it you choose memory and disk size and numbers of cores. A normal VM/CT for our use would be 8-16 G disk 1-2 G of ram and using 1-2 cores. Some services could run on as little memory as 512 k.
Name it and set the IP (or add the mac to your static addresses).
Give your hosta a proper name and in the vm/ct do:
sudo set-hostname vm100.example.org or set-hostname moon
Copy your ssh-key to the new server ssh-copy-id -i ~/.ssh/id_ed25519.pub [email protected] now re-login by ssh. If it works all is golden. Remeber to block use of password for login.
I have used up to 32-64 G disk for local storing image based services. A better way is to set a path in fstab to a NAS shared volume or some other shared disk. Use NFS not CIFS, you need the speed advantage.
You could also use a dedicated 10/100 G network as a backbone and use iSCSI.
- Disk size: according to size of storage needed on added apps. This is not a Desktop machine - it's a server so 8 - 16G is usually plenty.
- Memory size: 512 will run, but a limited number of user can be served concurrent, on server HW you have much of memory 64 or 128G.
- Cores: 1 is plenty but more users more load you might end up giving 2-4 cores. If you are running server HW you usually have plenty of cores
Set up the VM/LXC
Download the script - Read the script - Run the script - Read the logs
Newer ever run scrips from the web without reading them!
Download the latest script 1-install.sh (it will pull the next script automatically).
Download latest version by executing this:
wget https://raw.githubusercontent.com/nallej/MyJourney/main/1-install.sh
You can also download a older version from here
Installing the Docker-Stack:
- Run 1-install script updates, upgrades and add needed parts to the VM/LXC OS.
- Run2-install script installs the basic parts of our stack. You don't need to install everything!
- The third step you will secure the machine by configuring : firewall, fail2ban, sshd, ssh-keys, reverse proxy ...
- Finally you run some docker.compose.yml -files to install your favorite services
Securing with SSH 🔐
The quick one is ssh-keygen -t ed25519 The key goes to ~/.ssh -directory.
If you want to remember what it's for: ssh-keygen -t ed25519 -C "some comment of what it's for"
Better security is to use a passphrase, a complicated long one isbetter. For ease-off-use you can use the ssh-agent to remember it for you.
ssh-keygen -t ed25519 -N new_long_passphrase -C "comment what it's for"
Always use ssh-keys! 🔑
Send the ssh-key to yor server
ssh-copy-id -i ~/.ssh/id_ed25519.pub [email protected]
Set up ssh config
nano ~/.ssh/config
#All my vm's and node's I want to contrl by ssh
#
Host server1 # Web npm, authelia, wp for sites 1-5, dozzle
HostName 192.168.1.100
User boss
Port 55555
Host server2 # Running special sw for somthing
HostName s1.example.com
User admin
Port 22
Host server3 # Web npm, authelia, nextcloud, wp for sites 6-9, dozzle
HostName 192.168.1.22
User boss
Port 55555
Host s5 # off-site server
HostName s5.example.com
User admin
Port 22
IdentyFile /nfs/shared/users/keys/id_ed25519
Host node007 #User group 7 server
HostName 192.168.100.122
User boss
Port 12345Edit /etc/ssh/sshd_config and change/edit/add for example:
PubkeyAuthentication yes # and then block passwd login
PasswordAuthentication no # higly recommend
PermitRootLogin no # mandatory for security
UsePAM no
AllowUsers kirk # name of your user
Port 12345 # somthing else than the standard Port 22
Set up ssh config for ssh-agent
nano ~/.ssh/config
Save and restart the demon sudo systemctl restart sshd
Post install
- Read the install-log and see if you have any errors
- Reboot
Start the new containers by:
- cd into-directory-of-app and edit the docker-compose.yml
- Start by
dcupalias fordocker-compose up -d - Or you do it with Portainer, copy the code and set the parameters.
And now Your Journey begins - Bon voyage !
Members only area below
Script one
To update and upgrade is mandatory. We need to add the latest security paches always. Firts we upgrad, add some parts to the OS and reboot.
This bash script will pull from my GitHub and install the following:
1-install will
Add some aliases and addons to the os, upgrade the sw
- Adds the aliases to your. bashrc_aliases, restarts them
. ~/.bashrc - Downloads 2-install.sh from my github
- Runs
chmod +x pre-install.sh - Sets timezone TZ (I use Europe/Helsinki) and updates the system.
- Adds keys
- Sets up Unattended updates
- Updates and upgrades linux
- sets up watchtower to run every morning for new images
- installs Fail2ban - you need to setup (ssh is on by defaulth)
Reboot befor continuing
#!/bin/bash
#--------------------------------------------------------#
# 1-install.sh 2.0 #
# MyJourney project #
# #
# V.1 Created by Nalle Juslen on 10.01.2022 - 2 scripts #
# -review 9.2.2022 #
# #
# V.2 created 10.02.2022 Prepare VM for Docker-Stack #
# #
#--------------------------------------------------------#
# Declare function hyrraPyorii Show a activity spinner --#
hyrraPyorii (){ #
pid=$! # Process Id of the previous running command #
x='-\|/' # hyrra in its elements #
i=0 #
while kill -0 $pid 2>/dev/null #
do #
i=$(( (i+1) %4 )) #
printf "\r ${x:$i:1}" #
sleep .1 #
done #
printf "\r " #
} #
hyrraPyorii #
# -------------------------------------------------------#
clear
echo ""
echo "Starting upgrade/install of apps for the Docker-stack"
echo ""
echo " WARNING - DON'T run scripts without editing - WARNING"
echo ""
echo " Warning . This script will pull parts from my GitHub"
echo ""
lsb_release -a #print lsb-release info
echo ""
echo "Preparing the VM"
# Add bash_aliases ------------------------------------------------------------
wget https://raw.githubusercontent.com/nallej/MyJourney/main/.bash_aliases &> /dev/null
. ~/.bashrc
echo " - added .bash_aliases"
# running install -------------------------------------------------------------
echo ""
echo " - starting Install"
echo " WARNING Rread the code - edit and the run!"
read -rp " - Do you want to do the Install [y/n] " DPI
if [[ "$DPI" = [yY] ]]; then
echo " - Running VM updates and installs ..."
# This is the main part update-install-upgrade ------------------------------
echo "**** Initial install/upgrade stage **** **** ****" > ~/install.log
sudo ls &> /dev/null # dummy to get sudo psw
(sudo apt-get update &&
sudo apt-get install curl apt-transport-https ca-certificates software-properties-common fail2ban -y &&
sudo apt-get upgrade -y
) >> ~/install.log 2>&1 &
hyrraPyorii
echo " - starting Docker Pre-Install ..."
echo "**** Docker Pre Install stage **** **** ****" >> ~/install.log
wget https://github.com/nallej/MyJourney/raw/main/LICENSE &> /dev/null
# set your timezone ---------------------------------------------------------
echo " - TZ = Helsinki" # change to your timezone
sudo timedatectl set-timezone Europe/Helsinki
# ---------------------------------------------------------------------------
read -rp " - Are you ruinning 1-Focal 2-Hirsute 0=Quit : " OS # <<< add deb
echo " - LTS is recomended use Focal Stable"
if [[ "$OS" != [1] ]]; then
echo "You need to edit the script"
exit
else
echo " ... upgrading sw ..."
echo "**** second install/upgrade stage **** **** ****" >> ~/install.log
( # Add keys -----------------------------------------------------------
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - &&
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable" &&
# LTS is recomendded focal stable (hirsute, impish) ------------------
sudo apt-cache policy docker-ce &&
sudo apt-get update &&
sudo apt-get upgrade -y
) >> ~/install.log 2>&1 &
hyrraPyorii
fi
echo ""
echo " - Setting up unattenden updates = yes "
echo "**** Docker Pre Install stage **** **** ****" >> ~/install.log
(sudo systemctl enable unattended-upgrades --now) >> ~/install.log 2>&1 &
echo ""
echo " WARNING - Do you want to clear old updates?"
read -rp " - Clear old updates [y/n] " COU
if [[ "$COU" = [yY] ]]; then
echo " - Deliting old installs ..."
(sudo apt-get clean &&
sudo apt-get autoremove) >> ~/install.log 2>&1 &
fi
# This is only needed for Proxmox VM's ------------------------#
echo "" #
read -rp " - Install QGA for Proxmox VM [y/n] " QGA #
if [[ "$QGA" = [yY] ]]; then #
sudo apt-get install -q -y qemu-guest-agent &> /dev/null #
fi #
# qemu-quest-agent --------------------------------------------#
# Giv your stack a name or use default: docker-stack --------------------------
mkdir docker-stack
cd docker-stack
wget https://raw.githubusercontent.com/nallej/MyJourney/main/2-install.sh &> /dev/null
chmod +x docker-install.sh
# =============================================================================
echo ""
echo ""
echo ""
echo "All done ! "
echo " - Licens copied to current folder "
echo " - Next script is in ~/docker-stack/ - 2-install.sh "
echo " - install.log written, check for errors "
echo ""
echo ""
echo "Ready for the next script ? "
echo " - Read the code, change to your needs, add your stuff and passwords ! "
echo " - reboot is mandatory"
echo ""
echo "You should now: "
echo " - Power off and change VM settings "
echo " - Edit the script: ~/docker-stack/2-install.sh "
echo ""
sleep 2s
read -rp "Do you want to reboot? [y/n] " RB
if [[ "$RB" = [yY] ]]; then
sudo reboot
fi
# end of install y/n
else
echo "Exit - automated process"
echo " - Edit the script: ~/docker-stack/2-install.sh "
echo " - Read the code, change to your needs, add your stuff and passwords ! "
echo " - REBOOT befor running ./2-install.sh"
fi
Script two
This bash script will pull from my GitHub and install the following: Docker-ce, Docker-Compose, Portainer/Portainer Agent, Watchtower, Dozzle and Heimdall (optional). Heimdall is included but you only need it on one vm.
This part is to setup Docker and the basic tools you need. Sets up networks: frontend for anything and backend for internal trafic. The 2-install script was installed in the directory ~/docker-stack, by 1-install script. Please edit it! Do your changes, add your credentials and passwords.
Post install
- Read the install-log and see if you have any errors
- Reboot
Start the containers by:
- cd into-directory-of-app and edit the docker-compose.yml
- Start by
dcupalias fordocker-compose up -d - Or you do it with Portainer, copy the code and set the parameters.
#!/bin/bash
#--------------------------------------------------------#
# 2-install.sh 2.0 #
# MyJourney project #
# #
# V.1 created by Nalle Juslen on 20.11.2021 - 2 scripts #
# - revison 26.1.2022 #
# #
# V.2 created 15.2.2022 Install basic Docker-Stack #
# #
#--------------------------------------------------------#
# Declare function hyrraPyorii Show a activity spinner --#
hyrraPyorii (){ #
pid=$! # Process Id of the previous running command #
x='-\|/' # hyrra in its elements #
i=0 #
while kill -0 $pid 2>/dev/null #
do #
i=$(( (i+1) %4 )) #
printf "\r ${x:$i:1}" #
sleep .1 #
done #
printf "\r " #
} #
hyrraPyorii #
# -------------------------------------------------------#
clear
echo "This script will install:"
echo " - Docker-ce and Docker-Compose,"
echo " - Portainer/Portainer agent, "
echo " - Dozzle and Watchtower, "
echo " - if you need it Heimdall. "
echo ""
echo " WARNING - DON'T run scripts without editing - WARNING"
echo ""
echo " Warning . This script will pull parts from my GitHub"
echo ""
echo ""
# What to install -------------------------------------------------------------
echo "Chose apps to install:"
read -rp " Docker-ce [y/n] " DOCE
read -rp " Docker-Compose [y/n] " DOCO
echo "Use Portainer or the Agent "
read -rp " - Portainer-ce [y/n] " POT
read -rp " - Portainer Agent [y/n] " POTA
echo ""
echo "Recommended apps:"
read -rp " WatchTower [y/n] " WT
read -rp " Dozzle [y/n] " DOZ
echo ""
echo "Optional apps "
read -rp " - Heimdall [y/n] " HEIM
# Start installing Docker-ce --------------------------------------------------
if [[ "$DOCE" == [yY] ]]; then
#sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"
#apt-cache policy docker-ce
sudo ls &> /dev/null # dummy to get sudo psw
echo ""
echo "installing Docker-ce"
(sudo apt install -q docker-ce -y ) >> ~/install.log 2>&1 &
hyrraPyorii
echo ""
echo " - Docker-ce installed. "
echo " - Starting docker."
echo ""
sudo systemctl enable docker
sudo systemctl enable containerd.service
sudo systemctl start docker
sleep 5s
docker -v
sleep 5s
sudo usermod -aG docker "${USER}" # you do not need to sudo to run docker commands after re-login
sleep 2s
sudo docker network create -d bridge kadulla &> /dev/null #frontend
sudo docker network create -d bridge pihalla &> /dev/null #backbone
echo " $USER added to docker group"
echo ""
echo " Internal networks created:"
echo " - frontend "
echo " - backbone "
echo ""
fi
echo ""
# Start installing Docker-Copmpose --------------------------------------------
if [[ "$DOCO" == [yY] ]]; then
(sudo apt install -q docker-compose -y) >> ~/install.log 2>&1 &
hyrraPyorii
echo ""
echo " - Docker-Compose installed."
echo ""
fi
echo ""
# Start installing Portainer/Portainer agent ----------------------------------
echo ""
#---------------------------------------------#
# I use 9000 http, you can use 9443 for https #
#---------------------------------------------#
if [[ "$POT" == [yY] ]]; then
(sudo docker volume create portainer_data
sudo docker run -d \
-p 8000:8000 \
-p 9000:9000 \
--name=portainer \
--restart=always \
-v /var/run/docker.sock:/var/run/docker.sock \
-v portainer_data:/data \
portainer/portainer-ce) >> ~/install.log 2>&1 &
hyrraPyorii
echo " - Portainer @ x.x.x.x:9000"
echo ""
fi
if [[ "$POTA" == [yY] ]]; then
echo ""
echo ""
(sudo docker volume create portainer_data
sudo docker run -d \
-p 9001:9001 \
--name portainer_agent \
--restart=always \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /var/lib/docker/volumes:/var/lib/docker/volumes \
portainer/agent) >> ~/install.log 2>&1 &
hyrraPyorii
sleep 2s
echo ""
echo " - Portainer finds this agent @ x.x.x.x:9001"
echo ""
fi
sleep 2s
echo ""
# Start installing WatchTower, auto update ------------------------------------
if [[ "$WT" == [yY] ]]; then
mkdir ~/docker-stack/watchtower
cd ~/docker-stack/watchtower
wget https://raw.githubusercontent.com/nallej/MyJourney/main/Watchtower/docker-compose.yml &> /dev/null
# docker-compose up -d # uncomment if you want to start now
echo ""
echo "Watchtower installed"
echo " - Updating images daily 04.00 "
echo " - NOT started, edit befor starting"
echo ""
fi
echo ""
# Start installing Dozzle, log reader -----------------------------------------
if [[ "$DOZ" == [yY] ]]; then
mkdir ~/docker-stack/dozzle
cd ~/docker-stack/dozzle
wget https://raw.githubusercontent.com/nallej/MyJourney/main/Dozzle/docker-compose.yml &> /dev/null
# sudo docker-compose up -d # uncomment if you want to start now
echo ""
echo "Dozzle installed"
echo " - Not started, EDIT befor using"
echo " - ports: 9999"
echo ""
fi
echo ""
# Start installing Heimdall, internal portal ----------------------------------
if [[ "$HEIM" == [yY] ]]; then
mkdir ~/docker-stack/heimdall
cd ~/docker-stack/heimdall
wget https://raw.githubusercontent.com/nallej/MyJourney/main/Heimdall/docker-compose.yml &>/dev/null
# docker-compose up -d # uncomment if you want to start now
echo ""
echo "Heimdal installed"
echo " - Not started, EDIT befor using"
echo " - default user/group: 1000/1000"
echo " - ports: 9080 or 9088 "
echo ""
fi
echo ""
echo "Basic installation done!"
echo " - add services: prometheus, node-reporter, backup ..."
echo " - add your apps wp, ghost ...."
echo " - logout and login again to activate permissions"
echo " - reboot befor starting apps"
echo ""
echo "Rememper to have fun! Learn new things and love the CLI."```#docker #docker-compose #portainer #nginx #nextcloud #authelia #heimdall #dashy #fail2ban #dozzer #homelab #self-host #wordpress #reverseproxy
