eoli3n

Stop battery draining by Syncthing on Android

2021-12-29T00:00:00+01:00

In degoogling post, I explained how I switch to CalyxOS. One great relief is that, as most google services are not running, I should expect a lower battery consumption and so, a longer battery life.
It was true, until I install Syncthing to sync my data between my devices instead of using centralized Google Drive.The service was draining my whole battery, beeing the only app which eat it.
As I rarely use Wifi networks, I need Syncthing to run over mobile data. The problem is that it runs in background, constantly checking for new data.

The main Syncthing app from f-droid, doesn’t provide much battery saving options.

Syncthing Options / Disable RestartOnWakeUp : It is supposed to save battery, tried it without success.
Running conditions / Disable run on battery : It would be a good solution if I could be able to force a run, even on battery, in emergency case.

Syncthing fork is supposed to be a better solution.

The README says that “Battery eater problem is fixed.”, so I asked the dev for more explainations, his answer was a bit light.
Despite the fact that the fork gives some finer options to deal with battery consumption, it allowed me to save no more than 2 or 3 battery hours.

The solution would be to sync only on specific conditions, to reduce the running syncing window. Luckily, Syncthing has an option to Respect Android parameter about Data Syncing, and some automation tools exists.

Easer automation tool

Easer is an Android event driven automation tool.
I want my sync to be disabled when the screen is locked. When the screen is unlocked, I want to limit sync to 1 min.
As my main concern is battery consumption, I want to always sync when battery is charging.
I usually charge at home, which is the only place where I use Wifi, lets trigger it then.

Here’s my configuration.

Conditions

Conditions are long time events, based on states. I used it to check if the screen is unlocked or the battery is discharging.

Events

Events are what they are, short time changing states, used for exemple for timing.

Profils

Profils are set of actions.

Scripts

Scripts link events and conditions to profils.

Pivot

This is the global algorithm.

For 24 hours to … 5 days

After one night, here’s the estimated battery duration time. Syncthing doesn’t even appear in the list, which is normal because it was not syncing at all.

Degoogling

2021-12-21T00:00:00+01:00

Apps
Data
Emails, contacts, and agenda
Maps and navigation
OS
Hardware
CalyxOS installation
Security configurations
A good move

As many users, Google trapped me many years ago, with my first Android smartphone. Google services are like cotton candy, sweet, colorful, comfortable, but abuse it and you ruin you health. Before even thinking about privacy, I want to be able to choose what services I run on my smartphone. Android version upgrades and the overlay of the manufacturer android rom is heavy and it makes any mid-range smartphone unusable in less than 2 years.
Another problem is that you can’t use Android without a google account.

So I started degoogling, step by step, which was a one year trip…

Apps

I started by trying to find FOSS alternatives for each apps I use.
F-droid is a application store for FOSS on Android and you can find some alternative on Degoogle.

Here’s a cool app list :

Book Reader : Simple book reader
Catima : Loyalty Card Wallet
Etar Calendar : Agenda app which supports CalDav
FairEmail : Fully featured email client. Neat, intuitive user interface. Privacy friendly
Infinity : A beautiful, feature-rich Reddit client.
LibreOffice Viewer
Markor : Lightweight text editor, Markdown Notes & ToDo.
MuPDF : Minimalist PDF viewer
NewPipe : Lightweight YouTube frontend
OpenBoard : Look and feel of Gboard without the tracking
OpenKeychain : GPG key manager
OsmAnd : Map Viewing & Navigation for Offline and Online OSM Maps
Password Store : Password Manager
Shelter : Provides an isolated space that you can install or clone apps into
Simple Notes : Notes app with a clean widget
Termux : Terminal emulator with packages
Voice : Simple audiobook reader

Data

Google Drive allows you to access data on the cloud from any device. I used rclone to be able to sync my data to a local storage. But Google still owns it too and its security is dependant to my Google account.

Syncthing is a open, trustworthy and decentralized file synchronization. You can install it on any OS, it’s easy to configure and fast.

As you can sync any directory, I use it to sync my pictures to my nas and backup it.

Emails, contacts, and agenda

Fashion is to end-to-end encryption services, like Protonmail, or Tutanota.
Both of them are known backdoored, you can share encrypted mails only with users of the same service (if you don’t use gpg manually) and on Android, you need to use specific apps to access your synced contacts and agendas.
Some workaround are one the road, but I prefer more open solutions, because I don’t need that level of security.

For mail hosting, I chose runbox.com. The last version of the web app si open-source. It is hosted in Norway, which has a respectful privacy legislation, servers are powered with green energy, and it provides caldav sync for contacts and agendas.

I use the f-droid app Davx5 as Cal/Card Dav sync client.

FOSS alternative for Google Maps is OsmAnd. I tried to switch for my daily use, and I quickly realized that google maps was not only a simple navigation service. I use it to find opening hours of shops, phone numbers, shop based on a “meta” search… OsmAnd is based on openstreetmap which is a great service, if you know the full (and well typed) address. Another problem is that route calculation is done on the device, so it doesn’t know for the traffic, accidents, etc. To be honest, I think Google Maps as I use it, cannot be replaced for now by a FOSS app, so I would continue to use it.

OS

The main alternative OS is LineageOS. It is shipped defaultly without Google services.
Google services has been reimplemented open-source by the MicroG project.

Many apps and features depends on Google services : mainly push notifications with GCM.
Sadly, I still need those services, to be able to use any app from the regular store.
You can check compatibility of apps without Google services with Plexus.

LineageOS is great, but you need to find the right hardware to run it, and the hardware support is community-driven.
The flash process could be tricky, with a significant risk to brick your device.

Here comes the funny part : The only choice that allows you to unlock the bootloader and install a custom ROM without too much effort/risk are Google phones because those are designed for developpers !

Choosing a Google smartphone like a Pixel or a Nexus opens your choices to GrapheneOS or CalyxOS. Both of them as based on Android Open Source Project, which is Android without the Google commercial overlay.

I will not compare those 2 here, but I chose CalyxOS for those features :

Project driven by Calyx Institute
MicroG included (and can be disabled)
Focused on privacy
Security with Datura per app firewall
Auto backup with Seedvault

To go further : privacyguides.org/android

Hardware

I just switch to a 5G subscription, so the choice is pretty limited.
Despite the fact that the Google Pixel 6 is out since 4 month, I went for a Google Pixel 4a 5g, because it’s cheaper, the screen is smaller and it stills have a minijack :) I also wanted to be able to directly install CalyxOs without beeing stuck on the Google Pixel ROM. When I write those lines, CalyxOS doesn’t support Google Pixel 6 still.

CalyxOS installation

As said, a benefit for using a developer phone is that the flash procedure is pretty simple, and stressless.
You just need to download the flasher binary and the OS archive.
See Install on a Pixel 4a (5G).

On void linux, I needed to install some packages.

$ xbps-install -S android-tools android-udev-rules

At first, device-flasher didn’t detect the smartphone.
I got help on IRC libera.chat #calyxos, the community has been nice and helpful.
They adviced me to boot it in fastboot mode and to run device-flasher with sudo. It worked like a charm, 5 min and the phone rebooter under CalyxOS.

Security configurations

Aurora store

After the first boot, a configuration menu let you enable microG and it doesn’t ask you to attach a google account. To be able to install Play Store apps without account, it uses Aurora Store. It supports anonymous apk downloads and installations, and can silently auto upgrade for apps in background.

Shelter isolation

Those untrustable applications doesn’t need for the most part to be able to access my data, or network.
Shelter use the Android work profile feature to allow you to isolate apps from your data, disabling the ability to use or leak you contacts, for exemple.

To configure it, install shelter, and use it to activate your work profile. Then you can clone Aurora to the work profile, and use it from here to install untrusted apps.

Datura firewall

To isolate an app from network with datura, start the app firewall from the main profile.

SeedVault backups

Start Backup app and configure it to backup apps and its configurations on the local storage.
Encrypted backups will be stored in ~/.SeedVaultAndroidBackup.
Then I use Syncthing to spread it on my nas which snapshots it on ZFS.

A good move

For now, I didn’t face any issue, let’s see in daily use.
I still have many things to test :

CalyxOS provides two free VPNs by default, CalyxVPN and Rise Up VPN.
A default offline map app : Organic Maps
How location privacy works

Over The Air upgrade to Android 12 is coming soon.

For sure, it is a one way trip, may CalyxOS have a long life !

Monitoring

2021-12-10T00:00:00+01:00

Turnkey solution : Netdata
Declarative and lightweight solution : Monit
Web server monitoring

I was searching for a lightweight monitoring solution for my single freebsd home server.
It should be able to get storage, cpu and memory metrics, look at file changes, check up my services and send alerts.

I took a look at monitoring stacks like Prometheus-Node_exporter-Grafana or Telegraf-InfluxDB-Chronograf-Kapacitor. It looked like to me a rabbit hole, grafana dashboard is pretty, it does tons of things, but it can’t handle simplest of my needs. You still need AlertManager to send mails with your prometheus stack. Endless microservices chain…
Those are great solutions when managing large node/service farms.

Turnkey solution : Netdata

Netdata provides a nice dashboard with realtime metrics, and it supervises system health. It can raise a lot of alerts by default and mail it.

I had to configure DragonFly Mail Agent which is a small Mail Transfert Agent with SMTP authentication over TLS/SSL.

/etc/dma/dma.conf

SMARTHOST smtp.domain.com
PORT 587
AUTHPATH /etc/dma/auth.conf
SECURETRANSFER
STARTTLS
MAILNAME eoli3n.eu.org

/etc/dma/auth.conf

mail@domain.com|smtp.domain.com:P@ssw0rd!

Let’s configure mail forwarding from root to external email

/etc/aliases

root: jonathan.kirszling@runbox.com

I can now test netdata mail transfert by running a test script.

$ /usr/local/libexec/netdata/plugins.d/alarm-notify.sh test

Out of the box, netdata checks metrics every seconds, and stores it to RAM with 2 days of retention. With htop, I have noticed 1.7% of CPU and 0.7% of RAM.
Following recommendations for performance, netdata dropped to 0.0% of CPU, 0.4% of RAM, with 2 weeks of metrics retention.

/usr/local/etc/netdata/netdata.conf

 [global]
+    memory mode = dbengine
+    page cache size = 32
+    dbengine multihost disk space = 256
-    history = 86400
+    update every = 5
+    debug log = none
+    error log = none
+    access log = none

 [plugins]
     freebsd = yes

 [web]
     respect do not track policy = yes
     disconnect idle clients after seconds = 3600
     bind to = 127.0.0.1
     web files owner = netdata
     web files group = netdata

Netdata seems to be clever, it checks a lots of things, but I would like a more declarative solution, to check and alert anything I need.

Declarative and lightweight solution : Monit

Monit is a small utility for managing and monitoring processes, files, directories, filesystems, programs, scripts, hosts, system metrics… It conducts automatic maintenance and repair if you ask it to. It also embeed a clean WebUI to keep an eye on all monitored services.

Simply install monit through your package manager and start writing your monitrc file.
Here the jinja template I wrote for my server, explained in comments. I don’t even need to over-comment it, because the DSL syntax is human readable.

set log /var/log/monit.log

# Check every 30 seconds and delay 120s at start
set daemon 30
    with start delay 120

# Enable WebUI and configure it
set httpd
    port 8080
    use address 127.0.0.1
    allow localhost
    signature disable

# Recipient email for alerts
set alert 

# Configure SMTP server to use to send alert.
# Monit doesn't use system sendmail command.
set mailserver  port 
    username "" password "" using ssl

set mail-format {
    from: Monit <monit@>
    reply-to: noreply@
    subject: $ACTION $SERVICE
    message:
Date:        $DATE
Service:     $SERVICE
Event:       $EVENT
Action:      $ACTION
Description: $DESCRIPTION.
}

# Zpool health
check program zpool-status with path "/sbin/zpool status -x"
    if status != 0 then alert

# Check zpool usage with a custom script
check program usage-zroot with path "/tmp/zpool_usage.sh zroot"
    if status != 0 then alert
check program usage-dpool with path "/tmp/zpool_usage.sh dpool"
    if status != 0 then alert
check program scrub-zroot with path "/tmp/zpool_scrub.sh zroot"
    if status != 0 then alert
check program scrub-dpool with path "/tmp/zpool_scrub.sh dpool"
    if status != 0 then alert

# Resources
check system localhost
    if memory usage > 85% for 3 cycles then alert
    if loadavg (15min) > 4 then alert
    if cpu usage > 85% for 3 cycles then alert
    if swap usage > 25% for 3 cycles then alert

# SSHD
check process sshd with pidfile /var/run/sshd.pid
    start program = "/usr/sbin/service sshd start"
    stop program = "/usr/sbin/service sshd stop"
    if failed port  protocol ssh then restart
    if changed pid then alert
check file sshd_config path /etc/ssh/sshd_config
    if changed md5 checksum then alert

check file passwd path /etc/passwd
    if changed md5 checksum then alert

# Jails
check program jails with path "/usr/local/bin/bastille list -a"
    if content == "Down" then alert

# Updates
check program updates with path "/usr/bin/awk '/packages to be upgraded/ {v+=$NF}END{print v;if (v>=5) exit 1}' /tmp/check-updates"
    if status == 1 then alert

# Audit
check program audit with path "/usr/bin/awk '/found/ {v+=$1}END{print v;if (v>=5) exit 1}' /tmp/bastille-audit"
    if status == 1 then alert

# Web
check process nginx with pidfile /usr/local/bastille/jails/nginx/root/var/run/nginx.pid
    start program = "/usr/local/bin/bastille start nginx"
    stop program  = "/usr/local/bin/bastille stop nginx"
    if failed host photos.eoli3n.eu.org port 443 protocol https content = "Photos" then alert
    if failed host eoli3n.eu.org port 443 protocol https content = "… Blog …" then alert

# DNS
check process nsd with pidfile /usr/local/bastille/jails/nsd/root/var/run/nsd/nsd.pid
    start program = "/usr/local/bin/bastille start nsd"
    stop program  = "/usr/local/bin/bastille stop nsd"
    if failed host  port 53 use type udp protocol dns then restart
    if changed pid then alert

# Syncthing
check process syncthing with pidfile /usr/local/bastille/jails/syncthing/root/var/run/syncthing.pid
    start program = "/usr/local/bin/bastille start syncthing"
    stop program  = "/usr/local/bin/bastille stop syncthing"
    if changed pid then alert

# Backups
check directory backup-host1 path /data/zfs/backups/host1
   if timestamp > 24 hour then alert
check directory backup-host2 path /data/zfs/backups/host2
   if timestamp > 24 hour then alert

# Snapshots
check directory zfs-snapshots-slash path /.zfs/snapshot
    if timestamp > 2 hours then alert

# Sigal
check file sigal-check path /tmp/sigal-check
    if changed md5 checksum then alert

# Jekyll
check directory jekyll path /data/zfs/www/blog
   if changed timestamp then alert

zpool_usage.sh

#!/bin/sh

echo "cap: $(zpool list -o cap -H $1) used: $(zpool list -o alloc -H $1) free: $(zpool list -o free -H $1) size: $(zpool list -o size -H $1)"
cap="$(zpool list -o cap -H $1 | tr -d '%')"
if [ "$cap" -gt 90 ]
then
    exit 1
fi

zpool_scrub.sh

#!/bin/sh

scrub_expire="3110400" # 36days * 24hours * 60minutes * 60seconds
current_date="$(/bin/date +%s)"
scrub="$(/sbin/zpool status $1 | grep scrub | awk '{print $15 $12 $13}')"
scrub_date="$(date -j -f '%Y%b%e-%H%M%S' $scrub'-000000' +%s)"

echo "$(/sbin/zpool status $1 | grep scrub)"

if [ $(($current_date - $scrub_date)) -ge $scrub_expire ]
then
    exit 1
fi

Then run monit, and check http://localhost:8080. You will now receive a mail when a test fails !

Web server monitoring

Next step is to monitor my web server. Netdata provides a way to get real-time stats from /nginx_status.
If you don’t use Netdata, monitorix could be a good alternative.

Both of those are realtime, I prefer a solution with historization, which would parse access.log to produce some graphs.

GoAccess can parse logs and do it realtime though its own websocket server, developed by the same guy. The tool is impressive, it can be used cli or generate a html report !

To be able to consult the html report, I choose to generate it statically everyday with a cron and spread it with Syncthing.

- name: generate reports everyday at midnight
  cron:
    name: goaccess report
    job: '/usr/local/bin/goaccess /usr/local/bastille/jails/nginx/root/var/log/nginx/access.log --log-format=COMBINED -o /data/zfs/sync/docs/reports/nginx-goaccess.html'
    hour: '0'
    minute: '0'
    user: root

From any client which has the file synced.

$ firefox ~/share/docs/reports/nginx-goaccess.html

The site is responsive, I can also consult it with my smartphone.
To finish, let’s add a monit service to check that the report is well generated everyday.

# GoAccess
check file goaccess path /data/zfs/sync/docs/reports/nginx-goaccess.html
    if timestamp > 24 hours then alert

The loop is closed.

Managing FreeBSD Jails with Ansible - part 3

2021-06-14T00:00:00+02:00

Network role
Firewall role
Jails role
- Install and configure Bastille
- Bootstrap a release
Web role
Run the playbook
Test the service

In the second part, we adapted our Ansible project to manage Jails with vnet.
The two first posts was useful to understand basic Jail creation, let’s now wrap it with Bastille.
Bastille is the BSD Docker-like toolset for managing containers.
That solution has many advantages, it wraps useful commands to manage Jails without having to rewrite it with Ansible.
It also has a template feature to automate Jail provisionning.
The template automate Jails creation with a Bastillefile. Note the docker reference, even the syntax looks similar.

So why wrapping Bastille with Ansible ?

Using Bastille let you provision a Jail easily, but it does not wrap Jails creation nor automate the configuration of your services. My goal is that Ansible manages automatically everything related to your service, if you need to update a config file or anything, just do it in your project and run ansible-playbook, to deploy and restart everything properly.

Following Bastille documentation, we will configure the server as if it was in DMZ, using pf as firewall to expose containers ports.

Clean all roles in the Ansible project, and configurations on the server, from previous parts.
By the way, this post shows how Ansible is kind of auto documented, as each tasks has a name, the Ansible code is pretty clean to read, even if you don’t know the tool.

Network role

We need to match network requirements.
Create a network role.
meta: flush_handlers task triggers handlers without waiting the end of the play.

roles/network/tasks/main.yml

---
- name: Add lo1 interface
  community.general.sysrc:
    name: cloned_interfaces
    state: value_present
    value: "lo1"

- name: Name lo1 interface bastille0
  community.general.sysrc:
    name: ifconfig_lo1_name
    value: "bastille0"
  notify: netif cloneup

- meta: flush_handlers

roles/network/handlers/main.yml

---
- name: netif cloneup
  shell: service netif cloneup

Firewall role

Create a firewall role.
roles/firewall/tasks/main.yml

---
- name: enable pf
  community.general.sysrc:
    name: pf_enable
    value: "YES"
  notify: start pf

- name: enable pflog
  community.general.sysrc:
    name: pflog_enable
    value: "YES"
  notify: start pflog

- name: template pf.conf
  template:
    src: pf.conf.j2
    dest: /etc/pf.conf
  notify: reload pf

- meta: flush_handlers

roles/firewall/templates/pf.conf.j2

ext_if="{{ ansible_default_ipv4.interface }}"

### Default block policy is to return a reset packet
set block-policy return
### Reassemble fragmented packets
scrub in on $ext_if all fragment reassemble
### Ignore loopback interface
set skip on lo

### Allow empty table to exist
table <jails> persist
### Nat in jails table
nat on $ext_if from <jails> to any -> ($ext_if:0)

### Static rdr
# rdr pass inet proto tcp from any to any port {80, 443} -> 10.17.89.45

### Enable dynamic rdr (see below)
rdr-anchor "rdr/*"

### Block on incoming traffic
block in all
### Allow outgoing, skip others rules if match, and track connections
pass out quick keep state
### Block all incoming traffic from the $ext_if subnet which is not from $ext_if interface
### And block incoming traffic from $ext_if IP on $ext_if interface
antispoof for $ext_if inet
### Allow SSH
pass in inet proto tcp from any to any port ssh flags S/SA keep state

We use async on pf start handler to keep ansible connection up.
For the reload pf handler, we first test that the config file is valid with -n and apply the configuration only if it succeed.
roles/firewall/handlers/main.yml

---
- name: start pf
  service:
    name: pf
    state: started
  async: 45
  poll: 5

- name: start pflog
  service:
    name: pflog
    state: started

- name: reload pf
  shell: pfctl -nf /etc/pf.conf && pfctl -f /etc/pf.conf

Jails role

Install and configure Bastille

Create a role jails.
Bastille will be configured to use ZFS.

roles/jails/tasks/main.yml

---
- name: install bastille
  pkgng:
    name: bastille

- name: enable bastille
  community.general.sysrc:
    name: bastille_enable
    value: "YES"

- name: add bastille devfs rule
  blockinfile:
    path: /etc/devfs.rules
    marker: "<!-- {mark} ANSIBLE MANAGED vnet -->"
    create: yes
    block: |
      [bastille_vnet=13]
      add path 'bpf*' unhide

- name: enable zfs for bastille
  community.general.sysrc:
    name: "{{ item.name }}"
    value: "{{ item.value }}"
    path: /usr/local/etc/bastille/bastille.conf
  loop:
    - { name: "bastille_zfs_enable", value: "YES" }
    - { name: "bastille_zfs_zpool", value: "zroot" }

Bootstrap a release

Bootstrap the latest realease and configure it to use latest pkgs.
Releases in Bastille is the template which will be use to layer up your jails.
So each configuration made to a release will be applied to all new jails created from this release.

Add a var to group_vars/all.yml

release: 13.0-RELEASE

Then, add tasks to bootstrap the release from that var.

- name: bootstrap {{ release }} release
  shell: "bastille bootstrap {{ release }}"
  args: creates="/usr/local/bastille/releases/{{ release }}"

- name: configure bootstrap to use latest pkgs
  replace:
    path: "/usr/local/bastille/releases/{{ release }}/etc/pkg/FreeBSD.conf"
    regexp: '^(.*)quarterly(.*)$'
    replace: '\1latest\2'

- name: update bootstrap
  shell: "bastille update {{ release }}"

Web role

Prepare the nginx template

Create a role nginx.

Here’s the interesting part. With a Bastillefile, you automate your service provisionning.
Here we tell the template to install nginx and enable it. Then we create our /data/www dir in the jail, to bind the one from the host in it. We also overlay the nginx config file with CP usr .. Finally we check if the config file is valid and then restart the service.
The RDR line dynamically generate a rule for pf to redirect the http port from the host to the jail.

roles/nginx/files/Bastillefile

PKG nginx
SYSRC nginx_enable=YES
CMD mkdir -p /data/www
CP usr .
CMD nginx -t
SERVICE nginx restart
FSTAB /data/www data/www nullfs ro 0 0
RDR tcp 80 80

roles/nginx/tasks/main.yml

---
- name: create services template dir
  file:
    path: "/usr/local/bastille/templates/services/{{ role_name }}"
    state: directory
    recurse: yes

- name: copy template config files
  copy:
    src: Bastillefile
    dest: "/usr/local/bastille/templates/services/{{ role_name }}/"

- name: create config path
  file:
    path: "/usr/local/bastille/templates/services/{{ role_name }}/usr/local/etc/nginx/"
    state: directory
    recurse: yes

- name: copy config file
  copy:
    src: nginx.conf
    dest: "/usr/local/bastille/templates/services/{{ role_name }}/usr/local/etc/nginx/"

- name: create data/www dataset
  community.general.zfs:
    name: zroot/www
    state: present
    extra_zfs_properties:
      mountpoint: /data/www

roles/nginx/files/nginx.conf

http {
    server {
        listen       80;
        server_name  localhost;

        location / {
            root   /data/www;
            index  index.html index.htm;
        }
    }
}

Add the task to copy the website to the host dir, mounted in the jail.

- name: copy index.html
  copy:
    src: index.html
    dest: /data/www/

roles/nginx/files/index.html

<html>
  <p>A website without any JS !</p>
</html>

Create a nginx jail

We set, at jail creation, its static IP in any private subnet which differs from your gateway one, following the advice of the Bastille README.md.

Pick any private address and be done with it. These are all isolated networks. In the end, what matters is you can map host:port to container:port reliably, and we can.

Add your Jail IP to group_vars/all.yml

jails:
  nginx: 10.0.0.1

roles/nginx/tasks/main.yml

- name: create jail
  shell: "bastille create {{ role_name }} {{ release }} {{ jails[role_name] }}"
  args:
    creates: /usr/local/bastille/jails/{{ role_name }}

- name: start jail
  # https://github.com/BastilleBSD/bastille/issues/342
  shell: bastille start {{ role_name }} || true

Template the nginx jail

- name: template jail
  shell: "bastille template {{ role_name }} services/{{ role_name }}"

Run the playbook

$ ansible-playbook playbook.yml -t network,firewall,jails,nginx

PLAY [host-test] ***************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************
ok: [host-test]

TASK [network : Add lo1 interface] *********************************************************************************
changed: [host-test]

TASK [network : Name lo1 interface bastille0] **********************************************************************
changed: [host-test]

TASK [network : meta] **********************************************************************************************

RUNNING HANDLER [network : netif cloneup] **************************************************************************
changed: [host-test]

TASK [firewall : enable pf] ****************************************************************************************
changed: [host-test]

TASK [firewall : enable pflog] *************************************************************************************
changed: [host-test]

TASK [firewall : template pf.conf] *********************************************************************************
changed: [host-test]

TASK [firewall : meta] *********************************************************************************************

RUNNING HANDLER [firewall : start pf] ******************************************************************************
changed: [host-test]

RUNNING HANDLER [firewall : start pflog] ***************************************************************************
changed: [host-test]

RUNNING HANDLER [firewall : reload pf] *****************************************************************************
changed: [host-test]

TASK [jails : install bastille] ************************************************************************************
changed: [host-test]

TASK [jails : enable bastille] *************************************************************************************
changed: [host-test]

TASK [jails : add bastille devfs rule] *****************************************************************************
changed: [host-test]

TASK [jails : enable zfs for bastille] *****************************************************************************
changed: [host-test] => (item={'name': 'bastille_zfs_enable', 'value': 'YES'})
changed: [host-test] => (item={'name': 'bastille_zfs_zpool', 'value': 'zroot'})

TASK [jails : bootstrap 13.0-RELEASE release] **********************************************************************
changed: [host-test]

TASK [jails : configure bootstrap to use latest pkgs] **************************************************************
changed: [host-test]

TASK [jails : update bootstrap] ************************************************************************************
changed: [host-test]

TASK [nginx : create services template dir] ************************************************************************
changed: [host-test]

TASK [nginx : copy template config files] **************************************************************************
changed: [host-test]

TASK [nginx : create config path] **********************************************************************************
changed: [host-test]

TASK [nginx : copy config file] ************************************************************************************
changed: [host-test]

TASK [nginx : create data/www dir] *********************************************************************************
changed: [host-test]

TASK [nginx : copy index.html] *************************************************************************************
changed: [host-test]

TASK [nginx : create jail] *****************************************************************************************
changed: [host-test]

TASK [nginx : start jail] ******************************************************************************************
changed: [host-test]

TASK [nginx : template jail] ***************************************************************************************
changed: [host-test]

PLAY RECAP *********************************************************************************************************
host-test                  : ok=26   changed=25   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Test the service

From the server.

$ curl http://10.0.0.1
<html>
  <p>A website without any JS !</p>
</html>

From a client in the gateway subnet, if the dynamic RDR worked, it should be reachable.

$ curl http://192.168.0.100
<html>
  <p>A website without any JS !</p>
</html>

If your server is in your DMZ, then your service in reachable from internet too.
You can now easily add new services by creating one role per service and use the nginx one as exemple.

Managing FreeBSD Jails with Ansible - part 2

2021-06-09T00:00:00+02:00

Clean environment
Install jib script
Configure network stack in jails
Configure epair interface in jails template
Run the playbook and test connectivity

In the first part, we created the Ansible project to manage Jails with shared IP. In this post, we will adapt our playbook to create vnet jails.

vnet gives to jails their own network stacks. Each Jail will have a specific network interface with epair connected to a bridge. Let’s quote something I read in a forum which resume well how it works:

Analogous to a physical network, a bridge interface works like a software switch, an epair works like a virtual network cable and a jail acts as a virtual computer.

The bridge creation will be automated with jib script.
Let’s adapt our playbook to create that kind of jail networking.

There is only few steps, but before all, reset our first configurations.

Clean environment

Remove the task which creates aliases, and remove them from your system.

$ ansible host-test -m shell -a "service jail stop bind && service jail stop nginx"
$ ansible host-test -m lineinfile -a 'path=/etc/rc.conf regexp=".*alias.*" state=absent'
$ ansible host-test -m file -a 'path=/etc/jail.conf state=absent'
$ ansible host-test -m zfs -a 'name=zroot/jails/bind state=absent'
$ ansible host-test -m zfs -a 'name=zroot/jails/nginx state=absent'
$ ansible host-test -m raw -a "service netif restart"
$ ansible host-test -m raw -a "service routing restart"

Install jib script

Add a task to copy the script in your $path with execution perms.

- name: install jib script
  copy:
    src: /usr/share/examples/jails/jib
    dest: /usr/local/bin/
    remote_src: yes
    mode: 0755

Configure network stack in jails

At jail startup, jib will create a bridge if non existent, create epairs and automatically attach them to the bridge. Stopping the jail will destroy interfaces but not the bridge.
Let’s change our task to declare to use that script in exec.prestart and exec.poststop.

 - name: declare jails
-  vars:
-    alias_ip: "{{ inet | ipmath(ansible_loop.index) }}"
   blockinfile:
     path: /etc/jail.conf
     marker: "# {mark} ANSIBLE MANAGED: {{ item }}"
     block: |
       {{ item }} {
           host.hostname = "{{ item }}.domain.local";
           path = "/usr/local/jails/{{ item }}";
           exec.consolelog = "/var/log/jail_{{ item }}.log";
-          ip4.addr = {{ alias_ip }};
+          vnet;
+          vnet.interface = "e0b_{{ item }}";
+          exec.prestart += "jib addm {{ item }} {{ ansible_default_ipv4.interface }}";
+          exec.poststop += "jib destroy {{ item }}";
       }
   loop: "{{ jails | sort | flatten(levels=1) }}"
-  loop_control:
-    extended: yes

Configure epair interface in jails template

Edit the bsdinstall template to add epair interface configuration and default gateway.

 - name: template bsdinstall script
+  vars:
+    jail_ip: "{{ inet | ipmath(ansible_loop.index) }}"
   copy:
     dest: "/usr/local/jails/{{ item }}.template"
     content: |
       DISTRIBUTIONS="base.txz"
       export nonInteractive="YES"
       #!/bin/sh
       sysrc sshd_enable="YES"
+      sysrc ifconfig_e0b_{{ item }}="inet {{ jail_ip }} netmask 255.255.255.0"
+      sysrc defaultrouter="{{ gateway }}"
       pkg install -y python37
       mkdir /root/.ssh
       chmod 600 /root/.ssh
   loop: "{{ jails | sort | flatten(levels=1) }}"
+  loop_control:
+    extended: yes

Run the playbook and test connectivity

$ ansible-playbook playbook.yml
[...]

TASK [jails : create zfs per jail dataset] ***************************************
changed: [host-test] => (item=bind)
changed: [host-test] => (item=nginx)

TASK [jails : template bsdinstall script] ****************************************
changed: [host-test] => (item=bind)
changed: [host-test] => (item=nginx)

TASK [jails : bsdinstall jails] **************************************************
changed: [host-test] => (item=bind)
changed: [host-test] => (item=nginx)

TASK [jails : authorize your ssh key] ********************************************
changed: [host-test] => (item=bind)
changed: [host-test] => (item=nginx)

TASK [jails : permit root login] *************************************************
changed: [host-test] => (item=bind)
changed: [host-test] => (item=nginx)

TASK [jails : set default jails config] ******************************************
changed: [host-test]

TASK [jails : declare jails] *****************************************************
changed: [host-test] => (item=bind)
changed: [host-test] => (item=nginx)

TASK [jails : start jails at startup] ********************************************
ok: [host-test]

TASK [jails : start jails] *******************************************************
changed: [host-test] => (item=bind)
changed: [host-test] => (item=nginx)

PLAY RECAP ***********************************************************************
host-test                  : ok=11   changed=8    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

jib created a bridge and attached jails interfaces in it.

$ ansible host-test -m shell -a "ifconfig vtnet0bridge | grep member"
host-test | CHANGED | rc=0 >>
	member: e0a_nginx flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	member: e0a_bind flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	member: vtnet0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>

Let’s check if jails are running.

$ ansible host-test -m shell -a "jls"
host-test | CHANGED | rc=0 >>
   JID  IP Address      Hostname                      Path
     3                  bind.domain.local             /usr/local/jails/bind
     4                  nginx.domain.local            /usr/local/jails/nginx

As IP is configured in the Jail, host doesn’t know which one it is.
e0b_* interfaces are attached to jails.

$ ansible host-test -m shell -a "jexec bind ifconfig | grep e0b"
host-test | CHANGED | rc=0 >>
e0b_bind: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500

$ ansible host-test -m shell -a "jexec nginx ifconfig | grep e0b"
host-test | CHANGED | rc=0 >>
e0b_nginx: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500

$ ansible bind:nginx -m ping
nginx | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/local/bin/python3.7"
    },
    "changed": false,
    "ping": "pong"
}
bind | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/local/bin/python3.7"
    },
    "changed": false,
    "ping": "pong"
}

Jails are now exposed on my private subnet with their own network stack and reachable by Ansible.

We learned how to automate shared IP and vnet Jails provisionning with Ansible.
Managing Jails is more than just provision it, you’ll need to maintain it, by upgrading packages or releases.
In the network side, you would use a private subnet to isolate your jails from your local private subnet, and NAT to have the abilitiy to allow/deny access, or forward a port to the right Jail.

Full raw Ansible management is possible, but there already exists some clever wrapper with great features, like cbsd, iocage, or bastille.
In the next part, I’ll focus on bastille and how to automate our jails management with it, still wrapped by Ansible.

Managing FreeBSD Jails with Ansible - part 1

2021-06-08T00:00:00+02:00

Init Ansible
Shared IP Jail

When it comes to manage an OS configuration, I always fully automate the process, it is a oneshot work, you write it, test it, forget it, until you need to modify the process. I kind of use Ansible to make the work concrete as I use my blog to clarify my ideas.

Jails are like containers for FreeBSD, it lets you isolate your services from each others. Each Jail has its own IP, there are different ways to manage networking, let’s explore automation for each.

Init Ansible

Create project and configure

Create your Ansible project root and git it.

$ mkdir jails && cd jails
$ git init
$ mkdir roles group_vars

Create inventory, if your host is configured with DHCP, check its IP.

$ echo "host-test ansible_host=192.168.0.44" > hosts

Configure Ansible.

$ cat << EOF > ansible.cfg
[defaults]
stdout_callback = yaml
inventory = hosts
remote_user = root
interpreter_python = auto_silent
EOF

And create the playbook, with a role jails.

$ cat << EOF > playbook.yml
---
- hosts: host-test
  roles:
    - { role: jails, tags: jails }
EOF

$ mkdir -p roles/jails/tasks

Install community.general collection, it contains sysrc ansible module to safely edit rc.conf

$ ansible-galaxy collection install community.general

Make your project configurable

Add a configuration file as group_vars/all.yml, with 3 vars. We will increment the static IP of the host in tasks, so choose an IP which can be incremented as many time as you have jails.

inet: 192.168.0.100
netmask: 255.255.255.0
gateway: 192.168.0.254

Prepare the jails host

Bootstrap FreeBSD for Ansible.

$ ansible host-test -m raw -a "pkg install -y python37"

Configure your default network interface.

$ ansible host-test -m sysrc -a 'name="ifconfig_{{ ansible_default_ipv4.interface }}" value="inet {{ inet }} netmask {{ netmask }}"'
$ ansible host-test -m sysrc -a 'name="defaultrouter" value="{{ gateway }}"'

Restart netif and routing services to read you network configuration for rc.conf.

$ ansible host-test -m raw -a "service netif restart"
$ ansible host-test -m raw -a "service routing restart"

Adapt your inventory with the new static IP and you’re ready.

$ echo "host-test ansible_host=192.168.0.100" > hosts

Shared IP Jail

The simpler way to create a Jail is to add IP alias to you network interface and then bind that IP to your Jail.
For the exemple, let’s create 2 jails, bind and nginx.
First we need to create two IP aliases, default IP is incremented with ipmath which needs netaddr python package installed on the controller.

Let’s declare our jails in a list in group_vars/all.yml.

jails:
  - bind
  - nginx

Configure IP aliases

We use extended loop vars to increment inet IP with ipmath which needs netaddr python package installed on the controller. ansible_loop.index0 starts index at 0 instead of 1.
To be sure that the list will always be processed in the same order, it needs to be explicitly sorted.

In roles/jails/tasks/main.yml.

---
- name: create IP aliases for jails
  vars:
    alias_ip: "{{ inet | ipmath(ansible_loop.index) }}"
  community.general.sysrc:
    name: "ifconfig_{{ ansible_default_ipv4.interface }}_alias{{ ansible_loop.index0 }}"
    value: "inet {{ alias_ip }} netmask {{ netmask }}"
  loop: "{{ jails | sort | flatten(levels=1) }}"
  loop_control:
    extended: yes
  notify: restart netif

You need a handler to trigger the netif service restart on configuration update.

In roles/jails/handlers/main.yml.

---
- name: restart netif
  service:
    name: netif
    state: restarted

Run your playbook, and test.

$ ansible-playbook playbook.yml

PLAY [host-test] ***************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************
ok: [host-test]

TASK [network : create IP aliases for jails] ***********************************************************************
changed: [host-test] => (item=bind)
changed: [host-test] => (item=nginx)

RUNNING HANDLER [network : restart netif] **************************************************************************
changed: [host-test]

PLAY RECAP *********************************************************************************************************
host-test                  : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Your host now has two aliases configured.

$ ansible host-test -m shell -a "ifconfig vtnet0 | grep inet"
host-test | CHANGED | rc=0 >>
	inet 192.168.0.100 netmask 0xffffff00 broadcast 192.168.0.255
	inet 192.168.0.101 netmask 0xffffff00 broadcast 192.168.0.255
	inet 192.168.0.102 netmask 0xffffff00 broadcast 192.168.0.255

Provision jails environments

- name: create zfs jails dataset
  community.general.zfs:
    name: zroot/jails
    state: present
    extra_zfs_properties:
      mountpoint: /usr/local/jails

- name: create zfs per jail dataset
  community.general.zfs:
    name: "zroot/jails/{{ item }}"
    state: present
  loop: "{{ jails | sort | flatten(levels=1) }}"

Install FreeBSD on the jails with a custom bsdinstall script.
I faced an issue here, bsdinstall uses its jail argument to target the specific environment, and its script argument to automate the process. Using both make it ignore the second one.

I improved the jail script to be able to use a SCRIPT env var with the script path, to automate in jail provisionning and created a PR.

Now let’s create templates for automated provisionning.
Let’s install python3 and enable sshd to be able to use Ansible on our jails.

- name: template bsdinstall script
  copy:
    dest: "/usr/local/jails/{{ item }}.template"
    content: |
      DISTRIBUTIONS="base.txz"
      export nonInteractive="YES"
      #!/bin/sh
      pkg install -y python37
      sysrc sshd_enable="YES"
      mkdir /root/.ssh
      chmod 600 /root/.ssh
  loop: "{{ jails | sort | flatten(levels=1) }}"

Trigger the provisionning with the shell module and args: creates: to make it idempotent.

- name: bsdinstall jails
  shell: bsdinstall jail /usr/local/jails/"{{ item }}"
  environment:
    SCRIPT: "/usr/local/jails/{{ item }}.template"
  args:
    creates: "/usr/local/jails/{{ item }}/bin"
  loop: "{{ jails | sort | flatten(levels=1) }}"

Authorize your ssh public key in jails.

- name: authorize your ssh key
  copy:
    src: ~/.ssh/id_rsa.pub
    dest: "/usr/local/jails/{{ item }}/root/.ssh/authorized_keys"
    mode: 0600
  loop: "{{ jails | sort | flatten(levels=1) }}"

We finally need to permit root login.

- name: permit root login
  replace:
    path: "/usr/local/jails/{{ item }}/etc/ssh/sshd_config"
    regexp: '^#(PermitRootLogin).*'
    replace: '\1 yes'
  loop: "{{ jails | sort | flatten(levels=1) }}"

Declare your jails

Last thing we need is to declare jails in /etc/jail.conf.

- name: set default jails config
  blockinfile:
    path: /etc/jail.conf
    create: yes
    marker: "# {mark} ANSIBLE MANAGED: default"
    block: |
      exec.start = "/bin/sh /etc/rc";
      exec.stop = "/bin/sh /etc/rc.shutdown";
      exec.clean;
      mount.devfs;

- name: declare jails
  vars:
    alias_ip: "{{ inet | ipmath(ansible_loop.index) }}"
  blockinfile:
    path: /etc/jail.conf
    marker: "# {mark} ANSIBLE MANAGED: {{ item }}"
    block: |
      {{ item }} {
          host.hostname = "{{ item }}.domain.local";
          path = "/usr/local/jails/{{ item }}";
          exec.consolelog = "/var/log/jail_{{ item }}.log";
          ip4.addr = {{ alias_ip }};
      }
  loop: "{{ jails | sort | flatten(levels=1) }}"
  loop_control:
    extended: yes

Let’s tell rc.conf to run jails at startup.

- name: start jails at startup
  community.general.sysrc:
    name: "jail_enable"
    value: "YES"

Finally, add the tasks to start the jails now. We can’t use service module here, because args argument don’t pass its value to service jail $action. The service has a rc of 0 if the service is already running, so it’s not a problem to trigger the start at each playbook run.

- name: start jails
  shell: service jail start "{{ item }}"
  loop: "{{ jails | sort | flatten(levels=1) }}"

Run and test

Run the playbook to provision jails.

$ ansible-playbook playbook.yml
[...]

TASK [jails : create zfs jails dataset] ****************************************************************************
changed: [host-test]

TASK [jails : create zfs per jail dataset] *************************************************************************
changed: [host-test] => (item=bind)
changed: [host-test] => (item=nginx)

TASK [jails : template bsdinstall script] **************************************************************************
changed: [host-test] => (item=bind)
changed: [host-test] => (item=nginx)

TASK [jails : bsdinstall jails] ************************************************************************************
changed: [host-test] => (item=bind)
changed: [host-test] => (item=nginx)

TASK [jails : authorize your ssh key] ********************************************
changed: [host-test] => (item=bind)
changed: [host-test] => (item=nginx)

TASK [jails : permit root login] *************************************************
changed: [host-test] => (item=bind)
changed: [host-test] => (item=nginx)

TASK [jails : set default jails config] ****************************************************************************
changed: [host-test]

TASK [jails : declare jails] ***************************************************************************************
changed: [host-test] => (item=bind)
changed: [host-test] => (item=nginx)

TASK [jails : start jails at startup] ******************************************************************************
changed: [host-test]

TASK [jails : start jails] *****************************************************************************************
changed: [host-test] => (item=bind)
changed: [host-test] => (item=nginx)

PLAY RECAP *********************************************************************************************************
host-test                  : ok=4    changed=10    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

List your running jails and check their IP.

$ ansible host-test -m shell -a "jls"
host-test | CHANGED | rc=0 >>
   JID  IP Address      Hostname                      Path
     1  192.168.0.101   bind.domain.local             /usr/local/jails/bind
     2  192.168.0.102   nginx.domain.local            /usr/local/jails/nginx

$ ansible host-test -m shell -a "jexec bind ifconfig | grep inet"
host-test | CHANGED | rc=0 >>
	inet 192.168.0.101 netmask 0xffffff00 broadcast 192.168.0.255

$ ansible host-test -m shell -a "jexec nginx ifconfig | grep inet"
host-test | CHANGED | rc=0 >>
	inet 192.168.0.102 netmask 0xffffff00 broadcast 192.168.0.255

You can now reach your jails with Ansible, just add their entries in your inventory.

$ cat << EOF >> hosts
bind ansible_host=192.168.0.101
nginx ansible_host=192.168.0.102
EOF

$ ansible nginx:bind -m ping
bind | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/local/bin/python3.7"
    },
    "changed": false,
    "ping": "pong"
}
nginx | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/local/bin/python3.7"
    },
    "changed": false,
    "ping": "pong"
}

In the next part, we will see how to provision vnet jails.

Git pre commit hooks

2021-06-02T00:00:00+02:00

Git Hooks are shell scripts integrated to your git directory. They are triggered before: pre- or after: post- some actions, commit, push, checkout and run locally. The most useful function is pre-commit which can check your code syntax, lint to improve the code quality, or trigger any local test.

Hooks are mostly always the same, users share them on github and pre-commit tool automates hooks deployment.

$ pip install pre-commit

Deploy a hook

Create a test directory and git it.

$ mkdir git-commit-test && cd git-commit-test

$ git init
Initialized empty Git repository in /home/user/dev/git-commit-test/.git/

Choose hooks projects you want to use in pre-commit hooks list.
For this exemple, we will use yaml and shell checkers.

https://github.com/pre-commit/pre-commit-hooks : check-yaml and trailing-whitespace
https://github.com/shellcheck-py/shellcheck-py : shellcheck

Pre-commit configuration file is .pre-commit-config.yaml, create it at the project root.

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
  rev: v4.0.1
  hooks:
    - id: check-yaml
    - id: trailing-whitespace
- repo: https://github.com/shellcheck-py/shellcheck-py
  rev: v0.7.2.1
  hooks:
    - id: shellcheck

To init hook deployment, use install subcommand.

$ pre-commit install
pre-commit installed at .git/hooks/pre-commit

Test it

Let’s add a shell script to the project and a yaml file with some errors.

$ cat << EOF > test.sh
#!/bin/bash
var=1
echo $var
EOF

$ cat << EOF > test.yml
- key1
  key2:
EOF

Add those files to the index, pre-commit defaulty ignore files outside the index.

$ git add .

$ git status
On branch main

No commits yet

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
	new file:   .pre-commit-config.yaml
	new file:   test.sh
	new file:   test.yml

And just commit to fire your hooks !

$ git commit -m "added a great shell script and his yaml file"
Check Yaml...............................................................Failed
- hook id: check-yaml
- exit code: 1

mapping values are not allowed in this context
  in "test.yml", line 2, column 6

Trim Trailing Whitespace.................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook

Fixing test.sh

shellcheck...............................................................Failed
- hook id: shellcheck
- exit code: 1

In test.sh line 2:
var=1
^-^ SC2034: var appears unused. Verify use (or export if used externally).

For more information:
  https://www.shellcheck.net/wiki/SC2034 -- var appears unused. Verify use (o...

As pre-commit hooks failed, the commit was not created.

$ git log
fatal: your current branch 'main' does not have any commits yet

Trailing whitespace hooks automatically fixed his errors.
If you fix your shell script and yaml file, your commit will be ok.

$ sed -i 's/$var/\"$var\"/' test.sh

$ sed -i 's/key1/key1:/' test.yml

$ git add test.*

$ git commit -m "first sane commit"
Check Yaml...............................................................Passed
Trim Trailing Whitespace.................................................Passed
shellcheck...............................................................Passed
[main (root-commit) c689536] first sane commit
 3 files changed, 15 insertions(+)
 create mode 100644 .pre-commit-config.yaml
 create mode 100644 test.sh
 create mode 100644 test.yml

Custom Ansible Hook

I wanted to be able to check ansible syntax before.

- repo: local
  hooks:
    - id: ansible-syntax-check
      name: Ansible syntax check
      entry: "ansible-playbook --syntax-check playbook.yml"
      pass_filenames: no
      types_or: [yaml, jinja]
      language: system

Create the playbook.

cat << EOF > playbook.yml
- hosts: test
  tasks:
    - name: copy test.sh
      copyy:
        src: ./test.sh
        dest: dir
EOF

Then try to commit it.

$ git add .
$ git commit -m "added an insane playbook"
[user@osz git-commit-test]$ git commit -m "added an insane playbook"
Check Yaml...............................................................Passed
Trim Trailing Whitespace.................................................Passed
shellcheck...........................................(no files to check)Skipped
Ansible syntax check.....................................................Failed
- hook id: ansible-syntax-check
- exit code: 4

[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
ERROR! couldn't resolve module/action 'copyy'. This often indicates a misspelling, missing collection, or incorrect module path.

The error appears to be in '/home/user/dev/git-commit-test/playbook.yml': line 3, column 7, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

  tasks:
    - name: copy test.sh
      ^ here

As no shell script was in the index, shellcheck was skipped, but pre-commit detected our mispelled copy ansible module.

Nas project - part 3

2021-05-30T00:00:00+02:00

In part 2, I setup my nas, but I didn’t notice that I created ZFS pool without mirroring.

Let’s try to fix this.

$ zpool remove dpool sdc
$ zpool status
  pool: dpool
 state: ONLINE
 remove: Evacuation of /dev/sdc1 in progress since Sun May 30 08:51:38 2021
 4.23G copied out of 202G at 80.1M/s, 2.10% done, 0h42m to go
    566K memory used for removed device mappings
config:

        NAME          STATE     READ WRITE CKSUM
        dpool         ONLINE       0     0     0
          sdb         ONLINE       0     0     0
          sdc         ONLINE       0     0     0

errors: No known data errors

After it completes, I attach now the newly free disk to mirror the first one.

$ zpool attach dpool sdb sdc
$ zpool status
  pool: dpool
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun May 30 09:38:12 2021
        404G scanned at 2.43G/s, 1.52G issued at 9.41M/s, 404G total
        1.53G resilvered, 0.38% done, no estimated completion time
remove: Removal of vdev 1 copied 202G in 0h44m, completed on Sun May 30 09:36:27 2021
    566K memory used for removed device mappings
config:

        NAME          STATE     READ WRITE CKSUM
        dpool         ONLINE       0     0     0
          mirror-0    ONLINE       0     0     0
            sdb       ONLINE       0     0     0
            sdc       ONLINE       0     0     0  (resilvering)

errors: No known data errors

It ended without any stress/warning/error… how much do I love ZFS !

DNS day

I want to configure a authoritative DNS for my eoli3n.eu.org domain.
This DNS server will not be used as a recursive one, even locally, I just want it to resolv *.eoli3n.eu.org.
Following bind9 debian wiki page, I setup a bind authoritative server in a chroot. Here’s the bind config

$ cat /etc/bind/named.conf.options
options {
    directory "/var/cache/bind";

    // Listen on ipv4 interfaces only
    listen-on {
        127.0.0.1;
        192.168.0.253;
    };
    listen-on-v6      { none; };

    // Hide version
    version "[SECURED]";

    // disable zone transfert
    allow-transfer    {"none";};

    // disable recursion
    recursion no;

    // allow query
    allow-query       { any; };
    allow-query-cache { any; };
};

$ cat /etc/bind/named.conf.local
// https://serverfault.com/a/306109/339917
include "/etc/bind/zones.rfc1918";

zone "eoli3n.eu.org." IN {
    type master;
    file "selfhost.zone";
};

$ cat /var/bind9/chroot/var/cache/bind/selfhost.zone
$TTL 86400
@       IN      SOA     ns1 root (
                        1622390729 ; serial number
                        8h ; Refresh
                        2h ; Retry
                        8h ; Expire
                        4m ; Min TTL
                        )

@       IN      NS      ns1
ns1     IN      A       $public_ip

*       IN      A       $public_ip

Lets test it

$ dig @127.0.0.1 ns1.eoli3n.eu.org
[...]
;; ANSWER SECTION:
ns1.eoli3n.eu.org.	86400	IN	A	$public_ip

;; AUTHORITY SECTION:
eoli3n.eu.org.		86400	IN	NS	ns1.eoli3n.eu.org.
[...]

Get my own domain on eu.org

I asked for eoli3n.eu.org domain by creating a request on the free domain registrar eu.org.
To configure it, just add ns1.eoli3n.eu.org glue record with your public IP and redirect TCP/UDP on port 53 to your DNS server on your ISP router.

Password store

2021-05-30T00:00:00+02:00

Password managment is the cornerstone of services security. If you use web, you would create one random password per service, and activate Time-based One-Time Password for all important one, like mail.

I used firefox lockwise to manage my web passwords, and FreeOTP on Android.
Firefox Lockwise can be configured to asks for a master password at every accesses.
Easy synchronization between multiple hosts, including my phone, sticked me to it.

Problem is that I don’t really own my data, that I can’t drop Firefox if I would like to, and I can’t manage non web passwords.

Pass defines itself like “the standard unix password manager”.
It uses gnupg to encrypt passwords as .gpg files, and git to distribute.
A clever extension lets you use it with rofi : rofi-pass
But pass defaulty miss some comfortable features:

Automatic commit/push/pull, if you forget to commit a change, you can’t access it on other devices
TOTP generator, but pass-otp exists

Gopass handles those by default, and even more.
You can manage multiple password stores and share them with your teams, with access control lists based on gpg.
A password leak and quality checker is embeeded, desktop notifications, etc..

Setup and configure GnuPG

First, you need to create a gpg keypair. Ensure that you use gnupg2, some linux distros has gnupg v1.x in a different package.

On Voidlinux:

$ gpg2 --full-generate-key
# (1) RSA et RSA
# size: 3073 bits
# set expiration
# set your real name, email and comment

Generate a revoke certificate and store it in a safe place, check your keyid with gpg2 --list-keys

$ gpg2 --gen-revoke --output revoke.asc "$key_id"

To backup your key, export you secret key and the trust db

$ gpg2 --export-secret-keys --armor "$key_id" > secret.asc
$ gpg2 --export-ownertrust > trustdb-backup.txt

Gnupg defaulty use a ncurses pinentry, you would change it for a graphical one, I chose pinentry-gtk and configured gpg by creating the file ~/.gnupg/gpg-agent.conf

pinentry-program /bin/pinentry-gtk-2

To test gpupg, you can try to sign a file with gpg2 --sign test.file.

I use keychain to autostart gpg agent at first shell login, with fish

$ cat ~/.config/fish/conf.d/keychain.fish
# https://stackoverflow.com/questions/39494631/gpg-failed-to-sign-the-data-fatal-failed-to-write-commit-object-git-2-10-0
set -x GPG_TTY (tty)

# https://github.com/fish-shell/fish-shell/issues/4583
if status --is-interactive
    keychain --eval --agents ssh --quiet -Q id_rsa | source
    keychain --eval --agents gpg --quiet --gpg2 -Q | source
end

I don’t target the key to load, when I will use it, it will be added after typing my passphrase.
Gpg agent is not like SSH agent, it forgets the passphrase token after 600 seconds by default.

Configure Git with Gnupg

Now, you need a centralised git repository, and the ability to sign your commits.
I created mine on my nas server with gitolite, but you can use a Github private repository.
To make signing work, I needed to edit my git-config to match my gpg binary and my gpg key.

$ cat ~/.gitconfig
[user]
    name = eoli3n
    email = jonathan.kirszling@runbox.com
    signingkey = eoli3n
[pull]
    rebase = false
[gpg]
    program = gpg2

Let’s try to sign a commit:

$ mkdir git-gpg-test
$ cd git-gpg-test/
$ echo "testing" > README.md
$ git init
$ git add README.md
$ git commit -S -m "testing to sign this commit"
[master (commit racine) c8fb990] testing to sign this commit
 1 file changed, 1 insertion(+)
 create mode 100644 README.md

Ok, great, now let’s create the password store.

Password Store

$ gopass setup
   __     _    _ _      _ _   ___   ___
 /'_ '\ /'_'\ ( '_'\  /'_' )/',__)/',__)
( (_) |( (_) )| (_) )( (_| |\__, \\__, \
'\__  |'\___/'| ,__/''\__,_)(____/(____/
( )_) |       | |
 \___/'       (_)

🌟 Welcome to gopass!
🌟 Initializing a new password store ...
🌟 Configuring your password store ...
🎮 Please select a private key for encrypting secrets:
[0] gpg - 0xFEEDBEEF - Jonathan Kirszling <jonathan.kirszling@runbox.com>
Please enter the number of a key (0-12, [q]uit) (q to abort) [0]: 0
❓ Do you want to add a git remote? [y/N/q]: y
Configuring the git remote ...
Please enter the git remote for your shared store []: git@nas.domain.fr:passwords.git
✅ Configured

Add your first password

Use insert subcommand to create password entries

$ gopass insert work/test
Enter password for work/test:
Retype password for work/test:

Querying it will prompt for you gpg passphrase.

$ gopass work/test
Secret: work/test

Passw0rd

Add a TOTP

To add a TOTP key, you need to use --multiline argument

$ gopass insert -m work/vpn
totp: XXXXXXXXXXXXXXXXXXXXXX

$ gopass totp work/vpn
069268 lasts 8s 	|----------------------========|

Synchronization with Android

On Android, you need OpenKeychain, to add your previously exported gpg secret key.
To be able to use TOTP and your password store, use Password Store with your ssh key to reach you git passwords repository. No need to use FreeOTP anymore.
Password Store does not pull/push commit automatically, don’t forget to sync from the app !

Configure a new node

Import your gpg key and your trust db

$ gpg2 —-import secret.asc
$ rm ~/.gnupg/trustdb.gpg
$ gpg2 --import-ownertrust < trustdb-backup.txt

Then gopass wrap your passwords reporistory git clone

$ gopass clone git@nas.domain.fr:passwords.git

You project with be stored in .local/share/gopass/stores/root/.

Go a bit further

On my desktop hosts, I now need to find an alternative to rofi-pass for Wofi.
gopass should be fully compatible with pass by default, except for totp generator which differs a bit.

gopass has a great plugin to manage passwords from browser: gopassbridge.
It would be a great firefox lockwise replacement.

20/06/23 edit

This is the way to get wofi-pass working, with autocopy and totp mode.

Improve Backups

2021-05-21T00:00:00+02:00

After my first post about how to setup BorgBackup and a first install of a home nas, I do have now two borg servers, lets spread our backups !

Drop the shell backup script

Borgmatic is an overlay to BorgBackup which let you configure everything with a yaml file and wrap Borg command to let you interact with your repository easierly.

Here a /etc/borgmatic/config.yaml file exemple

11/06/23 edit: use patterns instead of exclude_patterns to be able to backup directories in a declarative way
See borg patterns help

location:
  source_directories:
    - /

  repositories:
    - root@nas:/data/backups/osz

  exclude_caches: true

  patterns:
    - R /
    - '- **/lost+found'
    - '- **/*.iso'
    - '- **/*.mkv'
    - '- **/*.vmdk'
    - '- **/*.pyc'
    - '- root/.cache'
    - '- home/*/.cache'
    - '- home/*/.var/app/*/cache' # flatpak caches
    - '- home/*/.local/share/Steam' # steam installed games
    - '- home/*/.local/share/Trash'
    - '+ etc/**'
    - '+ root/**'
    - '+ home/**'
    - '+ var/log/**'
    - '! re:^(dev|proc|run|sys|tmp)'
    - '- **'

storage:
  encryption_passphrase: "***************************"
  compression: zstd
  archive_name_format: '{hostname}-{now:%Y-%m-%dT%H:%M:%S}'
  ssh_command: ssh -i /root/.ssh/backup_rsa
  relocated_repo_access_is_ok: true

retention:
  prefix: '{hostname}-'
  keep_daily: 7
  keep_weekly: 4
  keep_monthly: 6
  keep_yearly: 2

consistency:
  check_last: 3

On the repository server, you need to add a restricted authorized_key as explained in my previous post.
Then you can create the repository and start your first backup.

$ sudo borgmatic init --encryption=repokey-blake2
$ sudo borgmatic -v2

As the process is now standard, you can write an ansible task to add an anacron and automate backups more nicely.

- name: automate daily backups
  copy:
    dest: /etc/cron.daily/backup
    mode: 0755
    content: |
      #!/bin/bash
      borgmatic -v1

The borgmatic toolbox let you interact with the repository from the client.

$ sudo borgmatic info
root@nas:/data/backups/osz: Displaying summary info for archives
Repository ID: 664b076398e3d4ef96031d33f99ec0df1bb98a8ca39b181052f5dbc6c335f70e
Location: ssh://root@nas/data/backups/osz
Encrypted: Yes (repokey BLAKE2b)
Cache: /root/.cache/borg/664b076398e3d4ef96031d33f99ec0df1bb98a8ca39b181052f5dbc6c335f70e
Security dir: /root/.config/borg/security/664b076398e3d4ef96031d33f99ec0df1bb98a8ca39b181052f5dbc6c335f70e
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
All archives:                1.90 TB              1.08 TB            149.50 GB
                       Unique chunks         Total chunks
Chunk index:                  961954              8286300

$ sudo borgmatic list
root@nas:/data/backups/osz: Listing archives
osz-2021-05-14T18:19:19-voidlinux-install Fri, 2021-05-14 18:19:20 [dd21865bff728fdf4751cdc0e1f714164436eb5863452298b72952093dfbad4c]
osz-2021-05-15T11:50:05              Sat, 2021-05-15 11:50:06 [a38b92d57f58c97195e42047611679aa24a065a092da93d6ed9a68d7d94a52ad]
osz-2021-05-16T12:21:03              Sun, 2021-05-16 12:21:03 [b2e2f061939bb4818cb7be33e7da2c572ce7b60ebb6f9482ed317e18cf01895f]
osz-2021-05-17T09:23:14              Mon, 2021-05-17 09:23:15 [183c1b2f399999012cfa977a3b5f67ca3b8b0299384adfad486e3858a469659e]
osz-2021-05-18T18:36:34              Tue, 2021-05-18 18:36:35 [b71d73f6d62a328ff09c5728ddb042b6858148fcfbfe078828178abd34a10795]
osz-2021-05-19T08:16:24              Wed, 2021-05-19 08:16:24 [514b13dcdfbca936930adf67066bb7fcb5e668f5301c0d5ec1a98915c9926bb9]
osz-2021-05-20T09:40:59              Thu, 2021-05-20 09:40:59 [de40b1ca2cfe99893cb8023b2c496ba56695f3596199a126ac79ccf36ee566d0]

$ sudo borgmatic mount --archive osz-2021-05-19T08:16:24 --mount-point /mnt

$ sudo ls /mnt
bin   dev  etc	lib	lib64  mnt  proc  run	sys	tmp  var
boot  efi  home  lib32	media  opt  root  sbin	sysroot  usr

Spread your Backups

I use Syncthing to sync my backups over the network between two repository servers.
The important line in the client configuration is relocated_repo_access_is_ok: true which lets you access your backups from the second server.
Syncing a borg repository is not the recommended way to spread your backups, because if a data corruption occurs on one side, it is stupidly replicated.
You should prefer to add a second repository in the yaml config, borgmatic will trigger two separated backups.

I chose to sync with Syncthing because one of my repository is accessible only from a OTP secured vpn. I can’t automate VPN connection on all clients that I backup.

The replication is done in two ways, from server 1 to server 2 and vice-versa.

02/01/22 edit

As borg documentation says, borg repositories are not designed to be synced.
When I switched to redundant backups, I had to debug my repositories for few hours… So, just ,don’t.
Using two backup locations in borgmatic config will took twice the time, but security worth it.

  repositories:
    - root@nas:/data/backups/osz
    - root@borgbase:/repo

Online Backups

If you backup at home and at a different location, that’s pretty solid. I was annoyed by the fact that I backup my personnal data at work as second place, and wanted, for my most important data, to be safe to move in another city, and changing work without to be worried about my backups.

BorgBase describes itself as “Simple and Secure Offsite Backups” service.
To use it, simply open an account for free, to test. Then you will be able to upgrade your plan to the 100G small plan. That’s enough for me, for my most important data, and only costs 2€/month under 100G, and then 0.01€/Go/month.

Repositories support encryption, and the web UI is secured with 2fa TOTP authentication.
I upload my backups at 12Mo/s, so I’m fully satisfied with the service.
You can enable alerts when repositories didn’t get any backup since some days.
Let’s see with time.

11/06/23 edit

Per repository configurations

Borgbase storage is not free, you can then define a shorter retention time on that repository.
To do this, we will use include feature of configuration files, which is documented in Borgmatic multiple backup configurations.

You need to create a /etc/borgmatic/config-main.yaml file which will have configurations that apply to both repostiories. Then two /etc/borgmatic.d/config-{nas,borgbase}.yaml for repository specific configurations, which will include the main configuration file. You need to put the main configuration file outside of /etc/borgmatic.d or it will be processed as a third configuration file.

The borgmatic/config-main.yaml will contains in my case excludes and storage configurations.

location:
  exclude_caches: true
  patterns:
    - R /
    - '- **/lost+found'
    - '- **/*.iso'
    - '- **/*.mkv'
    - '- **/*.vmdk'
    - '- **/*.pyc'
    - '- root/.cache'
    - '- home/*/.cache'
    - '- home/*/.var/app/*/cache' # flatpak caches
    - '- home/*/.local/share/Steam' # steam installed games
    - '- home/*/.local/share/Trash'
    - '+ etc/**'
    - '+ root/**'
    - '+ home/**'
    - '+ var/log/**'
    - '! re:^(dev|proc|run|sys|tmp)'
    - '- **'

storage:
  encryption_passphrase: "********************************"
  compression: zstd
  archive_name_format: '{hostname}-{now:%Y-%m-%dT%H:%M:%S}'
  relocated_repo_access_is_ok: true

borgmatic.d/config-nas.yaml and borgmatic.d/config-nas.yaml will contain location with source path, the repository url, consistency and retention policies.

<<: !include /root/.config/borgmatic/config-main.yaml

location:
  source_directories:
    - /

  repositories:
    - path: ssh://borg-base/./repo
      label: borgbase

retention:
  prefix: '{hostname}-'
  keep_daily: 7
  keep_weekly: 4

consistency:
  check_last: 3

<<: !include /root/.config/borgmatic/config-main.yaml

location:
  source_directories:
    - /

  repositories:
    - path: ssh://nas/backups/osz
      label: nas

consistency:
  check_last: 3

retention:
  prefix: '{hostname}-'
  keep_daily: 7
  keep_weekly: 4
  keep_monthly: 6
  keep_yearly: 1

You can now run borgmatic and it will proceed as configured.
Note that borgmatic now supports labels for repositories.