rbpod Appliance Setup Work Log

Task List




Ugh, I can't believe that I just wasted that much time trying to make the "fab" commands work with Pelican. As soon as I tried the makefile equivelants, it worked perfectly.

Hooray, I now have an empty site. Next I get to actually add content.


Hooray, my first article is done.



I ordered my raspberry pi today, case, and sd card today. Here's the damage:

I still need a power supply and an external hard drive. For the power supply, I just plan on buying a usb phone charger at Walgreen's or something since that's what I'm currently using with my existing Raspberry Pi. I have a bunch of old external hard drives, so thankfully that's a sunk cost.

I'm really looking forward to getting this stuff early next week so I can get started building my backup appliance.



I just bought a charger for 9.99 at bhphotovideo.com. Shipping was 3.00 so my total for that is $13. When you add that to my previous total, the final total for all parts is now roughly $77. That's about 8 months' worth of Dropbox, so if this works out I will pay it off in less than a year.



Hooray, I got my new Pi! I put in the SD card with Noobs on it and just booted it and I see the Noobs install screen. Here's what I'm choosing:


Ok, thanks system rebooted automatically and now I see raspi-config. First I'm going to choose option 1.

Oh, it looks like that wasn't necessary since I am using NOOBS.

As for #2, I'll change the pi password after I set up an SSH server so that I can copy and paste from KeepassX.

I don't want to enable #3 because a GUI takes up a lot of resources and we don't need it.

Let's see wat #4 has.

I1 - Locale

Choose -> en_US.UTF-8 UTF-8 Unchoose -> en_GB.UTF-8 UTF-8

I chose a default locale of en_US.UTF-8 because I'm the only one using this system.

I2 - Change Timezone

I'm going to set the timezone with a Puppet module, so I don't need to change that now.

I3 - Change Keyboard Layout

I'm a Colemak user, so I definitely need to change this :-)

For keyboard model, I choes the default Generic 105-key (Intl) PC.

For the layout, I chose *English (UK) - English (UK, Colemak)

On the next screen, I chose "The default for the keyboard layout".

On the Compose key screen, I chose No compose key.

I chose not to have Ctrl+Alt+BS terminate the X server.

I don't need 5, 6, or 7.

I clicked on 8 - Advanced Options

A2 - I would like to set the hostname to rbpod01.

A4 - I am also enabling the SSH server.

Finally, I'm choosing "finish".



Now I'm changing my password:

    pi@rbpod01 ~ $ passwd
    Changing password for pi.
    (current) UNIX password: 
    Enter new UNIX password: 
    Retype new UNIX password: 
    passwd: password updated successfully

Now that I can log in via SSH it's much easier to use the system.


Next I need to update all of my packages:

    $ sudo apt-get update && sudo apt-get upgrade

Next I need to reboot:

    $ sudo shutdown -r now

Next we need to install the default Puppet package:

    $ sudo apt-get install puppet

I can't install the newer package that is maintained by Puppet Labs because it doesn't work out-of-the-box on a Raspberry Pi. Also, I would rather not install using Rubygems because that is fairly difficult by comparison.



My goal today is to make monit notify me every time rbpod01 is rebooted. However, I first need to install the librarian-puppet package since I need to start keeping track of packages.

    $ cd /etc/puppet
    $ sudo vi manifests/site.pp

I'm adding the following to this new file:

    package { "librarian-puppet":
        ensure   => installed,
        provider => gem,

Next I need to run this command:

    $ sudo puppet apply /etc/puppet/manifests/site.pp

Super duper. That worked.

Next I would like to install the nullmailer package. First I need to create my Puppetfile:

    $ cd /etc/puppet
    $ sudo librarian-puppet init

This creates /etc/puppet/Puppetfile. I need to make a few changes. First, I need to delete these lines:

    # use dependencies defined in Modulefile
    $ modulefile

Now I'm going to add this so we can install the nullmailer module.

    mod "akumria/nullmailer"

Now I can install the module with this command:

    $ cd /etc/puppet
    $ sudo librarian-puppet install

I got the following error messages but I think I can ignore them:

    /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': iconv will be deprecated in the future, use String#encode instead.
    Using plain http as your version of Puppet 2.7.23 can't download from forge.puppetlabs.com using https
    Using plain http as your version of Puppet 2.7.23 can't download from forge.puppetlabs.com using https

Great, the nullmailer module is now in the right place:

    $ ls /etc/puppet/modules
    nullmailer  stdlib

Now I can actually provision the nullmailer application on my machine. First I need to open the /etc/puppet/manifests/site.pp file:

    $ vi /etc/puppet/manifests/site.pp

Here's what I'm adding for nullmailer:

    package { "mailutils": ensure => "installed" }

    class { 'nullmailer':
      adminaddr   => "me@me.com",
      remoterelay => "smtp.mandrillapp.com",
      remoteopts  => "--port=587 --starttls --user=me@me.com --pass=somethingRandom",

    # After installing nullmailer, I also had to edit the /etc/nullmailer/defaultdomain
    #>file. I just added "me.com" and that magically made Mandrill accept my email.

I'm using Mandrill as my SMTP provider. LOTS of reasons why this is a good idea on a remote device that you have limited control of. There's a free plan that's excellent that I recommend highly.

Plase note that the --pass option is my API key.

Now we can provision the software:

    $ sudo puppet apply /etc/puppet/manifests/site.pp

Great, that worked too. Now unfortunately I have to do one manual step to make this work before I can test. I need to open the following empty file:

    $ sudo vi /etc/nullmailer/defaultdomain

Now add the domain of your Mandrill login address. Here's what I have:


Now reload the nullmailer service:

    $ sudo service nullmailer reload

Awesome. Now we can finally test the ability to send mail from the command line:

    $ echo test | mail -s "foo" me@me.com

Now check you mail :-)

Next I need to provision monit.


Ok, here's what I need to add to the Puppetfile to install my custom monit package:

    mod 'tompurl/monit',
      :git => "ssh://tom@supersecret.tompurl.com:1234/gitrepo/tompurl-monit.git"

Currently this is only hosted on a private git server. I need to fix this soon. Now I can download it:

    $ cd /etc/puppet
    $ sudo librarian-puppet install

Here's what I added do provision monit in site.pp:

    ### START Monit
    package { 'elinks': ensure => "installed" }
    class { 'monit': 
      alert_address => "me@me.com",
      web_username  => "admin",
      web_password  => "monit",
    ### END Monit

...and I can install it with this command:

    $ sudo puppet apply /etc/puppet/manifests/site.pp

I can now visit the web interface using the following command:

    $ elinks http://localhost:2812

The user id is admin and the password is monit.

I'm now going to reboot to see if I get an email message.


Wow, I had to do a lot of work to set up email. I basically had to add all of my smtp info to the monit config file and update the from address. However, now I can send email.

Now I need to determine if I want to update my monit module with this new information or use the example42 module.



This btsync puppet module looks perfect:

I will definitely try this one out.


I need to remove my custom monit puppet package so I can install a different one. To do this, I ran the following commands:

    $ cd /etc/puppet
    $ sudo rm -rf modules/
    $ sudo rm -rf Puppetfile.lock

I then removed my custom monit package reference from the Puppetfile and ran the following command:

    $ cd /etc/puppet
    $ sudo librarian-puppet install

I'm now going to add the mcanevet-btsync and example42-monit packages to puppet by adding this to the Puppetfile:

    mod 'mcanevet/btsync'
    mod 'example42/monit'

And then I installed it with this command:

    $ cd /etc/puppet
    $ sudo librarian-puppet install

I also removed the monit Debian package:

    $ sudo apt-get remove monit
    $ sudo apt-get autoremove


Now I need try out the new monit package. First I created the following directory:

I then staged monitrc file that I want to use. Now I need to make the following changes to my site.pp file.

Delete this:

    class { 'monit':
      alert_address => "tom@tompurl.com",
      web_username  => "admin",
      web_password  => "monit",

...and replace it with this:

    class { 'monit':
      source => "puppet:///modules/rbpod/monit/monit.conf",

And now I can provision the software:

    $ sudo puppet apply /etc/puppet/manifests/site.pp


Whoops, it looks like I made a mistake with my dir structure. Here's the directory that I needed to create:

I then moved my monitrc file to that dir and renamed it monit.conf.


Hooray, the proper file was staged but once again email isn't working. Grumble grumble. I have no idea what's going on there :-)


Oh duh, I just realized that my domain name is being transferred. Mandrill is telling me that it's delivering the mail, so it must be a temporary domain issue.



I'm really happy that the example42-monit package is working so well for me. Next I would like to set the timezone. Here's what I'm adding to my Puppetfile:

    mod 'saz/timezone'

Then I run this:

    $ sudo librarian-puppet install

...and here's what I'm adding to site.pp:

    class { 'timezone':
      timezone => 'America/Chicago',

...and here's how I'm applying it:

    $ sudo puppet apply /etc/puppet/manifests/site.pp

Sweet, that worked. Now I'm dying without a few basic packages like tmux and vim. I'm therefore adding the following to my manifest:

    $enhancers = [ "vim",
    package { $enhancers: ensure => "installed" }

Now I'm applying that change in puppet.

It's also killing me that I have to enter my password every time I want to log in over ssh. Let's fix that by adding the public key on my laptop to the authorized_keys file. And of course we'll be doing this with puppet :-)

Here's what I'm adding to my init.pp file:

    ### START ssh keypair
    ssh_authorized_key { 'pi_for_tom':
      name    => "rbpod01 key",
      ensure  => present,
      key     => "thisIsn/tReallySecretthisIsn/tReallySecretthisIsn/tReallySecretthisIsn/tReallySecretthisIsn/tReallySecretthisIsn/tReallySecretthisIsn/tReallySecretthisIsn/tReallySecretthisIsn/tReallySecretthisIsn/tReallySecret"
      type    => "ssh-rsa",
      user    => "pi",
    ### END ssh keypair

Now I'm applying that change. Sweet, that worked too.


While I'm grabbing all of this low hanging fruit I should also secure SSH. Here's what I'm going to add to the manifest:

    ### START SSH-hardening
    package { 'openssh-server':
        ensure => "installed",

    service { "ssh":
        ensure => running,
        hasstatus => true,
        require => Package["openssh-server"],

    augeas { "configure_sshd":
        context => "/files/etc/ssh/sshd_config",
        changes => [  "set Port 9000",
                      "set PasswordAuthentication no",
                      "set PermitRootLogin no",
                      "set Protocol 2",
        require => Package["openssh-server"],
        notify  => Service["ssh"]
    ### STOP SSH-hardening

I'm applying that change now. Sweet, that worked too.


I know this makes me a wet blanket but I'd also like to harden sudo access. Currently, I don't have to enter a password when I run a sudo command. That seems like a big security hole to me, so I'm going to change that now. First I need a third party module so it's time to add this to my Puppetfile:

    mod 'saz/sudo'

After installing that module I'm going to add the following to my manifest:

    ### START sudo hardening
    class { 'sudo': }
    sudo::conf { 'pi':
        priority => 60,
        content  => "pi        ALL=(ALL:ALL) ALL
                     pi        ALL=NOPASSWD: /usr/bin/apt-get upgrade
                     pi        ALL=NOPASSWD: /usr/bin/apt-get update",
    ### STOP sudo hardening

This does the following:

Cool, that change worked too.


Ok, it looks like I'm done with low-hanging fruit :-) Now I'm going to create a really simple tor package that does the following:

Easy peasy. I'll start on this now.


Great, the torssh module is done and it's uploaded to my private git server. I added it to my Puppetfile and here's how I'm going to install it:

    ### START torssh
    class { 'torssh' :
        ssh_port => "9000",
    ### END torssh

...and here's what I got when I tried to run it:

    pi@rbpod01 /etc/puppet $ sudo puppet apply /etc/puppet/manifests/site.pp
    notice: /Stage[main]/Torssh::Prereqs/Apt::Source[torproject]/Apt::Key[Add key: 886DDD89 from Apt::Source torproject]/Apt_key[Add key: 886DDD89 from Apt::Source torproject]/ensure: created
    notice: /Stage[main]/Torssh::Prereqs/Apt::Source[torproject]/File[torproject.list]/ensure: created
    notice: /Stage[main]/Apt::Update/Exec[apt_update]: Triggered 'refresh' from 1 events
    notice: /Stage[main]/Torssh::Install/Package[deb.torproject.org-keyring]/ensure: ensure changed 'purged' to 'latest'
    notice: /Stage[main]/Torssh::Install/Package[tor]/ensure: ensure changed 'purged' to 'latest'
    notice: /Stage[main]/Torssh::Config/File[/etc/tor/torrc]/content: content changed '{md5}a9184f0a4f4675f6332d67f9895ac022' to '{md5}26ed04026f62b7615751b3679376b2d6'
    err: /Stage[main]/Torssh::Service/Service[tor]/ensure: change from stopped to running failed: Could not start Service[tor]: Execution of '/etc/init.d/tor start' returned 1:  at /etc/puppet/modules/torssh/manifests/service.pp:5
    notice: /Stage[main]/Torssh::Service/Service[tor]: Triggered 'refresh' from 1 events
    notice: Finished catalog run in 143.94 seconds

Hmmm...let me see why the service didn't start.


Ok, it looks like the Tor project people don't maintain a Raspberry Pi build. I therefore updated my module to use the tor package that's maintained by the Raspbian people.


All right, I'm now able to connect from my laptop to my rbpod over ssh using Tor. This is fantastic. Here's how I have configured my client.

First, I install the tor package. I don't yet know if this is necessary, but here's how I did it:

Look at the Client section at the bottom.



I got an email message from monit today telling me that something was wrong with my sshd and sftpd processes and that it wasn't going to monitor them any more. I updated those binaries this morning, and apparently if that happens while monit is running it doesn't like it very much.

To fix that, here's what I had to do:

  1. Restart monit
  2. Manually initialize monitoring of those files again using the web client.

Easy peasy.

The next step is to monitor tor. Then I can start dorking around with btsync.


I'm setting up a hard drive now. After plugging it in I deleted all of the old partitions using fdisk. I then added the btrfs-tools package to my $enhancers list in the puppet manifest and applied that change. I then formatted the partition with the following command:

    $ sudo mkfs.btrfs /dev/sda1

Next I will add this partition to my /etc/fstab. I wonder if I can do this with puppet.


Nice, of course you can!

    ### START manage external hard drive
    file { "/mnt/sda1":
        ensure => "directory",
        owner  => "root",
        group  => "root",
        mode   => "755",
    mount { "/mnt/sda1" :
        device  => "/dev/sda1",
        fstype  => "btrfs",
        ensure  => "mounted",
        options => "defaults",
        atboot  => "true",
        require => File["/mnt/sda1"],
    ### STOP manage external hard drive

Now we can install btsync.


Here's what I added to the manifest:

    ### START btsync config
    include btsync
    include btsync::repo
    btsync::instance { 'pi':
        storage_path => '/home/pi/.sync',
        webui        => {
            listen   => '',
            login    => 'admin',
            password => 'password',
    ### END btsync config

This isn't the most secure setup in the word. For example, I may want to add a "btsync" user in the future without a home directory to make it harder to escalate privileges if this service is cracked. However, for now this works! I'm able to visit the following url on my laptop with Iceweasel:

Shoot, I can't visit it in the console using elinks. Lame. I really don't want to expose this thing with Tor.

I just realized that I will also need a data directory like /mnt/sda1/backups that's owned by the user running the btsync process. Let me add that now.



I was able to set everything up and backup my videos and mp3's yesterday. Here's a few lessons that I learned:

My next step is to finalize my monit changes. Here's what I would like to do:

Once I'm done with these three tasks I think my system will meet all of my requirements. The final step will be publishing my code on github.



Urgh, I just found out that it's fairly difficult to secure the Btsync web ui using SSL:

Seems a bit ironic to me, but this is a deal breaker for exposing the ui as a Tor hidden service.



Today something funny happened. I was applying a security patch to tor via apt-get and this forced the service to restart, which kicked me off of the rbpod01. I then could not re-connect because it did not have a chance to start again. I had to call my host and ask him to unplug it and plug it back in again (basically do a hard reboot). This makes me want to do two things:

  1. If a tmux instance isn't open when the user logs in I want to start one. This will at least keep the patching process going if the user is kicked off of the server in this scenario.
  2. I need to have monit monitor the tor service so that it will automatically restart the service if it is accidentally shut down again.

Today I'm going to continue adding the mail and monit packages to the new rbpod package.


I just finished pushing a couple of puppet modules to github:

The docs aren't great yet, but I'll update those soon.


Great, I just added the torssh package to the rbpod package too. I'll add that to github too.


All right, I push all of today's changes to rbpod01 and it all seemed to work! Not a bad night's work.



The ssh hardening part of the rbpod module still isn't working. Here's what I get:

    # tom at pam in ~/Dev/Vagrant/rbpod/custom_modules [18:32:42]
    $ vp
    [default] Running provisioner: Vagrant::Provisioners::Puppet...
    [default] Running Puppet with /tmp/vagrant-puppet/manifests/default.pp...
    stdin: is not a tty
    Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera defaults

    Notice: /Stage[main]/Rbpod/Exec[apt-update]/returns: executed successfully

    Notice: /Stage[main]/Rbpod::Enhancers/Package[libaugeas-ruby1.9.1]/ensure: ensure changed 'purged' to 'latest'
    Notice: /Stage[main]/Rbpod::Ssh::Hardening/Augeas[configure_sshd]/returns: executed successfully

    Notice: /Stage[main]/Rbpod::Ssh::Service/Service[ssh]: Triggered 'refresh' from 1 events
    Notice: Finished catalog run in 16.48 seconds

However, the sshd_config file doesn't change.


Actually, I was mistaken. The ssh hardening script did work.