VMWare upgrade 5.1 -> 6.0 notes

In 2013, the fine folks at Interconnekt sold us a fine Dell PowerEdge T420 server to run VMWare and a bunch of virtual machines. VMWare came installed on a dual SD module, and this came in handy later. We quickly discovered that VMWare ESXi 5.1 has a RAM limit of 32GB, so we removed the other 32GB and sailed on.

Three years on, and we have bumped up against that limit, our VMs exhausting available supply. We could pare down our use on some machines, but knowing there is an upgrade available and some DIMMs in the cupboard, I’m upgrading.

Dell recommend their customised version of VMWare, so I downloaded that through the Drivers & Downloads part of their website. I was stumped finding the link for a while, but eventually found I had to ‘change OS’ from Windows to VMWare, and found it under Enterprise Solutions. There is nothing to suggest skipping 5.5 is a bad idea, so I grabbed 6.0 and burned it to CD.

Backup the VMs

I installed ghettoVCB and did a test backup run of all machines:

$ wget -O ghettoVCB.vib https://github.com/lamw/ghettoVCB/blob/master/vghetto-ghettoVCB.vib?raw=true
$ scp ghettoVCB.vib root@server:
esxi # esxcli software vib install -v /ghettoVCB.vib
 [DependencyError]
 VIB virtuallyGhetto_bootbank_ghettoVCB_1.0.0-0.0.0 violates extensibility rule checks: [u'(line 23: col 0) Element vib failed to validate content']
 VIB virtuallyGhetto_bootbank_ghettoVCB_1.0.0-0.0.0's acceptance level is community, which is not compliant with the ImageProfile acceptance level partner
 To change the host acceptance level, use the 'esxcli software acceptance set' command.
 Please refer to the log file for more details.
esxi # esxcli software vib install -v /ghettoVCB.vib -f
Installation Result
   Message: Operation finished successfully.
   Reboot Required: false
   VIBs Installed: virtuallyGhetto_bootbank_ghettoVCB_1.0.0-0.0.0
   VIBs Removed: 
   VIBs Skipped: 

GhettoVCB creates a snapshot, does a backup of that snapshot, then deletes the snapshot. It fails if a snapshot already exists.

esxi # vim /etc/ghettovcb/ghettoVCB.conf
esxi # /opt/ghettovcb/bin/ghettoVCB.sh -a -c /etc/ghettovcb/ghettoVCB.conf -l /ghettoVCB-`date +%Y%m%d`.log

Some VMs had old unused snapshots, and some were in an old format, probably from when they were copied across from previous versions of ESXi. Therefore those backups failed. Right click, upgrade VM worked. Delete old snapshots. A successful test backup took 24 hours to our NAS – slow, but that’s okay in this case.

I didn’t want to manage changes in the event of failure, so I shutdown all VMs before running the actual backup. 24 hours later, I was ready to upgrade ESXi and install more RAM.

Upgrade ESXi to 6.0.0 Update 2

I put ESXi in maintenance mode so it wouldn’t automatically boot VMs on next boot. I first wanted to see that all is well. Shutdown ESXi, physically removed all HDDs and network cable from NAS – for paranoia’s sake, boot up with the CD I previously burned. Select upgrade and the dual SD module. Dammit, the upgrade failed because there’s a CommunitySupported VIB installed – ghettoVCB! Boot up, remove it:

esxi # esxcli software vib remove --vibname=ghettoVCB

Shutdown and boot the CD again. The upgrade was simple as can be, and for interest I timed it: 8 minutes of processing time. ESXi boots, complains about missing HDDs, and works just fine.

I’d heard of another benefit that I am now going to realise: I don’t have to boot my Windows VM to run vSphere Client any more! I might still for some things, because web clients sometimes suck, but not having to is great.

Install 4 x 8GB 2Rx4 1333MHz RAM DIMMs

Now, on to the RAM upgrade. The machine is nicely laid out, with easy access to the DIMMs. Naturally I want to put two of my DIMMs into each bank of slots, but where? There are two used in each out of six. The information panel on the case was helpful but not conclusive. Over to the Owner’s Manual.

“Populate all sockets with white release tabs first and then black.”
“Advanced ECC mode extends SDDC from x4 DRAM based DIMMs to both x4 and x8 DRAMs”
“Memory Optimized (Independent Channel) Mode … supports SDDC only for memory modules that use x4 device width and does not impose any specific slot population requirements”

A bit of discussion led me to believe that I should just place them closes to the CPU and with x4 DIMMs it didn’t make a lot of difference. It worked. Sadly I left out the big plastic baffle from the case when I reassembled – not a huge problem, but I need to schedule another outage to reinstall it.

Job done. I’m happy.

FreeBSD and pg_upgrade

Thank you kreynolds, your post makes my current job look dead easy. I want to upgrade PostgreSQL from 9.3.12 to 9.5.4, and after checking through all of the release notes between the two versions, I have narrowed the relevant ones down to the two biggest releases: 9.4 and 9.5 (thanks in part to our devs not using the darkest corners of PostgreSQL features… yet).

A side note: PostgreSQL is a shining example of how to do release notes. All in one place, linked properly between versions, available back as far as the eye can see, to pre-1.0 in the 1990s!

Another win for documentation: the FreeBSD handbook. Curated by members of the project, a simple URL, and quality information.

I could use pg_dumpall > backup.sql, upgrade the package, then psql -f backup.sql postgres, or do it in place faster with pg_upgrade. FreeBSD doesn’t simply allow both old and new packages to be installed together, so enter jails.

# jailroot=/usr/tmp/pg_upgrade
# bsdinstall jail $jailroot
# pkg -r $jailroot install postgresql93-server

In the dialog boxes, choose a nearby mirror and no extra components. Wait for the installation to complete.

# su pgsql -c 'pg_dumpall -c | bzip2' > /usr/tmp/pgdump.sql.bz2
# service puppet stop
# service postgresql onestop
# pkg delete postgresql93-client postgresql93-contrib postgresql93-server
# mv /usr/local/pgsql/data{,.93}
# pkg install postgresql95-client postgresql95-contrib postgresql95-server
# service postgresql oneinitdb
# pg_controldata -D /usr/local/pgsql/data.93

“Latest checkpoint location” needs to be the same on master and replication slaves. pg_upgrade(1) has good information on the 16 step process.

# su -l pgsql -c "pg_upgrade -b $jailroot/usr/local/bin -B /usr/local/bin -d /usr/local/pgsql/data.93 -D /usr/local/pgsql/data -j 16 -k --check"
Checking for reg* system OID user data types       fatal
Your installation contains one of the reg* data types in user tables.
These data types reference system OIDs that are not preserved by
pg_upgrade, so this cluster cannot currently be upgraded. You can
remove the problem tables and restart the upgrade. A list of the problem
columns is in the file:
    tables_using_reg.txt
[root@flora ~]# wc -l ~pgsql/tables_using_reg.txt 
 38 /usr/local/pgsql/tables_using_reg.txt

Bother. I can’t use pg_upgrade. Oh well, I’ll talk to the devs for next time, and get on with dump and restore.

Tribute to Nice

A professional cyclist and a pretty decent orator, I’ll let Tom Dumoulin’s words speak for themselves.

It’s a beautiful stage win on a very very sad day
We all woke up with the news or maybe you saw it last night
but I woke up with it
and then cycling is not important any more huh, for a few a moments
and maybe it still isn’t
but the organisation decided to race and I think it was a good decision
We cannot let terrorists decide our lives I think
I’m happy and very sad at the same time

PostgreSQL backup and replication systems

There are too many PostgreSQL backup and replication systems for me to keep track in my head, so here’s a list, with a few details relevant to me. The high availability matrix helped, and I found BDR performance discussion helpful, even if I didn’t pay close attention to which-one-is-fastest.

Name Installation Summary
BDR  from source Custom psql binary, asynchronous multi-master replication, high performance
Bucardo  ports tree Asynchronous multi-master or master-slave replication
OmniPITR  github Scripts to manage and monitor WAL replication, hot backups from master or slave servers
pg-rman  ports tree
pgbarman  ports tree Hot physical backup, point-in-time recovery management
pgbarman plugin: pgespresso  ports tree Enables backup from standby server
pglogical  ports tree Logical (cf physical) master-slave replication, postgresql >= 9.4, high performance, flexible, no hot standby state
pgpool-II  ports tree Synchronous multi-master replication, requires conflict resolution
postgresql hot standby  built in
repmgr  ports tree Manage and monitor replication and failover to standby server
Slony-I  ports tree
Skytools, Londiste  ports tree Asynchronous master-slave replication

I reckon pgbarman plus pgespresso plus repmgr (all from 2ndQuadrant) will do what I want. OmniPITR probably would too, with what looks like easier setup against potentially fewer restore options.

How to test a FreeBSD port

I am testing a new port written by Palle Girgensohn (https://github.com/girgen). He sent me the link https://github.com/girgen/freebsd-ports/tree/master/sysutils/filebeat to test.

I’ve never done this before, so I started thinking of how I might do so.

https://github.com/girgen/freebsd-ports being a fork of https://github.com/freebsd/freebsd-ports which in turn appears to be the mainline FreeBSD ports tree, I thought it would be possible to just get sysutils/filebeat into my own mainline ports tree (which I maintain with portsnap). So, combining my svn knowledge and git ignorance, I tried this:

root@travis:/usr/ports/sysutils # git clone
https://github.com/girgen/freebsd-ports/tree/master/sysutils/filebeat filebeat
Cloning into 'filebeat'...
fatal: repository 'https://github.com/girgen/freebsd-ports/tree/master/sysutils/filebeat/' not found

A bit of searching tells me I can’t simply do a subtree clone or checkout. I decided to do this:

root@travis:~ # mv /usr/ports /usr/ports.bak
root@travis:~ # git clone https://github.com/girgen/freebsd-ports /usr/ports

This worked, but (unsurprisingly) took a long time. I could then test the new port with the usual commands:

root@travis:~ # make -C /usr/ports/sysutils/filebeat install

So, I’ve achieved my short term goal, but it has left me wondering – how do other people do this? People who regularly test new ports or similar activity on anything other than the mainline ports tree surely have more streamlined practices. Somewhat complicated git commands to do a ‘sparse checkout’? Is it normal practice that Palle Girgensohn provided me a fork of the entire ports tree? I could ask him to do it differently. Hmm, I wonder if I could fork a subset of his fork, and then clone my entire ‘subfork’.

Thoughts welcome.

Dear Prime Minister Turnbull, please fix foreign aid

I am writing in support of Australian foreign aid. I shouted agreement and gratitude at the TV when Charlie Pickering delivered a segment on the topic a few months ago, and I continue to be heartbroken that people of the world are dropping bombs on each other at vast expense.

I do not pretend to fully understand armed global activity, nor national budgets, but I do know that given a choice between spending time, effort and money on overseas military engagements versus well-thought out nation building projects (Oh. So. Very. Many. To. Choose), I choose the latter. This is not simply throwing money away, rather investing in the future of humanity, of which we are a part.

To be honest, I would prefer blindly giving a bomb’s-worth of food directly to anyone who even *might* be an enemy (in this complicated world of “the enemy of my enemy is, well, a friend of Russia”), than manufacturing, delivering and dropping that bomb upon them, their land, and the people around them.

I hope and trust that you are familiar with the goal of 0.7% GNP going to foreign aid. Please, help us move toward it, not away from it.

On behalf of TEAR Australia, I make the following two concrete requests:

  1. Stop the fourth consecutive cut to Australian aid so that we don’t reach our lowest ever levels of aid.
  2. Start the journey to fulfil our global promise to invest our fair share in a future free from poverty.

I can only assume TEAR Australia has sources indicating that a fourth cut is en route. If not, then I am grateful for small mercies.

(I sent this via http://dearprimeminister.org/ then added the links here.)