Sun and snow

I’ve been hearing that it’s unseasonably warm on top of the fact that season ends next week, so I was a bit apprehensive about how much snow I’d see and how good it would be. Sure enough, Whistler Village is looking very summery, and there’s a “compulsory download” from from the top of the main gondola, meaning there’s definitely not enough snow to ski back down to the village at the end of a day’s fun. Well the only way to know for sure was to get up there, via a beautiful tree-lined ride on that gondola.

Most of the time I was with experienced people taking it all for granted, but I just used that to pick up gossip and snow conditions. I quickly picked up that the place is awash in Aussies – second only to Canadians and a long way ahead of any other accent (mainly Brits, Kiwis, Japanese).

Happily, the snow is great. It’s patchy in places, but there are more lifts to go higher so I followed the small crowd up there. Also happily, I’m (barely) good enough to do better than just survive on blue runs, given that most of the beginner terrain is closed. It’s warm up there, and shorts turned out to be the right choice, except for sunburnt legs!

No injuries, and no particular muscles overtaxed, which is to say my entire body is exhausted. I put in a full day yesterday, and did almost as many runs today much more quickly, then decided I’d had enough mid afternoon. Early dinner, another soak in the hot tub, more (!) sleep, and I’m ready for another day tomorrow. I wonder if the weather will hold.



Sip slowly & mindfully

Sage advice on an overpriced bottle of juice. To be expected I suppose, in Sydney Airport. The upstairs view from the café was a nice escape from the crowds. Next stop Vancouver.

If I only had the three big photos I wouldn’t know it was an airport, but I would remember it as Vancouver. They’ve done a great job of making the place look good, and everything ran smoothly too. Apparently people didn’t trust the driverless trains when they installed them in the 90s, so they put in cardboard cut-outs of drivers to help. They’re gone, and I got to look out the front window right up until we hit the city centre. It got busy so I found a corner in which to cower, and we went underground anyway. I was curious about some floating logs… didn’t lumberjacks float them down the river in yesteryear?

Very tired after not quite enough sleep on the aeroplane, I wandered aimlessly through Vancouver for a while, and eventually stumbled upon a walking tour (hey look, there’s me) which was perfect: chat with some travellers, learn a bit about the city, see some places. Logan was keen to tell us about Terry Fox, so I’ll honour him with a photo of his monument, outside the two big stadiums.

Solid lunch (“I’ll just have a salad…”) and cider put away, I found some Canadian rye whiskey (when in Rome) and dashed for my bus to the mountains, and sleep, glorious sleep.


I’m tired and grumpy. Kid 2 is anxious about school because she knows things are different today, so she’s cranky with us about every little thing. My wife is sick. I’ve still got more paperwork to do than I have motivation for. I’m not being as effective at work as I’d like to be. Other stuff that’s private. Our local community open space still hangs in the balance, although it feels like the balance is against us. People around the world are being nasty and insensitive to each other:

  • people are treating Barnaby Joyce poorly, and others around him
  • thoughts and prayers in the wake of a mass shooting
  • decades of something akin to genocide in Myanmar
  • dark web paedophile guy got caught, but he’s just one
  • Oxfam workers behaving poorly and the fallout from that
  • people walking 400km from Yemen to escape, to be attacked by Saudis

So, I’m grumpy. Not depressed, just grumpy. After deciding that I won’t, I’ll probably go eat some more crap food. It won’t help for long, but time will. I’ll look forward to Taekwon Do on the weekend.

Cat Empire helped on the way to work – no more gloomy news, just meandering lyrics to tease the brain onto wandering thoughts.

Puppet, OpenStack, where to start? Ceph

Ten months into this job, and I still feel like an OpenStack novice, but it feels better than a couple of months ago at least. In fact last week we had what I felt was a big automation win, where we deployed a Ceph OSD node from bare metal to joining the cluster without any ‘manual’ intervention. That automation needs more, well, automation, but at least it’s repeatable and consistent now. But I’ve leapt ahead of myself. This is a heavily abbreviated history of how we got here:

  • Luca deployed OpenStack with Fuel. Five short words which actually represent months of detailed work and a fair bit of complaining from his cubicle. Disk partitioning, network bonds, bridges, VLANs, GRE, VXLAN, MTU settings, bugs, confusing or missing or out of date documentation, people in the wrong timezone for proper conversations… oh my. I helped a bit.
  • I created an All-In-One (AIO) deployment with the puppet-openstack-integration (POI) project. I started comparing the (hiera) data between it and the Fuel-deployed stack.
  • Using POI I deployed a compute node almost to the point of working, but we managed to break our dev stack before we got to iron out the final kinks.
  • Luca got us started with MAAS, which proved a little more intuitive than xCAT and being built by Canonical it works well with Ubuntu. We customised the MAAS deployment process to suit our hardware and needs.
  • Ceph is not as much of a core integrated component of OpenStack as the other parts so it is another good candidate for early deployment tooling, and so we got started with Puppet-Ceph. In the end we found spjmurray’s Ceph module more intuitive and reliable, and it handled the new long term stable release 12.x Luminous almost as soon as it was released.

Here’s how we deploy a Ceph OSD node:

  • PXE boot the node:
$ ipmitool -I lanplus -H $IP -U $user -P $pass chassis power off
$ ipmitool -I lanplus -H $IP -U $user -P $pass chassis bootdev pxe
$ ipmitool -I lanplus -H $IP -U $user -P $pass chassis power on
  • Commission the node: Straightforward MAAS step from the documentation.
  • Customise the node: Network bridges, disk partitions, hostname. We have a hundred-line script to do this, and the main tools in use are the MAAS CLI and jq.
  • Prepare the curtin (curt installer) script (largely one-off work, although we continue to tweak it). Currently this just installs the Puppet Agent.
  • Deploy the node: Straightforward MAAS step from the documentation.

Once the node is deployed, it lets Puppet and our modules (which in turn use the Ceph module) take over, and we have more OSDs in our cluster!

$ ceph osd df tree
 -1       51.97385        - 53220G   177G 53043G 0.33 1.00   - root default
-21       20.06506        - 20547G 70573M 20478G 0.34 1.01   -     host new-node
 12   hdd  1.82410  1.00000  1867G  6458M  1861G 0.34 1.02  90         osd.23
 13   hdd  1.82410  1.00000  1867G  6433M  1861G 0.34 1.01  95         osd.24
 25   hdd  1.82410  1.00000  1867G  6344M  1861G 0.33 1.00  71         osd.25
 26   hdd  1.82410  1.00000  1867G  6429M  1861G 0.34 1.01  74         osd.26
 27   hdd  1.82410  1.00000  1867G  6394M  1861G 0.33 1.00 103         osd.27
 28   hdd  1.82410  1.00000  1867G  6412M  1861G 0.34 1.01  94         osd.28
 29   hdd  1.82410  1.00000  1867G  6429M  1861G 0.34 1.01 102         osd.29
 30   hdd  1.82410  1.00000  1867G  6559M  1861G 0.34 1.03 104         osd.30
 31   hdd  1.82410  1.00000  1867G  6343M  1861G 0.33 1.00  76         osd.31
 32   hdd  1.82410  1.00000  1867G  6474M  1861G 0.34 1.02  98         osd.32
 33   hdd  1.82410  1.00000  1867G  6293M  1861G 0.33 0.99  69         osd.33

CloudAtCost money grab

I was intrigued by the pay once model of Cloud At Cost and spent some money there a few years ago. The machine is good enough for my low level purposes, but I have seen a number of articles over the years complaining of their network performance and poor service. I figured you get what you pay for, and I would use it while it worked, and dump it if I hit problems. Besides, I could leave it lying idle and use it later when they’ve improved right? My one-time payment gives me a VPS for life.

So I thought.

I stopped using my machines a while back when there was a problem with a reboot, and I couldn’t get any help to get access again. The reimaging process was buggy, and of course without any effective support, that left my machine dead. Oh well, see above about leaving it idle until later.

A year or more later I’ve logged in to have another poke at it, and nothing has changed in the interface, including my inability to reimage my machines. I don’t think it’s worth submitting a support ticket, judging by the responses on other tickets and questions I can see in the system. On top of that, I now see an invoice for USD$9 for ‘annual service fee’, which is perhaps understandable given their description, but entirely unpalatable given my experience. Oh well, good luck to them I guess.

So long, and thanks for the very few fish, C@C.

Airline staff

Daddy, why is that gate so narrow?

So people don’t take their big bags onto the aeroplane themselves, and they check them in to go in the cargo hold.

So the pilot can put them in the aeroplane?

Yes, the pilot needs to put them in the place underneath. Well the pilot has just one special job of flying the plane. Other people do other special jobs like that one. It actually takes a lot of people to help get us and our things to fly from one place to another.

I don’t love flying – and I reckon few people do – but I am still grateful for the people who make it possible. I spend some hours packing, getting the timing and the papers all right, lining up, waiting and then waiting some more. For that, I get to be somewhere else much faster than by other ways. They get a day’s pay, which I guess they consider a fair trade. I don’t know how many flights per week a cargo loader, air steward or pilot might do, or how often they meet that tiny number of passengers who make things difficult, or what they have to deal with behind the scenes. So, thank you. I have time to sit and feel grateful, because the kids are settled.

(This is actually from November 2015, but the words haven’t dated. I’m still grateful to sit still when the kids are settled, and to anyone who helps that happen!)

Thank you, honest strangers

Life has been heavy going for the past year, more so than ever before. It’s had ups and downs, and yesterday was largely a down – I forgot too many things and failed to communicate with people properly so that only made things worse. Anyway, they were all very good about it and I’m lucky to have great caring people around me helping make it all work.

Today was better. I had time to get things done which will mercifully remain done. Washing, cleaning the chook cage, getting the kids to brush their teeth… have to be done again and again, but it’s good to make some progress on the backlog.

Anyway, to the title of the post: I dealt with two unfamiliar situations today, and both of them turned out more smoothly than I dared hope: Selling a low-value car, and shipping a kitchen appliance across the country to a Gumtree buyer with just-in-time (significant) payment of postage. Both of these have scam potential, certainly evidenced by a couple of the messages I’d received from potential buyers (or scammers). I reckon the best response is to treat everyone as if they’re honest, but take protective steps along the way, like taking photos of goods, explicitly discussing how things are happening, and recording relevant information. There’s no sense in being paranoid and putting people off side.

To cut a long story short, one person handed me cash, let me photograph their drivers licence and took away the car; another sent me the full agreed payment plus enough to cover relevant costs, and I no longer have these things cluttering up my life. To both of you, thank you. We drank champagne tonight for a few reasons, and you helped.