win win habits

PMman was designed for long lived VM's (compare contra: ephemeral AWS or OpenShift type instances) , and so our communication needs vary from other cloud instance control interfaces.  This applies for the client tenant side views, but also for the sysadmin / devop side view

One of the convenience features of the PMman cloud product we run is the ease of communicating with 'tenants' on a given dom0.  When I came in this morning, I had this notice waiting for me:

From: mdadm monitoring
To: root@kvm-nNNN.pmman.com
Subject: DegradedArray event on /dev/md0:kvm-nNNN.pmman.com

This is an automatically generated mail message from mdadm
running on kvm-nNNN.pmman.com

A DegradedArray event had been detected on md device /dev/md0.

Faithfully yours, etc.

It is not the end of the world -- there are four members in that raid array; all mutable data is backed up nightly, and it is easy enough to fix with a hot drive swap.  BUT, we did a drive swap for a different raid member on that same unit within the last month.  I want to totally remove that unit from production, and put it into 'on the bench' mode, so I can see if there is some deeper hardware issue

The neat part from my point of view is that part of our design included a way for contacting the tenant VM owners on JUST that box alone.  It is as easy as clicking a couple buttons and typing

The following message will be sent to the to the list of users [on box: kvm-nNNN.pmman.com]. Here is the opportunity to fine tune the list.

From:     support@pmman.com
Subject:     Raid array member maintenance needed
Notice Level:     N
Message:     We have had some raid array member failures on the underlying dom0 of a machine you run at PMman. We replaced a drive hot three weeks ago on this same unit, and now have a new failure.

This may be a portent of a failing drive control subsystem, rather than the drive (although the previously removed drive tested bad and has been RMA'ed)

One new feature of PMMan of which you may not be aware is the 'no extra charge' weekly reboot and Level 0 backup. This is called a 'cyclic' backup with a 7 day repetition interval. To the extent you have NOT enabled this feature, I strongly suggest you test it and take advantage of this enhanced functionality of the interface

It is accessible off the 'Backups' button of the VM control web interface. Feedback is welcome of course. A nice side benefit from OUR point of view of offering this to customers is that it enables us to do invisible migrations from one dom0 to another as the backup and reboot are occurring, and so 'clear out' a dom0 host of running client instance VMs

Thanks

-- Russ herrold
User list: ... [elided] ...

Emails go out, and a history item gets dropped into each VM's logfile as well as in our admin side logs.  Optionally I can have the tool turn on an 'attention needed' flag in the end tenant's console, that will persist until they acknowledge it.  We already do that as to 'too long between updates' and' 'too long between reboots' and such

We can of course do invisible 'hot' migrations of machines around, but even safer is to encourage the good habit of encouraging tenant VM owners to  take (and we automatically test) Level zero backups

Win-win

Right, Left, down the middle

A couple weeks ago, a 'Derecho' blew through Columbus, on its way to the metro DC area. Amazon had some failures that cascaded through to people who did not have site redundancy. People know that the East Coast was hit hard, but as we are out in 'fly-over' country they did not perhaps realize that we had several hundred thousand people around here without electricity for a couple weeks as well

I've mentioned before that the primary datacenter that we run our PMman product out of is at the Tier IV level -- multiply redundant cooling, power grid, power backup, fiber entrances, carriers. The owner, a friend, is just a fiend that he does not HAVE outages

Me, too. In our after-event review, I see that one of our secondary sites here in town fell back to its generators, but the rest were all fine. But all sites we use are well covered, all fiber, all multi-homed. Planning for failure was in our deployment planning checklist; we pay for (and we charge for) that coverage; and I consider it worth it

A national footprint customer based in Canada agrees. And their lead technical person reports that our connectivity is haster than their datacenter eighty miles from their home ofice. Not surprising, as oAltantaur main DC is on a 'main line' fiber route between Chicago, NYC, DC, and Atlanta -- financial markets and federal government presences can help, that way

If the availability of your online presence matters to you, feel free to ask for a quote

LOPSA at the PMman DC

I went up to a meeting at our North datacenter for PMMan, where local group of system admins held a meeting, starting up a local LOPSA chapter. Food and soda were provided by the DC operator, along with salad ... since when did sysadmins starting more healthy food, rather than a diet of high sugar, high caffeine junk food?

The presentation slide deck was fine, and the presenter (a 'long timer' at a local credit-card clearance operation) ran through his bullet list of what to look for in the 'build vs. buy (lease space)' decision, and then a number of siting concerns.

Now I am familiar with his firm's site from prior visits, and it is adjacent to a major highway with regular closures for accidents; adjacent to a major rail yard where chemical spills have caused evacuations; and sole serviced into the power company grid

Our North site was chosen after a survey of all offerings within a radius we were willing to drive to for 'end of the world' 'hands on' intervention; is jacked into two independent power grids along with the on-site generators, is a premier demonstration location for the former Liebert (now owned by Emerson) power and site conditioning

I happen do drink coffee daily at Staufs with Liebert's representative here in town. My evaluation team suggested the location as a finalist, and when I checked, it turned out that I already knew the owner / developer from long, long ago telephony days, and when I have time, I'll go up and 'shoot the bull' with him on Saturday mornings at the DC

We have had a grand total of ZERO power related outages, and only one network connectivity issue in the last three years, that lasting less than 15 minutes, and that, due to human error in not handling a BGP fail-across migration properly [the cut-over protocol was changes, as I noticed the drop from my monitoring and called the owner's cell phone at once ;) ]. Well suited to our 'enterprise' customers

It is 'carrier neutral' and hugely connected -- multiple entrances of up to 88 x 100 G fiber spread across six or seven principal carriers. Native IPv6 to all drops we run through multiple carriers, along with the IPv4. I helped with the IPv6 design and cut-over some 18 months ago, and it has been seamless. The facility, and our services, just do not have outages except that human error causes

It is not the cheapest in town ... but it is fairly priced for the value we have received

I had not sat down and reflected on how satisfied I was with that shift of our center of operations to the DC, but as I think about it, I am well pleased

Change control in operations

This crossed the puppet-users mailing list earlier today:

We have an engineering environment of around 200 CentOS servers, plus a production environment of roughly the same size. Currently, when we roll out a new server, we do a 'yum update' so the new server has the latest packages; however this means that just about every server has a different set of package versions - a system rolled out today will have different versions from one rolled out last month, and that will have different versions from one rolled out last year.

...

Has anybody else been faced with this problem, and if so, how did you resolve it?

Let's consider just the problem of 'package version skew' in operations, and come up with a solution for it. [The questioner is also 'starting' with a couple of deployment targets that vary over time because of a poorly considered 'start image' creation ... An obvious approach here is to have a couple of stable base deployment image, and a set of defined transforms to produce a basic engineering workstation or server, per to specification, and is largely uninteresting here]

1. Set up a local mirror of the centos external mirrors, and call it 'incoming'
2. Optionally, set a sub-mirror of 'incoming' called 'vault', and mirror in a fashion that does NOT delete old content no longer present on 'incoming'
3. Set a third mirror called 'testing', which 'picks and chooses' selected packages to test, and their dependencies (see the package: yum-utils for some tools to permit confirming that one has 'closure' of those dependencies)
4. Test on your pre-deployment 'bench' against 'testing' until you have a change-set you wish to deploy throughout the universe of your boxes under management. Obviously, several 'testing' mirrors can be set up, for differing classes of machines
5. FINALLY, have a master distribution mirror called 'rtm' that has a change-set from a 'testing' mirror deployed to it. Remove the stock repository specification files from

           /etc/yum.repos.d/ 

   and deploy local variants to taste, that point at 'rtm'. Again, several 'rtm' mirrors can be set up, for differing classes of machines

Something like this to ensure coherency of a enterprise wide deployment is usually mandated by a Change Control Board (explicitly, or implicitly). Obviously, other aspects of an IT policy document will attend to getting the various mirrors properly recoverable in one's backup strategy. [there, the 'testing' mirrors are often NOT covered, as they are ephemeral as to their usefulness, and recoverable out of 'vault' (top down) or from a 'rtm' (bottom up)]

Rock and Roll never forgets ...

I came across this quote [pdf] today, from 2006 doing some research. From TFA:

Q: Why isn't Oracle a member of the OSDL?
A. ... We basically know where to go. We have a good relationship directly with people in the Linux community. We have all our partners. So there is no immediate advantage to being a member for us.
Not to sound arrogant, but we know how to deal with the Linux community

Gotcha, Wim ... 'does not play well with others'

... in a related vein, I cannot remember the last time I heard an Oracle representative on the weekly LSB conference call

Enterprise distributions ...

Where old software goes, to live out its declining days

see this bug: https://bugzilla.redhat.com/attachment.cgi?id=439002

Perhaps the motto is: 'we shall backport no patch before its time'