The problem with older packaging managers in two lines ...

There was a post from the NetBSD oriented 'pkgsrc' project into the Scientific-Linux mailing list earlier this week. Lots of skilled geeks following their muse ... but From the #pkgsrc IRC channel ('nick' changed to protect the poster)

09:11 wxzy> Ugh. I think I should have chosen the X from pkgsrc.
09:11 wxyz> The one I have now lacks fonts that applications need.

Yeah. Package 'Managers' that unroll tarballs, but lack a strong Requirements / Dependency system are like that

Naming Names and 'Shaming'

I customarily run all the updates for the Windows 7 (64 bit) box at home on Sunday, but was traveling then, and so just ran some. Adobe's Flash Player (each no doubt asserted as trademarks) had a update notice in the bunch. I refer to those Marks in this paragraph for identification and not for other purposes. Disparagement comes later

So I attained Administrator (the Windows role enabled for installing software in the customary security model), and went through the process of starting their updater. This is a small intermediate program which retrieves the larger update, and then hands off execution to that later payload. AT NO TIME, have I or any other person with the knowledge needed to reach Admin rights, ever consented to the (seemingly always 'defaulted on') "Also install some third party 'enhancement'". It seems MacAfee and its Security Suite are the current firm, co-marketing of their products with Adobe. Again, I refer to those Marks in this paragraph for identification and not for other purposes. Disparagement comes later

Notwithstanding no consent, the McAfee kit was part of the payload that the Adobe retrieved ... and installed ... with no consent to such by me

Let's be frank. Adobe's Flash Player is a gaping security hole, with a well documented track record of poor coding. There are 356 using the search link. I am all for cleaning up issues and not hiding them, but this is just a whorehouse of fail, Adobe

The McAfee record for 'Security Suite' is less clear, as it seems to me that the nameing of their products is designed to confuse, and segment purchases, rather than to provide long term clarity of what the names, used as Marks, mean. The CVE count at MITRE is 137 by that search. But I see from a more general search engine query that McAfee seems to have licensed for distribution and re-naming, its product: 'AT&T Internet Security Suite - powered by McAfee' for instance, so one cannot rely on the Marks to reliability identify offenders. AOL seems to have licensed it as well

McAfee seems to have other lapses as well

I am just not sure what the remedy is:

• Stop using Windows? This is pretty much done with the maturity of the Open Source ecosphere -- we have NONE at the office facing the internet, and only ONE left at home or in the family members I support. For commercial software use, Apple's OS/X and hardware have been remarkably complete and their use uneventful without need to load them down with defensive 'crapware'
• Stop using Flash? Is the pain worth the short remaining life Adobe's Flash seems destined to have. I recall there was an effort to re-implement to the published Flash specification, and Adobe (to their credit is better than most commercial vendors on publishing file format documentation -- think: PSF and friends). time to investigate what the FOSS replacement is
• And of course: 'Shame' bundlers, and providers of 'ride-along' 'crapware'. So: Adobe, McAfee, and your products: Flash Player, and Security Suite -- consider yourself identified as bad actors, worthy of avoidance

Well, a start then

About those doughnuts ...

In a recent post, I closed:
Dollars to doughnuts, there is 'more than one roach' lurking.
I'll cover a bet that there are not tested backups in that shop
I had occasion to speak with that person again the other day. No backups, no thought to plan for such by that person's predecessor, and so by definition, no Disaster Recovery Plan

I am looking forward to getting a box of doughnuts -- from the DK would be nice. Please make sure two Apple Fritters are in there, as the other coffee vultures always grab the one I wanted. The Doughnut Kitchen is close to Staufs

win win habits

PMman was designed for long lived VM's (compare contra: ephemeral AWS or OpenShift type instances) , and so our communication needs vary from other cloud instance control interfaces.  This applies for the client tenant side views, but also for the sysadmin / devop side view

One of the convenience features of the PMman cloud product we run is the ease of communicating with 'tenants' on a given dom0.  When I came in this morning, I had this notice waiting for me:

From: mdadm monitoring
To: root@kvm-nNNN.pmman.com
Subject: DegradedArray event on /dev/md0:kvm-nNNN.pmman.com

This is an automatically generated mail message from mdadm
running on kvm-nNNN.pmman.com

A DegradedArray event had been detected on md device /dev/md0.

Faithfully yours, etc.

It is not the end of the world -- there are four members in that raid array; all mutable data is backed up nightly, and it is easy enough to fix with a hot drive swap.  BUT, we did a drive swap for a different raid member on that same unit within the last month.  I want to totally remove that unit from production, and put it into 'on the bench' mode, so I can see if there is some deeper hardware issue

The neat part from my point of view is that part of our design included a way for contacting the tenant VM owners on JUST that box alone.  It is as easy as clicking a couple buttons and typing

The following message will be sent to the to the list of users [on box: kvm-nNNN.pmman.com]. Here is the opportunity to fine tune the list.

From:     support@pmman.com
Subject:     Raid array member maintenance needed
Notice Level:     N
Message:     We have had some raid array member failures on the underlying dom0 of a machine you run at PMman. We replaced a drive hot three weeks ago on this same unit, and now have a new failure.

This may be a portent of a failing drive control subsystem, rather than the drive (although the previously removed drive tested bad and has been RMA'ed)

One new feature of PMMan of which you may not be aware is the 'no extra charge' weekly reboot and Level 0 backup. This is called a 'cyclic' backup with a 7 day repetition interval. To the extent you have NOT enabled this feature, I strongly suggest you test it and take advantage of this enhanced functionality of the interface

It is accessible off the 'Backups' button of the VM control web interface. Feedback is welcome of course. A nice side benefit from OUR point of view of offering this to customers is that it enables us to do invisible migrations from one dom0 to another as the backup and reboot are occurring, and so 'clear out' a dom0 host of running client instance VMs

Thanks

-- Russ herrold
User list: ... [elided] ...

Emails go out, and a history item gets dropped into each VM's logfile as well as in our admin side logs.  Optionally I can have the tool turn on an 'attention needed' flag in the end tenant's console, that will persist until they acknowledge it.  We already do that as to 'too long between updates' and' 'too long between reboots' and such

We can of course do invisible 'hot' migrations of machines around, but even safer is to encourage the good habit of encouraging tenant VM owners to  take (and we automatically test) Level zero backups

Win-win

I am not Harry Truman

I received a email from a customer, followed by a phone call, to the effect they had received huge number of email 'return' bounces to a general intake email address.  He and I have had this discussion before

I have written about email sender forgery (There is probably NOT an email account: godzilla@microsoft.com) and its fallout ("Customer: My cousin says that his email to me is not going through") before.  So let's take the time to think it through yet again

Takeaway: He wanted me to stop such pieces from cluttering their email box, but he is unwilling to have 'heavy' spam filtering

As a personal matter, and also wearing my sysadmin hat, I would like to stop seeing this cruft as well

But as a technical matter, it seems that it cannot easily be done without constant 'tuning' of rejection rules or some other rather serious matching of 'Message-ID' of pieces sent against return pieces offered.  An attempt to do so through filtering tools with no prior knowledge of Message-ID's sent, is to always 'play defense' against the spammers, without an ability ever score a 'win'.  The effort to match Message-Id's in offered return pieces is perhaps more promising

But, so far, no-one has been sufficiently vexed by it in the FOSS community to publish such a tool and to commit do doing do the ongoing 'tuning' of message parsers needed.  Perhaps we can design around it with existing tools, and amending our outgoing pieces by adding a certification that a given candidate email is truly from us

As a design matter, building a milter, writing some procmail rules, and parsing sendmail logs, probably into a database backend, as my first thought as to how I would approach the matter.  The database constraint is troubling, though.  I have other work that I need to attend to first, but I went through the thought process.  I memorialize that process in part in case someone is interested.  Even more, I will provide webspace, mailing list support, and a VCS gratis, if someone 'feels the itch'.  It would be useful to have, but is not urgent to attain -- Seven Habits Quadrant 2 or 4 stuff.  Absent such a volunteer effort or a paying customer, for me, Quadrant 4

Or, version two, a trusted cohort of outbound mailservers could build a MAC MIME Multipart attachment for each outbound message, and also a second MIME attachment that is cryptographically validated 'clearsign' of that MAC part.  Possibly bundle this up into a Multipart Related set of structured attachments.  Add these two new MIME attachments to all messages on every outbound piece.  The first part -- the MAC part -- would be based a hash of the message body, plus a timestamp of seconds since Epoch or such, and other optional entropy, to avoid forgery and replay attacks

Later then, when a putative return is offered, only accept for further processing those returns that had a validating pair of MIME attachments, produced based on a re-hash the message body in chief, and that MAC section's timestamp; and  that had previously clearsigned by it. Discard stale stuff, and non-validating content. This gets rid of the need for the database and simplifies the procmail rules.  A well-formed candidate return piece can carry around all that is needed to known to decide if one will pass a mail return message along to human eyes

Not free, as it will burn up compute cycles on every send, and a few more at return time, but also complete and under controllable locally so resistant to spammers.  Avoids the database requirement, so it can scale out. Most of the needed tools already exist as FOSS.  hmmm


The protocols governing what constitutes: email permit a sender to enter whatever 'return address, and 'sender address' they wish on a piece of email.  It is trivial to find a 'open' relay to accept email to send to any third party.  Consider the analogy:

• All the while being careful to not leave a fingerprint or other biometric, I use cash to purchase a post card at the corner store, along with a stamp
• I address it to someone of tender sensibilities, and assert that I noticed that their car was parked outside the local 'adult entertainment' establishment
• I sign it: Harry S Truman
• I enter a 'return address' of:

  Harry S Truman
  President Emeritus
  1600 Pennsylvania Avenue
  Washington DC  20500

I mail it

The recipient is outraged to find such a libelous assertion, visible for their letter carrier to see, and demands that the person who did so be identified, and stopped.  Also, for good measure, they want the Postal Service Inspectors to get on the matter to prevent such heartbreaking assertions to never happen again

About all the Postal Service will offer to do in the usual case is to return the piece to its nominal sender. And he no longer receives mail at that address

(I note parenthetically that the Postal Service DOES seem to scan images of ALL paper mail passing through their system)


Stopping spam (here: bounce backsplatter and 'joe jobs') is just not going to turn out to have a durable, easy, and comprehensive solution, without re-thinking what we send looks like.  Spammers and legitimate receivers are in a 'arms race' and today's fix will rot if senders can re-engineer around the fixes.  If this state of affairs distresses a person greatly and until I can get that MIME solution going to test my hypothesis: stop reading email; hire a full time, 24x7 secretary to pre-read all email and toss the junk.; turn up the filtering and accept the false positives; grow a thick skin

Or, of course, start coding and beat me to it

Phone call: 'I've got this sick machine ...'

me:  well, why it is sick?

them:  yum complains about a missing signing key

me: so install the key; it is down in /etc/pki/rpm-gpg/, and rpm --import ... will do the trick

them: that directory is not there

me: who set up the machine?

them:  well, I was handed it, and ...

me: so, take a level zero backup and then clean up the machine before trying to work on it, or deploy a new one

them: well, I can't

I just got off that call from a friend in a new employment situation

The technical fix was outlined by me long ago, and I sent an email with the link along to the person calling

BUT: Fixing the mindset inside the caller's head: do not try to work in a undefined (here: broken) environment is harder

But the caller has a problem in their work-flow process; a fix has to be done; sooner is probably better than later; a broken machine in production is 'technical debt', pure and simple.  Fundamental expectations are not met; binary partition will not work well to isolate problems, as more than one piece is probably broken.  It will break again, and a perception may well form that the caller may be the problem, rather than the broken environment they were handed

Be sure to make a note to yourself to also address the broken process that permitted that machine to escape into production.  Dollars to doughnuts, there is 'more than one roach' lurking.  I'll cover a bet that there are not tested backups in that shop

Another pet died across the holidays

I wrote before about un-maintained and orphaned WordPress sites being exploited.  That same frantic user from two months ago, called again.  The TL;DR summary is:

• cPanel administration with multiple accounts in a single host without protections 
• OS Updates not being run
• WordPress updates not being run
• Random add-on's being used without an awareness of security issues
• No SELinux (disabled)

An exploit un-gzip-ping a hostile payload from cache was used, and the machine taken over

The absence of good sysadmin skills, well packaged content, and updates 'for the loss' ...

disable IPv6 DNS results

We had an end user appear in the main #centos IRC channel the other day with a IPv6 problem.  That person had leased a VPS somewhere, and their provider had included and enabled IPv6, at least partially.  Something was wrong in the network fabric, so that while some IPv6 services worked, others did not; DNS results returned with AAAA record results; but then the VM hoster was not transiting port 80 TCP traffic.  Very curious, and frustrating to the end user who just wanted yum to work so they could install updates and packages on their instance

The culprit is the grafting in of IPv6 readiness in man 2 getaddrinfo.  This is the way of the future, so there is no fighting it on a long term basis, but tactically having a means to be IPv4 only is appealing for people just wanting to work in the older address space.  The TL; DR uptake of that man page is that in a properly functioning system, name resolution answers under IPv6 are preferred, and only if not available, does one fall back to the older IPv4.  But this places a premium on IPv6 actually working when present.  We've shipped a full native IPv6 setup for customers at PMMan for a couple of years ago, but I assure you that we had some head-scratching as we rolled it out, and found customers using tunnels from HE or SixXs were also leaking advertisements to other VM's.  We added rules to filter out the false traffic after a bit of tracing with TCPDUMP

I have blogged about it before when IPv6 link-local address resolution (the ^FE family) was confusing distcc under Debian a couple of years ago.  There are links in the CentOS wiki for approaches on disabling IPv6 traffic, which vary between C5 and C6

That last mentioned article has an outlink to a bugzilla ticket that offers food for thought.  It mentions in passing that one can direct a nameserver to NOT deliver IPv6 results with a fairly simple tweak

Another option is to add this to /etc/sysconfig/named:

OPTIONS="-4"

... so, ... it should be possible to set up a local cacheing nameserver on the localhost, configured to NOT return IVv6 content, and so workaround the issue.  This smells sort of 'hackish', but it would have the benefit of being a single method that should work in the general case and not be tied to any particular kernel version, or other variable

Feeding the pet

We had a frantic call from a sometimes customer today.  Their self-administered WordPress-based website had a Trojan in it, and it was saturating their website traffic allocation.  "THE SITE WAS DOWN!!"  They had signed up at a CPanel mediated, shared hosting firm, and a plug-in they had installed turned out to contain a well-known trojan

We spent a couple of hours looking into it.  And then a couple hours looking into the WordPress security notification system.  Perhaps, I should say: non-notification system as to getting subscribed to a formal notification mailing list from the WordPress folks, proper

The WordPress model seems to be: treat your WordPress site as though it is a pet that needs daily feeding.  And to be 'put down' when you lose interest in it, move on, or forget about it -- Oops.  Log in daily as an administrator, and look for a notification

that you need to apply the 'latest and greatest' update.   Run the update process manually whenever it appears.  Oh yeah, did you remember to take a backup FIRST, and test that you can roll back to it if the 'update' breaks anything? Oops

This of course RULES OUT using a packaged approach to managing such sites, as the lag for stabilizing a new RPM package, accounting for potential database changes, and the like 'take too long'. Just unroll a tarball, and trust that it will not break any local customizations

I see fourteen open tabs in my browser panel still open, related to trying to track down a central and formal notification feed that I (or any person seeking to get 'push' notification) might subscribe containing only 'Security' notifications.  Weeding through the tabs, ...

• The 'Famous 5-Minute Install' for WordPress -- Nope, no useful outlink for hardening, nor to subscribe to notifications, beyond a pointer to a third-party Ubuntu appliance with an 'automatic security updates'.  That appliance's page has pointers to a tool to enable taking database backups, adding PHPMyAdmin, and Webmin.  Not good choices for a person caring about security
• Perhaps FAQ items tagged with: Security -- Nope, clearly incomplete, as for example a Google search turns up this third-party alert for version 3.3.2,  but the Release Notice does not get titled with: Security
• This bug (#10253) lingered for three years with a Security tag in their Trac issue tracker as to the current release series (3.4), and was amended ten days ago; But the latest release (for 3.4.2) was twenty days ago when this is written.  Should an update have been release?  Who Knows?
• Perhaps their FAQ Security -- Nope, no push notification link suggested there, but lots of clutter as to copyright infringement notification handling, and miscellaneous topics
• Perhaps watch the Releases News in an RSS reader - Oops, no sub-tag feed offered, and there has not been an "Important" Security release since December 2010, if one used that approach
• Run a Google search daily, and look for third-party commendary - Nope, although nuggets may be found, for it is not viable as: Not Authoritative, irregular and partial as to updates, and wading through search engine hit, or RSS feed clutter will kill your productivity

Clearly, one MUST configure the webserver to NOT permit off-site access to the credentials and configuration file: wp-config.php but I'll be darned if I can see instructions on the WordPress site, showing a novice administrator how to do this. In a shared hosting environment without 'root' level control, it is probably not even doable.  There is not hint of this rather elementary precaution on the official write-up concerning editting the file

A quick Google search for: turns up lots of vulnerable candidate installations, and a handy, dandy code fragment for parsing information out of potential victims so found, to automate take-overs. No criticism of the author of that code publishing his work; a knife can heal (as a scalpel), prepare dinner, or injure, depending on the intent of its holder

I see an official  recovery outline  suggestion, anyway

Worth repeating; Trust and Open Source

I first encountered Mark Shuttlesworth in person at an Ottawa Linux Symposium a few years ago, and passed along a reply from Dag, responding to some controversial comment Shuttlesworth had made at the time.  I choose not to use Ubuntu or Debian as my primary X desktop, but that said, there are 6 machines running one of those two distributions powered on in my office at the moment, so I am not a stranger there, either
He was being 'up front' about the fact that Amazon search results are being trialled for an upcoming Ubuntu version
He points out, and it bears repeating, the following:

[Question: ] Why are you telling Amazon what I am searching for?
[Answer: ] We are not telling Amazon what you are searching for. Your anonymity is preserved because we handle the query on your behalf. Don’t trust us? Erm, we have root. You do trust us with your data already. You trust us not to screw up on your machine with every update. You trust Debian, and you trust a large swathe of the open source community. And most importantly, you trust us to address it when, being human, we err.

The boldface are important, but I carry the context as well here.  When you use any computer operating system, you in the role of: user are  implicitly placing trust in the decisions and the commitment of those who put it together to 'do the right thing', or to make it right when things go awry
Do you trust your vendors?  Your actions may be pointing out a dissonance, if you said: no

More chickens, returning home to roost

I referred to the roosting chickens theme earlier today as to Oracle, and did not expect to be back to that topic for a while.  But the news does not wait
One of my long time concerns as a Cassandra, is that FOSS geeks firmly believe that  ignorance and  unworldly innocence, and a native sense of protection that 'just conduct' while participating in FOSS matters, will somehow insulate the 'good folks' from the world
To my experience, it does not work that way.  One needs to clearly disassociate from risky conduct.  A person needs to read EULAs and look for traps, like indemnification clauses
I see in the overnight news that RackSpace and Github face some hot water.  Drilling down, as to how this is likely to play out, I am aware that Github has an EULA, that provides in part:

Section F 3
You shall defend GitHub against any claim, demand, suit or proceeding made or brought against GitHub by a third party alleging that Your Content, or Your use of the Service in violation of this Agreement, infringes or misappropriates the intellectual property rights of a third party or violates applicable law, and shall indemnify GitHub for any damages finally awarded against, and for reasonable attorney’s fees incurred by, GitHub in connection with any such claim, demand, suit or proceeding; provided, that GitHub (a) promptly gives You written notice of the claim, demand, suit or proceeding; (b) gives You sole control of the defense and settlement of the claim, demand, suit or proceeding (provided that You may not settle any claim, demand, suit or proceeding unless the settlement unconditionally releases GitHub of all liability); and (c) provides to You all reasonable assistance, at Your expense.

Guess who is going to be added as a Defendant in that lawsuit next; guess who's pocketbook will ultimately be looked to, to pay for Github 'lawyering up.'  Github and RackSpace have every incentive in the world to 'point out' the poster of claimed infringing content to the Plaintiff and its lawyers, and so demonstrate 'good faith', a lack of intent to infringe, and their desire to mitigate any asserted damages

On my RADAR: Java to iOS Objective C

This crossed my RSS newsreader feed :

J2ObjC is an open-source command-line tool from Google that translates Java code to Objective-C for the iOS (iPhone/iPad) platform. This tool enables Java code to be part of an iOS application's build, as no editing of the generated files is necessary. The goal is to write an app's non-UI code (such as data access, or application logic) in Java, which is then shared by web apps (using GWT), Android apps, and iOS app

J2ObjC supports most Java language and runtime features required by client-side application developers, including exceptions, inner and anonymous classes, generic types, threads and reflection. JUnit test translation and execution is also supported

The project homepage is here, and along with Google's Web Toolkit, seems to be under an acceptable FOSS license
It does not strive to be a full replacement for all things Java, and as it notes:

when new projects first start working with it, they usually find new bugs to be fixed. Apparently every Java developer has a slightly different way of using Java, and the tool hasn't translated all possible paths yet

Long time readers of this blog will recall that I have been a critic of Java in FOSS projects particularly in the LSB standards context, and as I predicted, the Oracle anti-FOSS Java lawsuit [against Google it turns out, because they are a juicy target] was filed a couple years later. People working for nasty corporate masters earn their futures

This is not to say that translation layers are without warts and flaws, but at least this provides a partial roadway away from a Java dependency into a code generator that may emit freely licensable code.  I'll be tracking this with interest.  I have a sample conversion in mind ...

about this week's GoDaddy outage

It is not clear if a cabal of Anonymous hackers, or simple network administration issues, caused the GoDaddy outage of Monday past. I guess it does not really matter

What really would have hurt is if the root domain server constellation had been compromised, to well and truly take down the internet. A Domain Registrar sends along updates to those root servers periodically, and GoDaddy's outage, from the extent of our involvement with them, simply impaired our ability to renew domains, and set new nameservers (NS records). As we had no urgent renewals pending, that is to say, not at all

We do not rely on GoDaddy for DNS services, and really, never have relied on them for production purposes. For PMman and for our ISP and COLO services, we run three geographically diverse nameservers for most of our purposes. We also run a few others for customers' needs (PTR records for a couple of datacenters we are in, testing, demonstration units)

The true 'masters' of our externally visible DNS servers are simply not accessible from the public internet. We push out updates to our public nameservers by cryptographically protected rdnc transactions. Those transactions are logged, and the information causing a given RDNC transaction are created by queries into a local database with a custom written LAMP control interface based on the FOSS tools that are in a stock CentOS install. Compared to manually editing zone files, checking variants in and out of a version control system, and so forth, this more readily provides us with scalability, traceability and auditability. Why, I caught a piece of lint in a zone file just last week, reading the overnight error report emails

We also retrieve the state of the generated zone files at the client public nameservers, and check them for consistency and coherency, essentially after each update, to prevent errors from propagating. ACLs, transaction logging and other checks provide more tracability, and we closed the mouse hole that that 'lint' crept in through in short order

As a result of the GoDaddy outage, a couple of our 'alumni' tech support folks who have moved on in their careers to other employment, gave us a call Tuesday, because they remembered how paranoid I am on making sure DNS is available. I appreciate the calls, and we've some new customers as a result

People have strong opinions about GoDaddy, sometimes for reasons of political correctness; I like them, by and large, because they provide a workmanlike product for a price that is hard to beat. They sure beat the heck out of the old Network Solutions rates. I have something like 500 domains that I administer and renew and most are there, although some are at other registrars for both historical and other reasons

And while Danica Patrick is not my cup of tea, she is not hard on the eyes, either

RPM and YUM trick: synchronize packages installed on two hosts

The question was asked in IRC today:

hello folks, is there any way to install packages from a list written by yum list installed? I've two CentOS 6.3 hosts and I like to get them with the same packages installed (also versions)

Here is a quick (and accurate) answer:

1. Take and test backups to taste
2. Run:
   

   rpm -qa --qf '%{name} \n' | sort > MANIFEST

   Note: that is a backslash n -- the html markup makes it hard to see the distinction
3. Then copy that MANIFEST to the second unit and run:
   

   yum -y install `cat MANIFEST`

   Note: and here, backtick around the cat to get a sub-shell
4. Finish by running:
   

   yum -y update

   on each unit

For extra credit, re-run the MANIFEST creator on each unit, and use diff to find any variances

Thinking about Secure Boot

The Open Source community has a corner turn coming, in the upcoming roll out of UEFI enabled 'Secure Boot' hardware. And the 'Build Your Own' hobby computer builders are in for a rude shock as well, as they are going to have to start paying per each re-install of a commercial operating system, because UEFI provides a mechanism to reliably control (read: prevent free unlimited) reinstalls of Microsoft operating systems, as well as end user Application software

An integrated computer hardware or motherboard Manufacturer (Dell, HP, SuperMicro, etc) will have no choice but to conform to the new 'latest is greatest' approach that Windows 8 will bring to the market if it wants to keep selling new hardware. ... which means being able to run whatever Microsoft's latest is. And I don't want to be coy about this: it is in their economic interest as well to gain a way to limit the life of hardware, as it builds a ready market for re-selling to customers with older hardware

I simplify here, but once in full release, Windows 8, (and going forward, I think I can safely predict that high ticket proprietary Applications), will ONLY install on a system that has a secure chain of signed binaries and RELATED counter-signed Variables. That secure chain will be done by verifying a chain of checksums ('hashes') duly counter-signed by a Key-Exchange Key (KEK) public/private key pair, on back to a database itself counter-signed and verifiable by a Platform Key (PK). The PK may, but is not required to be able to, be wiped by an end consumer. For the reasons below, I think there WILL be an ability to reset the NVRAM on the general case

But perhaps not -- in the past applying some firmware updates required cracking open a case and moving a jumper. So I feel pretty comfortable that a mechanical 'jumper' option will appear as motherboards are designed, to prevent purely electronic (via executable code actions) key-wipes; it just makes sense in a corporate IT environment to prevent end employees from tampering with machines. And in some use cases, the jumper disappears altogether

Fresh from the fabrication plant, no PK exists in a UEFI 'bios' and it is said to be in a 'Setup' state. In the usual case, a PK will initially be generated and 'injected' into the NVRAM keystore at the time of manufacture (those cases where the end customer expects to receive a 'ready to run' computer). That is, while there may be provision for generating and injecting a PK keypair (either at initial receipt, or on a unit needing a 'wipe and reinstall'), in the general case the PK keypair injection will have already been done by the Manufacturer. And the Manufacturer is NOT going to readily supply the private side of that key with each chassis it ships ... the support load is too great, and an end user will be told: they are out of luck as it is not available

Those Variables I mentioned will almost certainly contain one or more 'unique per unit' UUID type hash, and as I say, these variables will be countersigned and held in the motherboard's non-volatile configuration ram. We have seen such approaches before -- old SGI hardware used NVRAM to hold a MAC address, that may have been signed as well, and SGI install disks looked for values in a range assigned to SGI

Part of an OS install process will be generating and authorizing a unique UUID, signed by a controlled KEK keypair chain, that chain running unbroken through a specific PK. The Manufacturers will generate a keypair, good for a given number of units and report sales to their Licensor. No more pesky COA stickers. If a KEK pair is compromised, they report it as such to their Licensor (here, Microsoft), and the public side is added to a blacklist database update. Computers relying on that trust chain simply stop booting, or stop running some node-locked Application, and instruct their owners to call technical support

Eventually Tech support says that this unit is out of warranty and end of life, and the Manufacturer or its competitor gets a new sale. Life is good for the Licensor and the Manufacturers as a group

As I read the UEFI specification, however, nothing requires being able to 'read back' those hashes, so that one could back them up, or have them at hand to be able to re-inject them. Indeed, the usual model is to NOT be able to read back enough information from a secure key store to be ABLE to back up and restore it. To do so would destroy the metering and regulation value of the KEK signed hashes and Variables

I assume the Open Source community will solve how to transition most hardware back into Setup state; I see that some of the kernel hackers have already solved wiping and then injecting new keys into the PK. I know that UEFI supports, but does not mandate use of x.509 certificates, so it is just a matter of time until StartSSL or some other Open Source friendly Certificate Authority documents and issues signing keys for both formal distributions, and end users compiling local kernel modules needing signing

Two futures are around the corner, here: Either way, BYO hobby users of commercial operating systems and applications lose a lot of usability to experiment. When it is too hard to 'play' around, they will need a new space to experiment and customize in. It is Open Source for the win as such a venue

And for the Open Source folks: 1. If the hardware permits an ability to re-set a UEFI 'bios' to Setup state and to re-inject an initial PK, Open Source wins, as it picks up new users when a person is presented with an estimate for re-purchasing all that they thought they had bought, but that need new activation keys, and asks: what else can I do? 2. If the hardware does not, this is readily apparent, and the manufacturer is shunned and their hardware avoided by folks in the know

I know it is the present fashion the Open Source community to damn UEFI and all, but the outlook is not all darkness. Just a turn around the corner to a new future

---

Updated: The interface at blogger.com (wasn't this formerly blogspot.com) has changed, and I missed that the final copy would 'lay our' so poorly; it is also eating random markup that formerly worked ... fixed

Right, Left, down the middle

A couple weeks ago, a 'Derecho' blew through Columbus, on its way to the metro DC area. Amazon had some failures that cascaded through to people who did not have site redundancy. People know that the East Coast was hit hard, but as we are out in 'fly-over' country they did not perhaps realize that we had several hundred thousand people around here without electricity for a couple weeks as well

I've mentioned before that the primary datacenter that we run our PMman product out of is at the Tier IV level -- multiply redundant cooling, power grid, power backup, fiber entrances, carriers. The owner, a friend, is just a fiend that he does not HAVE outages

Me, too. In our after-event review, I see that one of our secondary sites here in town fell back to its generators, but the rest were all fine. But all sites we use are well covered, all fiber, all multi-homed. Planning for failure was in our deployment planning checklist; we pay for (and we charge for) that coverage; and I consider it worth it

A national footprint customer based in Canada agrees. And their lead technical person reports that our connectivity is haster than their datacenter eighty miles from their home ofice. Not surprising, as oAltantaur main DC is on a 'main line' fiber route between Chicago, NYC, DC, and Atlanta -- financial markets and federal government presences can help, that way

If the availability of your online presence matters to you, feel free to ask for a quote

Yeah, it's a Monday ...

Last week, I made a trip up to the local computer store, the mothership of MicroCenter, and finally broke down and bought a USB/VGA four port KVM switch for my lead worksstation .... not DVI or HDMI on the video, but still VGA. The PS/2 mouse and keyboards were scheduled to be phased out, and a move to USB devices slated for this week

Well, the components to be affected .. the old KVM, the panel monitor, and so on must have held a pow-wow across the weekend, because when I came in this morning the panel monitor's backlight (a Westinghouse L2210NW, 1680x1050, datecode of April 2008) seems to be completely dead. This is of course the absolutely MOST inconvenient part of the display chain to die, because I need to run custom 'modeline' detail under X to squeeze the maximum resolution and sharpness out of the display. I had also purchased the three year 'no questions' replacement warranty on that four year old unit, so no help there

I grabbed a 'retired to the front bench' NEC AcuSync LCD223wxm, also nominally 1680x1050, off one of the benches and have been fiddling with the modeline settings to have a backup to limp through the day, but the horizontal height is wrong, the pixels puddgy, and the video muddled

No doubt I _could_ get it to stand up and dance, but the NEC has a datecode from early 2007 so that is a suckers game

Yeah, its a Monday

LOPSA at the PMman DC

I went up to a meeting at our North datacenter for PMMan, where local group of system admins held a meeting, starting up a local LOPSA chapter. Food and soda were provided by the DC operator, along with salad ... since when did sysadmins starting more healthy food, rather than a diet of high sugar, high caffeine junk food?

The presentation slide deck was fine, and the presenter (a 'long timer' at a local credit-card clearance operation) ran through his bullet list of what to look for in the 'build vs. buy (lease space)' decision, and then a number of siting concerns.

Now I am familiar with his firm's site from prior visits, and it is adjacent to a major highway with regular closures for accidents; adjacent to a major rail yard where chemical spills have caused evacuations; and sole serviced into the power company grid

Our North site was chosen after a survey of all offerings within a radius we were willing to drive to for 'end of the world' 'hands on' intervention; is jacked into two independent power grids along with the on-site generators, is a premier demonstration location for the former Liebert (now owned by Emerson) power and site conditioning

I happen do drink coffee daily at Staufs with Liebert's representative here in town. My evaluation team suggested the location as a finalist, and when I checked, it turned out that I already knew the owner / developer from long, long ago telephony days, and when I have time, I'll go up and 'shoot the bull' with him on Saturday mornings at the DC

We have had a grand total of ZERO power related outages, and only one network connectivity issue in the last three years, that lasting less than 15 minutes, and that, due to human error in not handling a BGP fail-across migration properly [the cut-over protocol was changes, as I noticed the drop from my monitoring and called the owner's cell phone at once ;) ]. Well suited to our 'enterprise' customers

It is 'carrier neutral' and hugely connected -- multiple entrances of up to 88 x 100 G fiber spread across six or seven principal carriers. Native IPv6 to all drops we run through multiple carriers, along with the IPv4. I helped with the IPv6 design and cut-over some 18 months ago, and it has been seamless. The facility, and our services, just do not have outages except that human error causes

It is not the cheapest in town ... but it is fairly priced for the value we have received

I had not sat down and reflected on how satisfied I was with that shift of our center of operations to the DC, but as I think about it, I am well pleased

It is always time to get back into the swing of a good habit

When I crushed my ankle, I really could not think straight for a couple months with the painkillers they had me on, and so I lost the habit

It's time to start blogging regularly again

Happy IPv6 test day, part 2

In my first post in this small series, I closed without addressing matters of securing an IPv6 connection, and in matters of debugging where a connection failure is occurring. Thinking about it, the diagnostic post needs to come first, because tightening down a conneciton can cause hard to diagnose symptoms. So, on to diagnosis ...

We examined the interface results last time. Looking at just the routing related to ipv6 is straightfowrard as well:

Some familiar tools:

/sbin/ifconfig eth0
/sbin/ifconfig sit1
/sbin/route -A inet6

So using those tools:

[herrold@hostname ~]$ /sbin/ifconfig sit1
sit1      Link encap:IPv6-in-IPv4
          inet6 addr: 2604:aa:bb:cc::2/64 Scope:Global
          inet6 addr: fe80::4cf2:1c/128 Scope:Link
          UP POINTOPOINT RUNNING NOARP  MTU:1480  Metric:1
          RX packets:1691 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1693 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:177195 (173.0 KiB)  TX bytes:210468 (205.5 KiB)

[herrold@hostname ~]$ /sbin/route -n -A inet6 | grep 2604
2604:aa:bb:cc::/64      ::          U     256    1658       0 sit1
2604:aa:bb:cc::/128     ::          U       0       0       2 lo
2604:aa:bb:cc::2/128    ::          U       0    1691       1 lo
[herrold@mailhub ~]$   

That is a pretty ordinary routing table for a non-gateway endpoint. Off-box traffic (to the '/64' netmask) is handed to the sit interface, and local traffic (to the '/128') retained on the local lo interface

We use a unfamiliar tool: ping6 -- The common '127.0.0.1' localhost has a new form under ipv6:

ping6 ::1 -c 2

and testing

[root@hostname ~]# ping6 ::1 -c 2
PING ::1(::1) 56 data bytes
64 bytes from ::1: icmp_seq=0 ttl=64 time=0.157 ms
64 bytes from ::1: icmp_seq=1 ttl=64 time=0.132 ms

--- ::1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.132/0.144/0.157/0.017 ms, pipe 2
[root@ostname ~]# 

And we can ping by IP on the 'Global' link, both on the local end, and remotely with differing transit times for the packets:

Looking at the network connections, we examine the tunelling interface:

[root@hostname ~]# /sbin/ifconfig sit1
sit1      Link encap:IPv6-in-IPv4
          inet6 addr: 2604:aa:bb:cc::2/64 Scope:Global
          inet6 addr: fe80::4cf2:1c/128 Scope:Link
          UP POINTOPOINT RUNNING NOARP  MTU:1480  Metric:1
          RX packets:1714 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1731 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:179587 (175.3 KiB)  TX bytes:215180 (210.1 KiB)

[root@hostname ~]# ping6 2604:aa:bb:cc::2 -c 2
PING 2604:aa:bb:cc::2(2604:aa:bb:cc::2) 56 data bytes
64 bytes from 2604:aa:bb:cc::2: icmp_seq=0 ttl=64 time=0.135 ms
64 bytes from 2604:aa:bb:cc::2: icmp_seq=1 ttl=64 time=0.137 ms

--- 2604:aa:bb:cc::2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.135/0.136/0.137/0.001 ms, pipe 2
[root@hostname ~]# ping6 2604:aa:bb:cc::1 -c 2
PING 2604:aa:bb:cc::1(2604:aa:bb:cc::1) 56 data bytes
64 bytes from 2604:aa:bb:cc::1: icmp_seq=0 ttl=64 time=55.1 ms
64 bytes from 2604:aa:bb:cc::1: icmp_seq=1 ttl=64 time=53.7 ms

--- 2604:8800:100:bb::1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 53.781/54.473/55.165/0.692 ms, pipe 2
[root@hostname ~]# 

Turning to DNS and name resoluton, it is quite familiar. One does not need an IPv6 link to query nameservers and receive back results, as they will answer questions _about_ ipv6 hostnames ('AAAA' records) to any authorized inquirant. Try these:

dig +trace www.ipv6.sixxs.net
dig www.kame.net aaaa 

I get answers like this:

[herrold@centos-5 ~]$ dig www.kame.net aaaa

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-16.P1.el5 <<>> www.kame.net aaaa
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45595
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 2, ADDITIONAL: 3

;; QUESTION SECTION:
;www.kame.net.                  IN      AAAA

;; ANSWER SECTION:
www.kame.net.           85164   IN      CNAME   orange.kame.net.
orange.kame.net.        85164   IN      AAAA    2001:200:dff:fff1:216:3eff:feb1:44d7

;; AUTHORITY SECTION:
kame.net.               85164   IN      NS      mango.itojun.org.
kame.net.               85164   IN      NS      orange.kame.net.

;; ADDITIONAL SECTION:
mango.itojun.org.       2364    IN      A       210.155.141.200
mango.itojun.org.       2364    IN      AAAA    2001:2f0:0:8800:206:5bff:fe8d:940
mango.itojun.org.       2364    IN      AAAA    2001:2f0:0:8800::1:1

;; Query time: 1 msec
;; SERVER: 10.16.1.112#53(10.16.1.112)
;; WHEN: Thu Jun  9 17:17:20 2011
;; MSG SIZE  rcvd: 195

[herrold@centos-5 ~]$

which is certainbly a mess to read -- let's trim out the interesting parts:

[herrold@centos-5 ~]$ dig www.kame.net aaaa
;; ANSWER SECTION:
www.kame.net.           85164   IN      CNAME   orange.kame.net.
orange.kame.net.        85164   IN      AAAA    2001:200:dff:fff1:216:3eff:feb1:44d7 

Which is the familiar information: a CNAME record is pointed in fact at a AAAA record at a ipv6 -type IP. We can ping (ping6) it by IP:

[root@hostname ~]# ping6 2001:200:dff:fff1:216:3eff:feb1:44d7 -c 2
PING 2001:200:dff:fff1:216:3eff:feb1:44d7(2001:200:dff:fff1:216:3eff:feb1:44d7) 56 data bytes
64 bytes from 2001:200:dff:fff1:216:3eff:feb1:44d7: icmp_seq=0 ttl=52 time=246 ms
64 bytes from 2001:200:dff:fff1:216:3eff:feb1:44d7: icmp_seq=1 ttl=52 time=256 ms

--- 2001:200:dff:fff1:216:3eff:feb1:44d7 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 246.833/251.705/256.577/4.872 ms, pipe 2
[root@hostname ~]#

or ping it by name, as DNS is working:

<[root@hostname ~]# ping6 www.kame.net -c 2
PING www.kame.net(2001:200:dff:fff1:216:3eff:feb1:44d7) 56 data bytes
64 bytes from 2001:200:dff:fff1:216:3eff:feb1:44d7: icmp_seq=0 ttl=52 time=227 ms
64 bytes from 2001:200:dff:fff1:216:3eff:feb1:44d7: icmp_seq=1 ttl=52 time=244 ms

--- www.kame.net ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1008ms
rtt min/avg/max/mdev = 227.291/235.678/244.066/8.401 ms, pipe 2
[root@hostname ~]# 

Some other examples to try as 'beacons' might include:

ping6 2604:8800:100:9a::1 -c 2
ping6 2001:200:0:8002:203:47ff:fea5:3085 -c 2
ping6 ftp.ipv6.uni-muenster.de -c 2
ping6 -I eth0 ipv6.google.com -c 2 

So the familiar diagnostic methods of examining interfaces, checking routing, testing connectivity by IP, and connectvity after resolution by name are all present