disable IPv6 DNS results

We had an end user appear in the main #centos IRC channel the other day with a IPv6 problem.  That person had leased a VPS somewhere, and their provider had included and enabled IPv6, at least partially.  Something was wrong in the network fabric, so that while some IPv6 services worked, others did not; DNS results returned with AAAA record results; but then the VM hoster was not transiting port 80 TCP traffic.  Very curious, and frustrating to the end user who just wanted yum to work so they could install updates and packages on their instance

The culprit is the grafting in of IPv6 readiness in man 2 getaddrinfo.  This is the way of the future, so there is no fighting it on a long term basis, but tactically having a means to be IPv4 only is appealing for people just wanting to work in the older address space.  The TL; DR uptake of that man page is that in a properly functioning system, name resolution answers under IPv6 are preferred, and only if not available, does one fall back to the older IPv4.  But this places a premium on IPv6 actually working when present.  We've shipped a full native IPv6 setup for customers at PMMan for a couple of years ago, but I assure you that we had some head-scratching as we rolled it out, and found customers using tunnels from HE or SixXs were also leaking advertisements to other VM's.  We added rules to filter out the false traffic after a bit of tracing with TCPDUMP

I have blogged about it before when IPv6 link-local address resolution (the ^FE family) was confusing distcc under Debian a couple of years ago.  There are links in the CentOS wiki for approaches on disabling IPv6 traffic, which vary between C5 and C6

That last mentioned article has an outlink to a bugzilla ticket that offers food for thought.  It mentions in passing that one can direct a nameserver to NOT deliver IPv6 results with a fairly simple tweak

Another option is to add this to /etc/sysconfig/named:

OPTIONS="-4"

... so, ... it should be possible to set up a local cacheing nameserver on the localhost, configured to NOT return IVv6 content, and so workaround the issue.  This smells sort of 'hackish', but it would have the benefit of being a single method that should work in the general case and not be tied to any particular kernel version, or other variable

Feeding the pet

We had a frantic call from a sometimes customer today.  Their self-administered WordPress-based website had a Trojan in it, and it was saturating their website traffic allocation.  "THE SITE WAS DOWN!!"  They had signed up at a CPanel mediated, shared hosting firm, and a plug-in they had installed turned out to contain a well-known trojan

We spent a couple of hours looking into it.  And then a couple hours looking into the WordPress security notification system.  Perhaps, I should say: non-notification system as to getting subscribed to a formal notification mailing list from the WordPress folks, proper

The WordPress model seems to be: treat your WordPress site as though it is a pet that needs daily feeding.  And to be 'put down' when you lose interest in it, move on, or forget about it -- Oops.  Log in daily as an administrator, and look for a notification

that you need to apply the 'latest and greatest' update.   Run the update process manually whenever it appears.  Oh yeah, did you remember to take a backup FIRST, and test that you can roll back to it if the 'update' breaks anything? Oops

This of course RULES OUT using a packaged approach to managing such sites, as the lag for stabilizing a new RPM package, accounting for potential database changes, and the like 'take too long'. Just unroll a tarball, and trust that it will not break any local customizations

I see fourteen open tabs in my browser panel still open, related to trying to track down a central and formal notification feed that I (or any person seeking to get 'push' notification) might subscribe containing only 'Security' notifications.  Weeding through the tabs, ...

• The 'Famous 5-Minute Install' for WordPress -- Nope, no useful outlink for hardening, nor to subscribe to notifications, beyond a pointer to a third-party Ubuntu appliance with an 'automatic security updates'.  That appliance's page has pointers to a tool to enable taking database backups, adding PHPMyAdmin, and Webmin.  Not good choices for a person caring about security
• Perhaps FAQ items tagged with: Security -- Nope, clearly incomplete, as for example a Google search turns up this third-party alert for version 3.3.2,  but the Release Notice does not get titled with: Security
• This bug (#10253) lingered for three years with a Security tag in their Trac issue tracker as to the current release series (3.4), and was amended ten days ago; But the latest release (for 3.4.2) was twenty days ago when this is written.  Should an update have been release?  Who Knows?
• Perhaps their FAQ Security -- Nope, no push notification link suggested there, but lots of clutter as to copyright infringement notification handling, and miscellaneous topics
• Perhaps watch the Releases News in an RSS reader - Oops, no sub-tag feed offered, and there has not been an "Important" Security release since December 2010, if one used that approach
• Run a Google search daily, and look for third-party commendary - Nope, although nuggets may be found, for it is not viable as: Not Authoritative, irregular and partial as to updates, and wading through search engine hit, or RSS feed clutter will kill your productivity

Clearly, one MUST configure the webserver to NOT permit off-site access to the credentials and configuration file: wp-config.php but I'll be darned if I can see instructions on the WordPress site, showing a novice administrator how to do this. In a shared hosting environment without 'root' level control, it is probably not even doable.  There is not hint of this rather elementary precaution on the official write-up concerning editting the file

A quick Google search for: turns up lots of vulnerable candidate installations, and a handy, dandy code fragment for parsing information out of potential victims so found, to automate take-overs. No criticism of the author of that code publishing his work; a knife can heal (as a scalpel), prepare dinner, or injure, depending on the intent of its holder

I see an official  recovery outline  suggestion, anyway

Worth repeating; Trust and Open Source

I first encountered Mark Shuttlesworth in person at an Ottawa Linux Symposium a few years ago, and passed along a reply from Dag, responding to some controversial comment Shuttlesworth had made at the time.  I choose not to use Ubuntu or Debian as my primary X desktop, but that said, there are 6 machines running one of those two distributions powered on in my office at the moment, so I am not a stranger there, either
He was being 'up front' about the fact that Amazon search results are being trialled for an upcoming Ubuntu version
He points out, and it bears repeating, the following:

[Question: ] Why are you telling Amazon what I am searching for?
[Answer: ] We are not telling Amazon what you are searching for. Your anonymity is preserved because we handle the query on your behalf. Don’t trust us? Erm, we have root. You do trust us with your data already. You trust us not to screw up on your machine with every update. You trust Debian, and you trust a large swathe of the open source community. And most importantly, you trust us to address it when, being human, we err.

The boldface are important, but I carry the context as well here.  When you use any computer operating system, you in the role of: user are  implicitly placing trust in the decisions and the commitment of those who put it together to 'do the right thing', or to make it right when things go awry
Do you trust your vendors?  Your actions may be pointing out a dissonance, if you said: no

More chickens, returning home to roost

I referred to the roosting chickens theme earlier today as to Oracle, and did not expect to be back to that topic for a while.  But the news does not wait
One of my long time concerns as a Cassandra, is that FOSS geeks firmly believe that  ignorance and  unworldly innocence, and a native sense of protection that 'just conduct' while participating in FOSS matters, will somehow insulate the 'good folks' from the world
To my experience, it does not work that way.  One needs to clearly disassociate from risky conduct.  A person needs to read EULAs and look for traps, like indemnification clauses
I see in the overnight news that RackSpace and Github face some hot water.  Drilling down, as to how this is likely to play out, I am aware that Github has an EULA, that provides in part:

Section F 3
You shall defend GitHub against any claim, demand, suit or proceeding made or brought against GitHub by a third party alleging that Your Content, or Your use of the Service in violation of this Agreement, infringes or misappropriates the intellectual property rights of a third party or violates applicable law, and shall indemnify GitHub for any damages finally awarded against, and for reasonable attorney’s fees incurred by, GitHub in connection with any such claim, demand, suit or proceeding; provided, that GitHub (a) promptly gives You written notice of the claim, demand, suit or proceeding; (b) gives You sole control of the defense and settlement of the claim, demand, suit or proceeding (provided that You may not settle any claim, demand, suit or proceeding unless the settlement unconditionally releases GitHub of all liability); and (c) provides to You all reasonable assistance, at Your expense.

Guess who is going to be added as a Defendant in that lawsuit next; guess who's pocketbook will ultimately be looked to, to pay for Github 'lawyering up.'  Github and RackSpace have every incentive in the world to 'point out' the poster of claimed infringing content to the Plaintiff and its lawyers, and so demonstrate 'good faith', a lack of intent to infringe, and their desire to mitigate any asserted damages

On my RADAR: Java to iOS Objective C

This crossed my RSS newsreader feed :

J2ObjC is an open-source command-line tool from Google that translates Java code to Objective-C for the iOS (iPhone/iPad) platform. This tool enables Java code to be part of an iOS application's build, as no editing of the generated files is necessary. The goal is to write an app's non-UI code (such as data access, or application logic) in Java, which is then shared by web apps (using GWT), Android apps, and iOS app

J2ObjC supports most Java language and runtime features required by client-side application developers, including exceptions, inner and anonymous classes, generic types, threads and reflection. JUnit test translation and execution is also supported

The project homepage is here, and along with Google's Web Toolkit, seems to be under an acceptable FOSS license
It does not strive to be a full replacement for all things Java, and as it notes:

when new projects first start working with it, they usually find new bugs to be fixed. Apparently every Java developer has a slightly different way of using Java, and the tool hasn't translated all possible paths yet

Long time readers of this blog will recall that I have been a critic of Java in FOSS projects particularly in the LSB standards context, and as I predicted, the Oracle anti-FOSS Java lawsuit [against Google it turns out, because they are a juicy target] was filed a couple years later. People working for nasty corporate masters earn their futures

This is not to say that translation layers are without warts and flaws, but at least this provides a partial roadway away from a Java dependency into a code generator that may emit freely licensable code.  I'll be tracking this with interest.  I have a sample conversion in mind ...

about this week's GoDaddy outage

It is not clear if a cabal of Anonymous hackers, or simple network administration issues, caused the GoDaddy outage of Monday past. I guess it does not really matter

What really would have hurt is if the root domain server constellation had been compromised, to well and truly take down the internet. A Domain Registrar sends along updates to those root servers periodically, and GoDaddy's outage, from the extent of our involvement with them, simply impaired our ability to renew domains, and set new nameservers (NS records). As we had no urgent renewals pending, that is to say, not at all

We do not rely on GoDaddy for DNS services, and really, never have relied on them for production purposes. For PMman and for our ISP and COLO services, we run three geographically diverse nameservers for most of our purposes. We also run a few others for customers' needs (PTR records for a couple of datacenters we are in, testing, demonstration units)

The true 'masters' of our externally visible DNS servers are simply not accessible from the public internet. We push out updates to our public nameservers by cryptographically protected rdnc transactions. Those transactions are logged, and the information causing a given RDNC transaction are created by queries into a local database with a custom written LAMP control interface based on the FOSS tools that are in a stock CentOS install. Compared to manually editing zone files, checking variants in and out of a version control system, and so forth, this more readily provides us with scalability, traceability and auditability. Why, I caught a piece of lint in a zone file just last week, reading the overnight error report emails

We also retrieve the state of the generated zone files at the client public nameservers, and check them for consistency and coherency, essentially after each update, to prevent errors from propagating. ACLs, transaction logging and other checks provide more tracability, and we closed the mouse hole that that 'lint' crept in through in short order

As a result of the GoDaddy outage, a couple of our 'alumni' tech support folks who have moved on in their careers to other employment, gave us a call Tuesday, because they remembered how paranoid I am on making sure DNS is available. I appreciate the calls, and we've some new customers as a result

People have strong opinions about GoDaddy, sometimes for reasons of political correctness; I like them, by and large, because they provide a workmanlike product for a price that is hard to beat. They sure beat the heck out of the old Network Solutions rates. I have something like 500 domains that I administer and renew and most are there, although some are at other registrars for both historical and other reasons

And while Danica Patrick is not my cup of tea, she is not hard on the eyes, either

RPM and YUM trick: synchronize packages installed on two hosts

The question was asked in IRC today:

hello folks, is there any way to install packages from a list written by yum list installed? I've two CentOS 6.3 hosts and I like to get them with the same packages installed (also versions)

Here is a quick (and accurate) answer:

1. Take and test backups to taste
2. Run:
   

   rpm -qa --qf '%{name} \n' | sort > MANIFEST

   Note: that is a backslash n -- the html markup makes it hard to see the distinction
3. Then copy that MANIFEST to the second unit and run:
   

   yum -y install `cat MANIFEST`

   Note: and here, backtick around the cat to get a sub-shell
4. Finish by running:
   

   yum -y update

   on each unit

For extra credit, re-run the MANIFEST creator on each unit, and use diff to find any variances

Thinking about Secure Boot

The Open Source community has a corner turn coming, in the upcoming roll out of UEFI enabled 'Secure Boot' hardware. And the 'Build Your Own' hobby computer builders are in for a rude shock as well, as they are going to have to start paying per each re-install of a commercial operating system, because UEFI provides a mechanism to reliably control (read: prevent free unlimited) reinstalls of Microsoft operating systems, as well as end user Application software

An integrated computer hardware or motherboard Manufacturer (Dell, HP, SuperMicro, etc) will have no choice but to conform to the new 'latest is greatest' approach that Windows 8 will bring to the market if it wants to keep selling new hardware. ... which means being able to run whatever Microsoft's latest is. And I don't want to be coy about this: it is in their economic interest as well to gain a way to limit the life of hardware, as it builds a ready market for re-selling to customers with older hardware

I simplify here, but once in full release, Windows 8, (and going forward, I think I can safely predict that high ticket proprietary Applications), will ONLY install on a system that has a secure chain of signed binaries and RELATED counter-signed Variables. That secure chain will be done by verifying a chain of checksums ('hashes') duly counter-signed by a Key-Exchange Key (KEK) public/private key pair, on back to a database itself counter-signed and verifiable by a Platform Key (PK). The PK may, but is not required to be able to, be wiped by an end consumer. For the reasons below, I think there WILL be an ability to reset the NVRAM on the general case

But perhaps not -- in the past applying some firmware updates required cracking open a case and moving a jumper. So I feel pretty comfortable that a mechanical 'jumper' option will appear as motherboards are designed, to prevent purely electronic (via executable code actions) key-wipes; it just makes sense in a corporate IT environment to prevent end employees from tampering with machines. And in some use cases, the jumper disappears altogether

Fresh from the fabrication plant, no PK exists in a UEFI 'bios' and it is said to be in a 'Setup' state. In the usual case, a PK will initially be generated and 'injected' into the NVRAM keystore at the time of manufacture (those cases where the end customer expects to receive a 'ready to run' computer). That is, while there may be provision for generating and injecting a PK keypair (either at initial receipt, or on a unit needing a 'wipe and reinstall'), in the general case the PK keypair injection will have already been done by the Manufacturer. And the Manufacturer is NOT going to readily supply the private side of that key with each chassis it ships ... the support load is too great, and an end user will be told: they are out of luck as it is not available

Those Variables I mentioned will almost certainly contain one or more 'unique per unit' UUID type hash, and as I say, these variables will be countersigned and held in the motherboard's non-volatile configuration ram. We have seen such approaches before -- old SGI hardware used NVRAM to hold a MAC address, that may have been signed as well, and SGI install disks looked for values in a range assigned to SGI

Part of an OS install process will be generating and authorizing a unique UUID, signed by a controlled KEK keypair chain, that chain running unbroken through a specific PK. The Manufacturers will generate a keypair, good for a given number of units and report sales to their Licensor. No more pesky COA stickers. If a KEK pair is compromised, they report it as such to their Licensor (here, Microsoft), and the public side is added to a blacklist database update. Computers relying on that trust chain simply stop booting, or stop running some node-locked Application, and instruct their owners to call technical support

Eventually Tech support says that this unit is out of warranty and end of life, and the Manufacturer or its competitor gets a new sale. Life is good for the Licensor and the Manufacturers as a group

As I read the UEFI specification, however, nothing requires being able to 'read back' those hashes, so that one could back them up, or have them at hand to be able to re-inject them. Indeed, the usual model is to NOT be able to read back enough information from a secure key store to be ABLE to back up and restore it. To do so would destroy the metering and regulation value of the KEK signed hashes and Variables

I assume the Open Source community will solve how to transition most hardware back into Setup state; I see that some of the kernel hackers have already solved wiping and then injecting new keys into the PK. I know that UEFI supports, but does not mandate use of x.509 certificates, so it is just a matter of time until StartSSL or some other Open Source friendly Certificate Authority documents and issues signing keys for both formal distributions, and end users compiling local kernel modules needing signing

Two futures are around the corner, here: Either way, BYO hobby users of commercial operating systems and applications lose a lot of usability to experiment. When it is too hard to 'play' around, they will need a new space to experiment and customize in. It is Open Source for the win as such a venue

And for the Open Source folks: 1. If the hardware permits an ability to re-set a UEFI 'bios' to Setup state and to re-inject an initial PK, Open Source wins, as it picks up new users when a person is presented with an estimate for re-purchasing all that they thought they had bought, but that need new activation keys, and asks: what else can I do? 2. If the hardware does not, this is readily apparent, and the manufacturer is shunned and their hardware avoided by folks in the know

I know it is the present fashion the Open Source community to damn UEFI and all, but the outlook is not all darkness. Just a turn around the corner to a new future

---

Updated: The interface at blogger.com (wasn't this formerly blogspot.com) has changed, and I missed that the final copy would 'lay our' so poorly; it is also eating random markup that formerly worked ... fixed

Right, Left, down the middle

A couple weeks ago, a 'Derecho' blew through Columbus, on its way to the metro DC area. Amazon had some failures that cascaded through to people who did not have site redundancy. People know that the East Coast was hit hard, but as we are out in 'fly-over' country they did not perhaps realize that we had several hundred thousand people around here without electricity for a couple weeks as well

I've mentioned before that the primary datacenter that we run our PMman product out of is at the Tier IV level -- multiply redundant cooling, power grid, power backup, fiber entrances, carriers. The owner, a friend, is just a fiend that he does not HAVE outages

Me, too. In our after-event review, I see that one of our secondary sites here in town fell back to its generators, but the rest were all fine. But all sites we use are well covered, all fiber, all multi-homed. Planning for failure was in our deployment planning checklist; we pay for (and we charge for) that coverage; and I consider it worth it

A national footprint customer based in Canada agrees. And their lead technical person reports that our connectivity is haster than their datacenter eighty miles from their home ofice. Not surprising, as oAltantaur main DC is on a 'main line' fiber route between Chicago, NYC, DC, and Atlanta -- financial markets and federal government presences can help, that way

If the availability of your online presence matters to you, feel free to ask for a quote

Yeah, it's a Monday ...

Last week, I made a trip up to the local computer store, the mothership of MicroCenter, and finally broke down and bought a USB/VGA four port KVM switch for my lead worksstation .... not DVI or HDMI on the video, but still VGA. The PS/2 mouse and keyboards were scheduled to be phased out, and a move to USB devices slated for this week

Well, the components to be affected .. the old KVM, the panel monitor, and so on must have held a pow-wow across the weekend, because when I came in this morning the panel monitor's backlight (a Westinghouse L2210NW, 1680x1050, datecode of April 2008) seems to be completely dead. This is of course the absolutely MOST inconvenient part of the display chain to die, because I need to run custom 'modeline' detail under X to squeeze the maximum resolution and sharpness out of the display. I had also purchased the three year 'no questions' replacement warranty on that four year old unit, so no help there

I grabbed a 'retired to the front bench' NEC AcuSync LCD223wxm, also nominally 1680x1050, off one of the benches and have been fiddling with the modeline settings to have a backup to limp through the day, but the horizontal height is wrong, the pixels puddgy, and the video muddled

No doubt I _could_ get it to stand up and dance, but the NEC has a datecode from early 2007 so that is a suckers game

Yeah, its a Monday

LOPSA at the PMman DC

I went up to a meeting at our North datacenter for PMMan, where local group of system admins held a meeting, starting up a local LOPSA chapter. Food and soda were provided by the DC operator, along with salad ... since when did sysadmins starting more healthy food, rather than a diet of high sugar, high caffeine junk food?

The presentation slide deck was fine, and the presenter (a 'long timer' at a local credit-card clearance operation) ran through his bullet list of what to look for in the 'build vs. buy (lease space)' decision, and then a number of siting concerns.

Now I am familiar with his firm's site from prior visits, and it is adjacent to a major highway with regular closures for accidents; adjacent to a major rail yard where chemical spills have caused evacuations; and sole serviced into the power company grid

Our North site was chosen after a survey of all offerings within a radius we were willing to drive to for 'end of the world' 'hands on' intervention; is jacked into two independent power grids along with the on-site generators, is a premier demonstration location for the former Liebert (now owned by Emerson) power and site conditioning

I happen do drink coffee daily at Staufs with Liebert's representative here in town. My evaluation team suggested the location as a finalist, and when I checked, it turned out that I already knew the owner / developer from long, long ago telephony days, and when I have time, I'll go up and 'shoot the bull' with him on Saturday mornings at the DC

We have had a grand total of ZERO power related outages, and only one network connectivity issue in the last three years, that lasting less than 15 minutes, and that, due to human error in not handling a BGP fail-across migration properly [the cut-over protocol was changes, as I noticed the drop from my monitoring and called the owner's cell phone at once ;) ]. Well suited to our 'enterprise' customers

It is 'carrier neutral' and hugely connected -- multiple entrances of up to 88 x 100 G fiber spread across six or seven principal carriers. Native IPv6 to all drops we run through multiple carriers, along with the IPv4. I helped with the IPv6 design and cut-over some 18 months ago, and it has been seamless. The facility, and our services, just do not have outages except that human error causes

It is not the cheapest in town ... but it is fairly priced for the value we have received

I had not sat down and reflected on how satisfied I was with that shift of our center of operations to the DC, but as I think about it, I am well pleased

It is always time to get back into the swing of a good habit

When I crushed my ankle, I really could not think straight for a couple months with the painkillers they had me on, and so I lost the habit

It's time to start blogging regularly again