Another pet died across the holidays

I wrote before about un-maintained and orphaned WordPress sites being exploited.  That same frantic user from two months ago, called again.  The TL;DR summary is:

• cPanel administration with multiple accounts in a single host without protections 
• OS Updates not being run
• WordPress updates not being run
• Random add-on's being used without an awareness of security issues
• No SELinux (disabled)

An exploit un-gzip-ping a hostile payload from cache was used, and the machine taken over

The absence of good sysadmin skills, well packaged content, and updates 'for the loss' ...

disable IPv6 DNS results

We had an end user appear in the main #centos IRC channel the other day with a IPv6 problem.  That person had leased a VPS somewhere, and their provider had included and enabled IPv6, at least partially.  Something was wrong in the network fabric, so that while some IPv6 services worked, others did not; DNS results returned with AAAA record results; but then the VM hoster was not transiting port 80 TCP traffic.  Very curious, and frustrating to the end user who just wanted yum to work so they could install updates and packages on their instance

The culprit is the grafting in of IPv6 readiness in man 2 getaddrinfo.  This is the way of the future, so there is no fighting it on a long term basis, but tactically having a means to be IPv4 only is appealing for people just wanting to work in the older address space.  The TL; DR uptake of that man page is that in a properly functioning system, name resolution answers under IPv6 are preferred, and only if not available, does one fall back to the older IPv4.  But this places a premium on IPv6 actually working when present.  We've shipped a full native IPv6 setup for customers at PMMan for a couple of years ago, but I assure you that we had some head-scratching as we rolled it out, and found customers using tunnels from HE or SixXs were also leaking advertisements to other VM's.  We added rules to filter out the false traffic after a bit of tracing with TCPDUMP

I have blogged about it before when IPv6 link-local address resolution (the ^FE family) was confusing distcc under Debian a couple of years ago.  There are links in the CentOS wiki for approaches on disabling IPv6 traffic, which vary between C5 and C6

That last mentioned article has an outlink to a bugzilla ticket that offers food for thought.  It mentions in passing that one can direct a nameserver to NOT deliver IPv6 results with a fairly simple tweak

Another option is to add this to /etc/sysconfig/named:

OPTIONS="-4"

... so, ... it should be possible to set up a local cacheing nameserver on the localhost, configured to NOT return IVv6 content, and so workaround the issue.  This smells sort of 'hackish', but it would have the benefit of being a single method that should work in the general case and not be tied to any particular kernel version, or other variable

Feeding the pet

We had a frantic call from a sometimes customer today.  Their self-administered WordPress-based website had a Trojan in it, and it was saturating their website traffic allocation.  "THE SITE WAS DOWN!!"  They had signed up at a CPanel mediated, shared hosting firm, and a plug-in they had installed turned out to contain a well-known trojan

We spent a couple of hours looking into it.  And then a couple hours looking into the WordPress security notification system.  Perhaps, I should say: non-notification system as to getting subscribed to a formal notification mailing list from the WordPress folks, proper

The WordPress model seems to be: treat your WordPress site as though it is a pet that needs daily feeding.  And to be 'put down' when you lose interest in it, move on, or forget about it -- Oops.  Log in daily as an administrator, and look for a notification

that you need to apply the 'latest and greatest' update.   Run the update process manually whenever it appears.  Oh yeah, did you remember to take a backup FIRST, and test that you can roll back to it if the 'update' breaks anything? Oops

This of course RULES OUT using a packaged approach to managing such sites, as the lag for stabilizing a new RPM package, accounting for potential database changes, and the like 'take too long'. Just unroll a tarball, and trust that it will not break any local customizations

I see fourteen open tabs in my browser panel still open, related to trying to track down a central and formal notification feed that I (or any person seeking to get 'push' notification) might subscribe containing only 'Security' notifications.  Weeding through the tabs, ...

• The 'Famous 5-Minute Install' for WordPress -- Nope, no useful outlink for hardening, nor to subscribe to notifications, beyond a pointer to a third-party Ubuntu appliance with an 'automatic security updates'.  That appliance's page has pointers to a tool to enable taking database backups, adding PHPMyAdmin, and Webmin.  Not good choices for a person caring about security
• Perhaps FAQ items tagged with: Security -- Nope, clearly incomplete, as for example a Google search turns up this third-party alert for version 3.3.2,  but the Release Notice does not get titled with: Security
• This bug (#10253) lingered for three years with a Security tag in their Trac issue tracker as to the current release series (3.4), and was amended ten days ago; But the latest release (for 3.4.2) was twenty days ago when this is written.  Should an update have been release?  Who Knows?
• Perhaps their FAQ Security -- Nope, no push notification link suggested there, but lots of clutter as to copyright infringement notification handling, and miscellaneous topics
• Perhaps watch the Releases News in an RSS reader - Oops, no sub-tag feed offered, and there has not been an "Important" Security release since December 2010, if one used that approach
• Run a Google search daily, and look for third-party commendary - Nope, although nuggets may be found, for it is not viable as: Not Authoritative, irregular and partial as to updates, and wading through search engine hit, or RSS feed clutter will kill your productivity

Clearly, one MUST configure the webserver to NOT permit off-site access to the credentials and configuration file: wp-config.php but I'll be darned if I can see instructions on the WordPress site, showing a novice administrator how to do this. In a shared hosting environment without 'root' level control, it is probably not even doable.  There is not hint of this rather elementary precaution on the official write-up concerning editting the file

A quick Google search for: turns up lots of vulnerable candidate installations, and a handy, dandy code fragment for parsing information out of potential victims so found, to automate take-overs. No criticism of the author of that code publishing his work; a knife can heal (as a scalpel), prepare dinner, or injure, depending on the intent of its holder

I see an official  recovery outline  suggestion, anyway

Worth repeating; Trust and Open Source

I first encountered Mark Shuttlesworth in person at an Ottawa Linux Symposium a few years ago, and passed along a reply from Dag, responding to some controversial comment Shuttlesworth had made at the time.  I choose not to use Ubuntu or Debian as my primary X desktop, but that said, there are 6 machines running one of those two distributions powered on in my office at the moment, so I am not a stranger there, either
He was being 'up front' about the fact that Amazon search results are being trialled for an upcoming Ubuntu version
He points out, and it bears repeating, the following:

[Question: ] Why are you telling Amazon what I am searching for?
[Answer: ] We are not telling Amazon what you are searching for. Your anonymity is preserved because we handle the query on your behalf. Don’t trust us? Erm, we have root. You do trust us with your data already. You trust us not to screw up on your machine with every update. You trust Debian, and you trust a large swathe of the open source community. And most importantly, you trust us to address it when, being human, we err.

The boldface are important, but I carry the context as well here.  When you use any computer operating system, you in the role of: user are  implicitly placing trust in the decisions and the commitment of those who put it together to 'do the right thing', or to make it right when things go awry
Do you trust your vendors?  Your actions may be pointing out a dissonance, if you said: no

More chickens, returning home to roost

I referred to the roosting chickens theme earlier today as to Oracle, and did not expect to be back to that topic for a while.  But the news does not wait
One of my long time concerns as a Cassandra, is that FOSS geeks firmly believe that  ignorance and  unworldly innocence, and a native sense of protection that 'just conduct' while participating in FOSS matters, will somehow insulate the 'good folks' from the world
To my experience, it does not work that way.  One needs to clearly disassociate from risky conduct.  A person needs to read EULAs and look for traps, like indemnification clauses
I see in the overnight news that RackSpace and Github face some hot water.  Drilling down, as to how this is likely to play out, I am aware that Github has an EULA, that provides in part:

Section F 3
You shall defend GitHub against any claim, demand, suit or proceeding made or brought against GitHub by a third party alleging that Your Content, or Your use of the Service in violation of this Agreement, infringes or misappropriates the intellectual property rights of a third party or violates applicable law, and shall indemnify GitHub for any damages finally awarded against, and for reasonable attorney’s fees incurred by, GitHub in connection with any such claim, demand, suit or proceeding; provided, that GitHub (a) promptly gives You written notice of the claim, demand, suit or proceeding; (b) gives You sole control of the defense and settlement of the claim, demand, suit or proceeding (provided that You may not settle any claim, demand, suit or proceeding unless the settlement unconditionally releases GitHub of all liability); and (c) provides to You all reasonable assistance, at Your expense.

Guess who is going to be added as a Defendant in that lawsuit next; guess who's pocketbook will ultimately be looked to, to pay for Github 'lawyering up.'  Github and RackSpace have every incentive in the world to 'point out' the poster of claimed infringing content to the Plaintiff and its lawyers, and so demonstrate 'good faith', a lack of intent to infringe, and their desire to mitigate any asserted damages

On my RADAR: Java to iOS Objective C

This crossed my RSS newsreader feed :

J2ObjC is an open-source command-line tool from Google that translates Java code to Objective-C for the iOS (iPhone/iPad) platform. This tool enables Java code to be part of an iOS application's build, as no editing of the generated files is necessary. The goal is to write an app's non-UI code (such as data access, or application logic) in Java, which is then shared by web apps (using GWT), Android apps, and iOS app

J2ObjC supports most Java language and runtime features required by client-side application developers, including exceptions, inner and anonymous classes, generic types, threads and reflection. JUnit test translation and execution is also supported

The project homepage is here, and along with Google's Web Toolkit, seems to be under an acceptable FOSS license
It does not strive to be a full replacement for all things Java, and as it notes:

when new projects first start working with it, they usually find new bugs to be fixed. Apparently every Java developer has a slightly different way of using Java, and the tool hasn't translated all possible paths yet

Long time readers of this blog will recall that I have been a critic of Java in FOSS projects particularly in the LSB standards context, and as I predicted, the Oracle anti-FOSS Java lawsuit [against Google it turns out, because they are a juicy target] was filed a couple years later. People working for nasty corporate masters earn their futures

This is not to say that translation layers are without warts and flaws, but at least this provides a partial roadway away from a Java dependency into a code generator that may emit freely licensable code.  I'll be tracking this with interest.  I have a sample conversion in mind ...

about this week's GoDaddy outage

It is not clear if a cabal of Anonymous hackers, or simple network administration issues, caused the GoDaddy outage of Monday past. I guess it does not really matter

What really would have hurt is if the root domain server constellation had been compromised, to well and truly take down the internet. A Domain Registrar sends along updates to those root servers periodically, and GoDaddy's outage, from the extent of our involvement with them, simply impaired our ability to renew domains, and set new nameservers (NS records). As we had no urgent renewals pending, that is to say, not at all

We do not rely on GoDaddy for DNS services, and really, never have relied on them for production purposes. For PMman and for our ISP and COLO services, we run three geographically diverse nameservers for most of our purposes. We also run a few others for customers' needs (PTR records for a couple of datacenters we are in, testing, demonstration units)

The true 'masters' of our externally visible DNS servers are simply not accessible from the public internet. We push out updates to our public nameservers by cryptographically protected rdnc transactions. Those transactions are logged, and the information causing a given RDNC transaction are created by queries into a local database with a custom written LAMP control interface based on the FOSS tools that are in a stock CentOS install. Compared to manually editing zone files, checking variants in and out of a version control system, and so forth, this more readily provides us with scalability, traceability and auditability. Why, I caught a piece of lint in a zone file just last week, reading the overnight error report emails

We also retrieve the state of the generated zone files at the client public nameservers, and check them for consistency and coherency, essentially after each update, to prevent errors from propagating. ACLs, transaction logging and other checks provide more tracability, and we closed the mouse hole that that 'lint' crept in through in short order

As a result of the GoDaddy outage, a couple of our 'alumni' tech support folks who have moved on in their careers to other employment, gave us a call Tuesday, because they remembered how paranoid I am on making sure DNS is available. I appreciate the calls, and we've some new customers as a result

People have strong opinions about GoDaddy, sometimes for reasons of political correctness; I like them, by and large, because they provide a workmanlike product for a price that is hard to beat. They sure beat the heck out of the old Network Solutions rates. I have something like 500 domains that I administer and renew and most are there, although some are at other registrars for both historical and other reasons

And while Danica Patrick is not my cup of tea, she is not hard on the eyes, either