05 December 2008

Only you can prevent forest fires

You say you want a revolution
Well, you know
We all want to change the world
You tell me that it's evolution
Well, you know
We all want to change the world
-- John Lennon, The Beatles

Ted Tso has a blog post commenting about the market capitalization of Sun (stock ticker: JAVA) being 'underwater' relative to its cash on hand:
Sun’s current market cap: $2.40 billion USD. Sun’s cash on hand: $2.63 billion USD
and advancing a concern that a firm hostile to Software Freedom might purchase and kill it. [A bit ironic, thinking about the Cobalt Qube; a PFY I trained a few years ago moved to Portland to work supporting the Qube software, only to have Sun functionally kill the Cobalt product within the year ... oops]

Not surprisingly, there is a maze of intellectual property rights to navigate in considering using an asset so valuable that a company renames its Wall Street 'nickname' from 'SUNW' ("Stanford University Network Workstation") to JAVA

Hopefully, Ted Tso was just playing 'Devil's Advocate' earlier this year in a thread of debate surrounding a proposed adoption of a java (Sun's 'Java' or otherwise) into the Linux standards base upcoming ver. 4.0 release stirred concern

Reviewing the bidding;
  • LSB weekly conference call chair Jeff Licquia stated the non-Free (OSI or FSF wise) Java Test Conformance Kit concisely in the post call minutes
Jeff: summary, Ted: do a checkpoint two weeks from now, to see that the features are getting in. Jeff: agenda item for August 13th call? Ted: yes. Mats: Java is also an issue. Ted: issues with Java? Mats: which spec? Required methods, classes, etc. Test suite question is also a big deal. Ted: should follow up on these issues before Ron gets back
LSB Committee work happens in part on the mailing list. Note: If one wants the flavor of the thread, read the posts by Tso, Cox and Herrold out of the pipermail archive for August 2008. At one point in that thread, I was asked: Why do you care. I care because FOSS culture matters and needs defenders to stand up and advocate for it. I can live with the balloting results for the 4.0 release

Software Freedom issues are an 'elephant in the room' at the Linux Foundation (current 'owners' of the LSB), and they try to thread a delicate balance to draw the commercial into the FOSS world

My personal opinion is that LF need not worry so: The 'should we have a FOSS strategy' discussion is over; 'of course,' being the outcome. Market forces will solve adoption because the firms in an given industry NOT using FOSS properly will have higher cost structures, and go extinct

Oh, and you too can make a difference; by showing up, participating with considered intent, doing justice, loving kindness, and walking humbly, all the while remembering that "politics ain’t beanball"

13 November 2008

Behind Blue Eyes

Behind Blue Eyes -- The Who
Dateline: U.S. Department of Labor
The largest increases in initial claims for the week ending Nov. 1 were in Ohio (+3,885), Michigan (+2,619), Pennsylvania (+2,155), Wisconsin(+2,119) ...

A friend wrote:
All this proves is that when someone crosses a state line to register to vote it is just as easy to register for unemployment while you're at it.

I think it is probably much worse than that

It is easy enough for anyone to set up a (several!) new 'employers' and then walk away from them 8 weeks later with no individual financial responsibility for the 'tail' -- after all, we encourages formation of 'small business, the engine of economic growth' and barriers to entry should be small, right?

Unemployment benefits may be had at full rate for 6 months after 6 weeks employment at a given 'employer' if one is otherwise qualified; when an 'employer' goes out of business, the employees are eligible for benefits Several telephone poles in central Ohio had signs, with differing phone numbers, for what appeared to be short term 'jobs' working to elect Obama and 'make Change'. I snapped a picture with my mobile device, and will see if I can find it for the exact text; I recall thinking at the time:
-- Don't the 'employee candidates' KNOW they will be let go the day after the election

Now, I think the answer is:
-- Sure -- indeed they were TOLD by the recruiter at the other end of the phone, that this was a way to get rid of a pesky 'termination for cause' {disqualifying} black mark which was keeping them from what they were 'entitled to'

As the One won, and 'We can do it!' if the system is properly 'gamed', I think there will be no investigations after Jan 20 to 'connect the dots', and the Lame One will just snooze out his term. 'No law will prevent it'

And so the Republic was lost. "Meet the new boss; same as the old boss"

10 November 2008

rpm -import of GPG keys, revisited

Dbacks at the BOB
Color commentary guy

Okay, I guess I covered too much too fast last time I discussed adding a signing key to RPM. Let's do it again with more annotation and color commentary.

The RPM package manager (see: the old RPM.ORG website, which I maintained as 'rpm.org' for several years; JBJ's 'way forward' for RPM development site; and the rather sparse, intentionally stale, and to me useless site controlled by and populated to suit the Red Hat corporate agenda -- details of the fork in RPM are out of scope here) has the capability to verify through strong cryptography that a package is intact, and is counter-signed by a person in possession both halves of an asymmetric public and private keypair. Assuming that reasonable care (where 'reasonable' is a very large and paranoid number) is used to protect the confidential nature of the private half, the chances of a successful substitution are vanishingly small.

Anyone can examine and inventory the keys in RPM's trusted keystore. The process of additions, changes, and deletions of keys is an operation requiring root level privileges, and so assuming a machine can be trusted (both network level and local physical level attacks need to be considered)

Enumerate the keys present:

$ rpm -qa gpg\*

Examine a specific key:

$ rpm -qi gpg-pubkey-e8562897-459f07a4

If we know or can determine the 'fingerprint' of the public half of a signing key, and if that key has been placed at a public keyserver, we can retrieve it, examine it, or even directly import it. For the sake of this example, we again consider the Raw Hide SRPM signing key (with the re-organizations over time, Red Hat presently signs Raw Hide content with key: 0x4F2A6FD2 which the MIT keyserver identifies thus)

The CGI query on the link above used the 'op=index' modifier; the next uses the 'op=get' -- one assumes 'op' is shorthand for the type of query operation made -- terse, or key-bearing. In any event, we retrieve the key into a local file thus:

$ wget -O fedora-key "http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x4F2A6FD2"

and then may examine it with the conventional 'nix tools:

$ less fedora-key
<title>Public Key Server -- Get ``0x4F2A6FD2
<h1>Public Key Server -- Get ``0x4F2A6FD2
Version: PGP Key Server 0.9.6

... snip ...

The important thing to notice, amid the HTML markup, is that the key is 'armoured text' well set off with start and end markers, so that GnuPG (and also RPM) may pick the key out of the chaff.

We discussed previously the chain of steps we used to decide that the key was authentic, and worthy of trust; as such we do not repeat them here.

Then, using the 'sudo' command to temporarily attain 'root' rights for the importation step, we can insert (import into the RPM database) the locally checked key:

$ sudo rpm -import fedora-key

Or, assuming that we will do a post-insertion check, we can do the import directly from the keyserver:

$ sudo rpm -import "http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x4F2A6FD2"

Then we can re-inventory keys, and see the new one present, and the full name under which it may be found; part of the name is, conveniently, the 'fingerprint' of that key.

$ rpm -qa gpg\*
$ rpm -qi gpg-pubkey-4f2a6fd2-3fcdf8c9

Hopefully this clears things up a bit.

08 November 2008

Going, going, gone ...

They paved paradise and put up a parkin' lot
With a pink hotel, a boutique, and a swingin' hot spot
Don't it always seem to go
That you don't know what you got till it's gone
They paved paradise and put up a parkin' lot

-- Joni Mitchell
I flew into Phoenix for a week's rest, and was unpleasantly confronted with America West USAir's 15 dollar baggage check flying in. Took the SuperShuttle up to Scottsdale, and had a driver on his third day on the job for them. We were seeing part of the Valley new to him, anyway.

He drove a route up Scottsdale past the site of the former Raw Hide Western Town and Steakhouse. It has been gone three years now. The old site was bull-dozed flat, but it looks as though the developer who started condo construction ran out of money half-way through the project.

Dinner early in the week up in Carefree was to be at Crazy Ed's 'Satisfied Frog' -- its menu has the old saw about how 'it is so popular, no-one goes there anymore'. Last time, we ate at 'The Horny Toad' (as I recall Ed lost the Toad in a divorce; his ex kept running the place). Drove in from the west after a visit to Fry's Electronics on Thunderbird. Gone -- an imposter with new signage in its place -- google says I missed the close by a month.

Drove up to Page AZ for a wonderful nature hike in a private 'slot canyon' with Overland Canyon Tours and photo session [highly recommended and well worth the premium price]; on the way back decided to stop in Sedona for a nice dinner, and found that the high end restaurant we dined at a year ago May had closed doors; at least there is a new bank branch in its place.

Oh yes -- and two traffic circles -- a new concept there; their City Engineer must have heard how great they are -- on State Route 89 inside the city limits, and another four or five on the way back toward I-17 south of town. As traffic circles are a foreign concept, each cardinal entry point had illuminated, gas generator powered signage explaining their use. The fumes and noise are only remporary, right?

Well, it's the last night in town, and so we decided to go to Pinnacle Peak Patio; the resturant may have opened to the public in 1957 (per its website last updated 2007), but their display cabinets show postcards and envelopes from WWII simply addressed to 'John Doe; Pinacle Peak; Phoenix'. The waitress, proud of her new leather belt, and 'new in town' still bearing a California accent, came to the table. Once we indicated we had been eating there for nearly 20 years, she blurted out that the place is 'to close next March, or mebbe a year later as the developer "gave an extension"' before it is to be knocked down for yet another 'resort community' near Troon. She had heard about those other places, 'though. Seeing my reaction, she offered that 'perhaps they'll rebuild inside the new facility, but it will probably look like all the other chain restaurants'. Yeah, probably.

Well, at least a pack of coyotes woke me up at 2 in the morning, midweek, delighting at the moon; the red rocks watched silently as they have forever, and with any luck will continue to do.

29 October 2008

... and then there were none

When the Nazis came for the communists,
I remained silent;
I was not a communist.

When they locked up the social democrats,
I remained silent;
I was not a social democrat.

When they came for the trade unionists,
I did not speak out;
I was not a trade unionist.

When they came for the Jews,
I remained silent;
I was not a Jew.

When they came for me,
there was no one left to speak out.
-- Pastor Martin Niemöller (1892–1984)

We had a truck strike the power pole for the building hit last week; it took out the transformer with a most satisfying 'pop'. It also had the secondary effect of a power surge, which caused a 'fried' monitor, so that I had occasion to need a new one to get us back up to full complement.

New monitors offer an occasion to play 'monkey move up,' it is my turn for the upgrade, and the $200 price point has a nice Westinghouse L2210NW panel display [1680 x 1050 pixels, 22" diagonal] at the moment. I have had a Westinghouse LTV 19W3 [1440 x 1050, 19"] which I have enjoyed using since January 2006, and it seemed to make sense to stay in the brand. (I bought the 3 year service plan on that one for an extra 25% on the price, as I was unsure as to durability of this, by first panel, but that has never been needed)

One trial and tribulation (and geeky challenge) of a new resolution is the need to adjust the video card driver to support the new Modeline, and to squeeze every ounce of performance out of the monitor. I am an old hand with the Intel Modeline tool, 810resolution, and its successor, 915resolution, for my present X desktop chassis' video card.

Over time, 'progress' has removed the tools for a 'nix admin to configure a display for the X window manager:
  • Xconfigurator
  • xf86setup
  • a working X -configure
  • kudzu
  • system-configure-display
  • manual configuration of /etc/X11/xorg.conf

I find that the new panel has consumed 6 hours of setup time at this point, and is still not working, edge to edge at full resolution. Unpleasantly I was surprised to find kudzu erroring and dying; ddcprobe --raw returns nothing; X -configure and system-config-display seem to know only how to turn the screen blank and lock up the keyboard so that a power cycle is needed to regain the unit (I'll write more on this later); and manual edits of xorg.conf have so far succeeded in getting only an off center, mis-sized image up.

This is not at the magnitude of the atrocities of which Niemöller wrote so well; I see the battle raging about making a gratuitous change to VT's over on the Fedora-devel mailing list with false statistics abounding, and the usual 'don't bother us with the facts, kid; our mind is made up' on knowing what you need and want.

Dax Kelson wrote well with diagnosis and action plan, but it seems to have fallen on deaf ears; 'pearls before swine', and 'the tragedy of the commons' again. We must fight the good fight anyway for
"The punishment of wise men who refuse to take part in the affairs of government is to live under the government of unwise men"

-- Plato

Summary, for those still listening: I want fallback (and degraded but partial performance) modes when a tool is not working as determined by the person looking at it; I want diversity rather than monoculture in tools; I want a upstream community which does not 'break expectation' by 'feeping creaturism' (or 'creeping featurism').

I'll take a stroll to Stauf's (the coffee shop down the street) to lower my blood pressure.

22 October 2008

stopping the next ssh leapfrog chained attack

For want of a nail the shoe was lost,
for want of a shoe the horse was lost,
for want of a horse the knight was lost,
for want of a knight the battle was lost.
So it was a kingdom was lost - all for want of a nail.
It is sensible to assume that the 'black hat' side is just a smart as the 'defense', indeed that they read the open literature and mailing lists, and think about where unseen holes might remain. They share and collaborate, albeit covertly and imperfectly.

The end case of this train of thought is that using a 'security through obscurity' approach is simply to 'hide and hope', ostrich-like, that the counter-party chooses another target.

So we end up with the case for openly discussed and developed security. It may not be possible to 'wash the linen' publicly at first, but if a project does not provide a frank and open 'root cause analysis' and response to its clientele, when an exploit has occurred, one has to question why one should trust them prospectively.

Part of basic system administration is inventorying the hosts under management. Based on review of some found cracker scripts, it is clear that some scripts 'phone home' information about the target or compromised host. At first, generic drop box accounts might have been used for transport, but of course those have to be retrieved, or forward along information, and as such can be traced in some cases. Game over.

So methods to anonymously place, and retrieve content emerge on the 'cracker' side:
  • encrypted IRC networks for command, control and transport;
  • computer mediated one-time pads and drop boxes which enforce proper use and are provable secure (at pg. 5), see also Schneier on the topic [we differ from his assertion that OTP are: 'also pretty much useless. Because the key has to be as long as the message, it doesn't solve the security problem.' While correct so far as it goes, that objection merely clarifies the remaining problem to solve];
  • strong asymmetric [public, private keypair] cryptography with DH key transfer can permit truly untraceable secure communication.
The three preceding forms of root level access are taken from the news.
  • for convenience, backups are customarily not strongly keyed with one time keys -- backup processes are customarily scheduled to run in slack activity periods, and so run at night when no-one is there to provide the keying; automated hardware one time keying systems that meet FIPS 140-2 standards are hard to do properly and expensive when certified to NIST standard levels
  • locking bolts to control chassis access (the 'Kensington cable' chassis frame slot), BIOS lockdown, and tamper switch audit are routinely left unused and unmonitored
  • the 'minimal' case of 'cracker' compromise
Presently Red Hat derived distributions carry too much gratuitous 'plain-text treasure' for a person in possession of an unencrypted backup, or with unchecked physical access to hardware, or who has root level read access.

I am thinking here particularly of harvesting 'known_hosts' and residual 'known_hosts2' for cleartext 'next hop' targets. I have speculated on this vector in the past.

Quick test to play along: run:
sudo find / -name 'known_hosts*' -print 2> /dev/null | grep [s2]$
and then as a non-privileged user, cat a few files. For extra credit and extra heartburn, repeat the inventory thus:
sudo find / -type d -name '*gnupg' -print 2> /dev/null

I certainly do not like what I see on my systems in reviewing the contents of the found files. It is clear that my practice (before authoring this piece) of rsyncing disk-to-disk backups around without cleaning up; and leaving working files on host transfers and migrations around are not well thought out as to security implication.

[herrold@centos-5 ~]$ wc /tmp/transferiso/1/root/.ssh/known_hosts
54 162 13967 /tmp/transferiso/1/root/.ssh/known_hosts
[herrold@centos-5 ~]$

Enough. I will not continue such a state of affairs. The default global ssh and sshd settings need to be altered in /etc/ssh/

man ssh_config provides:
Indicates that ssh should hash host names and addresses when they are added to ~/.ssh/known_hosts. These hashed names may be used normally by ssh and sshd, but they do not reveal identifying information should the file’s contents be disclosed. The default is “no”. Note that hashing of names and addresses will not be retrospectively applied to existing known hosts files, but these may be manually hashed using ssh-keygen(1).
and the tool for a system-wide cleanup and conversion is in the default open-ssh already:

man ssh-keygen contains the following option:
-H Hash a known_hosts file. This replaces all hostnames and addresses with hashed representations within the specified file; the original content is moved to a file with a .old suffix. These hashes may be used normally by ssh and sshd, but they do not reveal identifying information should the file’s contents be disclosed. This option will not modify existing hashed hostnames and is therefore safe to use on files that mix hashed and non-hashed names.
We just need to have the will and the time to make the changes, write the scripts, do the work to secure content, adopt better habits, and push those habits into scripted repetitive tasks. Yeah -- that's all ... hmmm

081023 typo, layout, and grammar fix

15 October 2008

If I have seen further ...

"If I have seen further, it is by standing on the shoulders of giants."

-- Isaac Newton

Jon Postel
August 6, 1943 - October 16, 1998

Jon Postel served as editor of the RFC series from April 7, 1969 (its inception) until his death in October 1998. Full details of the debt we all have are outlined in the eulogy by Vint Cerf.

He died a decade ago, now -- we are poorer without him.

07 October 2008

"Back, to the Future"

Doc: "You see, Marty, this time I really, really know what I am doing, so you can trust me on this one"
Marty: "Gee, I dunno, Doc"

Fannie Mae Eases Credit To Aid Mortgage Lending - 30th September 1999 (New York Times)

... In moving, even tentatively, into this new area of lending, Fannie Mae is taking on significantly more risk, which may not pose any difficulties during flush economic times. But the government-subsidized corporation may run into trouble in an economic downturn, prompting a government rescue similar to that of the savings and loan industry in the 1980's.

Yeah ... but THAT will never happen again. That 'S and L bailout' thing was a once in a lifetime event. Six Sigma, and all that. We're smarter than that now. It's different this time.

"Mr. Anderson. Welcome back; we missed you"

When I came home, I found a couple pieces of paper mail. One from Scottsdale AZ, and the other from Bologna Italy. Google Maps indicates a separation of 5,973 miles. Another source makes it a great circle distance of 5,981 miles.

Either way, they are each venue recently visited by family members, authorized to use my credit card.

It appears, also, that each venue has an efficient traffic citation issuance system, and I will have the privilege to dispute a citation for driving in excess of ten miles over the speed limit (Scottsdale), and for improperly parking a vehicle (Bologna).

At least it is late enough in the day for a single malt Scotch.

06 October 2008

Sledding down the slippery slope

Mr Dooley reads the paper:

08:52 Facing shortfall, Massachusetts inquires about a Federal loan - NY Times

NY Times reports the Massachusetts state treasurer has asked the federal government about lending the state money under the same favorable terms given to banks and investment firms during the financial crisis ...

Call me old fashioned, but wasn't this result perfectly predictable [to the Fed, to Treasury, and to the Joint Economic Committee], once starting down the 'moral hazard' path?

It is too early for strong drink, but ...

30 September 2008

"latest and greatest" disease

RIMM, the maker of the popular 'Blackberry' smartphone, seems to have forgotten what we all have known for a long, long time. In recent models, they moved from a roller clickwheel at the right thumb position, to a set of 'up and down' buttons, or a trackball.

Less precise, and less capable: The clickwheel could be operated by touch alone; the new variants cannot.

Making matters worse, seemingly no-one at RIMM ever is in a environment where their hands pick up grime. This grime, of course, transfers to the trackball.

The trackball is not field cleanable; compare contra the 'remove and clean' capabilities of a computer mouse. I see: Broken BlackBerry Blues over at thestreet.com today.

Time to 'stock up' with a couple unlocked '8700's off ebay, in advance of the day I lose or damage my current device.

Bus test

A recent questioner in the CentOS forums complained about the decision to carry older versions of Unix network services, rather than the 'latest and greatest'.

That person said the package was:
> dead stable
as though it was a criticism. CentOS is not and will never be anything but boring, predictable, and dead stable.

The use case was:
> it will be used as an email server for a small business
without any rationale on why a later version is needed. Security matters are backported, as part of the CentOS upstream's approach on an Enterprise grade operating system. CentOS dutifully issues said updates after a quick trip of a SRPM through the build process (trade mark elidement, signing, and other validity checking; announcement), and sends it off to the 'updates' mirror network, for yum to find.

One goal of consulting for third party clients is to hold costs down and yet meet requirements. Anything more, and the consultant may be learning, but the client's best interest is not being served.

Either the client is being charged while the consultant 'plays', or if not being charged, their business is (unknowingly) being used as a 'crash test dummy' venue for such exploration. Not good.

Consider the 'bus test':

What happens when a bus sideswipes the consultant, who is then out of commission for a month during recovery? [I know of other formulations of the 'bus test', of a more gruesome nature, but they are not needed for this hypothetical ;) ]

By forking as little as possible, nothing bad happens to the client.

Seems like a win to me.

Now, where did my coffee cup disappear to?

29 September 2008

'whipped like a rented mule'

... pretty well describes the way I felt most of last week.

One of my children commented that I had 'racoon eyes' when they stopped by the house; I was collapsing on the couch after dinner, only to awaken after nightfall, and retire to bed.

Finally the weekend arrived, and I made up the sleep deficit a bit. Made a run to the liquor store as well. I was looking for Seagram's 7 Crown, a favorite Blended Canadian Whisky, for a tall, cool '7 and 7'

None to be found. The store had a 'special rebate coupon' on Canadian Club in the 1.75 L size, so I picked up that instead as a 'tide me over', and a Wild Turkey 101, my sentimental favorite for an 'on the rocks' drink.

The store does have Lagavulin 16 at a fairly reasonable price, but in taking inventory before my trip to the store, I have two pleasant 'straight up' choices, and three bottles of less appealing Scotch to work through first. Truth be known, those last three will probably be consigned to the kitchen for holiday 'Scotch ball' cookies.

... later ... that Canadian Club is drinkable, but not something I'll be buying over 7 Crown. At least there is that rebate.

09 September 2008

Adding a signing key to RPM

A common (and commonly ignored) step when rebuilding Source RPMs from a remote archive is that of verification of the authenticity of the content.

An archive maintainer may choose to sign, or to not sign RPM (and thus SRPM) content it releases. Implicitly, an archive which does sign its content provides a way for a consumer of that content, remote in time or at another site, to verify the authenticity, integrity, and provenance of that package. An earlier post discussed using GPG to verify signed content generally.

The RPM package manager has long had the ability (similar to GnuPG) to receive GPG public keys into its trusted store, and then to test assertions about the presence, absence, and validity of a given signing. It can retrieve a remote key with the usual RPM network retrieval capabilities, or perhaps better to avoid MitM ('Man in the Middle') compromises across a network, from the local filesystem, or a local piece of immutable media, such as a CD which has had its md5sum verified.

Hughesjr covered it when 'near' to the install process. I covered CentOS key imports, but did not there mention checking key fingerprints and adding keys for other archives, which provoked some grouchiness later in that thread.

Let's take another swing at it here, and work an example, to manually retrieve a Red Hat 'RawHide' signing public key, of which the matching private key was assumedly used to 'sign' a package from the RawHide archive. This article uses the term: we to mark steps which the reader might safely attempt on their local box. From following Fedora, and from prior work on the old RHL testers-list team, we may believe that Fedora signatures on Red Hat content in RawHide are not unexpected. Let's be a bit more formal.

[herrold@centos-5 rpmlint]$ rpm -Kv rpmlint-0.82-3.fc9.src.rpm
Header V3 DSA signature: OK, key ID 4f2a6fd2
Header SHA1 digest: OK (8d753729ecfc59620fca2ccc1018e767787da9a6)
MD5 digest: OK (7d8b212cc68ff675fd64b124d832c423)
V3 DSA signature: OK, key ID 4f2a6fd2
[herrold@centos-5 rpmlint]$

So RPM informs us that the the 'fingerprint' of the keypair element which was used to sign that package is: 0x4f2a6fd2 Please note that in this piece, as with hexadecimal numbers generally, upper and lower A through F are interchangeable. Also, this piece flags 'hex' numbers with the conventional: 0x

Part of the RPM package signing process requires unlocking a GPG private key (if so protected with a passphrase), and so to permit RPM to sign content using that key. If a private key were NOT so protected with a passphrase, even more obvious (and potentially undetectable) problems exist, as a non-passphrase protected key might be silently copied and thereafter used by a malicious third party.

Aside on passphrases and signing keys: In recent days, both Red Hat and Fedora have recently revoked many of their signing keys, due to separate potential compromises of the signing processes for these two entities.

Some press reports suggest that an unauthorized insertion was made into the Red Hat CVS server [a trapdoor bypass after the 'pam' authentication checking return], and that a binary RPM package containing that exploit was presented to, and made it through the signing process. One might speculate that a developer may have unknowingly used least one cracked sshd server [consider, say, e.g., via a 'small sshd keyspace' Debian gateway, then exploited] through which passwords, passphrase keystrokes and so forth may have transited. For a person with 'root' level access to a Linux host, it is reasonably straightforward to add a malicious silent capability to also silently log keystrokes.

Later on, with credentials in hand, doing a malicious CVS commit in the account rights of another is trivial; non-trivial to understand is how a build came to be enqueued to the signing process was authorized and happened and a signing occurred on the Enterprise product at Red Hat. It further appears from some statements that the signed trojaned binary was accessible within the general internal network at Red Hat, and may have been able to 'escape into the wild' or was harvested by persons presently unknown. Under such a possible fact pattern, the revocation of the signing keys is a wholly responsible and proper response by Red Hat. The Fedora details are less well explained publicly presently, but one hopes a more complete description will emerge as time passes.

Using Google and the search argument: 4f2a6fd2 key fingerprint, I find a seemingly authoritative statement of Fedora keys, matching the key fingerprint of the package in question. A check of the domain registration indicates that it is under the control of Red Hat (owner of the Fedora project):

[herrold@centos-5 ~]$ whois fedoraproject.org
[Querying whois.publicinterestregistry.net]
... snip ...
Domain ID:D101496757-LROR
Created On:24-Sep-2003 10:32:11 UTC
Last Updated On:10-Jun-2008 20:25:33 UTC
Expiration Date:24-Sep-2009 10:32:11 UTC
Sponsoring Registrar:Network Solutions LLC (R63-LROR)
Registrant ID:41295926-NSI
Registrant Name:Red Hat, Inc.
Registrant Organization:Red Hat, Inc.
Registrant Street1:1801 Varsity Drive
Registrant City:Raleigh
Registrant State/Province:NC
Registrant Postal Code:27606
Registrant Country:US
Registrant Phone:+1.919754370
Registrant FAX:+1.919754370
Registrant Email:domainadmin@redhat.com
... snip ...

Both new and former keys seem to be enumerated at that webpage. The SSL certificate respecting this website seems to have been counter-signed by a commercial third party in the business of running a certificate Authority, Equifax. At the date of writing, this signing and the certificat itself are each unrevoked.

As such, we can reasonably conclude this is an authoritative website.

The Fedora project administrators additionally published a key yielding that fingerprint to the GPG keyserver network. Querying with: 0x4F2A6FD2, we are offered both the key content, and also a record of countersigning of persons asserting that they have been furnished sufficient credible evidence that the Fedora key is authentic.

Let's retrieve a candidate key into a local file:

[herrold@centos-5 rpmlint]$ wget -O Fedora-key \
[herrold@centos-5 rpmlint]$

We locally compute the 'fingerprint' of that key file, to give us some assurance that there was not some hostile MitM when we did this non-SSL retrieval.

[herrold@centos-5 rpmlint]$ gpg Fedora-key
pub 1024D/4F2A6FD2 2003-10-27 Fedora Project
sub 1024g/FB939E34 2003-10-27
[herrold@centos-5 rpmlint]$

and the anticipated 'pub' fingerprint value is returned: 0x4F2A6FD2

But is it really, really that of Fedora? Looking further at the MIT keyserver website, I see that I 'know' and trust another of the cross-signers of that key:

Public Key Server -- Verbose Index ``0x4F2A6FD2 ''

Type bits /keyID Date User ID
pub 1024D/4F2A6FD2 2003/10/27 Fedora Project
... snip ...
sig DB42A60E Red Hat, Inc
... snip ...

I happen to have a chain of pressed original distribution CDs from Red Hat (which have carried its then current key on that immutable media for many years, over many generations). It makes sense, at a trade show, to obtain and keep such a CD, often handed out as a 'sample' for the key it contains. Since Red Hat Security is a endorsing signer of that Fedora key, and since I can verify an unbroken chain forward to the present values at the Red Hat Security key page, I am comfortable at the authenticity of the key offered at the MIT keyserver under that fingerprint.

That is, I am now willing to concur that content signed with a key producing that fingerprint is authentic content from the Fedora project.

As such, we temporarily assume root rights to get write access to the RPM database, and import the key held in a local file into that RPM database:

[herrold@centos-5 rpmlint]$ sudo rpm --import  Fedora-key
[herrold@centos-5 rpmlint]$

[herrold@centos-5 rpmlint]$ rpm -qa gpg\*
[herrold@centos-5 rpmlint]$ rpm -qi gpg-pubkey-4f2a6fd2-3fcdf8c9
Name : gpg-pubkey Relocations: (not relocatable)
Version : 4f2a6fd2 Vendor: (none)
Release : 3fcdf8c9 Build Date: Tue 09 Sep 2008 12:52:19 PM EDT
Install Date: Tue 09 Sep 2008 12:52:19 PM EDT Build Host: localhost
Group : Public Keys Source RPM: (none)
Size : 0 License: pubkey
Signature : (none)
Summary : gpg(Fedora Project )
Description :
Version: rpm-4.4.2 (beecrypt-4.1.2)


[herrold@centos-5 rpmlint]$

RPM is being careful there, dumping the entire key value.

Let's see that process again without the key dump:

[herrold@centos-5 rpmlint]$ rpm -Kv rpmlint-0.82-3.fc9.src.rpm
Header V3 DSA signature: OK, key ID 4f2a6fd2
Header SHA1 digest: OK (8d753729ecfc59620fca2ccc1018e767787da9a6)
MD5 digest: OK (7d8b212cc68ff675fd64b124d832c423)
V3 DSA signature: OK, key ID 4f2a6fd2
[herrold@centos-5 rpmlint]$ rpm -K rpmlint-0.82-3.fc9.src.rpm
rpmlint-0.82-3.fc9.src.rpm: (sha1) dsa sha1 md5 gpg OK
[herrold@centos-5 rpmlint]$ sudo rpm -e gpg-pubkey-4f2a6fd2-3fcdf8c9
[herrold@centos-5 rpmlint]$ rpm -K rpmlint-0.82-3.fc9.src.rpm
rpmlint-0.82-3.fc9.src.rpm: (SHA1) DSA sha1 md5 (GPG) NOT OK (MISSING KEYS: GPG#4f2a6fd2)
[herrold@centos-5 rpmlint]$ sudo rpm --import Fedora-key
[herrold@centos-5 rpmlint]$ rpm -K rpmlint-0.82-3.fc9.src.rpm
rpmlint-0.82-3.fc9.src.rpm: (sha1) dsa sha1 md5 gpg OK
[herrold@centos-5 rpmlint]$

Notice that difference in how RPM reports matters, with the key present [gpg OK], and removed [(GPG) NOT OK (MISSING KEYS: GPG#4f2a6fd2)] from the RPM database.

A careful automated build system can complain and stop, when offered a SRPM bearing an unknown signature. I outlined (and implemented) such a process some years ago for the cAos community distribution effort.

Recent efforts to reach a buildsystem to pull 'on the fly' from CVS head [I am thinking here of Mark Shuttlesworth's keynote address at OLS this summer, and of some comments within Fedora's -devel mailing list] or other version control system are all well and good, but when the time comes for the official external build for record, I prefer a system where the entire chain of binaries produced are only from signed packages and signed by persons I trust.

28 August 2008

... letters, we get letters

To : (elided)
Cc : CentOS security role account
Subject : vulnerability cache poisoning in bind-9.3.4-6.0.2.P1.el5_2
----- Message Text -----
On Thu, 28 Aug 2008, (elided) wrote:

> I haven't found any update to the bind software in the
> repositories. Is it necessary to download the source of bind
> version 9.5.x and compile it?
> S.O CentOS 5.2

No; CentOS uses the RPM packaging management system, and 'yum' (which itself uses the 'rpm' programs). This issue has been addressed already for people running updates regularly.

You do not mention the CVE you are concerned about. This is
the process to see the most recent updates as to CVE's.

The RPM package manager permits you to view what has been
addressed in recent time thus:

~]$ rpm -q --changelog bind | \
  grep -i cve | tac | tail
- added upstream patch for correct SIG handling - CVE-2006-4095
- added fix for #225229 - CVE-2007-0494 BIND dnssec denial of service
- added fix for #224445 - CVE-2007-0493 BIND might crash after
- fixed cryptographically weak query id generator (CVE-2007-2926)
- CVE-2007-6283 (#419421)
- CVE-2008-0122 (small buffer overflow in inet_network)
- CVE-2008-1447

and then viewing:


Obviously, I used some command line tools to winnow down the
mass of Changelog; one could feed it to '| less' as well.

Placing: 2008-0122 into the: Search Master Copy of CVE, we

(under review)

Learn more at National Vulnerability Database (NVD)
• Severity Rating • Fix Information • Vulnerable Software
Versions • SCAP Mappings

The DNS protocol, as implemented in (1) BIND 8 and 9 before
9.5.0-P1, 9.4.2-P1, and 9.3.5-P1; (2) Microsoft DNS in Windows
2000 SP4, XP SP2 and SP3, and Server 2003 SP1 and SP2; and
other implementations allow remote attackers to spoof DNS
traffic via a birthday attack that uses in-bailiwick referrals
to conduct cache poisoning against recursive resolvers,
related to insufficient randomness of DNS transaction IDs and
source ports, aka "DNS Insufficient Socket Entropy
Vulnerability" or "the Kaminsky bug."


which is the recent Kaminsky bug. As it is mentioned, we see
it was addressed by CentOS in:

~]$ rpm -q bind
~]$ rpm -q bind

Thanks for asking.

22 August 2008

GnuPG -- A few minutes on using detached and clearsigned content

This is a re-formatted [and typo reduced ;) ] version, re-laid for the blogging software, of a post I made to the main CentOS mailing list earlier today. A test copy to verify of this which will properly verify is here, and may be retrieved with wget.

A few minutes on using detached and clearsigned content.

In light of today's CVE-2007-4752 by the CentOS project's upstream:

I issue this brief piece on using GnuPG

1. View a proposed key to use, at the MIT keyserver

from: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x650D5882

2. Copy and create a local instance

[herrold@centos-5 redhat]$ vi rht-key

[herrold@centos-5 redhat]$ gpg --import rht-key
gpg: key 650D5882: duplicated user ID detected - merged
gpg: key 650D5882: public key "Red Hat, Inc. (Security Response Team)
" imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: 3 marginal(s) needed, 1 complete(s) needed, classic trust model
gpg: depth: 0 valid: 2 signed: 5 trust: 0-, 0q, 0n, 0m, 0f, 2u
gpg: depth: 1 valid: 5 signed: 2 trust: 0-, 0q, 0n, 1m, 4f, 0u
gpg: next trustdb check due at 2009-03-14

3. Compute a local fingerprint of the candidate

[herrold@centos-5 redhat]$ gpg --fingerprint 650D5882
pub 1024D/650D5882 2001-11-21
Key fingerprint = 9273 2337 E5AD 3417 5265 64AB 5E54 8083 650D 5882
uid Red Hat, Inc. (Security Response Team)

sub 2048g/7EAB9AFD 2001-11-21

[herrold@centos-5 redhat]$

4. Compare and validate the fingerprint of the candidate against the RHT statement of the same fingerprint:


5. You do NOT need to accept a key permanently to check signed content purportedly with it; consider the Red Hat notice at:

6. We can retrieve the checking script

wget https://www.redhat.com/security/data/openssh-blacklist-1.0.sh

and the (presumptively) signed checksum of that file

wget https://www.redhat.com/security/data/openssh-blacklist-1.0.sh.asc

This is called a detached signature

7. And then we can validate ('--verify') that the signature and the file were signed by a person in possession of the private key.

Hopefully that private key is itself protected, as behind one way firewalls, and with a 'pass phrase' which matches a known public (which we retrieved and added earlier). This procedural security process is followed by me [one way firewalls, and pass phrases, and other CentOS team members], along with other measures.

[herrold@centos-5 redhat]$ gpg --verify openssh-blacklist-1.0.sh.asc openssh-blacklist-1.0.sh

gpg: Signature made Fri 22 Aug 2008 05:02:29 AM EDT using DSA key ID
gpg: Good signature from "Red Hat, Inc. (Security Response Team)
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the
Primary key fingerprint: 9273 2337 E5AD 3417 5265 64AB 5E54 8083 650D 5882
[herrold@centos-5 redhat]$

8. As we have not indicated to gpg that we permanently trust this key, gpg adds the WARNING -- this is expected and correct under this outline. The validation checks out.

9. This file can be clearsigned -- the process we will follow is this:

[herrold@centos-5 .gnupg]$ gpg --clearsign import-key-howto.txt

You need a passphrase to unlock the secret key for
user: "R P Herrold "
1024-bit DSA key, ID 9B649644, created 2003-02-09

File `import-key-howto.txt.asc' exists. Overwrite? (y/N) y
[herrold@centos-5 .gnupg]$

10. That is, import-key-howto.txt is clearsigned, and a new file,
import-key-howto.txt.asc, is produced. As I did it twice, to add this text, the warning about Overwriting a file appeared.

11. This is a non-detached (clearsigned, file, and might also be tested by retrieving the indicated key contents, and doing a '--verify'

12. As I have previously certified my own key, I can do it more simply locally:

[herrold@centos-5 .gnupg]$ gpg --verify import-key-howto.txt.asc
gpg: Signature made Fri 22 Aug 2008 12:37:39 PM EDT using DSA key ID
gpg: Good signature from "R P Herrold "
[herrold@centos-5 .gnupg]$

Note that the TIME of the signing will vary, as I have to resign the file after adding this content.

13. Previously (prior to 22 Aug 2008), I have included my PGP details in every piece of email I send. Starting today, as to email originate; I will add another line with my GPG details as well. I will send this document to the main centos mailing list.

Date: Thu, 21 Aug 2008 17:43:28 -0400 (EDT)
From: R P Herrold
To: trading-shim general mailing list
Subject: segmentation faults
In-Reply-To: <1219351509.12150.18.camel@gb07>
References: <200808202117.m7KLH4rf011059@pippin.first.lan>

User-Agent: Alpine 1.999 (LRH 1145 2008-08-19)
X-M: Go Blue
X-OpenPGP-Key-ID: 0x7BFB98B9
MIME-Version: 1.0

In pine (alpine), one does this with Customized X-headers:

Customized Headers = X-M: Go Blue
X-GnuPG-GPG-Key-ID: ox9B649644
X-OpenPGP-Key-ID: 0x7BFB98B9

[hmmm -- a typo: o for 0 in the GnuPG line -- I'll fix that in alpine]

This piece intentionally does not address CentOS response; a preliminary statement on this has been posted in the /topic of the IRC channel #centos on irc.freenode.org, and I have done a blog posting which is up at: http://planet.centos.org/

- -- Russ herrold

CVE-2007-4752 and CentOS

wearing my 'security@centos.org' hat, I have changed the IRC topic temporarily:

11:47 orc_orc changed the topic of #centos to: updated 22 Aug 2008 CentOS acknowledge CVE-2007-4752 and are reviewing our build and signing processes and hosts for signs of tampering subsequent to retrieval of SRPMs. // DO NOT PASTE IN HERE (unless asked; 1 line MAX), use http://pastebin.centos.org/ | See http://centos.org/irc | How to ASK a question: http://tinyurl.com/anel | CentOS mirrors:
http://centos.org/mirrors | Understanding Backporting:

and had to temporarily omit:

Current Releases: CentOS 5.2, 4.6, 3.9, 2.1 | CentOS 5.2 now released

20 August 2008

Let's get rid of disclaimers like this ...

... on mailing lists, as well. Or just subscribe and post from another email account. Or use more than a subject line to ask a question.

Email must be too hard for mere mortals to figure out.

Date: Wed, 20 Aug 2008 08:17:39 -0400
From: Mark T. Kennedy
To: quickfix developers
Subject: quickfix-d] is there a new bug/issue tracker?

QuickFIX Documentation:
QuickFIX Support: http://www.quickfixengine.org/services.html


This communication and any attachments may contain confidential/proprietary
information and is intended for information purposes only. It is not an
invitation or offer to purchase interests from Diamondback. Any
representation to the contrary is unintentional. This communication is
intended only for the person(s) to whom it is addressed. If you are not the
intended recipient you are hereby notified that you have received this
document in error and that any review, dissemination, distribution, or
copying of this message or any attachments is not permitted. If you have
received this in error, please notify the sender immediately by e-mail and
delete this message. All e-mails sent to or received from this address will
be received by Diamondback's company e-mail system and is subject to
archival and possible review by someone other than the recipient. This
notice is automatically appended to each e-mail message leaving Diamondback.

Where is my coffee cup, anyway?

12 August 2008

If a tree falls in the forest, and no one hears it, ...

... does it still make a sound?
-- folk equivalent of a Zen koan

hmmm. No smiley. Clearly sent by someone with a keen perception of the obvious:

Date: Tue, 12 Aug 2008 08:56:29 -0400
From: spamtools-owner @ lists.abuse.net
To: herrold @ owlriver.com
Subject: Spamtools recipient validation for herrold @ owlriver.com

This is a probe message to check the distribution of the spamtools list. Please let me know immediately if you did not receive this message.

John Levine, list meister

I'll hop right on getting that message out, right after another cup of coffee.

11 August 2008

"We're going to need another Timmy!"

Mr. Lizard, Dinosaurs

A running gag on that show, and in IRC, the same.

04:37 msivak> umga9pej
04:37 msivak> hups
04:37 msivak> time to change another password ;)

10 August 2008

score: pen one, orc zero

'Out, damned spot! out, I say.'

-- Lady Macbeth, Macbeth, Act V, Scene 1, Shakespeare

Came back from a trip out of town, and as is my usual custom, had all the dirty clothing on the top of the suitcase [for the TSA to appreciate digging through]. Now I am usually pretty careful to pull stray paper, change, and writing implements out of clothing as I disrobe. I missed a ball point pen this time, and in loading the laundry bin, missed it a second time.

We all know how this comes out, and indeed once the pen moved from the washer to the dryer, it opened up. Spots everywhere. Dr Suess would be proud, but no 'Voom' seems to be in our house. It spotted and gave its distinctive blue-black hue to the good towels, napkins, and other items which went through with a white summer weight cotton shirt in which pocket the pen was riding. There is probably nothing in the future of those towels than promotion to the 'rag box.'

But the issue remained of removing the ink from the dryer drum interior. I consulted Google, and a couple of commercial products were suggested, but it is Sunday, and I am not likely to go out again today. Household agents such as acetone (sometimes found in nail polish remover), denatured alcohol, Comet brand dry bleach powdered abrasive cleaner came to mind. Digging through the garage, I also came across an ether based starting fluid, and WD-40 brand spray lubricant.

Down to the dryer, and spot testing {ahem} began. Bottom line, alcohol on a paper towel and a bit of elbow grease triumphed.

Too late in the day for coffee, not late enough for Scotch. I'll go find a Miller Genuine Draft (bottled) in the 'fridge.

17 July 2008

"... I've got a bad feeling about this"

said Han to the Wookie.

When my aged mother starts asking about blogs, and how she can start using one, it is probably time to do a trial run for a while. But it doesn't mean I have to like it.

I am pretty non-emotional when participating in, and chan-op'ing in the #centos IRC channel at: irc.freenode.org. But we do get the same questions, day after day, by people who either won't read, or won't take the time to do anything more than leech the channel for a quick answer. Once fed their answer without having had to go through the process of learning the skills to dig an answer out, they promptly disappear ... until the next random problem happens to them.

All the expected resources of a mature FOSS project are there; and signs are posted as to channel expectations: a /topic banner, a web page specific to the IRC channel, a wiki, a forum, documentation, a bug tracker, man pages and .... Release Notes. But it is easier to burden an entire channel of several hundred people, than to look for the answer first. Classic 'tragedy of the commons'.

Sadly, even some channel regulars feed such bad behaviour, and so encourage it. It is just like leaving food open on the counters, not washing dishes, and then being surprised that roaches appear.

Don't even get me started on people seeking, and channel participants giving, answers on software we do not ship. I'm sure I'll see a writing prompt on that soon enough.

I think I'll go get a coffee. Dark roast, no cream, no sugar, and a large portion please.