Naming Names and 'Shaming'

I customarily run all the updates for the Windows 7 (64 bit) box at home on Sunday, but was traveling then, and so just ran some. Adobe's Flash Player (each no doubt asserted as trademarks) had a update notice in the bunch. I refer to those Marks in this paragraph for identification and not for other purposes. Disparagement comes later

So I attained Administrator (the Windows role enabled for installing software in the customary security model), and went through the process of starting their updater. This is a small intermediate program which retrieves the larger update, and then hands off execution to that later payload. AT NO TIME, have I or any other person with the knowledge needed to reach Admin rights, ever consented to the (seemingly always 'defaulted on') "Also install some third party 'enhancement'". It seems MacAfee and its Security Suite are the current firm, co-marketing of their products with Adobe. Again, I refer to those Marks in this paragraph for identification and not for other purposes. Disparagement comes later

Notwithstanding no consent, the McAfee kit was part of the payload that the Adobe retrieved ... and installed ... with no consent to such by me

Let's be frank. Adobe's Flash Player is a gaping security hole, with a well documented track record of poor coding. There are 356 using the search link. I am all for cleaning up issues and not hiding them, but this is just a whorehouse of fail, Adobe

The McAfee record for 'Security Suite' is less clear, as it seems to me that the nameing of their products is designed to confuse, and segment purchases, rather than to provide long term clarity of what the names, used as Marks, mean. The CVE count at MITRE is 137 by that search. But I see from a more general search engine query that McAfee seems to have licensed for distribution and re-naming, its product: 'AT&T Internet Security Suite - powered by McAfee' for instance, so one cannot rely on the Marks to reliability identify offenders. AOL seems to have licensed it as well

McAfee seems to have other lapses as well

I am just not sure what the remedy is:

• Stop using Windows? This is pretty much done with the maturity of the Open Source ecosphere -- we have NONE at the office facing the internet, and only ONE left at home or in the family members I support. For commercial software use, Apple's OS/X and hardware have been remarkably complete and their use uneventful without need to load them down with defensive 'crapware'
• Stop using Flash? Is the pain worth the short remaining life Adobe's Flash seems destined to have. I recall there was an effort to re-implement to the published Flash specification, and Adobe (to their credit is better than most commercial vendors on publishing file format documentation -- think: PSF and friends). time to investigate what the FOSS replacement is
• And of course: 'Shame' bundlers, and providers of 'ride-along' 'crapware'. So: Adobe, McAfee, and your products: Flash Player, and Security Suite -- consider yourself identified as bad actors, worthy of avoidance

Well, a start then

Yeah, it's a Monday ...

Last week, I made a trip up to the local computer store, the mothership of MicroCenter, and finally broke down and bought a USB/VGA four port KVM switch for my lead worksstation .... not DVI or HDMI on the video, but still VGA. The PS/2 mouse and keyboards were scheduled to be phased out, and a move to USB devices slated for this week

Well, the components to be affected .. the old KVM, the panel monitor, and so on must have held a pow-wow across the weekend, because when I came in this morning the panel monitor's backlight (a Westinghouse L2210NW, 1680x1050, datecode of April 2008) seems to be completely dead. This is of course the absolutely MOST inconvenient part of the display chain to die, because I need to run custom 'modeline' detail under X to squeeze the maximum resolution and sharpness out of the display. I had also purchased the three year 'no questions' replacement warranty on that four year old unit, so no help there

I grabbed a 'retired to the front bench' NEC AcuSync LCD223wxm, also nominally 1680x1050, off one of the benches and have been fiddling with the modeline settings to have a backup to limp through the day, but the horizontal height is wrong, the pixels puddgy, and the video muddled

No doubt I _could_ get it to stand up and dance, but the NEC has a datecode from early 2007 so that is a suckers game

Yeah, its a Monday

Coping with xz under the RPM tools in CentOS 5

So there I am, minding my own business, building a SRPM from Red Hat's 'rawhide' archive, and it fails. They are cutting over to 'xz' compression for the tarballs they ship. Their archive, and so their call, and not the end of the world

The symptom shows up when rpmbuild goes to uncompress such:

...
+ rm -rf clamav-0.96.4
+ tar -xf /home/herrold/rpmbuild/SOURCES/clamav/clamav-0.96.4-norar.tar.xz
tar: This does not look like a tar archive
tar: Skipping to next header
tar: Archive contains obsolescent base-64 headers
tar: Read 6508 bytes from /home/herrold/rpmbuild/SOURCES/clamav/clamav-0.96.4-norar.tar.xz
tar: Error exit delayed from previous errors
error: Bad exit status from /var/tmp/rpm-tmp.36477 (%prep)

RPM build errors:
Bad exit status from /var/tmp/rpm-tmp.36477 (%prep)
[herrold@centos-5


The error messages could be better, but the older compression methods that are known to the 'file' program available to rpmbuild that ships with CetnOS 5 do not contain the relevant 'magic numbers' yet. Progress is like that, and so until and unless Red Hat backports support into its RHEL sources, CentOS will not pick up the fix in its version 5 mainline

One perfectly suitable response is to use the RPM5 branch of the package manager, which DOES know. But some people cannot relax that constraint for various non-technical reasons

This issue is rather like the old cutover from md5sums to shasums which RPM did a while ago, and that I wrote aboutI wrote about

The fix is straightforward:

1. Install the compressed tarball, spec file and any patches with rpm in the usual fashion
2. Uncompress from the unknown compression format and re-compress with a known one
3. Amend the spec file; here, I use grep to look, and as there is just one edit, sed to edit
4. Rebuild using the '-ba' option from the revised .spec file with the tools of the target environment (here, CentOS 5)
5. The resulting SRPM will be portable and as a result of the second step, uses a known compression

Lets look:

[herrold@centos-5 clamav]$ unxz /home/herrold/rpmbuild/SOURCES/clamav/clamav-0.96.4-norar.tar.xz
[herrold@centos-5 clamav]$ gzip /home/herrold/rpmbuild/SOURCES/clamav/clamav-0.96.4-norar.tar
gzip: /home/herrold/rpmbuild/SOURCES/clamav/clamav-0.96.4-norar.tar.gz already exists; do you wish to overwrite (y or n)? y

[herrold@centos-5 clamav]$


That question about over-writes happened because it appears the sources from a prior build of clamav-0.96.4 were not re-rolled into a 0.96.5 tarball by the upstream packager at RawHide, but may have been patched instead. I've not expressly looked

[herrold@centos-5 clamav]$ cp ~/rpmbuild/SPECS/clamav.spec .
[herrold@centos-5 clamav]$ rpmbuild -ba clamav.spec
error: File /home/herrold/rpmbuild/SOURCES/clamav/clamav-0.96.4-norar.tar.xz: No such file or directory
...


As we have not yet fixed the .spec file, this was expected, but is shown here so the diagnosis path is clear

[herrold@centos-5 clamav]$ grep xz clamav.spec
Source0: %name-%version%{?prerelease}-norar.tar.xz
[herrold@centos-5 clamav]$ sed -i -e 's@xz@gz@g' clamav.spec


And now the .spec file is ready as well

[herrold@centos-5 clamav]$ rpmbuild -ba clamav.spec
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.47404
+ umask 022
+ cd /home/herrold/rpmbuild/BUILD
+ LANG=C
+ export LANG
+ unset DISPLAY
+ cd /home/herrold/rpmbuild/BUILD
+ rm -rf clamav-0.96.4
+ /bin/gzip -dc /home/herrold/rpmbuild/SOURCES/clamav/clamav-0.96.4-norar.tar.gz
+ tar -xf -
...
checking host system type... x86_64-redhat-linux-gnu
checking target system type... Invalid configuration `noarch-redhat-linux-gnu': machine `noarch-redhat' not recognized
configure: error: /bin/sh config/config.sub noarch-redhat-linux-gnu failed
error: Bad exit status from /var/tmp/rpm-tmp.86669 (%build)
...


The build fails for other reasons out of scope for this post, in that a new configure 'target' is emitted. This is similar to a later compression format addition, but a different problem, solved elsewhere. Such a change is another part of distribution and brand management matters at Red Hat's part. I'll note the solution for this (putting to side seriously amending the rpm build environment macros, which is the 'one way' path into later versions) in a later post

Once all the changes are done, and the 'scratch' test builds and will install cleanly, I go in with an editor, manually bump the release value by one, and add a note in the changelog stanza. Then I repeat the build 'for record', signing, and distribution. The Release 'bump' is needed so the NEVR (name, Epoch, Version, and Release comparison which librpm does, and that yum calls through librpm to do can detect the fact that a later updated version is in an updates repository in due course

All done

lost in the bowels of Google Groups

A post I made earlier today to a mailing list seems to have been held up for an hour, even though I am a subscriber to the mailing list in question, have proper and meticulously preened DNS A, PTR, MX, and even TXT records, publishing SPF details properly, because of prior problems with Google's mailservice's erroneous markings of some pieces as 'spammy' in the past ...

Received: by 10.90.14.22 with SMTP id 22mr127029agn.36.1285171616911;
        Wed, 22 Sep 2010 09:06:56 -0700 (PDT)
X-BeenThere: puppet-users@googlegroups.com
Received: by 10.91.83.8 with SMTP id k8ls391483agl.0.p; Wed, 22 Sep 2010
 09:06:54 -0700 (PDT)
Received: by 10.150.51.21 with SMTP id y21mr255924yby.58.1285171614696;
        Wed, 22 Sep 2010 09:06:54 -0700 (PDT)
Received: by 10.229.192.137 with SMTP id dq9mr33711qcb.14.1285167411800;
        Wed, 22 Sep 2010 07:56:51 -0700 (PDT)
Received: by 10.229.192.137 with SMTP id dq9mr33709qcb.14.1285167411749;
        Wed, 22 Sep 2010 07:56:51 -0700 (PDT)
Received: from bronson.owlriver.com (bronson.owlriver.com [198.49.244.50])
        by gmr-mx.google.com with ESMTP id
    c41si5677929qcs.12.2010.09.22.07.56.51;
        Wed, 22 Sep 2010 07:56:51 -0700 (PDT)
Received-SPF: pass (google.com: best guess record for domain of
    herrold@owlriver.com designates 198.49.244.50 as permitted sender)
    client-ip=198.49.244.50;
Received: from localhost (localhost.localdomain [127.0.0.1])
    by bronson.owlriver.com (8.13.8/8.13.8) with ESMTP id o8MEumOR020433
    (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
    Wed, 22 Sep 2010 10:56:49 -0400
Date: Wed, 22 Sep 2010 10:56:48 -0400 (EDT)

... anti-spam measures, one assumes. I understand taking such measures, but sure wish the scoring 'downticks' Google was marking were published and findable (compare, to the good: AOL's current practices)

But then, I am told from time to time that my world view and some of my approaches are 'too utopian'. Humph -- a little bit 'utopian is all right, but one can overdo it? Who knew?'

"Okay, not a problem"

It drives me nuts in a store or when contacting telephone support somewhere, when the clerk or call center denizen replies to my social courtesy of thanking them for some service, to receive in return:

Okay, not a problem

D*mn it -- In such a circumstance, I have usually just made a purchase, or have previously paid good money to get their firm's attention. I could care less if they were pleased to not to have had to work hard doing their appointed tasks. I know darn well they are drawing some salary to boot

I rather feel that I am entitled, instead, to:

Thank you

or perhaps,

You are welcome and it was a pleasure

as the back and forth of the interaction suggests

Oh, yes, and "No worries" usually works just about as well with me, except when used as an affirmation that all is well

Running down stray errors

Part of my daily routine is to check the logwatch summary, note and address any security matters, and then to chip away at the friction and non-working parts of the compute environments in which I can effect change

This one has been on my radar for a while, but it is on a protected interior machine, not disabling, and so not critical. From a configuration file review, and with reading of the sendmail and openssl documentation, and some 'googleing' I just could not see where the error was.

**Unmatched Entries**
STARTTLS=client, error:
SSL_CTX_use_certificate_file(/etc/mail/certs/xps400.first.owlriver.net-10.pem) failed: 173 Time(s)
STARTTLS=client, error: SSL_CTX_check_private_key
failed(/etc/mail/certs/xps400.first.owlriver.net-10.key): 0: 173 Time(s)

To test if sendmail is compiled with STARTTLS support, we can run the following command:


$ sendmail -bt -d0.8 < /dev/null

Compiled with: DNSMAP HESIOD HES_GETMAILHOST LDAPMAP LOG MAP_REGEX
MATCHGECOS MILTER MIME7TO8 MIME8TO7 NAMED_BIND NETINET NETINET6
NETUNIX NEWDB NIS PIPELINING SASLv2 SCANF STARTTLS TCPWRAPPERS
USERDB USE_LDAP_INIT

more concisely:


[root@xps400 certs]# sendmail -bt -d0.8 < /dev/null | grep -i tls
NETUNIX NEWDB NIS PIPELINING SASLv2 SCANF SOCKETMAP STARTTLS

And, yet when we connect to the mailserver to test if STARTSSL is advertised

[herrold@centos-5 ~]$ telnet xps400 25
Trying 10.16.1.112...
Connected to xps400.first.lan (10.16.1.112).
Escape character is '^]'.
220 xps400.first.owlriver.net ESMTP Sendmail 8.14.3/8.14.3; Thu, 8 Apr 2010 14:41:48 -0400
EHLO localhost
250-xps400.first.owlriver.net Hello centos-5.first.lan [10.16.1.101], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE
250-DSN
250-ETRN
250-AUTH GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN PLAIN
250-DELIVERBY
250 HELP
quit
221 2.0.0 xps400.first.owlriver.net closing connection
Connection closed by foreign host.
[herrold@centos-5 ~]$

openssl has the rather interesting sub-tool s_client 'SSL/TLS client program' which knows how to talk several protocols though a transition into a secure sockets mode as well

[root@xps400 ~]# openssl s_client -connect localhost:25 -starttls smtp
CONNECTED(00000003)
didn't found starttls in server response, try anyway...
2005:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:583:
[root@xps400 ~]#

so ... starttls is linked in sendmail as present, but is not working

Let's run this down by making sure all the needed moving parts are present:

[root@xps400 mail]# cd /etc/init.d/
[root@xps400 init.d]# ls *sasl*
saslauthd
[root@xps400 init.d]# chkconfig --list saslauthd
saslauthd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
[root@xps400 mail]# /sbin/service saslauthd restart
Stopping saslauthd: [ OK ]
Starting saslauthd: [ OK ]
[root@xps400 mail]# /sbin/chkconfig saslauthd on
[root@xps400 mail]# /sbin/service sendmail restart

and from another panel watching the log files:

# tail -f /var/log/maillog
Apr 8 11:39:30 xps400 sendmail[3536]: STARTTLS=server, error: SSL_CTX_use_certificate_file(/etc/mail/certs/xps400.first.owlriver.net-10.pem) failed
Apr 8 11:39:30 xps400 sm-msp-queue[3547]: starting daemon (8.14.3): queueing@01:00:00

.. so ... sendmail is telling us that it refuses to use: /etc/mail/certs/xps400.first.owlriver.net-10.pem Looking at the certificate countersign:

# less /etc/mail/certs/xps400.first.owlriver.net-10.pem
N CERTIFICATE-----
MIIHATCCBemgAwIBAgICFokwDQYJKoZIhvcNAQEFBQAwgYwxCzAJBgNVBAYTAklM
MRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRh
...

a HA !!

Looks like it was a bad 'scrape and paste' by me when I retrieved and installed the counter-signing of the certificate from startssl. The start of that file should look like:


-----BEGIN CERTIFICATE-----
MIIHATCCBemgAwIBAgICFokwDQYJKoZIhvcNAQEFBQAwgYwxCzAJBgNVBAYTAklM
MRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRh
bCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAy
...

As always, such mistakes are only obvious once found.

---

A side observation. A recent blog bost "Securing the Enterprise" by Eddy Nigg of Startssl pointed out the willingness of some Certificate Authorities to sign whatever they are offered, and some admins to not consider this attack vector in submitting such, in the case of hosts in an RFC-1918 IP assignment block or non-DNS mediated formal namespace. Undetected forgeries are facilitated, and I am certain such Man in the Middle compromises occur in coffee-shops with wireless access all the time. As such the host: xps400.first.lan at 10.16.1.112 also appears with a internal split DNS PTR and A record as: xps400.first.owlriver.net The certificate for it countersigned by startssl is useful here for verifications

[herrold@centos-5 ~]$ host xps400.first.owlriver.net
xps400.first.owlriver.net has address 10.16.1.112
xps400.first.owlriver.net mail is handled by 20 mailhub.owlriver.net.
xps400.first.owlriver.net mail is handled by 10 new.owlriver.com.
[herrold@centos-5 ~]$

The trick to getting the mailserver to answer as xps400.first.owlriver.net was even easier -- just some DNS work, and a quick addition of a non-customary line in the /etc/mail/sendmail.mc, a rebuild, and a restart:

[herrold@xps400 mail]$ grep xps400.first.owlriver.net sendmail.mc | head -1
define(`confDOMAIN_NAME', `xps400.first.owlriver.net')dnl
[herrold@xps400 mail]$

Are you using SSL certificates where you can and should? ... Is the namespace of network they protect thoughtfully designed? StartSSL makes it easy to do, for a person willing to be minimally 'validated' as to their identity and their right to administer a given domain. Once that identity check is done, the process is essentially free of any marginal cost to roll out as many certificates as one wishes, and to NOT 'cop out' or cut corners here

Rainy Days & Mondays

Karen Carpenter made the song famous for its authors, but clearly none of them were sysadmins

The rule, long known, for sysadmins is:

Never make a major change on a Friday, nor before leaving for vacation

I've been wrestling with the fallout from a violation of the sysadmin's rule by an upstream provider -- the vendor pushed in some change on Friday in the preparation of CDR -- Call Detail Records. For four days running, my sub-processes which manage the account have been failing for want of data. Those processes retrieve and apply CDR data, to emit accounting detail for customers, and have not been working

I've filed five or six sub-issue tickets which that primary change exposed, in trying to get the matter resolved: The current Firefox cannot open tickets under the current Windows XP, current SP [no problems with CentOS and FireFox or konqueror]; my 'closed' tickets were not visible; tickets were being closed by upstream before I confirmed a fix worked, so I ended up essentially re-opening the same ticket three times as each day's CDR pull failed; I was not receiving email updates of tickets; and so on. I am quite sure they consider me a 'stickler for details' and something of a pedantic pest at the moment, but dammit, I'm paying their bills. The PHB supervisor may want tickets closed quickly; but I want my issues fixed first

... as no one likes to be called into work on the weekend to revert a change, the sysadmin's rule must be faithfully applied

Afraid of experimentation

The #centos IRC channel at irc.freenode.net never ceases to amaze me. We get questions that would take at least 30 seconds of reading a man page and experimention to answer, asked over and over again.

Here is one of the latest:

16:14 clueless> I installed a Windows Vista Business x64 VM on 5.2. Is it possible to get hibernate/sleep to work?
16:15 clueless> I want only want the VM up occasionally and I'd rather not wait for a full boot every time

Firing up a xen virtual machine in a root panel, and popping open another to read the xm man page, I find:

# cd /etc/xen
# ls
# xm create win-2000pro
Using config file "./win-2000pro".
Started domain win-2000pro
# virt-viewer win-2000pro

and a Windows 2000 session appears. I let it boot to the login prompt, and then:

# cd /var/lib/xen
# xm list
# xm pause win-2000pro
# xm save win-2000pro win-2000pro-save.img

Which of course as the man page promises, terminates the running image. Then:

# xm restore win-2000pro-save.img
# xm list
# xm unpause win-2000pro
# virt-viewer win-2000pro

And we are right were we left off, at the initial log in prompt.

Is it so hard to at least pretend to look first?

IPv6 eats kittens (and distcc) on Debian Testing

This can only end badly

I spent a good 5 hours this week, tracking down a problem with distcc hanging up in our Debian Testing build farm. We use distcc to speed up compilation of the c++ sources in the development of the trading shim. Interestingly, our end user community forced us to this decision of developing on Debian testing, as they are using later gcc versions than we were on CentOS, and it was useful to be able to see their errors, BEFORE they reported them to us

On the new compile farm, sometimes we would get a compile in, say, 44 seconds; other times it would drag out for several minutes. This is a problem as we had just slotted a new unit into harness, and expected better results

In checking the logs in the client doing the distribution of compilation tasks, we were seeing a symptom of 'segfaults' in that client's process; other times, the client would stall, seemingly blocked waiting for a compilation result to come back from a remote buildfarm peer, that never came back. Checking on the remote build unit, one of the distccd children would die for mysterious reasons, leaving a message in the dmesg record. Once that failed build timed out, the needed file would be built locally, and the build proceed. Checking the log files nothing obvious jumped out

The obvious debugging technique is to get a minimal reproducer, and then to partition the problem into smaller and smaller possible causes using that reproducer tool. the issue will manifest on one setup, but not the other, ans so one can rule out more and more issues, until the answer is left, staring you in the face

Looking at my Debian helper tool, it had rotted, and was in sorry need of removal of some constraints: It did not use distcc when available; it did not use proper -J parallel compiles; it did not use -O3 optimization in the compiles. My test tool was not set up to see what I needed to see

Time to pay down some 'technical debt' (If you've not read martinfowler piece, and viewed Ward Cunningham's video, stop now, and do so). And so I made some payment there. After testing, I got these results:

Master	Clients	Elapsed time (real)
pippin	nfs2, 10.16.1.231	0m23.281s
nfs2	10.16.1.231, pippin, localhost	0m23.702s
10.16.1.231	pippin, nfs2, localhost	0m22.551s

My first thought looking at this: Well, that pretty conclusively rules out machine specific errors, or network path issues. It must be something different in the setup of the user provoking the issue that my tool does not duplicate. NOTE: This is wrong-headed, of course, as: 'An absence of evidence is not evidence of absence of a problem' but was an easy trap to fall into

For every complex problem, there is a solution that is simple, neat, and wrong.

  — H. L. Mencken

For every problem there is a solution which is simple, obvious, and wrong."

  — Albert Einstein

I tossed my results at that user for their thoughts on the results, and went back to work on another issue

Later in the day, doing some thought experiments with the user, we could not pin down where to look yet. But as a team, I had him provoke the issue with his setup, while I watched the logs on the various machines through several consoles. And the error appeared, and then jumped out and tickled my eyeballs. I was watching nothing in particular, until I saw the failure on process 29673, and then traced that back up. A successful and a failed session looked like this, respectively:

distccd[29673]	(dcc_check_client)	connection from :ffff:10.16.1.249:41771
distccd[29673]	(dcc_r_file_timed)	909179 bytes received in 0.078651s, rate 11289 kB/s
distccd[29627]	(dcc_collect_child)	cc times: user 1.132070s, system 0.144009s, 23039 minflt, 0 majflt
distccd[29673]	(dcc_collect_child)	cc times: user 1.092068s, system 0.104006s, 22481 minflt, 0 majflt
distccd[29673]	(dcc_check_client)	connection from ::ffff:10.16.1.249:41775
distccd[29673]	(dcc_r_file_timed)	818437 bytes received in 0.071648s, rate 11155 kB/s
distccd[31248]	(dcc_check_client)	connection from ::ffff:10.16.1.249:41779
distccd[31248]	(dcc_r_file_timed)	886761 bytes received in 0.076688s, rate 11292 kB/s
distccd[29627]	(dcc_collect_child)	cc times: user 1.068066s, system 0.112007s, 23890 minflt, 0 majflt
distccd[29673]	(dcc_collect_child)	cc times: user 1.108069s, system 0.112007s, 22012 minflt, 0 majflt
distccd[29673]	(dcc_pump_sendfile)	Notice: sendfile: partial transmission of 15868 bytes; retrying 344332 @15868
distccd[1995]	(dcc_log_child_exited)	ERROR: child 29673: signal 11 (no core)

A-ha! Now we know what to look for:

dhcp-231:/var/log# grep dcc_pump_sendfile distccd-transition-log
distccd[29673] (dcc_pump_sendfile) Notice: sendfile: partial transmission of 15868 bytes; retrying 344332 @15868
distccd[31248] (dcc_pump_sendfile) Notice: sendfile: partial transmission of 15868 bytes; retrying 586732 @15868
distccd[30262] (dcc_pump_sendfile) Notice: sendfile: partial transmission of 15868 bytes; retrying 4655916 @15868
distccd[2005] (dcc_pump_sendfile) Notice: sendfile: partial transmission of 16384 bytes; retrying 74824 @16384
distccd[2128] (dcc_pump_sendfile) Notice: sendfile: partial transmission of 16384 bytes; retrying 286560 @16384
distccd[2170] (dcc_pump_sendfile) Notice: sendfile: partial transmission of 16384 bytes; retrying 97440 @16384
distccd[2129] (dcc_pump_sendfile) Notice: sendfile: partial transmission of 16384 bytes; retrying 301000 @16384
dhcp-231:/var/log#

The TCP process of shuttling code to compile, and the binary results of such compiles are failing the same way, over and over again: partial transmission of 15868 bytes is present every time. Looking at the log entry again, the form of the connecting hosts is unusual: ::ffff:127.0.0.1 and ::ffff:10.16.1.249. Why that is IPv6 notation? And I reach back to my logs as I remember I had an issue like this a year or so on a Debian box

And so, Google with the search argument: debian ipv6 distcc confirms as its first result: 1. #481951 - distcc: zeroconf support broken wrt IPv6 - Debian Bug ... ... and the bug is still open. Killing off IPv6 is the obvious next step, and so, back to Google with: debian disable IPv6 to find: Disabling IPv6 under a 2.6 kernel. Reading the post, there is some back and forth, and the answer seems to be, there is not an 'official Debian answer', but this is what people are doing. Back to Google with: site:debian.org debian disable IPv6 seems to confirm that there is not a single well documented answer which has floated up in Google's searching

Compare: CentOS addresses the matter directly, and as the first Google hit with: site:centos.org disable IPv6
7. How do I disable IPv6?

* Edit /etc/sysconfig/network and set "NETWORKING_IPV6" to "no"
* Add the following to /etc/modprobe.conf :

alias ipv6 off
alias net-pf-10 off

* Run chkconfig ip6tables off to disable the IPv6 firewall
* Reboot the system

Alternative (which might be easier and works on any release with /etc/modprobe.d):
echo "install ipv6 /bin/true" > /etc/modprobe.d/disable-ipv6

---

Sadly, there is something else on Debian testing in play as well, and it is not just an IPv6 issue (although turning off IPv6 has drastically reduced the frequency of the issue). When I look in today to make sure the 'fix' is working


[74988.951989] distccd[8671]: segfault at 1 ip 7fdd2250e030 sp 7fff2b025da8 error 4 in libc-2.7.so[7fdd22493000+14a000]
[74989.017836] distccd[8651]: segfault at 1 ip 7fdd2250e030 sp 7fff2b025da8 error 4 in libc-2.7.so[7fdd22493000+14a000]
[74989.518050] distccd[8664]: segfault at 1 ip 7fdd2250e030 sp 7fff2b025da8 error 4 in libc-2.7.so[7fdd22493000+14a000]
[74994.152461] distccd[8659]: segfault at 1 ip 7fdd2250e030 sp 7fff2b025da8 error 4 in libc-2.7.so[7fdd22493000+14a000]

Where is that coffee cup? I knew this would not end well

People do go both ways

There is a scene depicted in the movie: 'Battle of the Bulge' (1965) about the 1944 attempted German breakout offensive through the Ardennes, where German commandos are tasked with and shown changing road signs to confuse Allied troops

When I started this blog, it was in response to a desire to make the CentOS internals a little more transparent to interested observers. We at the project do get the questions, and I think a thoughtful reader can pull connections from the little stories and examples I choose from the full breadth of the blog. While I might 'tag' something specifically 'CentOS', real life has no such natural boundaries, and these are just guide markers in the channel of life.

I added the blog into the CentOS aggregator at planet.centos.org, and set to writing. I cribbed the configs from an example of another CentOS member. I tried then to restrict the feed to the 'CentOS' label, but following the documentation just did not work. I settled for the default full feed, and resolved to solve the revisit the matter later

My friend toracat gently reminded me of the need to finish the job, this morning. Sigh ... back to wrestling markup

The example follows [there are annoying line breaks in the blog layout as rendered, and indeed in the doco upstream that need to be pasted back together, mentally]. Can you spot the error?

Full site feed: Atom 1.0: http://blogname.blogspot.com/feeds/posts/default RSS 2.0: http://blogname.blogspot.com/feed/post/default?alt=rss   ... Label-specific site feed: Atom 1.0: http://blogname.blogspot.com/feeds/comments/default/-/labelname RSS 2.0: http://blogname.blogspot.com/feeds/comments/default?alt=rss/-/labelname   ... Individual post comment feed: Atom 1.0: http://blogname.blogspot.com/feeds/postId/comments/default RSS 2.0: http://blogname.blogspot.com/feeds/postId/comments/default?alt=rss

There is the obvious need to s/comments/posts/g, but more is needed. I am accustomed to 'magic CGI directories' that accept variables. I use them myself. See, e.g., the expanded URL to the thumbnail of Mothra which is not just an image, but the filename, and a link to the full size one. No express CGI script is called out, as the index file for that directory is actually a smart CGI script

Enough clues, and on to the answer. I put bit of text around the answer so your eyes do not pick it out. The text at the fourth bullet above is malformed ... the part following: alt=rss needed to precede the question mark marker that identified the start of variables to the CGI script. We move before it the part: /-/labelname and add the desired label. Now a custom subfeed chosen by label is properly specified

But there are no road signs on the Blogger provided doco page to permit easily reporting errors, so that they might have be fixed

Nuts

Every step you take ...

I received the above email [which I converted to a maskable image], with embedded web link, seeking market research data. I have masked the full URL, to prevent 'ballot box stuffing' and to protect my privacy Now in doing good statistical sampling, customarily one assures the recipient / respondent that the responses are aggregated, and that no personally identifying information is available to the researcher. This is done to foster truthfulness and frankness from people responding to the survey, by reassuring them that no information leaks, say back to the entity covered by the survey can tie particular positive or negative 'pull comments' to a specific person

Other survey research techniques use 'calibration' questions, repeated in slightly varying form a couple of times in the survey, to make sure the respondent is actually reading the questions, is answering consistently, matches the 'shaped sample' desired demographic, and similar concerns

Here, I am solemnly (or perhaps, cheerfully) assured:

We will also gladly share the aggregate results of the survey with you, as it may be of interest to you.

All responses will remain anonymous and confidential.

What is does not say is that the author is not planning to use the data for selling 'individual drill down' detail by respondent

The sender is sort of aware of this, or perhaps it is just a boilerplate footer from SurveyMonkey:

This link is uniquely tied to this survey and your email address, please do not forward this message.

I think I will pass on this one. Time for more coffee

---

Revised to lay better in the top table 20 march 2009

Wrestling blog markup, Round Three

Lovely ... the large image in the prior post boogered the CentOS aggregator. Posting this to push it further down the page.

Where is that coffee cup?

Wrestling blog markup, Round Two

Perhaps I should have thought through what I was saying in preparing a posting a bit more to avoid getting bitten This should look better in the CentOS consolidator:

Stuck between a rock and a hard place here, it seems. Notice the ocean of whitespace to the left of the picture of the pregnant lady on the left image

than this mess:


PCO eats markup diffferently than Blogger. Point taken.

But, dang it: 'wrangling' through blogging software quirks by adding tables with:

<td>&nbsp;<img src="http://www.herrold.com/images/blog/spacer.gif" width="0" height="0">&nbsp;</td>

blocks littered through it sure feels nasty and a lot like fighting browser quirks in a by-gone era. This should be WYSIWYG

Where did my coffee cup get to?

Wrestling blog markup

Stuck between a rock and a hard place here, it seems. Notice the ocean of whitespace to the left of the picture of the pregnant lady on the left image


I had previously noticed that the Centos feed was honoring exposing the <hr> 'separator rules' in my markup, where the Blogger was not, on my OMG post earlier this week. Not a big deal, and I did a quick re-edit and report, but I dislike doing so. It exposes a problem in the Blogger preview engine as well, I guess

But darn it -- I should not have to set up a table to box image placement, at this late date. I had to do so to get the 'clown' picture at the bottom of Nine pregnant gals to lay right at the bottom in the Blogger client. Then I see these side-by-sides once it hits the CentOS consolidator:



Grrr ....

Oh ... my ... goodness ...

I've been mirroring, rebuilding, patching, filing bugs, and so on, against bleeding edge source package RPMs (SRPMs) out of 'RawHide' for at least a decade. It is one means I have used to push features from later Red Hat releases and hotfixes back into previous releases, to customers in our consulting (and on my own account).

We pushed post-RHL support, and commercial RPM-based support for side architectures including Netwinders [MIPS], PA-RISC, PPC, Sparc hardware, and Alphas from RawHide. We have built FOSS-based 'latest and greatest' LTSP forks of reduced package sets for commercial applications from RawHide

A largely unheralded change to a new RPM package file format for Raw Hide SRPMs coming out to the builder, at Red Hat's rpm-4.6 breaks all that, for the first time in at least a decade ... Jeff Johnson, a former lead developer and maintainer for Red Hat preceding the current incumbent lead, bent over backward and jumped through hoops, to make rpm a lingua Franca. Jeff was followed by a short-time incumbent, Paul Nasrat, who similarly did no harm to rpm

[herrold@centos-5 ctrlproxy]$ cp /mnt/nfs/var/ftp/pub/mirror/redhat/rawhide/SRPMS/ctrlproxy-3.0.8-2.fc11.src.rpm .
[herrold@centos-5 ctrlproxy]$ ls
ctrlproxy-3.0.8-2.fc11.src.rpm
[herrold@centos-5 ctrlproxy]$ rpmbuild --rebuild \
   ctrlproxy-3.0.8-2.fc11.src.rpm
Installing ctrlproxy-3.0.8-2.fc11.src.rpm
warning: user mockbuild does not exist - using root
warning: group mockbuild does not exist - using root
error: unpacking of archive failed on file /home/herrold/rpmbuild/SOURCES/ctrlproxy-3.0.8.tar.gz;49aeb249: cpio: MD5 sum mismatch
error: ctrlproxy-3.0.8-2.fc11.src.rpm cannot be installed
[herrold@centos-5 ctrlproxy]$ rpm -Vp ctrlproxy-3.0.8-2.fc11.src.rpm
Unsatisfied dependencies for ctrlproxy-3.0.8-2.fc11.src: rpmlib(FileDigests) <= 4.6.0-1
missing ctrlproxy-3.0.8.tar.gz
missing ctrlproxy.config
missing ctrlproxy.init
missing ctrlproxy.spec
[herrold@centos-5 ctrlproxy]$

Amid all that, the part which is important is that: a new rpmlib of at level 4.6.0-1 is needed (marked in red), and that without it, it produces a rather unhelpful cpio md5sum error message (marked in blue)

For the short term, until I get the matter sorted better, I'll set up a domU Raw Hide xen instance (which has the later rpmlib, and so can manipulate the package), such domU will be upgraded just enough to handle rpmlib(FileDigests) <= 4.6.0-1, then frozen against other breakage from later other updates

Further I'll grant that domU RW access to the NFS export that contains my build tree (/home/herrold/rpmbuild/), so that I can position a SRPM into that tree, and unpack it rpm -U packagename.src.rpm

With that unpacked set in the SOURCES subdirectory -- the tarball, patches, and such; and the SPECS subdirectory .spec file, I can then (hopefully) switch into an earlier rpm variant on an older unit, and rebuild the package to write a new SRPM with the older rpmlib form

---

We'll see. Change requires that the caterpillar moult and break out of the crysalis. It does not require that an angry Mothra result, and destroy the surrounding city in concert with Godzilla.

It might be asserted that this is some sort of performance or speed optimiszation.

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. -- Donald Knuth

rpm has had the instrumentation capabilities in place to see about where performance penalties lie, and has them for some time, so that one can easily test where load is. This is not rocket science; similarly, we drilled in micro-second instrumentation and time-stamping in the shim, as we are accustomed to looking for code bottlenecks, and people using our software, and trading in the financial markets really care about where lags are. I wrote a bit about this last month in the New Future Always Coming piece.

As proof of the pudding, take a moment and run rpm 4.6.0 with --stats on a large package built under the old format, and rebuild and repeat the test with the new 4.6.0 variant. Please feel free to get back to me at: timetrials at owlriver dot com as to where YOU find the true performance issues are, comparing a package build with SHA2, v. MD5. I have my preliminary stats for a later post, but welcome more data

---

This sure seems like a gratuitous and thoughtless format breakage to me, with no backward compatibility path announced. RHAS 2.1 is about to go out of support, of course (which dates back to a foundation including a fix for the as shipped RHL 8 rpm database locking issues), but RHEL 3, 4 and 5 just lost the ability to use Raw Hide so far as I can currently see

I may be missing something obvious by way of a workaround, and would be glad to be corrected, but ...

I really feel that the current incumbent Panu Matilainen has NOT clearly articulated this on the mailing list and bugs at Red Hat which I read very closely

This WILL cause at a minimum confusion, and also compatibility problems down the road with bi-directional interoperability at the SRPM level between both the LSB, and Red Hat's major enterprise Linux distribution competitor, Novell and its SLES line [seemingly being branded: SUSE Linux Enterprise, presently]. Oracle's UBL and its consumers are sort of off in a 'market niche' world of its own here, and the impact will be less pronounced. It will probably cause such SRPMs to 'just NOT work' in the SuSE buildservice, but I have not tested this yet

Heaven help the users of side distributions -- cAos, Mandriva, PLD, and the non-English fluent RPM file format using -- who try to rebuild and use a random found SRPM. It really seems that Red Hat marginalizes (or forces one to choose a camp to join) other formats with this move. Perhaps that was their intent. Fedora has been used by Red Hat as a 'wedge' for this purpose in the past [consider studied dis-interest in inter-archive compatability], and it may be just more of the same

Grrr ...

---

Disclaimer: These days, I am aligned with the RPM5 insurgents, and served as the long time maintainer of the old RPM website which Red Hat has since re-claimed

Ironically I was selected to take a survey on RIMM support ...

... and seemingly looking for places where BlackBerry support services can be priced:

Here is your answer: I'll pay almost anything for a fix of a non-functioning product that is 'almost there', and won't continue to pay for a product that cannot be fixed (I'll switch away from the vendor, if the doco is incomplete, the support bad, or the offering bogus; I'll eat my loss and move on).

The survey design NEVER asked if the support received from RIMM worked or had value; what my opinion of the existing RIMM web offerings was; how the responsiveness of support modes were; how RIMM is handling its external interactions doing support.

I assume I was contacted to take a satisfaction survey in light of interactions with RIMM support -- recent ones have been about getting sync through the PocketMac program [not recommended] to work in an OS/X environment. Never could get it to work.

(URL was: http://www.blackberry.com/redirect/rdr?sap-client=110& ... etc )

header was:
**********************************
Happy with your BlackBerry solution?
Please help us serve you better.
**********************************

and what looks like the contact data I enter when using the BB website.

Executive summary:

No -- I am not happy.

I have found RIMM support for the PocketMac backup of the BB device to be incredibly frustrating and 'live' support useless [on line doco is clearly incomplete, and email requests seeking clarification have gone ignored]

Ticket handling techs using canned replies, with absolutely NO interest in follow through, or ACTUALLY FIXING and confirming a fix for the issue. The RIMM emphasis seems to be on closing the ticket, seemingly within three days, without a confirmation that the issue is fixed.

This leaves a very bad taste behind -- ticket numbers on request.

And the product Pocket Mac so bad that I have abandoned it for third party vendors' approaches (MarkSpace, and Google Sync)

Still not right, but at least partially usable -- and what do you know -- MarkSpace is running a beta [of its upcoming update to its BB synchronization tool], and opening and following bugs I file.

ps -- The blackberry.com website very safely and skillfully hides email addresses, so that all contact can come in only through avenues (webform inserts into databases) likely to be able to 'close and ignore' issues. I assume there is no post-close QA review, as I have seen no sign of it.

Sad -- I thought the 'killer app' that lead to the BB was
pervasive contact capabilities.

I sent copies of this to each of the co-CEO's seeming email addresses for the exec suite pulled from a bit of google searching -- Please share or redirect as needed. Call if you want to talk; I'll answer any email as well. [I received a phone call from 'customer retention' and a 'follow the call email' with a 'role account' email address, and no telephone number for me to use. RIMM just does NOT want communication to be initiated to it except on its terms.]

For what it is worth, prior live support by RIMM, gated through T-Mobile (where I have the ability at the end of the transaction to say: 'We both see that it just does not work -- cancel the service as you cannot deliver that you have sold me') had been fine.

There's a message there, I think.

"latest and greatest" disease

RIMM, the maker of the popular 'Blackberry' smartphone, seems to have forgotten what we all have known for a long, long time. In recent models, they moved from a roller clickwheel at the right thumb position, to a set of 'up and down' buttons, or a trackball.

Less precise, and less capable: The clickwheel could be operated by touch alone; the new variants cannot.

Making matters worse, seemingly no-one at RIMM ever is in a environment where their hands pick up grime. This grime, of course, transfers to the trackball.

The trackball is not field cleanable; compare contra the 'remove and clean' capabilities of a computer mouse. I see: Broken BlackBerry Blues over at thestreet.com today.

Time to 'stock up' with a couple unlocked '8700's off ebay, in advance of the day I lose or damage my current device.

Let's get rid of disclaimers like this ...

... on mailing lists, as well. Or just subscribe and post from another email account. Or use more than a subject line to ask a question.

Email must be too hard for mere mortals to figure out.

Date: Wed, 20 Aug 2008 08:17:39 -0400
From: Mark T. Kennedy 
To: quickfix developers 
Subject: quickfix-d]  is there a new bug/issue tracker?

QuickFIX Documentation:
http://www.quickfixengine.org/quickfix/doc/html/index.html
QuickFIX Support: http://www.quickfixengine.org/services.html

/mark

This communication and any attachments may contain confidential/proprietary
information and is intended for information purposes only. It is not an
invitation or offer to purchase interests from Diamondback.  Any
representation to the contrary is unintentional.  This communication is
intended only for the person(s) to whom it is addressed.  If you are not the
intended recipient you are hereby notified that you have received this
document in error and that any review, dissemination, distribution, or
copying of this message or any attachments is not permitted.  If you have
received this in error, please notify the sender immediately by e-mail and
delete this message.  All e-mails sent to or received from this address will
be received by Diamondback's company e-mail system and is subject to
archival and possible review by someone other than the recipient.  This
notice is automatically appended to each e-mail message leaving Diamondback.

Where is my coffee cup, anyway?