orc_orc's sharp edge

Happy IPv6 test day, part 2

2011-06-09T21:51:00.008+01:00

In my first post in this small series, I closed without addressing matters of securing an IPv6 connection, and in matters of debugging where a connection failure is occurring. Thinking about it, the diagnostic post needs to come first, because tightening down a conneciton can cause hard to diagnose symptoms. So, on to diagnosis ...

We examined the interface results last time. Looking at just the routing related to ipv6 is straightfowrard as well:

Some familiar tools:

/sbin/ifconfig eth0
/sbin/ifconfig sit1
/sbin/route -A inet6

So using those tools:

[herrold@hostname ~]$ /sbin/ifconfig sit1
sit1      Link encap:IPv6-in-IPv4
          inet6 addr: 2604:aa:bb:cc::2/64 Scope:Global
          inet6 addr: fe80::4cf2:1c/128 Scope:Link
          UP POINTOPOINT RUNNING NOARP  MTU:1480  Metric:1
          RX packets:1691 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1693 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:177195 (173.0 KiB)  TX bytes:210468 (205.5 KiB)

[herrold@hostname ~]$ /sbin/route -n -A inet6 | grep 2604
2604:aa:bb:cc::/64      ::          U     256    1658       0 sit1
2604:aa:bb:cc::/128     ::          U       0       0       2 lo
2604:aa:bb:cc::2/128    ::          U       0    1691       1 lo
[herrold@mailhub ~]$

That is a pretty ordinary routing table for a non-gateway endpoint. Off-box traffic (to the '/64' netmask) is handed to the sit interface, and local traffic (to the '/128') retained on the local lo interface

We use a unfamiliar tool: ping6 -- The common '127.0.0.1' localhost has a new form under ipv6:

ping6 ::1 -c 2

and testing

[root@hostname ~]# ping6 ::1 -c 2
PING ::1(::1) 56 data bytes
64 bytes from ::1: icmp_seq=0 ttl=64 time=0.157 ms
64 bytes from ::1: icmp_seq=1 ttl=64 time=0.132 ms

--- ::1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.132/0.144/0.157/0.017 ms, pipe 2
[root@ostname ~]#

And we can ping by IP on the 'Global' link, both on the local end, and remotely with differing transit times for the packets:

Looking at the network connections, we examine the tunelling interface:

[root@hostname ~]# /sbin/ifconfig sit1
sit1      Link encap:IPv6-in-IPv4
          inet6 addr: 2604:aa:bb:cc::2/64 Scope:Global
          inet6 addr: fe80::4cf2:1c/128 Scope:Link
          UP POINTOPOINT RUNNING NOARP  MTU:1480  Metric:1
          RX packets:1714 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1731 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:179587 (175.3 KiB)  TX bytes:215180 (210.1 KiB)

[root@hostname ~]# ping6 2604:aa:bb:cc::2 -c 2
PING 2604:aa:bb:cc::2(2604:aa:bb:cc::2) 56 data bytes
64 bytes from 2604:aa:bb:cc::2: icmp_seq=0 ttl=64 time=0.135 ms
64 bytes from 2604:aa:bb:cc::2: icmp_seq=1 ttl=64 time=0.137 ms

--- 2604:aa:bb:cc::2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.135/0.136/0.137/0.001 ms, pipe 2
[root@hostname ~]# ping6 2604:aa:bb:cc::1 -c 2
PING 2604:aa:bb:cc::1(2604:aa:bb:cc::1) 56 data bytes
64 bytes from 2604:aa:bb:cc::1: icmp_seq=0 ttl=64 time=55.1 ms
64 bytes from 2604:aa:bb:cc::1: icmp_seq=1 ttl=64 time=53.7 ms

--- 2604:8800:100:bb::1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 53.781/54.473/55.165/0.692 ms, pipe 2
[root@hostname ~]#

Turning to DNS and name resoluton, it is quite familiar. One does not need an IPv6 link to query nameservers and receive back results, as they will answer questions _about_ ipv6 hostnames ('AAAA' records) to any authorized inquirant. Try these:

dig +trace www.ipv6.sixxs.net
dig www.kame.net aaaa

I get answers like this:

[herrold@centos-5 ~]$ dig www.kame.net aaaa

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-16.P1.el5 <<>> www.kame.net aaaa
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45595
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 2, ADDITIONAL: 3

;; QUESTION SECTION:
;www.kame.net.                  IN      AAAA

;; ANSWER SECTION:
www.kame.net.           85164   IN      CNAME   orange.kame.net.
orange.kame.net.        85164   IN      AAAA    2001:200:dff:fff1:216:3eff:feb1:44d7

;; AUTHORITY SECTION:
kame.net.               85164   IN      NS      mango.itojun.org.
kame.net.               85164   IN      NS      orange.kame.net.

;; ADDITIONAL SECTION:
mango.itojun.org.       2364    IN      A       210.155.141.200
mango.itojun.org.       2364    IN      AAAA    2001:2f0:0:8800:206:5bff:fe8d:940
mango.itojun.org.       2364    IN      AAAA    2001:2f0:0:8800::1:1

;; Query time: 1 msec
;; SERVER: 10.16.1.112#53(10.16.1.112)
;; WHEN: Thu Jun  9 17:17:20 2011
;; MSG SIZE  rcvd: 195

[herrold@centos-5 ~]$

which is certainbly a mess to read -- let's trim out the interesting parts:

[herrold@centos-5 ~]$ dig www.kame.net aaaa
;; ANSWER SECTION:
www.kame.net.           85164   IN      CNAME   orange.kame.net.
orange.kame.net.        85164   IN      AAAA    2001:200:dff:fff1:216:3eff:feb1:44d7

Which is the familiar information: a CNAME record is pointed in fact at a AAAA record at a ipv6 -type IP. We can ping (ping6) it by IP:

[root@hostname ~]# ping6 2001:200:dff:fff1:216:3eff:feb1:44d7 -c 2
PING 2001:200:dff:fff1:216:3eff:feb1:44d7(2001:200:dff:fff1:216:3eff:feb1:44d7) 56 data bytes
64 bytes from 2001:200:dff:fff1:216:3eff:feb1:44d7: icmp_seq=0 ttl=52 time=246 ms
64 bytes from 2001:200:dff:fff1:216:3eff:feb1:44d7: icmp_seq=1 ttl=52 time=256 ms

--- 2001:200:dff:fff1:216:3eff:feb1:44d7 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 246.833/251.705/256.577/4.872 ms, pipe 2
[root@hostname ~]#

or ping it by name, as DNS is working:

<[root@hostname ~]# ping6 www.kame.net -c 2
PING www.kame.net(2001:200:dff:fff1:216:3eff:feb1:44d7) 56 data bytes
64 bytes from 2001:200:dff:fff1:216:3eff:feb1:44d7: icmp_seq=0 ttl=52 time=227 ms
64 bytes from 2001:200:dff:fff1:216:3eff:feb1:44d7: icmp_seq=1 ttl=52 time=244 ms

--- www.kame.net ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1008ms
rtt min/avg/max/mdev = 227.291/235.678/244.066/8.401 ms, pipe 2
[root@hostname ~]#

Some other examples to try as 'beacons' might include:

ping6 2604:8800:100:9a::1 -c 2
ping6 2001:200:0:8002:203:47ff:fea5:3085 -c 2
ping6 ftp.ipv6.uni-muenster.de -c 2
ping6 -I eth0 ipv6.google.com -c 2

So the familiar diagnostic methods of examining interfaces, checking routing, testing connectivity by IP, and connectvity after resolution by name are all present

Happy IPv6 test day

2011-06-08T15:59:00.021+01:00

There is a 'ipv6 readiness testing day' today, ~~April~~ June 8, 2011, and so it seems appropriate to post my personal checklist for putting a CentOS box up on that network fabric

Apply for an account with SixXs. Their reply takes a couple of days, as it is a volunteer run organization
Have a deployed, updated, and hardened unit at a routable static IPv4 address

Amend /etc/sysconfig/iptables to include a line passing the tunnelling protocol. I place the entry after the IPSEC protocol entries in a stock setup. Restart iptables

... 
-A RH-Firewall-1-INPUT -p 50 -j ACCEPT
-A RH-Firewall-1-INPUT -p 51 -j ACCEPT
# vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
-A RH-Firewall-1-INPUT -p ipv6 -j ACCEPT
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...

Strip out any previous efforts at disabling the ipv6 / net-pf-10 kernel modules from loading in /etc/modules.conf, and in the files sourced in /etc/modprobe.d/ . Then rebuild the modules dependency table: /sbin/depmod -a
Amend /etc/sysconfig/network to carry the following lines:
```
#
NETWORKING_IPV6=yes
IPV6INIT=yes
IPV6FORWARDING=yes
IPV6_DEFAULTDEV=sit1
#
```
Which anticipates that the configuration details for the ipv6 tunnel will live in a file: /etc/sysconfig/network-scripts/ifcfg-sit1

and add that mentioned file: /etc/sysconfig/network-scripts/ifcfg-sit1 -- I have elided site-specific details as to IP addresses with: aa.bb.cc and aa:bb:cc placeholders

#
DEVICE=sit1
BOOTPROTO=none
ONBOOT=yes
IPV6INIT=yes
IPV6_TUNNELNAME="SixXS"
#
IPV6_AUTOTUNNEL=yes
PHYSDEV=eth0
IPV6_ROUTER=yes
#
IPV6TUNNELIPV4="38.229.76.3"
#    38.229.76.3 is the remote end of the tunnel at the tunnel broker
IPV6TUNNELIPV4LOCAL="198.aa.bb.cc"
#    198.aa.bb.cc is the local ipv4 static IP
IPV6ADDR="2604:aa:bb:cc::2/64"
#    2604:aa:bb:cc::2/64 shows both the local gateway IP, and netmask
#    the remote end gateway IP is by convention, the :1 
IPV6_MTU="1280"
TYPE=sit
#

At this point, simply restarting networking should bring up the ipv6 link, and properly route it -- so: /sbin/service network restart

The interfaces will look something like this:

[herrold@nostname ~]$ /sbin/ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:01:02:aa:bb:cc
          inet addr:76.aa.bb.cc  Bcast:76.aa.bb.dd  Mask:255.255.255.248
          inet6 addr: fe80::201:aaff:bb05:cc16/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:11088057 errors:0 dropped:0 overruns:1 frame:0
          TX packets:10668738 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1726307345 (1.6 GiB)  TX bytes:3178496052 (2.9 GiB)
          Interrupt:3 Base address:0x6f80

[herrold@hostname ~]$ /sbin/ifconfig sit1
sit1      Link encap:IPv6-in-IPv4
          inet6 addr: 2604:aa:bb:cc::2/64 Scope:Global
          inet6 addr: fe80::bbf2:cc1c/128 Scope:Link
          UP POINTOPOINT RUNNING NOARP  MTU:1480  Metric:1
          RX packets:500 errors:0 dropped:0 overruns:0 frame:0
          TX packets:502 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:53331 (52.0 KiB)  TX bytes:62784 (61.3 KiB)

[herrold@hostname ~]$

To wrap this up, ipv6 hardening, and connection debugging are worthy topics, and it may well be that a cautious sysadmin wants to lock down /etc/sysconfig/ip6tables and examine how one has hardened /etc/hosts.deny ... But rather than rush out content (I have a couple of mailing list posts I need to re-work), I'll leave these for later posts, while you, gentle reader, go apply for an account at a tunnel broker

What not to buy: Dynex 1.3MP Webcam

2011-05-22T22:59:00.009+01:00

I've spent the time across the weekend, tinkering with a USB webcam -- particularly a Dynex 1.3MP Webcam (USB ID: 0x19ff:0x0102 ). As I recall, Dynex is a BestBuy house brand. The Linux USB device driver support table indicates that the device is supported under _some_ Linux variant

The need was occasioned because some small animal, probably a groundsquirrel, has been digging in the garden of missus, and she wanted confirmation on what to go after. The local cat, Malaki, heard it and darted to the door, but I was too late letting him out to track down the intruder ... this time

My laptop at home has been my primary compute platform there, since I crushed my ankle late last December. I still need to post a page with all the gory x-ray details, to go along with the twitter pictures I sent along the way with recovery. The medical bill cost was staggering as well, and I'll sanitize and post details of that as well. Back to the laptop -- it runs a reasonably stock CentOS 5 most of the time, except when I've been trialling rebuilds of part of Red Hat's '6' series SRPM rebuilds

The seemingly needed 'uvcvideo' video driver was present, and I forced it to load, at the cost of the machine locking up in short order thereafter. I had to power cycle the unit to recover use of it. Hmmm ...

So I went looking for an application to pull content off of the newly present /dev/video0, and turned to the native 'ekiga' that CentOS 5 carries. It refused to acknowledge anything useful at that device, and so ... I had to power cycle the unit to recover use of it. Hmmm ...

Perhaps it was 'ekiga'. So I set out to solve the needed packaging to attain a current 'zoneminder' ... a bit more complex chain:

06:42:44 PM libgcrypt-devel-1.4.4-5.el5
06:42:44 PM libgpg-error-devel-1.4-2
06:42:46 PM gnutls-devel-1.4.1-3.el5_4.8
06:42:47 PM pcre-devel-6.6-6.el5_6.1
06:44:49 PM perl-MIME-Types-1.19-2orc
06:46:55 PM perl-TimeDate-1.16-5.el5
06:47:02 PM perl-MailTools-1.74-1orc
06:47:38 PM perl-DateManip-5.44-1.2.1
06:47:59 PM perl-DBD-MySQL-3.0007-2.el5
07:19:55 PM perl-PHP-Serialization-0.27-4orc
07:20:50 PM perl-MIME-Lite-3.01-5orc
07:23:33 PM perl-IO-Stringy-2.108-3.orc
07:23:54 PM perl-MIME-tools-5.411a-12orc
07:34:34 PM perl-IO-Zlib-1.10-1orc
07:43:10 PM perl-Compress-Raw-Zlib-2.027-1orc
07:47:54 PM perl-Archive-Zip-1.16-1.2.1
07:48:05 PM perl-Archive-Tar-1.39.1-1.el5_5.2
07:49:54 PM php-pdo-5.1.6-27.el5_5.3
07:49:55 PM php-mysql-5.1.6-27.el5_5.3
07:50:28 PM perl-Module-Load-0.10-3orc
07:51:38 PM perl-Device-SerialPort-1.002-3orc
07:51:49 PM zoneminder-1.23.3-2orc

and went through the very nicely done configuration. Oops -- it wants a mysql database server running to save state:


07:59:18 PM mysql-server-5.0.77-4.el5_6.6
08:03:20 PM mysql-test-5.0.77-4.el5_6.6

Zoneminder was willing to admit it could read /dev/video0 but all it returned was a black image. Grrr. ... and then after a few minutes, the laptop locked up again, and I had to power cycle the unit to recover use of it. Hmmm ..

So I spent a few minutes with Google doing some research, and found what looks like a ratehr nice little application for USB frame grabbing called: gideo -- see: A GTK video grabber designed with spca5xx components. Building it dragged in the Gnome / GTK development environment of thirty or so packages, and I only had to fix up a dependency's .spec file to handle Red Hat's multilib conventions


05:27:56 PM libtiff-devel-3.8.2-7.el5_6.7
05:29:52 PM gideo-0.1-1orc
05:43:18 PM SDL_image-1.2.10-2orc
05:43:18 PM SDL_image-devel-1.2.10-2orc

But now, 'gideo' is unwilling to admit, or loading the module is unwilling to produce a live /dev/video0, and ... you guessed it: The laptop locked up again, and I had to power cycle the unit to recover use of it

I think perhaps I'll try a different video camera

hitting the ground running

2010-12-22T19:35:00.008Z

I've mentioned creating and injecting a root ssh key into a new instance. The images we run are deployed with ssh enabled and not TCP wrappered; we COULD wrapper them and use the 'add an exception for ssh from a single IP' tool which the PMman web control interface has, but we have found the support load fallout from people just getting started is too high. Once they have deployed and hardened a couple of boxes, they 'get it and can use the 'lock from all' web tool, and then add a single IP if their taste runs to web tools

The very next step I take as to each machine I administer, is to run a hardening script. While I have published an outline here, I use a script rather than reading and scrape and pasting from that outline. This step is done through a script, not because I think I WILL forget something, but because I know the script will NOT forget anything, and is written to perform the hardening process in an idempotent fashion -- that is, when done, finishing the same end result, time after time. One path to get to better host security is to have good processes, consciously designed, systematically applied, and continuously improved

herrold@centos-5 admin]$ ./hardening.sh hostname.pmman.net
The authenticity of host 'hostname.pmman.net (198.178.231.xyz)' can't be established.
RSA key fingerprint is 86:6e:84:e0:27:57:dd:4d:1f:88:82:fc:42:1d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hostname.pmman.net,198.178.231.xyz' (RSA) to the list of known hosts.
hosts.allow                                   100%  488     0.5KB/s   00:00
hosts.deny                                    100%  390     0.4KB/s   00:00
iptables                                      100% 1337     1.3KB/s   00:00
sshd_config                                   100% 3325     3.3KB/s   00:00
README                                        100%  897     0.9KB/s   00:00
rollup.pem                                    100%    0     0.0KB/s   00:00
openssl.cnf                                   100% 9682     9.5KB/s   00:00
arm-pmman.sh                                  100%  363     0.4KB/s   00:00
sa-update-local-NOTES                         100%  877     0.9KB/s   00:00
sa-update-local                               100%  117     0.1KB/s   00:00
logwatch.conf                                 100%   80     0.1KB/s   00:00
rollup.pem                                    100%    0     0.0KB/s   00:00
Package sendmail-8.13.8-8.el5.x86_64 already installed and latest version
Package 1:make-3.81-3.el5.x86_64 already installed and latest version
Package m4-1.4.15-2orc.x86_64 already installed and latest version
Package iputils-20020927-46.el5.x86_64 already installed and latest version
logwatch.conf                                 100%   80     0.1KB/s   00:00
Stopping crond: cannot stop crond: crond is not running.[FAILED]
Starting crond: [  OK  ]
Shutting down sendmail: [FAILED]
Starting sendmail: [  OK  ]
Starting sm-client: [  OK  ]
Flushing firewall rules: [  OK  ]
Setting chains to policy ACCEPT: filter [  OK  ]
Unloading iptables modules: [  OK  ]
Applying iptables firewall rules: [  OK  ]
Loading additional iptables modules: ip_conntrack_netbios_ns [  OK  ]
Stopping auditd: [FAILED]
Starting auditd: [  OK  ]
/etc/aliases: 76 aliases, longest 10 bytes, 765 bytes total

info: inspecting /etc/aliases  for a root email forwarder off the box
# Person who should get root's mail
#root:          marc

info: 1. do you want fail2ban -- if so, run: ./fix-fail2ban.sh hostname.pmman.net

info: 2. updates are not run by this script: consider running:
   ssh -l root hostname.pmman.net yum -y -q upgrade --enablerepo=pmman-mail

info: 3. verify that root's email is properly handled

info: 4. now:           ssh -l root hostname.pmman.net
                cd /root/hardening/

   and do some patching and service restarting ...
[herrold@centos-5 admin]$

Note: the IP is obscured, and the host name and ssh host key altered. The edit to add an opff-box alias entry for root's email is to centralize all the miscellaneous cron and asynchronous notifications off the box, to centrally monitored point

Then as noted before, this is a stock CentOS 5 image, and so needs some further tightening done and updates run. I have long since scripted that process:

[herrold@centos-5 admin]$ ./fix-fail2ban.sh hostname.pmman.net
local-fb-fix.sh                               100%  256     0.3KB/s   00:00
Stopping fail2ban: [FAILED]
Starting fail2ban: [  OK  ]
[herrold@centos-5 admin]$ ssh -l root hostname.pmman.net yum -y -q upgrade --enablerepo=pmman-mail
[herrold@centos-5 admin]$

Then, as suggested, the edits on the remote machine

[root@vm175551137 hardening]# netstat -pant
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      3641/sshd
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      2627/sendmail: acce
tcp        0    240 198.178.231.xyz:22          76.242.0.abc:41936           ESTABLISHED 3593/0
[root@vm175551137 hardening]# history
    1  cd /etc/
    2  joe aliases
    3  yum install ipsec-tools
    4  rm *~
    5  newaliases
    6  cd /root/hardening/
    7  ls
    8  joe /etc/ssh/sshd_config
    9  /sbin/service sshd restart
   10  ls
   11  cp hosts.allow hosts.deny /etc
   12  joe iptables /etc/sysconfig/iptables
   13  /sbin/service iptables restart
   14  netstat -pant
   15  history
[root@vm175551137 hardening]# reboot 

Broadcast message from root (pts/0) (Wed Dec 22 14:47:10 2010):

The system is going down for reboot NOW!
[root@vm175551137 hardening]# Connection to hostname.pmman.net closed by remote host.
Connection to hostname.pmman.net closed.
[herrold@centos-5 admin]$

I added the ipsec-tools in support of an application this particular unit will be performing. The reboot at the end is for good measure to apply any new kernel and libraries through a clean boot. The 'true' hostname will be picked up from DNS PTR records, once that has been done already, after a reboot, as well

two minutes on keyed ssh access

2010-12-22T18:38:00.014Z

In a Linux box, in BSD or derived such as under OS/X, or under Windows in Putty, a person can generate a LOCAL keypair which is used for keyed SSH access to such Virtual Machine instances. I have completely moved away from password based external access for all new instances, as keys carefully managed are demonstrably safer

So you know: There is some heat, but not a lot of thoughtful light about permitting and using root ssh access. Some not well though out security policies have a phobic avoidance of such. I'll address the matter in a later post, discussing hardening generally, sshd config file hardening, remote syslogs, TCP wrappers, iptables, and dynamic dictionary attack response

For the time being, let's put to one side and get past that security design choice rant, and accept that at least initially, a PMman instance has already made a short term choice for setting up access and for management of such units which are running CentOS or others following the Red Hat approach for management of instances [i.e., not SuSE/SLES, Debian family, nor the BSDs]

Under such Linux, this looks like this:

$ # create a high strength passphrase
$ # I have written of gen-pw.sh before 
$ gen-pw.sh  -a
a2Wa4aSaLWkRac
$ cd ~
$ cd .ssh
$ ssh-keygen -t dsa -f pippin.pmman.net.dsa

$ # there is a passphrase prompt here and 
$ # we use that: a2Wa4aSaLWkRac   
$ # -- also make a record of it in a safe place
$ # -- if one maintains multiple keys per box, it can be a 
$ # chore to manage this -- but see: man ssh-agent

$ #       this generated ~userid/.ssh/hostname.pmman.net.dsa    
$ #       [the private part] ... and 
$ #               ~userid/.ssh/hostname.pmman.net.dsa.pub   
$ #       [the PUBLIC part

$ cat ~userid/.ssh/hostname.pmman.net.dsa.pub

and scrape and place it in your mouse pastebuffer, and proceed to the web interface. The -f file's name 'hostname.pmman.net.dsa' is arbitrary, but chosen to be mnemonic

Then add a new stanza to: ~userid/.ssh/config like this:

#
Host                    hostname32.pmman.net hostname64.pmman.net
#  optionally one can make a note of the passphrase here, but 
#  at the risk of exposing such if a local dire read compromise
#  is experienced, or a backup of such falls into untrusted hands
IdentityFile            /home/userid/.ssh/hostname.pmman.net.dsa
PasswordAuthentication  no
Protocol                2
PubkeyAuthentication    yes
#

Note here a key may be used on more than one host; that is, we can add the same public key into /root/.ssh/authorized_keys of more than one unit -- here, both a 32 bit and a 64 bit instance with similar hostnames. But I get ahead of myself ...

Using a secure means, we need to transfer taht public key to a remote instance, and to add it to the right file; Here, we use the SSL protected web interface under the PMman machine management interface for a given machine, in the [more] menu, first item. By placing the public part into the web form box, the management backend at the datacenter, will be validated as to form, and then place that public key into ROOT's /root/.ssh/authorized_keys file

At that point, one can then ssh to that remote box as root, accept the host key, and take steps for hardening, adding a working userid, and so forth

loop -de- loop

2010-12-13T15:48:00.006Z

As I count it at the moment, I am building and using content from more than eight loop mounted ISOs on a principal NFS, TFTP and 'next', and FTP server in the internal network

Red Hat has updates for 4.9, and 5.6 in beta; CentOS is in QA on a initial '6' release; I am doing private builds for a Fortune 50 on some backports out of RawHide and from some local packaging; and I am working on a 'back of the envelope' design and test to try to get control of the huge bloat on Red Hat ISO space for installs, to see if I can get a trimmed minimal installer for i386, x86_64, ppc[64] and s390x down to a single piece of CD sized ISO media. Then there is my favorite of the minimal wire install image, again which I package up into an ISO

Going forward, we will see more of encrypted filsystems across loop devices, and that will also put load on here. It may be time for the kernel folks to consider bumping that limit to 16

As such I regularly crest over the stock eight provided loop devices. To address this without a reboot, one simply has to:

# shutdown all uses of loop devices, so we can remove the module
/sbin/rmmod loop
echo "options loop max_loop=255" > /etc/modprobe.d/loop.local.conf
/sbin/depmod -a
/sbin/modprobe loop

Note: that 'depmod' may not be strictly required, but will in any event be harmless, so I do it -- heck, I still type sync ; sync before rebooting a box, and I KNOW that is not needed any more. The force of habit ...

I add the suffix .conf on that file, because I was scolded by a Debian box a couple weeks ago on the topic; it seems that they are deprecating sourcing files in /etc/modprobe.d/ lacking such. Since when did Linux starting use file name suffixes to determine a file's content? -- at least it is not required to be in 8.3 format

Another approach is doing it with hard-coded values at boot time, with sysctl.conf or such

Tip of the hat to Paul Howarth on the SELinux rant I went off on last week; The interaction of loop mounted ISOs, and mounts in the FTP space of a filesystem can also be addressed with options to the mount command, and in the /etc/fstab with context= choices. He writes and points out:

I use context mounts to avoid it, e.g. in fstab:
/path/to/CentOS-5.5-x86_64-bin-DVD-1of2.iso /path/to/dvd1 iso9660 _netdev,ro,loop,fscontext=system_u:object_r:public_content_t:s0 0 0

... sorry about the funky line wrapping, but there is just no good way to display really long /etc/fstab entries

later, bye

2010-12-10T22:22:00.008Z

CentOS is not for some people -- I get it and there is no sense getting agitated about it

17:20 rictec> regret to informe that i will have to un-install Centos as soon as i find the un-installer
17:21 orc_emac> rictec: we do not publish one, but it is this:
as root:
dd if=/dev/zero of=/dev/sda bs=512 count=1
17:21 orc_emac> as it is so easy to remember, we don't bother publishing one

'later, bye'

10 Dec 2010: reformatted due to some rendering issue reports

ripping out the safeties

2010-12-10T17:24:00.020Z

There have been the endless bull session threads on the CentOS main mailing list, nominally on the subjects of SELinux and IPv6 the last couple of weeks. I am just not of a mind to tolerate cr*ppy content on mailing lists anymore. On one such list, a 'regular' whom I identify as '...' had the misfortune of being the 'designated bad example' of the day

On Fri, 10 Dec 2010, ... wrote:

Make sure iptables is off. Is selinux on and enforcing? If so, reverse that too (disabled).

and ... followed up with yet another ready description on how to disable the safeties

you know -- this is just a poor response all around ...

I understand that the poster produces a lot of content. I know that he is an old bull in the pen there. I also see that he races to post first, and I feel that others do not answer as a result. I see that he carries lots of URLs in all his posts so that Google's spiders crawling through the mailing list archive will widely index his business because it seems to have a lot of links

But quantity is not quality

The response SHOULD have been to sharpen the issue if it was unclear, and describe a diagnostic flow so that one can BOTH have iptables security, and SElinux protections, AND a working system

The question, trimmed to its essence:

I get this error:
(13)Permission denied: make_sock: could not bind to address [::]:8091

(13)Permission denied: make_sock: could not bind to address 0.0.0.0:8091

no listening sockets available, shutting down

Unable to open log

So the first question to know the answer to is:

did the process start and persist?

Check for this thus:

# netstat -pant | grep 8091

If that command, which looks for a listening socket returns some content, the process is running and its name is shown

If that command returns nothing, the process in question did not start, and we need to try to manually start it

IN NO CASE does iptables PREVENT a process from starting as it is running in the kernel and blocking parts of the network stack from transiting packets

I note in passing that it is a usual voodoo local folk wisdom in this project that one does not bind to ALL interfaces, [0.0.0.0] -- I do not know if this is the case or in play here, but note in passing that the usual practice ** here ** is to build to a specific interface, or more commonly a specific IP

Such constraints usually indicate a problem in the underlying application [ISO layer 7] not being sufficiently mature to reply to the IP that it received a service request from, and to let the routing tables manage routing at the proper ISO layer

Turning to how to amend iptables for socket based services:

If one needs to have 8091/tcp open (or 8091/udp), OPEN IT PROPERLY

Add to /etc/sysconfig/iptables the following entry:

[herrold@elided iptables]$ diff -u iptables-ORIG iptables
--- iptables-ORIG       2010-12-10 10:15:19.000000000 -0500
+++ iptables    2010-12-10 10:15:48.000000000 -0500
@@ -14,6 +14,8 @@
 -A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
 -A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
 -A RH-Firewall-1-INPUT -p tcp -m tcp --dport 631 -j ACCEPT
+-A RH-Firewall-1-INPUT -p udp -m udp --dport 8091 -j ACCEPT
+-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 8091 -j ACCEPT
 -A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
 -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
 -A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 37 -j
ACCEPT
[herrold@elided iptables]$

and then as root run:

# /sbin/service iptables restart

The rules are of a common form across all Linux variants, and exhaustively documented. There is not any valid reason NOT to manage iptables rules generation and maintenance 'right'

As to SELinux, there is full logging running when the auditd and the restorecond are present. One can identify, and add rules on the fly, to progressively add 'permit rules' all SELinux based 'intercepts'

Here is a sample script of general applicability, under the GPLv3+:

[root@elided bin]# cat selinux-fixup.sh
#!/bin/sh
#
#       selinux-fixup.sh
#       Copyright (c) 2010 R P herrold <info@owlriver.com>
#       License: GPLv3+
#
#       Additively build SELinux rule sets to investigate what
#       a new application needs
#
export
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
#
#       make sure we have all our tools, which may not install
#       in a stock CentOS 5 minimal installation
rpm -q diffutils       2> /dev/null || yum -y -q install diffutils
rpm -q audit           2> /dev/null || yum -y -q install policycoreutils
rpm -q policycoreutils 2> /dev/null || yum -y -q install policycoreutils
#
/sbin/chkconfig auditd on
/sbin/service   auditd restart
/sbin/chkconfig restorecond on
/sbin/service   restorecond restart
#
cd /root/bin/
#
/bin/echo "A"
/bin/touch oldlog
/usr/bin/audit2allow -i denial-log > oldlog
#
# /bin/grep ftp /var/log/audit/audit.log* > /root/bin/ftp_audit.log
/bin/grep "avc:  denied" /var/log/audit/audit.log* > /root/bin/denial-log
#
# echo A
# audit2allow -a -M ftpmirror
#
/bin/echo "B"
/usr/bin/audit2allow -i denial-log -M deniallog
#
/bin/echo "C"
/usr/sbin/semodule -i deniallog.pp
#
/bin/echo "D"
/usr/bin/audit2allow -i denial-log > newlog
/usr/bin/diff -u oldlog newlog
#
/bin/echo "E"
/bin/cat deniallog.te
#
/bin/echo "F"
#

which is used iteratively, running the application, and when the candidate under test stops for 'mysterious' reasons (more about 'mysterious' later), re-running it to see if some new SELinux denial has occurred --- the 'diff' is designed to make it perfectly clear what the new denial was, and to add and apply the needed allow rule

We don't KNOW IF there was a SELinux denial yet, and if so, what the denial was yet, as the sharpening question was not asked, but to wrap matters up

To make a set of local allow rules permanent, see:

$ man 8 semanage

for the methodology for making such permanent and persistent once the full set are known

But sometimes (here in my example case !! ) the fix NEEDS to be upstreamed so that others using FOSS gain from the fix -- let's examine that next

If the problem occurs in in a package from an upstream, one can 'file bugs' against the 'selinux' component, and that group are quite attentive to addressing such

A run of that script looks like this [and in point of fact, at one customer's premise, there ARE some fixes needed with Red Hat's DHCP client and VSFTPD when loop mounted ISOs are used, that I need to file a couple of bugs on]

[root@elided bin]# ./selinux-fixup.sh
diffutils-2.8.1-15.2.3.el5
audit-1.7.17-3.el5
policycoreutils-1.33.12-14.8.el5
Stopping auditd:                                           [  OK  ]
Starting auditd:                                           [  OK  ]
Shutting down restorecond:                                 [  OK  ]
Starting restorecond:                                      [  OK  ]
A
B
******************** IMPORTANT ***********************
To make this policy package active, execute:

semodule -i deniallog.pp

C
D
E

module deniallog 1.0;

require {
        type iso9660_t;
        type ftpd_t;
        type iptables_t;
        type initrc_t;
        class unix_stream_socket { read write };
        class lnk_file getattr;
        class unix_dgram_socket { read write };
        class dir { read getattr search };
        class file { read lock getattr };
}

#============= ftpd_t ==============
allow ftpd_t iso9660_t:dir { read getattr search };
allow ftpd_t iso9660_t:file { read lock getattr };
allow ftpd_t iso9660_t:lnk_file getattr;

#============= iptables_t ==============
allow iptables_t initrc_t:unix_dgram_socket { read write };
allow iptables_t initrc_t:unix_stream_socket { read write };
F
[root@elided bin]#

which rules are saying that iptables and ftp need help:

[root@elided bin]# cat /etc/sysconfig/iptables
# Firewall configuration written by system-config-securitylevel
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -i eth1 -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p 50 -j ACCEPT
-A RH-Firewall-1-INPUT -p 51 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 21 -j
ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s a.b.c.0/24 --dport 21 -j ACCEPT
#       permit the north data center
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s a.b.c.0/24 --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with  icmp-host-prohibited COMMIT
[root@elided bin]#

pretty bog standard, but for the:

-A RH-Firewall-1-INPUT -i eth1 -j ACCEPT

general allow rule on a backside RFC-1918 network on that interface

Note: a.b.c.0/24 is a replacement I did for privacy purposes, as the specific values do not matter

and the loop mounted ISO images:

[root@elided bin]# cat /etc/mtab
/dev/sda1 / ext3 rw 0 0
proc /proc proc rw 0 0
sysfs /sys sysfs rw 0 0
devpts /dev/pts devpts rw,gid=5,mode=620 0 0
tmpfs /dev/shm tmpfs rw 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
/var/ftp/pub/mirror/redhat/rhel/ISOS/rhel-server-6.0-i386-boot.iso
        /var/ftp/pub/mirror/redhat/rhel/loop/1 iso9660
        ro,loop=/dev/loop0 0 0
/var/ftp/pub/mirror/redhat/rhel/ISOS/rhel-server-6.0-i386-dvd.iso
        /var/ftp/pub/mirror/redhat/rhel/loop/2 iso9660
        ro,loop=/dev/loop1 0 0
/var/ftp/pub/mirror/redhat/rhel/ISOS/rhel-server-6.0-s390x-dvd.iso
        /var/ftp/pub/mirror/redhat/rhel/loop/3 iso9660
        ro,loop=/dev/loop2 0 0
/var/ftp/pub/mirror/redhat/rhel/ISOS/rhel-server-6.0-source-dvd1.iso
        /var/ftp/pub/mirror/redhat/rhel/loop/4 iso9660
        ro,loop=/dev/loop3 0 0
/var/ftp/pub/mirror/redhat/rhel/ISOS/rhel-server-6.0-source-dvd2.iso
        /var/ftp/pub/mirror/redhat/rhel/loop/5 iso9660
        ro,loop=/dev/loop4 0 0
/var/ftp/pub/mirror/redhat/rhel/ISOS/rhel-server-6.0-x86_64-boot.iso
        /var/ftp/pub/mirror/redhat/rhel/loop/6 iso9660
        ro,loop=/dev/loop5 0 0
/var/ftp/pub/mirror/redhat/rhel/ISOS/rhel-server-6.0-x86_64-dvd.iso
        /var/ftp/pub/mirror/redhat/rhel/loop/7 iso9660
        ro,loop=/dev/loop6 0 0
[root@elided bin]#

As a side note, I do not quite understand (it is a mystery to me) how Red Hat would have been able to test 'nightlies' for anaconda based FTP wire installs of loop mounted ISOs without that rule, but perhaps that case was not in their test coverage plan

SELinux has been around for eight years now I am told, and iptables longer (looking back to the ancestor packet filtering approaches (ipchains !!) under the 2.4 kernel and before) -- no business would run a box all 777 on permissions or on a root account with no password

This is not Dark Arts, or Black Magic, and one does not use voodoo methods of shaking a rubber chicken at such problems to solve them. Old dogs need to learn new tricks. Simply ripping out such protections is to be irresponsible. It is NOT proper sysadmin nor proper development

I see that composing this piece took over two hours, but it is durable content and accurate on the topics of iptables and SELinux for this project. THAT is a better answer than snapping out a quick:

turn off all the safeties

reply, I submit

Those who cannot remember the past are condemned to repeat it

2010-12-03T16:21:00.008Z

A member of the trade press, formerly at the Linux Foundation, has speculated at length as to the release date of a CentOS 6. I and at least one other member of the CentOS core group were approached for comment on this topic, coming into the US Thanksgiving holiday. We were discussing how to respond, but we had not issued a formal reply. That writer went to press with a piece that expresses a date not of any formal CentOS origin or estimate. His words, his choice, his opinion, and nothing more

Here is a statement which is perhaps more accurate:

CentOS really doesn't do pre-release interviews as to release dates and process, other than what anyone may read in and infer from the 'centos-devel' mailing list. Any CentOS 6 series will ship when it is ready and will be available when it is announced

CentOS is the successor in part by merger of Tao Linux ('Hi, David'). This quote comes from the Tao

Those who know, do not speak; those who speak, do not know

Coping with xz under the RPM tools in CentOS 5

2010-12-03T15:11:00.009Z

So there I am, minding my own business, building a SRPM from Red Hat's 'rawhide' archive, and it fails. They are cutting over to 'xz' compression for the tarballs they ship. Their archive, and so their call, and not the end of the world

The symptom shows up when rpmbuild goes to uncompress such:

 ... 
+ rm -rf clamav-0.96.4
+ tar -xf /home/herrold/rpmbuild/SOURCES/clamav/clamav-0.96.4-norar.tar.xz
tar: This does not look like a tar archive
tar: Skipping to next header
tar: Archive contains obsolescent base-64 headers
tar: Read 6508 bytes from /home/herrold/rpmbuild/SOURCES/clamav/clamav-0.96.4-norar.tar.xz
tar: Error exit delayed from previous errors
error: Bad exit status from /var/tmp/rpm-tmp.36477 (%prep)

RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.36477 (%prep)
[herrold@centos-5

The error messages could be better, but the older compression methods that are known to the 'file' program available to rpmbuild that ships with CetnOS 5 do not contain the relevant 'magic numbers' yet. Progress is like that, and so until and unless Red Hat backports support into its RHEL sources, CentOS will not pick up the fix in its version 5 mainline

One perfectly suitable response is to use the RPM5 branch of the package manager, which DOES know. But some people cannot relax that constraint for various non-technical reasons

This issue is rather like the old cutover from md5sums to shasums which RPM did a while ago, and that I wrote aboutI wrote about

The fix is straightforward:

Install the compressed tarball, spec file and any patches with rpm in the usual fashion
Uncompress from the unknown compression format and re-compress with a known one
Amend the spec file; here, I use grep to look, and as there is just one edit, sed to edit
Rebuild using the '-ba' option from the revised .spec file with the tools of the target environment (here, CentOS 5)
The resulting SRPM will be portable and as a result of the second step, uses a known compression

Lets look:

[herrold@centos-5 clamav]$ unxz /home/herrold/rpmbuild/SOURCES/clamav/clamav-0.96.4-norar.tar.xz
[herrold@centos-5 clamav]$ gzip /home/herrold/rpmbuild/SOURCES/clamav/clamav-0.96.4-norar.tar
gzip: /home/herrold/rpmbuild/SOURCES/clamav/clamav-0.96.4-norar.tar.gz already exists; do you wish to overwrite (y or n)? y

[herrold@centos-5 clamav]$

That question about over-writes happened because it appears the sources from a prior build of clamav-0.96.4 were not re-rolled into a 0.96.5 tarball by the upstream packager at RawHide, but may have been patched instead. I've not expressly looked

[herrold@centos-5 clamav]$ cp ~/rpmbuild/SPECS/clamav.spec .
[herrold@centos-5 clamav]$ rpmbuild -ba clamav.spec
error: File /home/herrold/rpmbuild/SOURCES/clamav/clamav-0.96.4-norar.tar.xz: No such file or directory
    ...

As we have not yet fixed the .spec file, this was expected, but is shown here so the diagnosis path is clear

[herrold@centos-5 clamav]$ grep xz clamav.spec
Source0:        %name-%version%{?prerelease}-norar.tar.xz
[herrold@centos-5 clamav]$ sed -i -e 's@xz@gz@g' clamav.spec

And now the .spec file is ready as well

[herrold@centos-5 clamav]$ rpmbuild -ba clamav.spec
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.47404
+ umask 022
+ cd /home/herrold/rpmbuild/BUILD
+ LANG=C
+ export LANG
+ unset DISPLAY
+ cd /home/herrold/rpmbuild/BUILD
+ rm -rf clamav-0.96.4
+ /bin/gzip -dc /home/herrold/rpmbuild/SOURCES/clamav/clamav-0.96.4-norar.tar.gz
+ tar -xf -
 ...
checking host system type... x86_64-redhat-linux-gnu
checking target system type... Invalid configuration `noarch-redhat-linux-gnu': machine `noarch-redhat' not recognized
configure: error: /bin/sh config/config.sub noarch-redhat-linux-gnu failed
error: Bad exit status from /var/tmp/rpm-tmp.86669 (%build)
 ...

The build fails for other reasons out of scope for this post, in that a new configure 'target' is emitted. This is similar to a later compression format addition, but a different problem, solved elsewhere. Such a change is another part of distribution and brand management matters at Red Hat's part. I'll note the solution for this (putting to side seriously amending the rpm build environment macros, which is the 'one way' path into later versions) in a later post

Once all the changes are done, and the 'scratch' test builds and will install cleanly, I go in with an editor, manually bump the release value by one, and add a note in the changelog stanza. Then I repeat the build 'for record', signing, and distribution. The Release 'bump' is needed so the NEVR (name, Epoch, Version, and Release comparison which librpm does, and that yum calls through librpm to do can detect the fact that a later updated version is in an updates repository in due course

All done

arrogance, personified

2010-10-14T22:42:00.010+01:00

I see this in the overnights from the OpenJDK front:
IBM joins the OpenJDK community, will help unify open source Java efforts, with the salient 'pull quote'

It became clear to us that first Sun and then Oracle were never planning to make the important test and certification tests for Java, the Java SE TCK, available to Apache

This dovetails with my prior post

It's official -- Oracle is a profit maximizer and could care a hoot about being anything but dictate what is best to FOSS. But then you already knew that, right?

Not to sound arrogant, but we know how to deal with the Linux community

... naw, that does not sound arrogant, at all, Wim

checklist RO rsync server

2010-10-08T19:53:00.016+01:00

Setting up a new RO RSYNC server setup

The primary usage case is we describe is how to deploy a read-only RSYNC server with no end user accounts, to be used for distribution of content (here, to move a builder result archive that is intentionally NOT 'visible' from the internet to a more capable transfer server) From there, the content is integrated into a internal archiving server, and after that, to a publicly accessible binary archive, accessible through ftp, rpm, or yum

As before, we start with a freshly deployed, and hardened PMman instance. At all times, we will strive to follow proper sysadmin 'best practices' discipline under SElinux, wrappers and iptables

Install and enable rsync, which is the package holding the stock rsync daemon. As rsync supports wrappers, we also need the xinetd which is the package holding the stock inetd in recent Red Hat derived distributions -- Let's get started:

yum can do the install trivially
yum install rsync xinetd
Then enable the needed services:
/sbin/chkconfig rsync on /sbin/chkconfig xinetd on
We need to do some configuration for the rsync daemon as to permissions and directories to serve:
[root@trap64 etc]# cd /etc [root@trap64 etc]# cat rsyncd.conf # motd file = /etc/rsyncd.motd log file = /var/log/rsyncd.log pid file = /var/run/rsyncd.pid lock file = /var/run/rsync.lock [trap64] path = /var/ftp/pub/local comment = x86_64 fruit uid = nobody gid = nobody read only = yes list = yes # auth users = username # secrets file = /etc/rsyncd.scrt hosts allow = 10.0.0.0/24 127.0.0.0/24 hosts deny = 0.0.0.0/0 [root@trap64 etc]#
Set up the iptables -- I do not recall the rsync daemon port off the top of my head, so I look it up:
[root@trap64 etc]# grep rsync /etc/services | head -2 rsync 873/tcp # rsync rsync 873/udp # rsync [root@trap64 etc]#
... so the port is 873
# localhost can do all ... -A RH-Firewall-1-INPUT -i lo -j ACCEPT # ... # rsync daemon -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s 10.0.0.0/8 --dport 873 -j ACCEPT # ...
Open the wrappers
# ALL: ALL@127.0.0.1 # # ... # rsync: ALL@10.0.0.0/255.0.0.0 #
Restart the wrappers enforcing daemon
[root@trap64 sysconfig]# /sbin/service xinetd restart
Test it:
[root@trap64 sysconfig]# rsync localhost:: trap64 x86_64 fruit [root@trap64 sysconfig]#
To put it into production on a client, we can use something like this:
#!/bin/sh # # this file: /root/bin/update-archive.sh # Copyright (c) 2010 R P Herrold # License: GPLv3+ # # ln -s /root/bin/update-archive.sh /etc/cron.hourly/ # export PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin umask 022 # -- non local content goes into the mirror constellation [ ! -e /var/ftp/pub/mirror/pmman/RPMS/x86_64/ ] && \ mkdir -p /var/ftp/pub/mirror/pmman/RPMS/x86_64/ # # export VERBOSE="-v " export QUIET="-q " # /usr/bin/rsync -a ${VERBOSE} ${QUIET} --exclude=working \ trap64.darkside.lan::trap64/pmman/RPMS/x86_64/. /var/ftp/pub/mirror/pmman/RPMS/x86_64/. chown -R root.root /var/ftp/pub/mirror/pmman/RPMS/x86_64 #
All done

Earlier in this series:

Function	Link
hardening	http://www.pmman.com/usage/hardening/
lftp	http://orcorc.blogspot.com/2010/08/mirroring-upstream-master-with-lftp-to.html
RO vsftpd	http://orcorc.blogspot.com/2010/07/checklist-ro-ftp-server-setup.html
RO NFS	http://orcorc.blogspot.com/2010/08/nfs-aide-to-memory.html

too funny

2010-10-06T18:44:00.005+01:00

I mentioned in my last blog post:

More importantly, it seems that the sending email account webteam (at) bhphotovideo.com is unmonitored, although one has to assume an e-commerce vendor DOES have a 'webteam'. How curious
Having a unmonitored email sending role account is fine, of course; driving responses into a webbish workflow is fine as well; but why not use something obvious not a monitored account like: noreply@ ... or unmonitored@ ... instead

After I submitted feedback through their web workflow form, I received a confirmation email containing this:

If for any reason you have any additional questions, thoughts or comments, please feel free to email us at webmaster (at) bhphotovideo.com, as we would be happy to hear from you.

So ... the right hand does not know what the left is doing?

Lost password #FAIL

2010-10-06T17:23:00.012+01:00

The file with my old saved password (a strong one: see: a prior post on the topic) for an e-commerce site was inadvertently deleted. No particular reason to chase the backup file out, as there was a lost password mailer. And so, I had occasion to use the 'lost password link' of that site today

Date: Wed, 6 Oct 2010 12:13:36 -0400 (EDT)
From: webteam (at) bhphotovideo.com
To: herrold (at) ... 
Subject: Your Password from bhphotovideo.com
----------------------------------------

Dear Russell Herrold

Thank you for your inquiry. Here's your password:

t3f38RbMMweRhg

We look forward to your next visit to our site. Please feel free 
to let us know if there's any other way we may assist you.

Thank you,
The B&H Web Team
www.bhphotovideo.com
NNN Ninth Avenue
New York, NY 10001, USA
800-606-asdf
212-444-qwer
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is an automated email response and cannot be replied to.

A couple threshold matters: I changed the password value they sent me to something similar. More importantly, it seems that the sending email account webteam (at) bhphotovideo.com is unmonitored, although one has to assume an e-commerce vendor DOES have a 'webteam'. How curious

Having a unmonitored email sending role account is fine, of course; driving responses into a webbish workflow is fine as well; but why not use something obvious not a monitored account like: noreply@ ... or unmonitored@ ... instead?

Back to the topic at hand. That is: The 'lost password' mailer sent me a unhashed, plaintext prior password, and when using so, did NOT require an immediate change of credential when I used it to log in

There was a 'feedback form' on their site, and so I sent along this:

email is inherently insecure as it cannot be protected from being read by people 'along the way' on the transfer (such as the ISP of the server that received the email)
Sending a 'reset your password' one time link, and noting a credential change in a permanent part of an account history, is pretty basic
Not having this in you user account management interface, and sending a prior password in plaintext are a big red warning sign. I am left to wonder: Would they also disregard credit card data security [CISP/now PCI] credit card 'hashing' and no saved plaintext' credentials restrictions ;(

There is an old saying: A chain is only as strong as its weakest link ... I think we found a weak one here

Unit test shepards

2010-09-27T19:30:00.016+01:00

I read with interest over the weekend this unit testing and TDD blog post from Douglas Hubler. I met him in real life a few weeks up in Chicago at the annual ClueCon, and was very impressed

I tracked down his email address and started to write a private email, but then as I re-read my draft and his piece, I noticed that it was a 'talking draft' by him. As such, I decided to surface my thoughts here

Hi -- Russ herrold (ex CentOS) here -- we met at cluecon

You put your finger on the problem well here:

"Project Maintainers" were always in fear of holding the bag on contributions that introduced bugs while not advancing their employer's goals

which is the well known 'capture by the employer problem' in FOSS. I am not saying (and would never suggest) that employer sponsorship of an interested 'Project Maintainer' is undesirable -- just the opposite, as it funds getting SOME motion in some cases (i.e., when it suits the employer's goals, or is not a clear 'CLM' -- career limiting move). Of course this path leads to 'freeze ups' similar to what we see in Debian Stable, where nothing short of dynamite (or a working remote exploit) seems to work to pry some forward progress into the main trunk

I put on my 'agile' thinking cap, to scope out the implications of your post

To work, the "Unit Test Sheppards" need to have global mandate to commit at least unit tests, via a Version Control System, and there needs to be a working Continuous Integration server. If this 'breaks the build' either the test is wrong, or the code is wrong. In the first response to 'breaking the build' the CI server has to revert the test, and file an exception report, to be owned by the UTS in the first instance, with a CC to the PM

This gets a 'heads up' in front of the PM, and a careful UTS will at a minimum either: 1) acknowledge that the test was ill-considered, withdraw it, and close the bug; 2) amend their code to correct misunderstanding that resulted in a broken test and re-attempt the commit [closing the bug, with the possibility of a 're-entry' of a new bug on the revised test], or 3) add documentation to the bug filing that indicates why the test is right [perhaps something as simple as pointing to a release target milestone, or part of the Requirements document] in preparation to handing the bug off to the PM (staying on the bug as a CC), and handing it along to the PM's queue

One problem is that when there is only a single PM, there is also only a single point of blockage, and 'real life' intervening, or a work-plan to do a substantial refactoring (perhaps even already partially working in a private tree), or even a non-public agenda on the part of one's employer may prevent the PM from ** wanting ** to respond 'just now' if a well-form test and bugreport gets dropped on them ...

... but if the unit test is 'right', usually it is proper to add to a test suite. I put to one side whether one should run all unit tests every pass; Tests do rot and one may well need to trim obsolete tests away, or refactor old ones to match code reorganizations; clearly one answer when the suite gets 'too big' is to start prioritizing, adding stochastic selection to generally omit tests related to rarely encountered failure modes and so forth

But a well written test never fully 'goes away' by default. At some predictable interval, of course, the 'full boat' of ALL tests, as well as more rigorous end to end functional tests are needed. Beck's TDD book glosses over this to some extent as his focus was development, but 'testing' means much more than 'unit testing'

One additional avenue toward a solution would be to convert the single PM 'person' into a trellised PM 'role' or 'team' containing two or more non-affiliated project members

By and large, FOSS works better when there is a consensus approach to management of a resource. It is basic group dynamics that achieving consensus is easier in a small team, able to consult in the 'stand up five minute meeting, and to come to a tactical 'what is the simplest thing that we can do' to conform to a well-formed test, write (or adopt) the unit test, apply it, and move on ;) With only two people in the PM trellis, or a senior and a junior relationship, the group dynamics may result in impasse, which is only visible to the 'outsider' UTS as 'nothing is happening on this bug'

Lots of inter-person political approaches exist here, but ultimately and in most projects, there is a agreed-to Release Manager team with global commits, that has to be willing and able to 'take up the reins', intervene to intentionally 'break the build' in HEAD when an impasse continues 'too long' [I assume here a model of a stable release, and a developmental HEAD], and force the PM to respond (perhaps by relinquishing participation as a co-PM)

I don't have an obvious candidate solution to suggest here, as there as many approaches are possible, and I've seen the issue as a project lead, as well as a mere participant, and sometimes simply as a concerned onlooker

lost in the bowels of Google Groups

2010-09-22T17:29:00.011+01:00

A post I made earlier today to a mailing list seems to have been held up for an hour, even though I am a subscriber to the mailing list in question, have proper and meticulously preened DNS A, PTR, MX, and even TXT records, publishing SPF details properly, because of prior problems with Google's mailservice's erroneous markings of some pieces as 'spammy' in the past ...

Received: by 10.90.14.22 with SMTP id 22mr127029agn.36.1285171616911;
        Wed, 22 Sep 2010 09:06:56 -0700 (PDT)
X-BeenThere: puppet-users@googlegroups.com
Received: by 10.91.83.8 with SMTP id k8ls391483agl.0.p; Wed, 22 Sep 2010
 09:06:54 -0700 (PDT)
Received: by 10.150.51.21 with SMTP id y21mr255924yby.58.1285171614696;
        Wed, 22 Sep 2010 09:06:54 -0700 (PDT)
Received: by 10.229.192.137 with SMTP id dq9mr33711qcb.14.1285167411800;
        Wed, 22 Sep 2010 07:56:51 -0700 (PDT)
Received: by 10.229.192.137 with SMTP id dq9mr33709qcb.14.1285167411749;
        Wed, 22 Sep 2010 07:56:51 -0700 (PDT)
Received: from bronson.owlriver.com (bronson.owlriver.com [198.49.244.50])
        by gmr-mx.google.com with ESMTP id
    c41si5677929qcs.12.2010.09.22.07.56.51;
        Wed, 22 Sep 2010 07:56:51 -0700 (PDT)
Received-SPF: pass (google.com: best guess record for domain of
    herrold@owlriver.com designates 198.49.244.50 as permitted sender)
    client-ip=198.49.244.50;
Received: from localhost (localhost.localdomain [127.0.0.1])
    by bronson.owlriver.com (8.13.8/8.13.8) with ESMTP id o8MEumOR020433
    (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
    Wed, 22 Sep 2010 10:56:49 -0400
Date: Wed, 22 Sep 2010 10:56:48 -0400 (EDT)

... anti-spam measures, one assumes. I understand taking such measures, but sure wish the scoring 'downticks' Google was marking were published and findable (compare, to the good: AOL's current practices)

But then, I am told from time to time that my world view and some of my approaches are 'too utopian'. Humph -- a little bit 'utopian is all right, but one can overdo it? Who knew?'

Change control in operations

2010-09-22T16:03:00.010+01:00

This crossed the puppet-users mailing list earlier today:

We have an engineering environment of around 200 CentOS servers, plus a production environment of roughly the same size. Currently, when we roll out a new server, we do a 'yum update' so the new server has the latest packages; however this means that just about every server has a different set of package versions - a system rolled out today will have different versions from one rolled out last month, and that will have different versions from one rolled out last year.
...
Has anybody else been faced with this problem, and if so, how did you resolve it?

Let's consider just the problem of 'package version skew' in operations, and come up with a solution for it. [The questioner is also 'starting' with a couple of deployment targets that vary over time because of a poorly considered 'start image' creation ... An obvious approach here is to have a couple of stable base deployment image, and a set of defined transforms to produce a basic engineering workstation or server, per to specification, and is largely uninteresting here]

Set up a local mirror of the centos external mirrors, and call it 'incoming'
Optionally, set a sub-mirror of 'incoming' called 'vault', and mirror in a fashion that does NOT delete old content no longer present on 'incoming'
Set a third mirror called 'testing', which 'picks and chooses' selected packages to test, and their dependencies (see the package: yum-utils for some tools to permit confirming that one has 'closure' of those dependencies)
Test on your pre-deployment 'bench' against 'testing' until you have a change-set you wish to deploy throughout the universe of your boxes under management. Obviously, several 'testing' mirrors can be set up, for differing classes of machines
FINALLY, have a master distribution mirror called 'rtm' that has a change-set from a 'testing' mirror deployed to it. Remove the stock repository specification files from
```
        /etc/yum.repos.d/ 
```
and deploy local variants to taste, that point at 'rtm'. Again, several 'rtm' mirrors can be set up, for differing classes of machines

Something like this to ensure coherency of a enterprise wide deployment is usually mandated by a Change Control Board (explicitly, or implicitly). Obviously, other aspects of an IT policy document will attend to getting the various mirrors properly recoverable in one's backup strategy. [there, the 'testing' mirrors are often NOT covered, as they are ephemeral as to their usefulness, and recoverable out of 'vault' (top down) or from a 'rtm' (bottom up)]

sitting in great connectivity ...

2010-09-21T15:27:00.018+01:00

... sure makes a difference, seemingly

I do daily checkouts from the FreeSwitch project, and run the same build script on a CentOS box inside our local network (which is nominally down a data link that is 3 x T-1 wide), and another that is up at a data center, and has the ability to sustain a 3.5 GByte/sec transfer rate indefinitely (it has been the disaster failover site for the periodic 'Victoria's Secret' soft pr0n 'strut their stuff' webcast)

I synchronized builds on the two boxes yesterday, so they happened to be at the exact same checkout from upstream's version control system level. Today, I opened a couple of consoles, and fired off the build commands within a second of one another. The first part of that script is to checkout current to HEAD, and then off into the builds. I've marked the two units in alternating colors so the comparisons stand out better

Unit A:

Unpacking objects: 100% (38/38), done.
From git://git.freeswitch.org/freeswitch
   184f395..f7d16ec  master     -> origin/master
Updating 184f395..f7d16ec
Fast-forward
 libs/freetdm/src/include/private/ftdm_types.h      |    2 +-
 src/mod/applications/mod_spandsp/mod_spandsp_fax.c |    6 +-
 src/mod/codecs/mod_codec2/Makefile                 |   14 ++
 src/mod/codecs/mod_codec2/mod_codec2.c             |  161 ++++++++++++++++++++
 src/mod/endpoints/mod_sofia/mod_sofia.c            |   23 +++
 src/mod/endpoints/mod_sofia/mod_sofia.h            |    1 +
 src/mod/endpoints/mod_sofia/sofia_glue.c           |   21 +++
 src/switch_ivr.c                                   |    4 +-
 8 files changed, 226 insertions(+), 6 deletions(-)
 create mode 100644 src/mod/codecs/mod_codec2/Makefile
 create mode 100644 src/mod/codecs/mod_codec2/mod_codec2.c

real    0m1.105s
user    0m0.425s
sys     0m0.090s
/home/herrold/vcs/git/freeswitch

Unit B:

Unpacking objects: 100% (38/38), done.
From git://git.freeswitch.org/freeswitch
   184f395..f7d16ec  master     -> origin/master
Updating 184f395..f7d16ec
Fast-forward
 libs/freetdm/src/include/private/ftdm_types.h      |    2 +-
 src/mod/applications/mod_spandsp/mod_spandsp_fax.c |    6 +-
 src/mod/codecs/mod_codec2/Makefile                 |   14 ++
 src/mod/codecs/mod_codec2/mod_codec2.c             |  161 ++++++++++++++++++++
 src/mod/endpoints/mod_sofia/mod_sofia.c            |   23 +++
 src/mod/endpoints/mod_sofia/mod_sofia.h            |    1 +
 src/mod/endpoints/mod_sofia/sofia_glue.c           |   21 +++
 src/switch_ivr.c                                   |    4 +-
 8 files changed, 226 insertions(+), 6 deletions(-)
 create mode 100644 src/mod/codecs/mod_codec2/Makefile
 create mode 100644 src/mod/codecs/mod_codec2/mod_codec2.c

real    0m15.607s
user    0m0.168s
sys     0m0.096s
/home/herrold/vcs/git/freeswitch

One box is running an 386 kernel, and the other x86_64; memory is somewhat smaller on the x86_64. The 'horsepower' of each is roughly the same

Unit A:

[herrold@centos-5 ~]$ ssh freeswitch.pmman.com uname -a
Linux freeswitch.pmman.com 2.6.18-194.11.3.el5PAE #1 SMP Mon Aug 30 17:02:48 EDT 2010 i686 i686 i386 GNU/Linux
[herrold@centos-5 ~]$ ssh freeswitch.pmman.com free
             total       used       free     shared    buffers     cached
Mem:       6226068    4427212    1798856          0     303156    3936312

Unit B:

[herrold@centos-5 ~]$ uname -a
Linux centos-5.first.lan 2.6.18-194.11.3.el5xen #1 SMP Mon Aug 30 16:55:32 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
[herrold@centos-5 ~]$ free
             total       used       free     shared    buffers     cached
Mem:       3072000    3036352      35648          0     291852    1790652

Unit A:

[herrold@centos-5 ~]$  ssh freeswitch.pmman.com dmesg \| grep -i bogo
Calibrating delay loop (skipped), value calculated using timer frequency.. 3990.15 BogoMIPS (lpj=1995079)
Calibrating delay using timer specific routine.. 3990.04 BogoMIPS (lpj=1995020)
Total of 2 processors activated (7980.19 BogoMIPS).

Unit B:

[herrold@centos-5 ~]$ dmesg | grep -i bogo
Calibrating delay using timer specific routine.. 6652.60 BogoMIPS (lpj=13305207)

Unit A:

[herrold@centos-5 ~]$ ssh freeswitch.pmman.com cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Xeon(R) CPU            5130  @ 2.00GHz
stepping        : 6
cpu MHz         : 1995.224
cache size      : 4096 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
apicid          : 0
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl vmx tm2 ssse3 cx16 xtpr lahf_lm
bogomips        : 3990.44

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Xeon(R) CPU            5130  @ 2.00GHz
stepping        : 6
cpu MHz         : 1995.224
cache size      : 4096 KB
physical id     : 0
siblings        : 2
core id         : 1
cpu cores       : 2
apicid          : 1
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl vmx tm2 ssse3 cx16 xtpr lahf_lm
bogomips        : 3990.02

Unit B:

[herrold@centos-5 ~]$ cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 CPU          6700  @ 2.66GHz
stepping        : 6
cpu MHz         : 2660.050
cache size      : 4096 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu tsc msr pae cx8 apic mtrr cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est ssse3 cx16 lahf_lm
bogomips        : 6652.60
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 CPU          6700  @ 2.66GHz
stepping        : 6
cpu MHz         : 2660.050
cache size      : 4096 KB
physical id     : 1
siblings        : 1
core id         : 0
cpu cores       : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu tsc msr pae cx8 apic mtrr cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est ssse3 cx16 lahf_lm
bogomips        : 6652.60
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

But ...

Unit A:

Wrote: /home/herrold/rpmbuild/RPMS/i386/freeswitch-sounds-0.0.20100921.git-1.i386.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.3898
+ umask 022
+ cd /home/herrold/rpmbuild/BUILD
+ cd freeswitch-20100921
+ '[' /var/tmp/freeswitch-0.0.20100921.git.root '!=' / ']'
+ rm -rf /var/tmp/freeswitch-0.0.20100921.git.root
+ exit 0

real    17m56.699s
user    13m18.982s
sys     3m10.880s

real    24m50.468s
user    18m2.521s
sys     4m56.827s
[herrold@freeswitch freeswitch]$

Unit B:

Wrote: /home/herrold/rpmbuild/RPMS/x86_64/freeswitch-sounds-0.0.20100921.git-1.x86_64.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.90424
+ umask 022
+ cd /home/herrold/rpmbuild/BUILD
+ cd freeswitch-20100921
+ '[' /var/tmp/freeswitch-0.0.20100921.git.root '!=' / ']'
+ rm -rf /var/tmp/freeswitch-0.0.20100921.git.root
+ exit 0

real    27m27.666s
user    8m27.160s
sys     3m25.909s

real    48m25.064s
user    11m34.027s
sys     5m15.264s
[herrold@centos-5 freeswitch]$

That is, the older, 2GHz Xeon is running away from the newer 2.6 GHz Core Duo. Quite the discrepency there, but the numbers don't lie. Perhaps due to the local load of being a X-desktop on 'centos-5' [no local xen domU are presently running on it], and NOT running X on the remote server. Interesting 'food for thought' of a problem to research as to the why's and wherefore's on causation

What do you discuss?

2010-09-12T13:01:00.002+01:00

Great minds discuss ideas
Average minds discuss events
Small minds discuss people
-- Eleanor Roosevelt

office background noise

2010-09-09T22:09:00.002+01:00

A question in IRC: Do you listen to music online?

17:07 =orc_orc> xmms is playing: Peshay / Pacific atm
17:08 =orc_orc> the library has more to 'rip' than I will ever be able to grow
tired of, for free
17:08 =orc_orc> NFS makes the OGG files available freely, throughout the LAN
17:08 =orc_orc> (through an RO export)

As I recall, I used 'grip' build under CentOS 4 to populate that music archive, which xmms randomly wanders through

an interesting forgery

2010-09-07T15:18:00.001+01:00

It is quite common for an online service provider to suggest adding their 'email sending address' to a end user, so that spam filters let pieces from know senders avoid spam filtering

This piece came in. Here are the headers:

Return-Path: 
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
    bronson.owlriver.com
X-Spam-Level:
X-Spam-Status: No, score=-87.1 required=4.0 tests=BAYES_05,
    HTML_IMAGE_ONLY_24,
    HTML_MESSAGE,MIME_HTML_ONLY,RCVD_IN_BL_SPAMCOP_NET,RCVD_IN_PSBL,
    SPF_HELO_PASS,
    T_SURBL_MULTI1,T_SURBL_MULTI2,T_SURBL_MULTI3,T_URIBL_BLACK_OVERLAP,
    URIBL_BLACK,URIBL_DBL_SPAM,URIBL_JP_SURBL,URIBL_OB_SURBL,URIBL_SC_SURBL,
    URIBL_WS_SURBL,USER_IN_WHITELIST autolearn=no version=3.3.1
Received: from shadow.apd.hu (shadow.apd.hu [195.70.36.72])
    by bronson.owlriver.com (8.13.8/8.13.8) with SMTP id o8224mbp009823
    for <rpm@owlriver.com>; Wed, 1 Sep 2010 22:04:50 -0400
Date: Thu, 2 Sep 2010 04:04:49 +0000
From: Twitter <twitter-notification-rpm=owlriver.com@postmaster.twitter.com>
Reply-To: noreply@postmaster.twitter.com
To: rpm@owlriver.com
Message-Id: <6aba5bca4c284_51e06cbd75096ceb8@mx001.twitter.com.tmail>
Subject: You have 5 unread direct messages from Twitter!
Mime-Version: 1.0
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: Quoted-printable
Content-Disposition: inline
X-Campaignid: twitter20100902312977
Errors-To: Twitter
    <twitter-notification-rpm=owlriver.com@postmaster.twitter.com>
Bounces-To: Twitter
    <twitter-notification-rpm=owlriver.com@postmaster.twitter.com>
X-Envelope-To:  rpm@owlriver.com
X-Munge: added X-Envelope-To
X-Orig-Subject: You have 5 unread direct messages from Twitter!
X-Loop: herrold@owlriver.com
X-ORC: antiloop

The body is heavily obsfucated HTML, but the clear text is:

HI, RPM.

You have 5 unread direct messages from Twitter!
http://twitter.com/account/messages/rpm/RKQYA-KU4GO-417167
[medicinete.info]

The Twitter Team

If you received this message in error and did not sign up for a
Twitter account, click not my account [medicinete.info].

Please do not reply to this message; it was sent from an unmonitored
email address. This message is a service email related to your use of
Twitter. For general inquiries or to request support with your
Twitter account, please visit us at Twitter Support
[medicinete.info].

Clever enough -- the "[medicinete.info]" is added by my MUA -- Mail (reading) User Agent, alpine, and so the link to a forged site is obvious. But the use of the forged sender address, and the fact that I have a global 'whitelist' pass rule on that mail server, rather than 'per user' pass rules for the custom spamassassin on this CentOS 5 box, means that the forgery was treated as though it was from a trusted sender and favorably scored 100 points

Of course there IS no such user 'rpm' here sending email, but that was scraped off a web page in the domain, and so it draws content from hopeful spammers

"Okay, not a problem"

2010-09-05T15:37:00.001+01:00

It drives me nuts in a store or when contacting telephone support somewhere, when the clerk or call center denizen replies to my social courtesy of thanking them for some service, to receive in return:

Okay, not a problem

D*mn it -- In such a circumstance, I have usually just made a purchase, or have previously paid good money to get their firm's attention. I could care less if they were pleased to not to have had to work hard doing their appointed tasks. I know darn well they are drawing some salary to boot

I rather feel that I am entitled, instead, to:

Thank you

or perhaps,

You are welcome and it was a pleasure

as the back and forth of the interaction suggests

Oh, yes, and "No worries" usually works just about as well with me, except when used as an affirmation that all is well

living at HEAD

2010-08-26T21:56:00.004+01:00

Mark Shuttlesworth gave a keynote a few OLS ago, proposing that some folks need to 'live at head' and file the bleeding edge bugs early so nasty stuff gets fixed early, and good stuff flows down

The assumption is that the need for patches, except for branding, etc will fade away, and the character of Open Source's code improves with 'a rising tide that floats all boats'. Apple is pretty far down this road already with Clang/LLVM

I already build much from nightlies from VCS anyway (including clang), and it is simple enough to first solve a selfhosting iso builder. Then continue to populate leaf nodes to taste. Auto-reinstall daily preserving configs or detecting breakages and writing idempotent configs migrators as needed

File bugs ruthlessly, or at least relentlessly

Wire in valgrind, llvm, drill in unit tests and end to end functional tests, buildbots, LSB conformance testers, more, and file more bugs off the daily exceptions reports after new commits appear upstream. Whitelist acknowledged bugs for a while (with timeouts to keep the upstream honest)

Keep the machines busy at night rather than letting them play cards with one another

This email drafted in 5 minutes but I've ranted the rant enough times it is familiar. How is that for a real world and useful goal? Certainly better than playing 'bikeshed' politics

What not to wear

2010-08-25T19:30:00.010+01:00

Me, quoting me from a private IRC conversation

14:29 =orc_orc> disabling selinux is like having perms of 777 or no root password at all or no wrappers or no iptables 14:29 =orc_orc> only weak minds should still be doing these things

Actually, one should set the root password to a unique and hard one for each box, and only use it for recovery [our practice per the 'blue ring notebook' of procedure at one site I ran. The 'per machine' passwords were 'remembered' and kept in a bound book journal in the CIO's office safe; usage logged, resets after use noted in said journal, audits performed]. Non-local root password based login should not be enabled. Rather, one should rely on pass-phrase protected, keyed SSH access the rest of the time

Rock and Roll never forgets ...

2010-08-25T19:02:00.008+01:00

I came across this quote [pdf] today, from 2006 doing some research. From TFA:

Q: Why isn't Oracle a member of the OSDL?
A. ... We basically know where to go. We have a good relationship directly with people in the Linux community. We have all our partners. So there is no immediate advantage to being a member for us.
Not to sound arrogant, but we know how to deal with the Linux community

Gotcha, Wim ... 'does not play well with others'

... in a related vein, I cannot remember the last time I heard an Oracle representative on the weekly LSB conference call

TTYtter move to OAth

2010-08-18T14:40:00.013+01:00

One of the command line tools of interest for light weight micro-notifications is TTYtter. In the CentOS-devel mailing list a few months, I described using it to issue completion, and build closure notices doing a long running, unattended distribution build for a IBM s390x architecture

Twitter has announced, and been testing a move to using OAuth -- Rather repeat details here as to why this is compelling solution to federated authentication, and how it is still secure, please to take a look at: http://oauth.net/, and the Twitter writeup on that topic

Cutting over to the latest TTYtter (which has the needed code to use OAuth) is as easy as doing a download, setting a file permission executable, and updating a test symlink. Some minor edits to the 'rc' file were needed -- the lynx seems to lack needed crypto hooks, and so we edit to cut over to using curl. The tool then leads one through generating, and injecting locally the needed OAuth keying

[herrold@centos-5 ttytter]$ ./ttytter -- using SSL for default URLs. trying to find cURL ... /usr/bin/curl -- checking version at http://www.floodgap.com/software/ttytter/01current.txt -- your version of TTYtter is up to date (1.1.3) ** warning: -user is ignored when -authtype=oauth (default) ++-------------------------------------------------------------------++ || WELCOME TO TTYtter: let's get you set up with an OAuth keyfile! || ++-------------------------------------------------------------------++ Twitter now requires all applications authenticating to it use OAuth, a more complex authentication system that uses tokens and keys instead of screen names and passwords. To use TTYtter with this Twitter account, you will need your own app key and access token. This requires a browser. The app key/secret and user access token/secret go into a keyfile and act as your credentials; instead of using -user, you use -keyf. THIS KEYFILE NEVER EXPIRES. YOU ONLY NEED TO DO THIS ONCE FOR EACH ACCOUNT. If you DON'T want to use OAuth with TTYtter, PRESS CTRL-C now. Restart TTYtter with -authtype=basic to use a username and password. THIS IS WHAT YOU WANT FOR STATUSNET, BUT WON'T WORK WITH TWITTER AFTER AUGUST 2010. If you need help with this, talk to @ttytter or E-mail ckaiser@floodgap.com. Otherwise, press RETURN/ENTER now to start the process. Start your browser. 1. Log in to https://twitter.com/ with your desired account. 2. Go to this URL (all one line). You must be logged into Twitter FIRST! http://dev.twitter.com/apps/key_exchange?oauth_consumer_key=credentialelided12345 3. Twitter will confirm. Click Authorize, and accept the terms of service. 4. Copy the entire string you get back. ck=BbEgsckKyR1234567890fw& cs=QhuEHoZoh1234567890rg5oZjCmaddogk kjhFnaYE&at1234567890n3lqqFjredbullX pyTM3iQH6I1234567890k8Wilz& ats=BVrFP1234567890ggODKHmTChME1234567890PCo9Y -- Paste it into this terminal, then hit ENTER and CTRL-D to write it --------- ck=BbEgsckKyR1234567890fw& cs=QhuEHoZoh1234567890rg5oZjCmaddogk kjhFnaYE&at1234567890n3lqqFjredbullX pyTM3iQH6I1234567890k8Wilz& ats=BVrFP1234567890ggODKHmTChME1234567890PCo9Y -- EOF ------------------------------------------------------------------------ Written new key file /home/herrold/.ttytterkey Now, restart TTYtter to use this keyfile -- it will use this one by default. (For multiple key files with multiple accounts, write them to separate filenames, and tell TTYtter where the key is using -keyf=... .)

Easy enough. The edits to the 'RC' file, post changes are shown thus:

[herrold@centos-5 ~]$ cat ~/.ttytterrc # # user=herrold:oldpass3.2.7172.14159word keyf=/home/herrold/.ttytterkey hold=1 ssl=1 # lynx=1 curl=1 # url=https://twitter.com/statuses/public_timeline.json vcheck=1 # [herrold@centos-5 ttytter]$

and let's fire it up again and test for function:

[herrold@centos-5 ttytter]$ ./ttytter -- using SSL for default URLs. trying to find cURL ... /usr/bin/curl -- checking version at http://www.floodgap.com/software/ttytter/01current.txt -- your version of TTYtter is up to date (1.1.3) (checking credentials) test-login SUCCEEDED! -- processing credentials: logged in as herrold ###################################################### ...

After showing recent and direct posts, ends up at a prompt ready for content

... -- notification: API rate limit is currently 350 req/hr -- your version of TTYtter is up to date (1.1.3) -- you are logged in as herrold TTYtter> hello world and @ttytter using OAuth credentials TTYtter> c9> hello world and @ttytter using OAuth credentials

Let's look a bit at the files with an eye to privacy. Oops

[herrold@centos-5 ~]$ cd ~ ; ls -al .ttytt* -rw-rw-r-- 1 herrold herrold 174 Aug 18 09:43 .ttytterkey -rwx------ 1 herrold herrold 161 Aug 18 09:48 .ttytterrc [herrold@centos-5 ~]$

I fix that thus, of course:

[herrold@centos-5 ~]$ chmod 600 .ttytterkey [herrold@centos-5 ~]$

I'll send a bug report to the author, suggesting use of a umask 077 before creating that keying file. A quick restart of the client indicates it is fine with that set of permissions

All done

Stupid RPM tricks: SOURCE files No. 2 in a series

2010-08-17T15:07:00.009+01:00

The #rpm channel has this question this morning:

how can I simple include a file placed in SOURCES dir in a pkg?

Locally on a development box, I maintain a huge archive of unpacked .spec files, either written from scratch, or with automated tools locally produced in house, or extracted from SRPMs (from a mirror collection of somewhat over a quarter-million such SRPMs)

This gives me a huge reservoir to look through for a clean example. Folks in a Unix ™ like environment are fortunate to live in a culture what favors portability by rebuilding from sources, and avoiding binary data stores in favor of plain text ones. This has been a part of the environment since the earliest days of that culture. The tool grep can act as one's eyes, to scout out examples

I know from experience reading and then writing .spec files, that a file living in the ./SOURCES/ directory might be listed, and then referred to by using the identifier: SOURCE. So, I look with grep and find the following

[herrold@centos-5 ~]$ cd rpmbuild/SPECS/ [herrold@centos-5 SPECS]$ grep SOURCE *spec | wc 623 2876 44222 [herrold@centos-5 SPECS]$ grep SOURCE *spec | less

As there were LOTS of matches, I used less to scan the results a screen at a time, looking for a well known, and readily available package to pull an example from. The ISC bind nameserver is a well known and prevasively available one

[herrold@centos-5 SPECS]$ grep -i SOURCE bind*spec | grep 29 Source29: named.conf.sample cp -fp %{SOURCE29} sample/etc/named.conf [herrold@centos-5 SPECS]$ grep -i sample bind*spec Source29: named.conf.sample Source30: named.rfc1912.zones.sample # sample bind configuration files for %doc: mkdir -p sample/etc sample/var/named/{data,slaves} cp -fp %{SOURCE29} sample/etc/named.conf cp -fp %{SOURCE30} sample/etc/named.rfc1912.zones cp -fp %{SOURCE31} sample/etc/ cp -fp %{SOURCE17} %{SOURCE18} %{SOURCE19} %{SOURCE20} %{SOURCE21} %{SOURCE22} %{SOURCE23} sample/var/named ns localhost.' > sample/var/named/$f; /usr/bin/tail -n '+'`/bin/egrep -n '\\$Id: rndc.conf,v' bin/rndc/rndc.conf | sed 's/:.*$/+1/' | bc` bin/rndc/rndc.conf | sed '/Sample rndc configuration file./{p;i\ ;d}' > sample/etc/rndc.conf; %doc sample/ ...

So we see that the process is:

List that file, living in ./SOURCES/ in the .spec file
Simply use the cp command (with some options preserving timestamps, etc.) to the install staging point, and
include the destination directory or the specific file in the list down in the %files stanza of the .spec file

By reading prior examples, a person can find both good and bad examples, of course, and so cross checking is sensible. This particular example was chosen as it is a nice clean one

This post is part of a continuing series. The content of this series might be indexed out with the tag: rpm It addresses usage questions and issues raised in the #rpm IRC channel on freenode. That channel is populated by folks who are part of the RPM branch represented at: http://rpm5.org/. Your author of this content formerly served as the editor of the RPM website, for many years, until the domain owner 'pulled' it back one day. The older content formerly at the old rpm.org site is now archived at: http://www.oldrpm.org/

Enterprise distributions ... 2010-08-16T21:03:00.005+01:00 Where old software goes, to live out its declining days see this bug: https://bugzilla.redhat.com/attachment.cgi?id=439002 Perhaps the motto is: 'we shall backport no patch before its time' Mirroring a upstream master with lftp to make a local yum repository 2010-08-16T20:01:00.006+01:00 I've been assembling parts for an automated builder, over on the ia64 (Intel Itanium architecture). It has been a while since CentOS had this in the active part of the updates rotation, and I've been working on builders again, as much for relaxation as for anything else. The old binary RPMs aged away to the archive systen under the vault.centos.org hostname, and I don't want to be repeatedly hitting and loading that link for local purposes As such I set to establishing a local mirror to run providing ftp access to that mirrored content for the local ia64 to draw from in populating a build chroot. I deployed a fresh host, gave it some drive space, and hardened it. Then I installed and turned up vsftpd. And I installed lftp which I use for mirroring remote FTP or WWW sites. It is more lightweight than rsync, and as this is anonymous mirroring, there is no security issue Setting up the control file to drive the mirror could not be easier. Note: we control permitted shell level access via wrappers and iptables, and use 'keyed' ssh access, and a measure of ~/.ssh/config , ssh-agent , and ssh-add and well named config files, to manage complexity, so it feels like reaching out to a host away in a data center is the same as to one inside the local trusted network. This is not a transitive trust, of course, and remote machines cannot reach in [herrold@centos-5 localcopy]$ scp root@198.178.231.209:/root/*conf . ... snip ... [herrold@centos-5 localcopy]$ cat c55-ia64.conf # mirror -c -e \ http://vault.centos.org/4.4/os/ia64/ \ /var/ftp/pub/mirror/centos/4/4.4/os/ia64 # [herrold@centos-5 localcopy]$ and running it just as easy: [root@ia64-c44-mirror ~]# lftp -f c55-ia64.conf I took a look to make sure it got a full measure of content: [root@ia64-c44-mirror ~]# du /var/ftp/pub/mirror/centos/4/4.4/os/ia64 10076 /var/ftp/pub/mirror/centos/4/4.4/os/ia64/images/pxeboot 59680 /var/ftp/pub/mirror/centos/4/4.4/os/ia64/images 2048812 /var/ftp/pub/mirror/centos/4/4.4/os/ia64/CentOS/RPMS 165296 /var/ftp/pub/mirror/centos/4/4.4/os/ia64/CentOS/base 2214116 /var/ftp/pub/mirror/centos/4/4.4/os/ia64/CentOS 7952 /var/ftp/pub/mirror/centos/4/4.4/os/ia64/repodata 25080 /var/ftp/pub/mirror/centos/4/4.4/os/ia64/headers 2307452 /var/ftp/pub/mirror/centos/4/4.4/os/ia64 [root@ia64-c44-mirror ~]# ... and then configured a custom set of yum repository files to point to that archive [root@ia64-builder hardening]# cd /etc/yum.repos.d/ [root@ia64-builder yum.repos.d]# grep -v ^# pmman-CentOS-Base.repo | grep -v ^$ [base] name=CentOS-$releasever - Base baseurl=ftp://ia64-c44-mirror.pmman.net/pub/mirror/centos/4/4.4/os/$basearch/ gpgcheck=1 gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-centos4 priority=1 protect=1 [root@ia64-builder yum.repos.d]# The proof of the pudding is in the eating, of course, so on that ia64, let's remove and then reinstall a leaf node package (here, the joe text editor) [root@ia64-builder ~]# yum -d 0 -e 0 clean all [root@ia64-builder ~]# rpm -e joe [root@ia64-builder ~]# rpm -q joe package joe is not installed [root@ia64-builder ~]# yum -d 0 -e 0 -y install joe [root@ia64-builder ~]# rpm -q joe joe-3.1-7.rhel4 [root@ia64-builder ~]# Seems fine Stupid RPM tricks, No. 1 in a new series 2010-08-16T15:53:00.009+01:00 In the #rpm channel on freenode, a new person has wandered in with what seems like a basic question when writing up a new .spec file. I looked into the back archive for this blog, and it seems there is a way to get a list of just the items I have tagged with "rpm" When I add: %config %{_builddir}/ossec-hids-%version/etc/ossec.conf .. where the rpm will put this file when i install that package? Kind of a strange question as a line begenning: %config ... is usually placed in the %files stanza to mark the character of a file as a configuration file. Those files usually live at down /etc/ or perhaps /etc/packagename/ The author will find that this construct is not likely to place content where a general system application striving to meet and to be conformant with the File Hierarchy Standard would place such Putting that to one side, let's 'solve' here it ends up when RPMBUILD processes that .spec file stanza $ rpm --showrc | grep _builddir [herrold@centos-5 ~]$ rpm --showrc | grep _builddir RPM_BUILD_DIR="%{u2p:%{_builddir}}" cd %{u2p:%{_builddir}} /usr/lib/rpm/find-debuginfo.sh %{?_missing_build_ids_terminate_build:--strict-build-id} %{?_find_debuginfo_opts} "%{_builddir}/%{?buildsubdir}" -14: __mono_provides /usr/lib/rpm/mono-find-provides %{_builddir}/%{?buildsubdir} %{buildroot} %{_libdir} -14: __mono_requires /usr/lib/rpm/mono-find-requires %{_builddir}/%{?buildsubdir} %{buildroot} %{_libdir} -14: _builddir %{_topdir}/BUILD RPM_BUILD_DIR="%{_builddir}" [herrold@centos-5 ~]$ Obviously here is a need to run another query to examine '_topdir' to fully answer that question: [herrold@centos-5 ~]$ rpm --showrc | grep _topdir -14: _builddir %{_topdir}/BUILD -14: _rpmdir %{_topdir}/RPMS -14: _sourcedir %{_topdir}/SOURCES/%{name} -14: _specdir %{_topdir}/SPECS -14: _srcrpmdir %{_topdir}/SRPMS -14: _topdir /home/herrold/rpmbuild [herrold@centos-5 ~]$ So that file will end up at: %{_topdir}/BUILD/ossec-hids-%version/etc/ossec.conf , or more precisely: /home/herrold/rpmbuild/BUILD/ossec-hids-%version/etc/ossec.conf The 'takeaway' here is that the "_builddir" variable is not customarily used down in the %files stanza of a .spec file stirring the pot with the same old spoon -- rpm options 2010-08-13T22:45:00.005+01:00 The same old question came up in an IRC channel yet again today, by a person who will not read, nor Google. Once again, with feeling, here is a quick script to study, as to stripping out excessive content on a Red Hat, CentOS, or Fedora derived 'multi-arch' box [root@centos-5 bin]# cat ./strip-non-x86_64.sh #/bin/sh # # strip-non-x86_64.sh # # strip all non x64_64 content (and also leave behind # noarch stuff) for more build dependency friendly # environment # # Copyright (c) 2010 R P Herrold # GPLv3+ # tossed together for education purposes for a blog post # # optionally run over or over, after enabling the removal # or one pass, if you uncomment two places # # not designed to be the more efficient one pass solution # which is possible # # NODEPS="--nodeps " # # see: man rpm, see '--qf' or: queryformat # or --querytags # for i in `rpm -qa --qf '%{name}.%{arch}\n' | sort | grep -v ^kernel | \ grep -v noarch$ | grep -v "86_64" | grep -v ^gpg-pubkey `; do echo "${i}" # rpm -e ${i} ${NODEPS} done # [root@centos-5 bin]# ./strip-non-x86_64.sh [root@centos-5 bin]# I am so bone tired of people who want to be fed from the same old dirty spoon For extra credit see: man yum.conf and add the needed multilib_policy = best line to etc/yum.conf 'your winnings, sir' 2010-08-13T16:18:00.014+01:00 From the movie: Casablanca Captain Renault: I'm shocked, shocked to find that gambling is going on in here! [a croupier hands Renault a pile of money] Croupier: Your winnings, sir. ... A person who was, I guess, looking to stir up something, was "shocked, shocked" in a CentOS mailing list to learn I have a direct commercial interest in the success of CentOS, and do not hide it, based on this post: one; and my reply As a bit of 'inside baseball', behind the scenes, I have not had access to certain parts of back side CentOS facilities for a while, and formally floated my request for getting this access right cleaned up, after informal efforts were not advanced by the party needing to take the needed steps. I was met with this reply in my mail spool yesterday: Subject: Giving Russ wider access to machines etc I have 2 concerns when it comes to giving Russ wider machine level access to [those facilities] 1) He has a direct commercial interest in CentOS. ... This is easy -- I co-founded this project long since, and have in no wise ever engaged in any act to damage it. The author proposes to exclude me as I have a 'direct commercial interest'. A slippery slope, not well thought through, and not a fight that makes ANY sense in an enterprise distribution. Form follows function Either the GPL means what it says, and so also similar licenses such as BSD, or they do not. I commercialize Linux and other Open Source, and have for many years long before CentOS existed, and don't hide it If some people don't like that they can say so, and I'll surface the discussions In other news: water is wet CentOS still has no corporate existence to protect its members, as part of an unincorporated association, from having to individually defend suits such as Oracle's today I am not going to hang around exposed, without a corporate form and protection, until people decide they want to move forward. It's been a year since the Lance letter and the time for 'all deliberate speed' is over If it turns out that only one person calls the shots these days in CentOS and to my exclusion, so be it. I'll not be a servant to another, and I'll not be improperly excluded Chickens, coming home to roost 2010-08-13T14:27:00.007+01:00 I see in the overnight news that the trades have picked up on the fact that Oracle (now owner of the former Sun's intellectual property in relevant part) has filed suit, on patent grounds against Google Big Deal, right? Doesn't affect me, right? Wrong Two years ago, the Linux Standards Base was talking about requiring Java functionalities in a 'conformant to the LSB standard' distribution. I spoke strongly against this as did Alan Cox, but as I am just one person; and Red Hat is largely uninterested in the LSB other than to conform to it to satisfy the 'checkbox' on an evaluation matrix, a 'trial use' of Java appeared in LSB 4.0 I asked the question and noted my response: So, exposing people to an NDA, and potentially unlimited liability, and defense costs, in order to be be able to test and demonstrate their distribution is LSB complaint is fine? I think not. I think the LSB cannot in good conscience place distributions in the line of fire, until and unless the testing tool is not a 'spring gun'. The pain comes in part from this: (f) you agree to defend and indemnify Sun and its licensors from and against any damages, costs, liabilities, settlement amounts and/or expenses (including attorneys' fees) incurred in connection with any claim, lawsuit or action by any third party that arises or results from (i) the use or distribution of your Operating System, or any part thereof, in any manner ... and it did not take a rocket scientist to see that patent threats were in the minefield: LSB needs to decide how much and what kinds of risk [contractual NDA, contractual indemnification, trademark infringement, patent, copyright, more?] it is willing to ask distributions to expose themselves to, to become 'LSB compliant'. The rubber has just hit the road; the chickens are on the glide path, coming home to roost. The hypotheticals just got instantiated, and the danger is real NFS aide to memory 2010-08-12T21:38:00.006+01:00 I had occasion to add NFS Read Only exports from several servers holding several generations of source RPMs to build, to be mounted on a build client. I always have to look at an existing setup, and replicate the configuration files ( /etc/exports and /etc/fstab ). Some RPM packages are needed as well On the server side, we need to install the package: nfs-utils as a 'keystone' that pulls in other dependencies it needs # yum install nfs-utils Do an edit in /etc/exports # /path/to exported/directory 10.85.0.0/16(ro) # And finally enable the services, and start them: # /sbin/chkconfig portmap on # /sbin/chkconfig nfs on # /sbin/service portmap start # /sbin/service nfs start Turning to the client side, we needs a running portmap and netfs # yum install nfs-utils Make any needed mountpoints: # mkdir -p /mnt/nfs/1 # mkdir -p /mnt/nfs/2 # mkdir -p /mnt/nfs/3 # mkdir -p /mnt/nfs/4 Add needed entries in the /etc/fstab [Note: I spread the content over two lines for each entry for presentation purposes] # 10.85.85.232:/var/ftp /mnt/nfs/1 nfs rsize=32768,wsize=32768,soft,nolock 0 0 10.85.85.253:/var/ftp /mnt/nfs/2 nfs rsize=32768,wsize=32768,soft,nolock 0 0 10.85.85.154:/var/ftp /mnt/nfs/3 nfs rsize=32768,wsize=32768,soft,nolock 0 0 10.85.85.133:/var/ftp /mnt/nfs/4 nfs rsize=32768,wsize=32768,soft,nolock 0 0 # And finally enable the services, and start them: # /sbin/chkconfig portmap on # /sbin/chkconfig netfs on # /sbin/service portmap start # /sbin/service netfs start Do the mounts: # mount -a Test using df -h and ls down in the mounts All done line noise and random numbers 2010-07-29T22:51:00.009+01:00 "Anyone who considers arithmetical methods of producing random digits is, of course, in a state of sin" -- John von Neumann Stipulated, but I am beset by closer devils, and I've tinkered with a mild solution I like. Let me tell you more I've been badgered as some web sites have moved to Java-script evaluation routines for site passwords. Some require mixed case; others punctuation; no doubled letters; minimum length. Contrariwise some limit the character set to prohibit what others require As such, for the last year or so I have been playing with a quick script on my CentOS 5 box, to generate a unique password per website, and keep a master index of the userid, email address used, and password used. This of course limits my ability to connect to those sites when away from that list. Open-ID roll-in seems to be coming however, and I have a rather clever device, backed by Verisign, and using an inexpensive OTP -- one time password -- hardware device as part of the authentication process The little generator I wrote is based on simple shell tools -- md5sum, df, date, ps, cut, tr and so forth. It gathers a bit of entropy from a few sources froma few systems around the office, which should be non-correlated from a theoretical basis in the time frames at issue. It does some hashing to get good dispersion. Then it expands into 3 or 4 character vectors each 16 characters wide, using the hexidecimal digits that md5sum emits, as translated by tr; the first three are letters, upper, lower, and digits; the fourth character set are selected specials and punctuation excluding some shell meta's. Depending on a limitation by an option to -a, that vector may be limited to the alpha-numerics only, or also stir in the specials That 'deck of characters' is handed off to a 'repeated cut of the deck' shuffler, and returned mixed once more just for good measure I then add a 'bumper' of a letter or digit at each end [one site prohibited starting with a special], and a second character of 'bang' to prevent a mouse slip from dropping a password into the bash history in the case of a panel slip The results are assembled, trimmed to an optionally specified length, and displayed, where I harvest them as mentioned above Really, passwords need to die, die, die, but that is for another post [herrold@centos-5 bin]$ for i in `seq 1 10 `; do ./gen-pw.sh ; done e!~YJAJ{e:sU[4 2!R5K*U#)LoH~2 c!T)T7A10RjS}7 1!cGJ5T@]YjW>4 5!Q+#)K8:@rT]2 8!^)S~FF:5lV<4 b!dJ:TcKK{tQ)9 2!1dEa:fe~mR{4 3!cD1:eH^6wO*d d!U*5(UEFWsI:e [herrold@centos-5 bin]$ for i in `seq 1 10 `; do ./gen-pw.sh -a ; done 5ec280RSY5wIfd 0ddQ31EdJGmIdb 7eb52645U1tH06 0bfb401eG1jUa5 c2cT85QY22pS2d ba8EALA9RRtR1f 35f59JRD6KpN04 7ed956UbA9pV59 402H3YLLR8hR3e f2a0Aa9J0JrPde [herrold@centos-5 bin]$Completely un-memorizable of course, so really only suitable to a protected physical environment where one may write them down Random number sin, of course, but the cyber-ninja can more readily put my thumb between pliers jaws, than predict the pseudo random source values I used, I'll readily spill the secret to access my LOL CATS site account. ... I still have to get around to building a few non-correlated hardware random number generators -- diode based, lava lamp based, dice tumbling machine for serious entertaining, I guess Letters, we get letters ... 2010-07-27T22:46:00.014+01:00 His mother must be so proud to have raised such a pottymouth: Date: Tue, 27 Jul 2010 15:01:39 +0000 From: BOBBY RAY MCALLISTER To: centosweb@centos.org Subject: www.centos.org - Contact the CentOS WebMaster Form BOBBY RAY MCALLISTER submitted the following Information: Email BLACKTHORNE4440@AOL.COM URL AIN'T GOT ONE ICQ NONE OF THEM, EITHER Company AIN'T WORKING Location U.S. OF FUCKING A. Comments NO MORE. NO FUCKING MORE OF YOUR BULLSHIT SOLICITATIONS FOR "THIRD PARTIES" TO ME, MOTHERFUCKERS. THE NEXT ONE GETS COPIED TO THE FTC AS WELL AS FBI FOR PROSECUTION UNDER ANTI-SPAM STATUTES. I HAVE ASKED YOU TO MAKE FUCKING WELL SURE TO DELETE MY ADDRESS, AND I STILL GET BULLSHIT FROM YOUR STUPID FUCKING ASSHIOLE MOTHERFUCKING CLIENTS. Mozilla/4.0 (compatible; MSIE 7.0; AOL 9.0; Windows NT 5.1; Trident/4.0; GTB6.5; .NET CLR 1.0.3705; .NET CLR 1.1.4322; Media Center PC 4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729) The template CentOS placeholder web page, which is found when people place IP addresses from mail headers into a web browser address bar and hit an unconfigured system, makes it quite clear that we merely supply an operating system, of course. I thought we went through all of this years and years ago with the Tuttle, Oklahoma to and fro. Reading with comprehension must be a rare skill; civility, rarer still The end headers indicate this person has 'hit the trifecta': Windows, AOL, and Media Center Sadly, the salesperson who sold him those fine products must have also sold him a teletype as the console to use as well, as it is in ALL CAPS Amazing stuff. I see that I need to amend that form to show originating IP, and perhaps put it under a 'captcha' to ensure at least some ability to read before posting This one needs to be written while it is fresh 2010-07-27T00:25:00.007+01:00 I write about CentOS, annoyances, and nuisances all the time. 'Lest people consider me a grumpy old man, never pleased, I'll drop my guard a bit here and now, and perhaps just this once I just posted this in Joe Landman's comments section to his blog, but it might get lost. It bears repeating Russ Herrold says: July 26, 2010 at 6:23 pm Hi, Joe I have not had the chance to write the blog post, but it will get written. My Dell laptop died, and I sent it to the following folks for refurb, based on some strong recommendations in a TX LUG mailing list It came back fine, but then died. http://twitpic.com/20p796 I was a slacker and did not ship it back right away. Even after the warranty interval expired, however they took it thru their intake, ID’d a bad video card they had added during the refurb, and swapped it out. gratis And then shipped it back to me FedEx ... again gratis One guess who I’ll be using from now on, for out of warranty repairs of Dell kit http://www.parts-people.com/ 512-339-1990 Tell them I sent you ;) Please say 'thanks again' for me – Russ herrold Checklist: RO FTP server setup 2010-07-20T18:36:00.018+01:00 Setting up a new RO FTP server setup The primary usage case is we describe is how to deploy a read-only FTP server with no end user accounts, to be used for distribution of content (here, to be a 'hotfix' archive for publicly accessible binary updates, accessible through yum). We need this to work around a temporarily broken update in CentOS space. We can also use it to add additioanl packages under and under the mediation of the rpm package database We start with a hardened PMman instance. A secondary purpose of this post is to work from first principles through adding a proper local 'forked packages' archive for CentOS users to follow as a worked example. At all times, we will strive to follow proper sysadmin 'best practices' discpline under SElinux, wrappers and iptables Install and enable vsftpd which is the package holding the stock ftp daemon -- yum can do this trivially yum install vsftpd Then enable the ftp server: /sbin/chkconfig vsftpd on and create a pilot file to look for in later testing: mkdir -p /var/ftp/pub/mirror echo test > /var/ftp/pub/mirror/README.txt Run updates, just 'because' and as a matter of good sysadmin yum update yum clean all Open wrappers to permit anonymous FTP connections. We edit /etc/hosts.allow and add: vsftpd: ALL@ALL Amend the iptables rules to allow ftp. The file /etc/services reminds us that FTP normally lives at TCP port 21 Add to /etc/sysconfig/iptables-config to include 'ip_conntrack_ftp' in the list of 'IPTABLES_MODULES=' IPTABLES_MODULES="ip_conntrack_ftp " and then, in /etc/sysconfig/iptables we add a line to pass FTP content: -A RH-Firewall-1-INPUT -m state --state \ NEW -m tcp -p tcp --dport 21 -j ACCEPT [Note: We use the backslash convention here, but iptables does not support this in its config files] Run the unit through a reboot, both to 'set' the updates by stopping use of any libraries held open through that update, and also to ensure that it works as expected after a 'hands off reboot' Test from a remote host that FTP works as expected [herrold@centos-5 ~]$ lftp 198.49.244.190 lftp 198.49.244.190:~> cd /pub/mirror cd ok, cwd=/pub/mirror lftp 198.49.244.190:/pub/mirror> ls -rw-r--r-- 1 0 0 5 Jul 20 16:56 README.txt lftp 198.49.244.190:/pub/mirror> cat README.txt test 5 bytes transferred lftp 198.49.244.190:/pub/mirror> exit [herrold@centos-5 ~]$ ... great At this point, we have a working RO anonymous ftp server, and can populate it with content. Free for some people just means they are not footing the bill ... maybe 2010-07-15T18:07:00.011+01:00 I see the following in the New York times today: Health Plans Must Provide Some Tests at No Cost By ROBERT PEAR Published: July 14, 2010 WASHINGTON — The White House on Wednesday issued new rules requiring health insurance companies to provide free coverage for dozens of screenings, laboratory tests and other types of preventive care. The new requirements promise significant benefits for consumers — if they take advantage of the services that should now be more readily available and affordable. In general, the government said, Americans use preventive services at about half the rate recommended by doctors and public health experts. The rules will eliminate co-payments, deductibles and other charges for blood pressure, diabetes and cholesterol tests; many cancer screenings; routine vaccinations; prenatal care; and regular wellness visits for infants and children. ... I assume that the reporter no longer believes in the tooth fairy. The article is tailored as news, and placed in that section of the paper (Page A16) by the Times editors. It has a laundry list of wonderful tests and services that no 'right thinking' person can deny are useful and desired But the suggestion is a 'promise [of new] significant benefits for consumers — if [only] they take advantage of the services' without a corresponding cost for getting there. No hint nor argument is made that such 'medical' services are unavailable for private purchase already Indeed, at the end of the day, there is no support for the headline writers assertion of 'no cost' and the reporter is well willing to disregard the pesky question of how to pay for this largess. Clearly these tests are not free and when accounts are settled; these costs will either pass through in a rate base, or the provider will exit the market it cannot make money in, or the insurance market will wither and die as 'the government' provides an 'option' that picks up the tab ... . But the problem is -- 'the govenment' at whatever level likewise needs to get the money to pay for such happy healthiness, and from the very same pool of people 'benefitted' It is not at all clear that the transaction friction of a single govenment payer works at all well, or that having no choice but 'insurance' through government once the private insurers die is a good thing at all. In watching the 'response' of the government to the oil spill in the Gulf, it is patently clear that government 'oversight' has slowed the response, as BP has become risk adverse to the (reasonable) prospect of being second-guessed at every turn, and so is seeking prior governmental approval before acting in the remediation. The ccase can be made that playing 'Mother may I?' has harmed the Gulf more than the prior approach Do we really think that a central government single point of control is going to react as well and quickly as a local doctor on the scene, when Aunt Minnie is lying, dying under an oxygen tent and needs some immediate surgery? Under the current system, the doc knows that he'll get paid, perhaps only in part of what is billed as a 'list price' for a prodedure, but eventually from the present model But that is the end game, anyway, right? Vote and mandate 'bread and circus entertainment' ... until the producers all surrender and act to stop being charged for 'free' benefits to the consumers 'The problem with socialism is that eventually you run out of the other peoples (willing to be robbed of their) money' SELinux other voices 2010-07-05T19:58:00.011+01:00 The RHCE in channel of my last post complains I was too hard on him or her. Also that person points out they used a differing approach for building the new policy file, which permits more atomicity in maintaining several policies (here, sorting by daemon). While I invited reply by way of a formal post to that person, it appears that this is their 'final word' ("topic closed") on the matter. As such I note it here for those of you playing along at home: grep vsftpd /var/log/audit/audit.log | \ audit2allow -M vsftpd semodule -i vsftpd.pp vi vsftpd.te checkmodule -M -m -o vsftpd.mod vsftpd.te semodule_package -o vsftpd.pp -m vsftpd.mod semodule -i vsftpd.pp More information that is accurate is better than less. Clearly there are many paths to rule generation and maintenance. The takeaway remains: Use, and do not disable, SELinux Thanks for the feedback SELinux sanity outline 2010-07-05T18:56:00.015+01:00 Rusty Coker mentioned in a recent blog post that he had not found a COLO facility or VM provider that enabled SELinux in its hosts by default. People regularly whine: It's too hard, and I don't need it and disable the SELinux protections. Foo I call: Bull on the latter As to the former I sent a private email to Rusty, and offered to 'comp' him an instance to break If anyone knows of a virtual hosting company that runs Xen or KVM virtual machines with SE Linux support then please let me know, I'll write a blog post comparing such companies if there are some. umm -- I would be embarrased to be a hosting provider which did NOT enable SElinux Please feel free to set up a 'comp' account at: http://www.pmman.com/signup/ at the green arrow. Use the [please do not repeat this] 'Offer Code' of: ... ... I repeated the offer at his blog's comment site And the question came up today in the #centos IRC channel 13:52 Andro1d> orc_orc: how can i recompile a pp from a te ? 13:53 Andro1d> checkmodule -M -m -o vsftpd.mod vsftpd.te gives a lot of errors :-/ 13:53 orc_orc> ehh? 13:53 wolfy> Andro1d: http://wiki.centos.org/HowTos/SELinux [CentOS wiki] 13:53 orc_orc> make a working dir -- say: mkdir -p /etc/selinux/targeted/foo and cd into it 13:54 orc_orc> Gather all the selinux noise: audit2allow -i /var/log/audit/audit.log* -m local > local.te 13:54 Andro1d> hm, I think I'm missing some types in my .te file 13:54 orc_orc> Note the '*' in that prior line, which reads all log files present 13:54 Andro1d> mom... 13:54 orc_orc> Install the selinux-devel package for the needed Makefile 13:54 Andro1d> don't wanna make a "huge" selinux policy :) 13:54 orc_orc> Then run: make -f /usr/share/selinux/devel/Makefile 13:55 orc_orc> and apply it: semodule -i local.pp 13:55 orc_orc> Test again 13:55 Andro1d> yop, mompl 13:55 orc_orc> When happy, be sure to save a versioned copy, because SELinux audit file ageing will cause you to forget what was needed in that merge 13:55 orc_orc> For extra credit, amend: /etc/audit/auditd.conf to retain a sensible universe of back logs 13:56 orc_orc> '4' is wayyy too small wolfy (a channel regular who offers reliable answers), pointed to the CentOS secondary source answer in the wiki; this post will also pass into our planet as yet another piece of documention and 'cheatsheet'. You saw a self-described RHCE (and he was proud of it coming into the channel today) doing that whimpering for his mommy as I read him the 'riot act'. I don't care in the least that this is new and 'hard' -- growing and learning new tools is part of the Unix culture, always has been, and always will be. That is why I try to make #centos a learning venue rather than a drive-by 'spoon-feeding' shop How many times do we need to bang the SELinux drum to get your attention? Yes, you lazy slogs of alleged sysadmins who simply disable SELinux, I am talking to YOU! yep - words are hard to memorize, but this is a basic 'lather, rinse and repeat' cycle which one can solve experimentally if not predictively from knowledge of what is happening. Run a tail -f /var/log/audit/audit.log if you must to see when the rule set needs to be rebuilt But stop disabling SELinux and stop making excuses lost memories 2010-06-27T02:56:00.011+01:00 From time to time, "we" 'clean house' and find the black trash bags. It is no surprise to me, of course for and earlier me have done to work of carefully tied packaging closed, and cacheing treasures up in the attic; from time to time, I am instructed to 'get rid of that clutter' as the now grown kids 'will never use those again' I am slow to act on this injunction The Brio trains, the metal Erector set, the cast lead soldiers and molds, the Duplo blocks, the stuffed animals, Lincoln logs, the McGuffey readers, the arrow and ax heads collected in the fields, have all fallen to head of the queue for disposition over time. Stuffed animals were in the dock this past weekend. At that point, I usually nod silently, carefully re-tie the sack, and set it to one side for a moment. Then my new task is to find a new hiding place for the bag in question after her attention turns to other matters But a grandchild's mother and the child were delighted with the animal figures from my preservation efforts, even if the spouse was not as well pleased to see 'those old things' again A few weeks ago, the Brio train set that was set aside in a cardboard box, up in the dark to rest almost two decades ago came out. It moved in with a grandson infatuated with rolling stock and was 'new' again; The Erector set, the melting pot and molds, all gone (not to return with current day safety rules — choking hazard of the nuts and bolts, heavy metal fumes). I am on the lookout for a replacement McGuffey reader set (that friend of books that taught me to read upstairs in a quiet room as the adults 'talked' downstairs), so I can 'seed' a room for young visitors The flints and shaped stones? I was not attuned to their disposition occurring; a 'sharpie' sweet-talked a sale for a pittance from a elderly family member when 'cleaning up' prior to closing down a house before sale. That lot of childhood treasures also carried out the door the minnie balls I dug from the earth at Gettysburg Entropy won a round that time; I know we'll battle again. [An earlier version of this appeared at Victor Niederhoffer's Daily Speculations, which aggregator I recommend] Debian mkfs is working again 2010-06-24T15:13:00.014+01:00 It's been a long June. I noticed early on that an update in Debian testing had moved mke2fs from one package to another without getting all the library dependencies right. As such I spent June without the ability to lay down a filesystem on a new partition with the 'proper' tool. Part of my series on logfile reading includes a task to review the 'percent full' for each partition (and to relocate or clean out fat ones) to avoid running out of room in a self-inficted denial of services attack I tried the obvious fallback to build a new filesystem: busybox but the version found in Debian Testing was lacking a needed build time switch. I filed the bug, and considered a local patch, or perhaps whether to rebuild of part of the chain needed to fork mkfs for a bit, but my need for space to reorganize a host's files was not that great nor urgent. Just pesky each day to see I knew from reading the bug reports that the fix had been committed and 'ageing' in the Debian fashion to its move from an Unstable 'nightly' to a mildly tested (or at least not black-balled) state and promotion into Testing nfs2:~# apt-get upgrade Reading package lists... Done Building dependency tree Reading state information... Done The following packages have been kept back: ksysguard libdevmapper1.02.1 The following packages will be upgraded: bsdutils e2fslibs e2fsprogs iptables iso-codes libblkid1 libcomerr2 libenchant1c2a libffcall1 libmime-tools-perl libnetpbm10 libss2 libuuid1 lockfile-progs mount mutt netpbm shared-desktop-ontologies util-linux 19 upgraded, 0 newly installed, 0 to remove and 2 not upgraded. Need to get 9,841kB of archives. After this operation, 115kB disk space will be freed. Do you want to continue [Y/n]? y ... nfs2:~# I've been running repository data update operations daily .. the Debian approach is more measured in its pace than we use with CentOS, and I think we may have something to learn there. It is a rare package update that cannot wait for a daily repo data update, push and mirror overnight in our space, and it would avoid much confusion to casual sysadmins Those bolded packages in that clutch of upgrades looks promising ... nfs2:~# mkfs /dev/sda12 mke2fs 1.41.12 (17-May-2010) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks 237568 inodes, 949835 blocks 47491 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=973078528 29 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736 Writing inode tables: done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 28 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override. nfs2:~# date Thu Jun 24 10:13:17 EDT 2010 nfs2:~# Lovely; I'm back in business Reading the logs, part 3 -- Run your updates 2010-06-19T14:26:00.016+01:00 It looks like I'll be writing these for a while as I clean up logfile noise. The earlier pieces are here and here. I say 'noise' here because they are not false positives, but neither are they material, just more a nuisance One the things every admin who reads log files sees are automated scanners looking for exploits in 'canned' packages that were installed but have not been updated, either because the admin for a given machine has neglected to run updates, because it is not a publicly known exploit, or because the upstream has not yet addressed the matter. A pattern that has emerged with our PMman with a data center with large contiguous swaths of IP space (and hosts scattered in assignment in that relatively compact range, said hosts reporting to me centrally) is as follows. The hostile exploit scanners are not even trying to be subtle any more -- they simply march sequentially through IP ranges, and inventory if a given weakness is present on every host to which they connect Today, I focus on one sample report stanza: --------------------- httpd Begin ------------------------ Requests with error response codes 400 Bad Request HTTP/1.1: 1 Time(s) 403 Forbidden /index.html: 1 Time(s) 404 Not Found /cms/e107_files/e107.css: 1 Time(s) /db/e107_files/e107.css: 1 Time(s) /e107/e107_files/e107.css: 1 Time(s) /e107_files/e107.css: 1 Time(s) /forum/e107_files/e107.css: 1 Time(s) /index.php: 1 Time(s) /manager/html: 1 Time(s) /portal/e107_files/e107.css: 1 Time(s) /site/e107_files/e107.css: 1 Time(s) /web/e107_files/e107.css: 1 Time(s) ---------------------- httpd End ------------------------- and apache can handle this trivially: # # file: noexploit.conf # # send scanners off to see the wizard # Redirect permanent /cms http://127.0.0.1/ Redirect permanent /db http://127.0.0.1/ Redirect permanent /e107 http://127.0.0.1/ Redirect permanent /forum http://127.0.0.1/ Redirect permanent /manager http://127.0.0.1/ Redirect permanent /mysql http://127.0.0.1/ Redirect permanent /phpmyadmin http://127.0.0.1/ Redirect permanent /phpMyAdmin http://127.0.0.1/ Redirect permanent /portal http://127.0.0.1/ Redirect permanent /site http://127.0.0.1/ Redirect permanent /user http://127.0.0.1/ Redirect permanent /users http://127.0.0.1/ Redirect permanent /web http://127.0.0.1/ # The obvious next step is to package deployment hardenings, and add them to a local RPM repository so that simply running updates, as with yum will get the current best approaches on hardening, en masse, on all the servers Reading the logs ... 2010-06-08T14:25:00.006+01:00 I see the following from logwatch in the overnight log file review: --------------------- httpd Begin ------------------------ Requests with error response codes 404 Not Found /crossdomain.xml: 1 Time(s) ---------------------- httpd End ------------------------- and so I go digging: [root@centos-5 httpd]# cat error_log [Sun Jun 06 04:02:04 2010] [notice] Digest: generating secret for digest authentication ... [Sun Jun 06 04:02:04 2010] [notice] Digest: done [Sun Jun 06 04:02:05 2010] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations [Mon Jun 07 14:20:39 2010] [error] [client 127.0.0.2] File does not exist: /var/www/html/crossdomain.xml Sure enough. It looks as though some piece of Flash code is hoping to 'leverage' a cross-domain permission to include something I may not have intentionally intended to allow. See the note at: http://kb2.adobe.com/cps/142/tn_14213.html For the sake of argument, assume you HAD to web view as root, as say with an operating system that required you use a browser front end to access system updates. Assume also that you improvidently viewed a 'seeder' of bad things that WROTE a hostile crossdomain.xml for later use by a second piece of hostile Flash to 'reap' Oops ... game over Running down stray errors 2010-04-08T19:33:00.024+01:00 Part of my daily routine is to check the logwatch summary, note and address any security matters, and then to chip away at the friction and non-working parts of the compute environments in which I can effect change This one has been on my radar for a while, but it is on a protected interior machine, not disabling, and so not critical. From a configuration file review, and with reading of the sendmail and openssl documentation, and some 'googleing' I just could not see where the error was. **Unmatched Entries** STARTTLS=client, error: SSL_CTX_use_certificate_file(/etc/mail/certs/xps400.first.owlriver.net-10.pem) failed: 173 Time(s) STARTTLS=client, error: SSL_CTX_check_private_key failed(/etc/mail/certs/xps400.first.owlriver.net-10.key): 0: 173 Time(s)To test if sendmail is compiled with STARTTLS support, we can run the following command: $ sendmail -bt -d0.8 < /dev/null Compiled with: DNSMAP HESIOD HES_GETMAILHOST LDAPMAP LOG MAP_REGEX MATCHGECOS MILTER MIME7TO8 MIME8TO7 NAMED_BIND NETINET NETINET6 NETUNIX NEWDB NIS PIPELINING SASLv2 SCANF STARTTLS TCPWRAPPERS USERDB USE_LDAP_INIT more concisely: [root@xps400 certs]# sendmail -bt -d0.8 < /dev/null | grep -i tls NETUNIX NEWDB NIS PIPELINING SASLv2 SCANF SOCKETMAP STARTTLSAnd, yet when we connect to the mailserver to test if STARTSSL is advertised [herrold@centos-5 ~]$ telnet xps400 25 Trying 10.16.1.112... Connected to xps400.first.lan (10.16.1.112). Escape character is '^]'. 220 xps400.first.owlriver.net ESMTP Sendmail 8.14.3/8.14.3; Thu, 8 Apr 2010 14:41:48 -0400 EHLO localhost 250-xps400.first.owlriver.net Hello centos-5.first.lan [10.16.1.101], pleased to meet you 250-ENHANCEDSTATUSCODES 250-PIPELINING 250-8BITMIME 250-SIZE 250-DSN 250-ETRN 250-AUTH GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN PLAIN 250-DELIVERBY 250 HELP quit 221 2.0.0 xps400.first.owlriver.net closing connection Connection closed by foreign host. [herrold@centos-5 ~]$ openssl has the rather interesting sub-tool s_client 'SSL/TLS client program' which knows how to talk several protocols though a transition into a secure sockets mode as well [root@xps400 ~]# openssl s_client -connect localhost:25 -starttls smtp CONNECTED(00000003) didn't found starttls in server response, try anyway... 2005:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:583: [root@xps400 ~]#so ... starttls is linked in sendmail as present, but is not working Let's run this down by making sure all the needed moving parts are present: [root@xps400 mail]# cd /etc/init.d/ [root@xps400 init.d]# ls *sasl* saslauthd [root@xps400 init.d]# chkconfig --list saslauthd saslauthd 0:off 1:off 2:on 3:on 4:on 5:on 6:off [root@xps400 mail]# /sbin/service saslauthd restart Stopping saslauthd: [ OK ] Starting saslauthd: [ OK ] [root@xps400 mail]# /sbin/chkconfig saslauthd on [root@xps400 mail]# /sbin/service sendmail restartand from another panel watching the log files: # tail -f /var/log/maillog Apr 8 11:39:30 xps400 sendmail[3536]: STARTTLS=server, error: SSL_CTX_use_certificate_file(/etc/mail/certs/xps400.first.owlriver.net-10.pem) failed Apr 8 11:39:30 xps400 sm-msp-queue[3547]: starting daemon (8.14.3): queueing@01:00:00.. so ... sendmail is telling us that it refuses to use: /etc/mail/certs/xps400.first.owlriver.net-10.pem Looking at the certificate countersign: # less /etc/mail/certs/xps400.first.owlriver.net-10.pem N CERTIFICATE----- MIIHATCCBemgAwIBAgICFokwDQYJKoZIhvcNAQEFBQAwgYwxCzAJBgNVBAYTAklM MRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRh ... a HA !! Looks like it was a bad 'scrape and paste' by me when I retrieved and installed the counter-signing of the certificate from startssl. The start of that file should look like: -----BEGIN CERTIFICATE----- MIIHATCCBemgAwIBAgICFokwDQYJKoZIhvcNAQEFBQAwgYwxCzAJBgNVBAYTAklM MRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRh bCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAy ... As always, such mistakes are only obvious once found. A side observation. A recent blog bost "Securing the Enterprise" by Eddy Nigg of Startssl pointed out the willingness of some Certificate Authorities to sign whatever they are offered, and some admins to not consider this attack vector in submitting such, in the case of hosts in an RFC-1918 IP assignment block or non-DNS mediated formal namespace. Undetected forgeries are facilitated, and I am certain such Man in the Middle compromises occur in coffee-shops with wireless access all the time. As such the host: xps400.first.lan at 10.16.1.112 also appears with a internal split DNS PTR and A record as: xps400.first.owlriver.net The certificate for it countersigned by startssl is useful here for verifications [herrold@centos-5 ~]$ host xps400.first.owlriver.net xps400.first.owlriver.net has address 10.16.1.112 xps400.first.owlriver.net mail is handled by 20 mailhub.owlriver.net. xps400.first.owlriver.net mail is handled by 10 new.owlriver.com. [herrold@centos-5 ~]$ The trick to getting the mailserver to answer as xps400.first.owlriver.net was even easier -- just some DNS work, and a quick addition of a non-customary line in the /etc/mail/sendmail.mc, a rebuild, and a restart: [herrold@xps400 mail]$ grep xps400.first.owlriver.net sendmail.mc | head -1 define(`confDOMAIN_NAME', `xps400.first.owlriver.net')dnl [herrold@xps400 mail]$ Are you using SSL certificates where you can and should? ... Is the namespace of network they protect thoughtfully designed? StartSSL makes it easy to do, for a person willing to be minimally 'validated' as to their identity and their right to administer a given domain. Once that identity check is done, the process is essentially free of any marginal cost to roll out as many certificates as one wishes, and to NOT 'cop out' or cut corners here Caller ID, wiretapping, call recording, and the federal Do Not Call list 2010-02-24T21:51:00.028Z There is a witches brew of rules that people making outbound telephone calls need to thread through. Also, the recipient of a call needs to observe some as well. Let's start in reverse order: Particularly, in the US, some states require consent from only ONE party to a telephone communication; others require TWO [or ALL, in the case of a conference call] participants to so consent. The asserted misconduct case of Linda Tripp in Maryland comes to mind. Linda got into some hot water for chatting up Monica's lovelife with some girltalk about Bill Clinton and recording it without needed consents from "that woman, Ms. Lewinsky" and then turning those recordings over to Kenneth Starr's office Neither side of the aisle is without stain in this space, it seems; recall that back earlier in the Clinton administration that a couple in Florida recorded a conference call bridge leg, on which the cell phone conversation of Representative John Boehner (R-Ohio), was connected. They later pled out to a criminal charge concerning this. That call (said to have been intercepted within the state of Florida through a common radio scanner) also included then-Speaker of the House Newt Gingrich and other House Republican leadership folks. The tape turned up, inter alia, into the possession of Representative James McDermott (D-Wash.), who then flipped the tape to The New York Times and the Atlanta Journal-Constitution. This drew a lawsuit from Boehner against McDermott, seeking to impose to civil liability for violation of the federal [anti-]wiretap law, alleging that no effective consent existed Stock brokers commonly record ALL calls, and I assume have paperwork in place at account opening time, that effectively and irrevocably obtain consent to such monitoring and recordation, and as I think it through, must contain some sort of representation and warranty by the customer that all parties connected from their side of the call brought in have also consented. Clearly, sometimes this turns out NOT to be the case, and yet I do not recall seeing any litigation as to improper recording of a conference bridge. Curious And then there is the federal Do Not Call list -- seemingly a shield for the consumer to ward off unwanted solicitation calls from unknown third parties. All the phone numbers under my control have been registered with the enforcing agency, the FTC, and should be showing up on the database tapes for telephone solicitors to elide. This does not happen of course -- sadly, anonymous VOIP calls, false and forged Caller ID information, and simple omission of caller ID data prevails; the ways to dodge the requirement are well know to telemarketers, it seems But I have been working in the caller ID adjunct industry -- if you need real time screen pop information of inbound callers, I have been a rep for TelComp -- for longer than I care to remember. Be sure to mention that Russ sent you if you call Larry directly, or contact me for a system design and suggested implementation I was on the phone with Larry earlier today. We have provided the web and email presence since the start. The domain registration says 1995, but I know we did a trade show in LA before that with a web presence up. I was doing a bit of debugging on SMTP AUTH issues with him. Commonly we will leave an open line when we do this, and I listened to him field calls for an hour or so. Larry is endlessly patient on support calls, and I hope to be as patient when I am doing support. ;) The call had discussed industry trends and practices, and in part the topics of this blog post were fresh in my mind, for we 'talked shop' during running down his email issue The next call, not two minutes later, went like this: Phone rings, and the caller ID has no name information, is from a number not known in a lookup to my real time 'whitelist' database, and is from out of the local area code --- a potential outbound solicitation call Me: Good afternoon. May I help you? Other party identifies himself as calling from "Merchant Services" and asks for 'the decision maker' at my business. Me: That's me, all right; we have a practice and policy of recording all calls for quality and training purposes. May I have your consent to such recording, please? Other party: (confused) uhh -- OK, I guess Me: Great, and thank you. How may I help you? Other party: Well, I am calling about your merchant services account. I was calling to make sure you were getting the best rate ... Me: (interrupting) Sure -- thanks. What is your firm's name and address please? Other party: ummm Me: (interrupting) ... you see, I need that because this is a residential number that is on the Do Not Call list, and I need that information to send the lawsuit papers to ... Other party: (click) Much more satisfying that simply silently hanging up at my end. Feel free to "clip and save" this handy outline. A copy to crib from at each phone just may come in handy CentOS 4 series on K6-II 2009-11-05T13:48:00.005Z Tru just pointed out http://i586.centos.org/ which is an archive of the fruit of the push to get the AMD K6-II / Intel i586 install ISO working. Nice stuff, and nice to know the effort was not wasted... ... I am the eggman 2009-10-08T15:50:00.015+01:00 One recent addition to Python modules packaging at Red Hat in its Fedora project, is carrying along an additional, and optional structured metadata about the contents of that module (package), held outside of the RPM database This additional information: egg-info came into Python at Python version 2.3 and following. More may be learned about these eggs at: egg-info (optional extra Python metadata about a Python module). See: Fedora specifics This new detail is a two edged sword. On one hand, it provides sufficient information that an ad hoc root level process, for when one using native Python tools that it makes for an easy_install [the skeptic in me suggests it might be 'easier', perhaps, along some skewed axis of metric of goodness]. See also the egg superset, Python setuptools which now work well in RPM-mediated space Sadly, down this path, modules which are outside of the protections and managae-ability of the RPM packaging system may "easily" and inadvertently be introduced by an incautious admin, and thus introduce of Python code into a otherwise controlled system. This is the horror of a mixed RPM and CPAN system, all over again. As I say, one needs to choose a 'metric of goodness' with care Incautious use of mixed packaging approaches in turn can lead to possible security and updates headaches. Using such non-packaging system tools can break the SPOT -- single point of truth -- to determine to what versions of binaries a given host is using. From sad experience, this way lies additional work, and madness But on the other edge of that blade, the egg-info adds descriptive narrative, and cautiously used, increases the usability of a system that does not (ab)use those native installation tools As noted, the FOSS world has faced this problem before with perl and CPAN. Weak and strong 'includes' versioning and security model questionable @INC 'include' path search practices in Python and perl are well known failings in their community archive models. I faced it recently in a packing push of CRAN modules for R -- hmmm, I still need to file a few bugs upstream to solve some problems I saw in some R module packaging choices that I consider poor ones There is not a single, objectively 'technically right' way to proceed, but rather just one consistent, or not consistent with packaging system design and usage choices Fedora helpfully offers a sample stanza to use in .spec files. I cruised my archive of .spec files to see what else turned up # See if there's any egg-info if [ -f %{buildroot}%{python_sitearch}/Conch*.egg-info ]; then echo %{buildroot}%{python_sitearch}/Conch*.egg-info | sed -e "s|^%{buildroot}||" fi > egg-info and later then using the %files stanza's -f file list option %files -f egg-info %defattr(-,root,root,-) %doc LICENSE NEWS README doc/* %{_bindir}/cftp ... It appears I need to do some work in my local archive of SRPMs. Never enough hours in the day Like a stake through the heart 2009-09-02T19:56:00.011+01:00 The CentOS 4 series point refresh has been released to the mirrors for a couple weeks now, and the updates it backlogged as well. But the AMD K6-II / Intel i586 install ISO was not right when we shipped, and we knew it Akemi 'toracat' Yagi had it working in her side archive, and kept working the issue with Johnny 'hughesjr' Hughes, and candidate ISOs have been in testing in the QA back channel. I get a 'heads up' on a new testing from hughesjr yesterday afternoon, and around 5 am today, a notice that a new candidate was ready for pulling and testing I put lftp to work, and burned the CD. Booted with the command line parameter: : i586 text and did a minimal install Eureka -- it works in mainline CentOS Coming soon to a mirror near you (for the four or five users of such old kit). The unit I am testing on was my workstation on 11 September 2001, and I long since consigned it to the boneyard 090902: fixed grammatical error A bit more on CentOS 4.8 and the K6-II 2009-08-11T15:45:00.011+01:00 Yesterday's post on the K6 covered getting a CentOS 4.8 beta candidate installed on ancient hardware; The careful reader may have noticed that I had an unexplained list item early on in that outline: Add to /etc/yum.conf exclude=kernel* This is not something that just occurred to me unbidden, but rather came from an awareness that the upstream has had the dreaded 'Regression' from time to time in its RHEL 4 series, where a patch needed to support the K6/i586 architecture was not consistently present. In reading the bug comment notes, it seems that the 'boneyard' available to the member of the kernel testing team tasked with this is not so full of carcasses as mine, and so he cannot test his fixes as well So, I took affirmative steps to preemptively 'partition away' the need for an updated working kernel from our 4.8 beta install candidate, and yet be able to get to a working chassis with the kernel from the 4.5 final image, which is known to work. Good thing. The regression is back in the 4.8 kernel SRPMs, and the needed patch got dropped, it seems (this from an initial workup -- detail testing will be needed to see) The workaround is straightforward; Akemi 'toracat' Yagi maintains a testing 'plus' archive, containing kernels with the needed patch, and I can confirm that her candidate works fine. see: http://centos.toracat.org/kernel/centos4/centosplus-testing/i386/ Thanks, toracat Advancement of technical skills with CentOS project tools 2009-08-11T15:07:00.013+01:00 I posted this piece inside a post on a runaway mailing list thread on the CentOS mailing list. It represents my opinions, and are not some policy statement of the CentOS project. To a degree it reprised earlier pieces on how to advance one's technical skills with CentOS, but it is worthwhile carving it out, so I have a reference point to discuss sub-pieces of, here. Others have other views If a person wishes to be advanced in the CentOS project, contribute to the project. [It is not clear to me WHY people think there is some huge benefit for being a 'project insider' as it is really just a chance to do more work. Early access to QA is just not that hard to earn] We are not likely to hold your hand much, but will answer questions well framed. Be a self starter. Do something material. Some things to do to gain my notice as a contributor of merit: The bug tracker is open self serve for people to sign up. Add its RSS feed, and read every one as it crosses. Start working through the bugs to replicate or note an inability to replicate issues; Work through the bug tracker from latest to earliest, seeing if there is a similar upstream bug, or a fix, or if an issue is CentOS local. Note your results. That would be useful The centos-docs ML is open for proposals of new content into the wiki. Add its RSS feed, and read every commit diff as it crosses. Fix broken stuff that can be fixed at once. Some even believe it is more useful to re-write documentation locally rather than feeding improvements upstream so that it flows back down and out into RHEL, Fedora, etc as well as just CentOS [I do not, and refer you to Fedora to push non-centOS specific content out more widerly] Set up a local mirror of SRPMs, not just of the released Enterprise sources of upstream, but its RawHide as well. I have a daily diff report in my email queue each morning to scan for new material to review. Start building and testing and filing bugs to make the .spec files more general and less distribution specific, so that cross pollination can occur. You may get rejected (I often am), but at least try to improve the breed The same problems repeat time and again in the Forums. Add its RSS feed, and read every new post as it crosses. Add pointers or content as needed, and 'cc' into updates on the thread. I have noticed a excellent trend, that lately the three or four regulars are moving content more to the correct tree location, and asking questioners to do their research, and dropping out-links to answers rather than doing so in line. I like to do this as well when I form an answer, there on on a mailing list that is archived, as it provides the linkage hints Google needs to note 'reputation' and to weave answers together Join the main IRC channel or mailing list, and confirm you can answer every question posed for a solid week; if not, fill in your knowledge gaps with experimentation. At that point, start thoughtfully pointing a person toward the answers. Spoon-feeding is NOT a good thing, and does not gain any points in my eyes, as that is not the stated purpose of the channel The mailing list is looser as to /on topic/ but when a person repeatedly recommends 'non-CentOS' approaches over acceptable CentOS product, I'll certainly notice ... and that is perhaps not a good thing for further advancement. I _USE_ tinydns some places where it is the right fit, but I don't mention it here Once you have demonstrated skills, ask to be admitted to the next QA effort (we get three of four point update chances a year), and do QA. People who sign up and are admitted often slack off [don't participate in the ML, don't file reports, are not in IRC], and by that inaction demonstrate they are are not interested in progressing further. People _do_ get busy with real life or have to rest from burnout and take time off Once you have demonstrated skills, ask for some special project to build some element of needed infrastructure that is not otherwise getting done, and do it. John Pierce's post earlier this week certainly caught my eye, as he demonstrated self-starter problem solving skills in a complex space I had not seen before. He is now on my 'watch list' to draw into the project More personal opinion: Will any of those 'earn' a centos.org mailing address as someone lamented they lacked earlier in this thread? Sometimes, but frankly, we don't give those out easily. I saw a remark earlier: In the meanwhile some things ... are getting a bit clearer so I guess we are on the right track 'We' can perhaps be read here as a generic 'things are on the right track' -- but frankly, the only 'we' that I would look to for authoritative statements as to the project are people with a '@centos.org' in their email address. There is back channel coordination, infrastructure, and much more Beta testing CentOS 4.8 with an AMD K6-II 2009-08-10T20:44:00.012+01:00 Painful does not begin to describe how laborious it seems, after using more modern kit. It appears that the AMD K6-II instruction set is a superset of that used on the i586 series. Some folks seem to be still running such, and we have a number of resolved bugs in the tracker, detailing various ways to get the units running Based upon exhortation and advice in the CentOS QA mailing list and some IRC banter, I was induced to drag one of these poor exhausted clunkers out of my boneyard, and do some testing on it These installation instructions SHOULD work on i586 as well, but I no longer have an examplar to confirm with: Download and install using 4.5 i386 ISO from vault.centos.org and start it up the following options Boot it with: i586 text nomce Manually install openssh-server, enable, and set up with iptables, so you can hop on the unit from a remote box to work on it Add to /etc/yum.conf exclude=kernel* Perform a general run updates against the intervening changes prior to 4.8 -- (seemingly 4.7 and intervening updates when I perform this testing) -- lots there, but get it close to current. Install 6 Package(s) Update 150 Package(s) Remove 0 Package(s) ... took forever as I only have 128k ram for this old beast --- 308 transaction steps Do an interim reboot Point at my local mirror of the CentOS 4.8 release test candidate and let it rip -- first pass only: ftp://ftp.first.lan/pub/mirror/centos/centos-qa/CentOS/4.8/os/i386/ without the later pending updates: ftp://ftp.first.lan/pub/mirror/centos/centos-qa/CentOS/4.8/updates/i386/ Install 1 Package(s) Update 83 Package(s) Remove 0 Package(s) Total download size: 117 M Do a second interim reboot Mysteriously, I got an 'unclean shutdown' FSCK required message as to /boot here ... no idea why Run yum again, for a second pass with the updates Install 0 Package(s) Update 9 Package(s) Remove 0 Package(s) Total download size: 9.8 M Do a final interim reboot I completed by my test suite without incident I am advised similar steps may work from later than a CentOS 4.5 ISO, and that i586 should work as well. As I lack the hardware to test this, your mileage may vary Poor old boxes. Let them rest. Save power. I need a shower. Yuck Life in the Fast Lane 2009-08-03T02:28:00.030+01:00 Slow down, you move too fast. You got to make the morning last. Just kickin' down the cobble stones. Looking for fun and feelin' groovy. -- Simon and Garfunkel I picked up my wife at the airport late Wednesday night as she returned from a trade association conference related to her job. As we drove home, she talked about the unusual that happened there. It seems 'New Media' and 'Social Networking' tools have popped up on their radar, but her peers are wrestling to understand the motivations, and how to participate. It seems she astonished them, describing the FOSS stories and tools used that I 'bring home' as I recount the day at the dinner table: websites, wiki, mailing lists public and private, user group meetings, IRC, blogging, Twitter, VOIP, and so forth. They were 'wowed' that an old guy like me had used Twitter and a quick Google tour into the Wikipedia, to answer a son's question raised by an Admiral in a meeting at his job consulting for the federal government in metro DC a while back in seconds of a question coming up I suppose I take the pervasive availability of such tools, which largely are implemented through a foundation on the fruit of the 'Software Libre' movement for granted, and live a comfortable existence in this virtual reality. Although my hair has been gray for a couple of decades, it is not the me of my self-image, where I still feel 25 and full of vigor. That I whistle, and know the words of pop tunes from 40 years ago and play word games on the tunes at the coffee shop with the barrista does not jar me, although if I get a young one, they clearly have no idea what I am riffing on That to one side, I still revel in the wonder of the tangible world; a world of taking the family to the State Fair, or working with my hands, wood, and tools repairing a grandchild's wagon. I wrote the first draft of this piece -- a blog post -- with pen and paper with no plan on my mind beyond reconnecting with myself after a hard week, not just the CentOS matters, but in my local physical world as well; I should perhaps rather say, this piece wrote itself, flowing out of my hand's motions, creating, and editing on the paper before me, with strike-through's, insertions, and circled blocks of test indicating movement of thoughts into a flow I got no deeds to do, no promises to keep. I'm dappled and drowsy and ready to sleep Let the morningtime drop all its petals on me Life, I love you, all is groovy! Thinking back as to how I write, I sorely miss the older times of a ready steno-typist, secretary trained in shorthand, and later a ready 'Dictaphone', and the 'gal Friday' legal secretary who helped organize my worklife for many years. I did the creative work, and she straw-bossed the rest behind the scene, as I turned to face the next 'fire'. Each hard work in its own right, and a great and productive partnership. She's dead of lung cancer now -- was a smoker. Ah well The economics of such luxuries are prohibitive to most in an era where a person who cannot touch-type is perhaps now considered not yet fully literate. Welcome to the next lap of the rat race in this brave new world When the positions of transcriptionist, book-keeper, and sales clerk, along with the others mentioned above disappeared, and 'progress' came to the smaller enterprise, they were replaced by the small individual computer, word-processing, Quick Books, etc. Oh, and a subtle transfer to self-service responsibility to do all the work with less facility for delegation. Layers of support costs disappeared, as did the middle management, as entities had to flatten the organizational chart, or be outraced by their competitor Of course, the workload did not go away, any more than a completely 'paperless office' has emerged, The load shifted up to what were formerly more 'knowledge work' folks -- supervisors, or in a small enough firm, the entrepreneur owner, or just was no longer done ... sometimes the customer is 'drafted' to scan bar-codes and pay a cold machine, and no human hand on the part of the vendor can be found. Just try to find a phone number for eBay or Amazon live support some time We as a culture have weakened and removed spare resource capacity needed to build and nurture long term repeat customers, in favor of cost efficient transactionalism. Gresham's Law, all over again Ba da da da da da da ba bap a dee... During the week I too must prioritize, and work away at the hottest items in Covey's Quadrant One, as my schedule dictates them to me, Less important dreams and promises, desires and goals are left for an open dated 'later.' In my heart of hearts, however, I know that later will never come. Those 'heart's desire' are left behind on the horizon of each new day, for dead I can offer no remedy, save a caution that when building that schedule, to not mistake a capability to act immediately, with a mandate to do so Where does the answer lie? Living from day to day If it's something we can't buy There must be another way We are spirits in the material world -- The Police, Sting 06 Aug 2009: edited for a typo/grammar fix, layout error sadly, an Open Letter to Lance Davis 2009-07-30T06:42:00.003+01:00 Open Letter to Lance Davis July 30, 2009 04:39 UTC This is an Open Letter to Lance Davis from fellow CentOS Developers It is regrettable that we are forced to send this letter but we are left with no other options. For some time now we have been attempting to resolve these problems: You seem to have crawled into a hole ... and this is not acceptable. You have long promised a statement of CentOS project funds; to this date this has not appeared. You hold sole control of the centos.org domain with no deputy; this is not proper. You have, it seems, sole 'Founders' rights in the IRC channels with no deputy ; this is not proper. When I (Russ) try to call the phone numbers for UK Linux, and for you individually, I get a telco intercept 'Lines are temporarily busy' for the last two weeks. Finally yesterday, a voicemail in your voice picked up, and I left a message urgently requesting a reply. Karanbir also reports calling and leaving messages without your reply. Please do not kill CentOS through your fear of shared management of the project. Clearly the project dies if all the developers walk away. Please contact me, or any other signer of this letter at once, to arrange for the required information to keep the project alive at the 'centos.org' domain. Sincerely, Russ Herrold Ralph Angenendt Karanbir Singh Jim Perrin Donavan Nelson Tim Verhoeven Tru Huynh Johnny Hughes Phat pipes 2009-06-08T15:12:00.009+01:00 Check the top row, right entry ... peaking at 44 megaBytes per second, and a lesser rate sustained over 8 hours; all relevant filtering bridges, and servers in the transfer at our end are running ... CentOS We've spent the last couple of months in the buildout of our (new) presence in the North data center. We have sites in the central city, on the Dublin fiber ring, and through the north end AT&T switching center, but each has had its faults over time. The downtown 'carrier hotel' was offline for four hours due to a lack of redundancy in its generators during last September's multi day power outage; the Dublin fiber ring peering exchange point had issues as well, but longer; our multi-site strategy saved the day as none of our customers lost inbound data nor went dark in their web presence; uplinks were not affected as we handle them over different routes. In the last couple weeks, AT&T's congestion issues have re-appeared at their plant as well when we were 'babysitting' a large CAD/FEA file transfer ... again multi-gig The new data center is pricey -- but in addition to the care at the physical layer, it is BGP multi-homed and has really fat pipes. The screenshot up top shows the inbound consumption on the green. Iniitally we had a hard cap on our switch to limit it to 10 MegaBytes/Sec inbound -- but we were doing a large (a multi hundred gigabyte pull), and dropped the cap once it was clear all was working well We are in the paperwork phase at the moment with ARIN, to clear up some 'lint' on our ASN, but with any luck by the end of the month, we'll have completed the cutover

Rainy Days & Mondays
2009-05-11T17:03:00.022+01:00

Karen Carpenter made the song famous for its authors, but clearly none of them were sysadmins
The rule, long known, for sysadmins is:
Never make a major change on a Friday, nor before leaving for vacation
I've been wrestling with the fallout from a violation of the sysadmin's rule by an upstream provider -- the vendor pushed in some change on Friday in the preparation of CDR -- Call Detail Records.  For four days running, my sub-processes which manage the account have been failing for want of data.  Those processes retrieve and apply CDR data, to emit accounting detail for customers, and have not been workingI've filed five or six sub-issue tickets which that primary change exposed, in trying to get the matter resolved:  The current Firefox cannot open tickets under the current Windows XP, current SP [no problems with CentOS and FireFox or konqueror]; my 'closed' tickets were not visible; tickets were being closed by upstream before I confirmed a fix worked, so I ended up essentially re-opening the same ticket three times as each day's CDR pull failed; I was not receiving email updates of tickets; and so on.  I am quite sure they consider me a 'stickler for details' and something of a pedantic pest at the moment, but dammit, I'm paying their bills.  The PHB supervisor may want tickets closed quickly; but I want my issues fixed first
... as no one likes to be called into work on the weekend to revert a change, the sysadmin's rule must be faithfully applied

Getting to a x86_64 build environment
2009-05-11T16:05:00.011+01:00
In the #centos IRC channel on freenode, today, a new user was trying to clean out the 'multi-lib' artifacts in his build environment, so that it was only generating 'x86_64' results
Tom mentioned:
10:26  Zathrus> realistically, just removing glibc.i?86 should nuke everything else...
and so I fired up a victim xen instance to test that hypothesis
sudo su -
cd /etc/xen
ls
cp centos-5-x86_64-test centos-5-x86_64-victim
joe centos-5-x86_64-victim
# the edit is to rename the instance name, and the image to be used
cd /var/lib/xen
cp centos-5-x86_64-test.img centos-5-x86_64-victim.img
xm create centos-5-x86_64-victim
virt-viewer centos-5-x86_64-victim
Then inside the instance as root, I ran:
rpm -qa --qf '%{name} \t %{arch} \n' | sort > pre-remove.txt
yum remove glibc.i?86
rpm -qa --qf '%{name} \t %{arch} \n' | sort > post-remove.txt
grep -v x86 post-remove.txt  | grep -v noarch
getting the result:
gpg-pubkey       (none)
libaio   i386
libgcc   i386
python-devel     i386
That's a pretty good result for a first pass, and a quick hack.  I think I'll go down for some coffee, and think about it a bit more

Revenge of the Jedi (Part II)
2009-05-05T14:54:00.004+01:00
I wrote last week from memory about the use by the Basel II standard of a method, additively combining non-linear and correlated risk events
I see a post on the R-SIG-Fin mailing list from conference organizer Jeff Ryan, that the presentations from R/Finance 2009 are up
Page 4 on the PDF of the Klaus Rheinberger, et al. presentation nicely states the executive summary that this is 'problematic'.  The work then shows a worked example
Let's call Basel II what it is -- a top down pronouncement on meaningless rules, written in a fashion that is willfully ignorant of the lessons from the US S & L 'hot money' and actuarially un-sound deposit insurance debacle 20 years ago, of LTCM as to correlated risks and 'being the market', and of the recent Credit Default Swap insurance blowup of AIG

May Day celebration
2009-05-01T14:30:00.010+01:00
I see a tweet from KB about the monthly mailman mailing list reminder emails;  I took steps long ago to use procmail to watch for these, and re-mark their subject line.  I then sort my mailspool by subject in alpine, and delete this noise all in one pass.
# mailing list memberships reminder
#
:0 fw
* ^Subject: \/.*mailing list memberships reminder
*!^X-Reminder
  | formail -i "Subject: mlmr] $MATCH"                \
    -A "X-Reminder:$MATCH"    \
    -A "X-Munge: moved mailing list memberships reminder"
All power to the people

Revenge of the Jedi
2009-04-27T21:13:00.010+01:00
"The pen is mightier than the sword"
I posted a bit earlier today about the forgotten religion of Monetarism in the context of my weekend at a conference in Chicago.  I had not heard mention of the faith, nor seen anyone but myself doing analysis using the old tools for a long, long time
I blogged a bit back about Jim Chanos' critique on CNBC of the new Mark to Market 'requirements' and the  artificiality of the Basel II reserve requirement target value, Chanos suggesting a relaxation to a transition value of say 1.5 percent to 'conform' to Basel II in the short term.  Two weeks ago, I had mentioned 'A Monetary History of the United States' (Friedman, Schwartz) to a friend wanting to understand how we got where we are; yesterday evening, I was discussing the Nixonian repudiation of Bretton Woods and the need to revisit Basel II, as I saw a clear demonstration that Basel II is defective at the conference.  I do not have my notes at hand, the slide decks are not up yet, but I believe it was in the mixed currency risk analysis (Austrian, Swiss, and back to Central Europe as to residential property loans), which I believe Rheinberger gave entitled: 'VEC and GVAR Models using R' which exposed quite clearly that the 'experts' are using simple additive risk summing in Basel II, seemingly oblivious of the concept of the non-linear nature of correlated risks
The afternoon's email brings a report that Anna Schwartz is still out there as well
The old craft will live so long as a single practitioner remembers them
"Hokey religions and ancient weapons are no match for a good blaster at your side, kid."
 The good fight continues;  I'll keep swinging with the tools I know, thanks
p.s.: I do know the regular titles for the third and sixth released Star Wars films

'R' you experienced?
2009-04-27T16:39:00.033+01:00
" ... To confer, converse, and otherwise hob-nob with my brother wizards ..."
I spent a productive weekend up in Chicago, at the R/Finance 2009: Applied Finance with R conference, which billed itself as the "first annual R/Finance conference for applied finance using R".  The conference organizers and hosts are the 'usual suspects' on the 'R sig fin' mailing list; Jeffrey Ryan, Dirk Eddelbuettel, Dale Rosenthal, Brian Peterson, Peter Carl, Gib Bassett, and John Miller, assisted by the talented and imperturbable Holly Griffin of UIC.  Pretty clearly most of this group code together regularly; see the committer list on the blotter module. The venue was at the 'other school' in Chicago, the one with a more practical interest in Economics and Finance
An aside about Chicago: Long ago, and far away, I was trained as a acolyte 'monetarist' by disciples of Herb Stein's CEA and the Fed, in the [University of] Chicago school [a fad, seemingly long forgotten by recent Economics and Finance grads, so far as I can tell].  Monetarism is a forgotten religion these days; the Fed stopped formally publishing its M3 series a few years ago, in light of the rise of what Bill Gross calls the 'shadow banking' system.
Luke: [The robot] claims to be the property of an Obi-Wan Kenobi. Is he a relative of yours? Do you know what he's talking about?
Obi-Wan: Obi-Wan Kenobi. Obi-Wan... Now, that's a name I've not heard in a long time. A long time.
Luke: I think my uncle knows him. He said he was dead.
Obi-Wan: Oh, he's not dead... Not yet.
Luke: You know him?
Obi-Wan: But of course I know him
Many of the organizers were known to me from my email correspondence or from observing their packages, and I had spoken with one (Dirk) two or three years ago briefly after the trading shim was first usable
Dirk seems to be a bundle of unbounded energy.  His tools had solved a lot of data storage and visualization issues for me early on in our project.  He led a push a couple years back to drill in many of the R add-on modules into the main Debian archives. I still hope to emulate his example in rpm space using R2spec and some post-processing scripts (Dependency enumeration is not quite perfect yet).  Dirk has already mentioned in his blog the 'after-sessions' at Jak's; we closed the place down Saturday with useful brainstorming happening long into the night
I had resolved to travel to the conference to learn, and to stay quiet as to matters of FOSS and advocacy. I was almost even able to keep to that intent, save at the interstitial times.  The formal presentations were amazing in their quality of content, competence of the presenters, and challenging to my old knowledge of Statistics and Mathematics.  I even understood most of what the presenters were doing, and why, on the formal finance side and will re-read the slide decks with great interest when they appear to fill in the holes. One would have probably had to be there to draw much more from the decks, as the presenters were not doing the occasional 'stand and read' presentation one finds at some conferences, but rather largely used their decks as reminders of the points they wanted to hit and elaborate on in their presentation, and to state exactly the code and formulae in play.  The committee have really set a high bar to reach for next year's event to top, and I look forward to it already
The pre-conference tutorials were worthwhile.  I knew Jeff Ryan's work from xts and IBrokers of course, and gained insight into his mental roadmap on where the code is going and how it will get there.  I think the enhancements he is trialling in xts will pretty clearly flow back upstream into zoo in general form; I had not heard of Dale before, but his breakout and presentation of an analytic approach on addition and testing of single constraints (I have covered scientific method and epistemology here before, and will again) served as a fine warm-up to the formal sessions
During the session breaks, at meals, and into the night, I had a chance for give and take at length with several of the committee, presenters, and attendees, to bridge what Patrick Burns spoke on -- the chasm between Practice and Theory
Part of the trip and my need for listen, was to get a handle on how to match the shim and R as a pair of heavy-weight co-processes, so that the user of the shim can hook in and use the wonderful tools already in R space.  We'll most likely get there, but the timing is not clear.  Having said 'we', permit me to make it clear that the heavy lifting will be done, if and when done, by Bill, and not me.  At the prodding of Peter, who as I understand it regularly team codes with Brian, I have started the sign-up process for an account at r-forge, and will 'cut my teeth' on a simple connector or module, to warm up my skills as a co-development tester and 'guinea pig' consumer of the major task of integrating the shim
FIX rules the roost for being the 'lingua Franca' for interchange to exchange order, position and fill data with counter-party upstream brokers or exchanges (thanks here to the CME Foundation for partially funding the event). We will not soon be adding a compressed FIX connector to the shim, and certainly not before we attain our major milestone of a formal 'complete' first release.
  Finally, a couple folks asked why we were playing down in retail space with the TWS and its vendor specific API.  For a researcher, and for a small proprietary trader, we still find IB's API and services the most affordable, and substantially complete.  It is a gateway to enable any interested researcher to do material research (the 'Theory') and strategy development and execution (the 'Practice').  For student academics, the availability of IB's 'trading Olympiad' program and the shim, and R offer all one needs for a better than free price
Update:  We see also the summary of the event at Revolution Computing, an R vendor

Afraid of experimentation
2009-04-16T21:27:00.008+01:00
The #centos IRC channel at irc.freenode.net never ceases to amaze me.  We get questions that would take at least 30 seconds of reading a man page and experimention to answer, asked over and over again.
Here is one of the latest:
16:14  clueless> I installed a Windows Vista Business x64 VM on 5.2. Is it possible to get hibernate/sleep to work?
16:15  clueless> I want only want the VM up occasionally and I'd rather not wait for a full boot every time
Firing up a xen virtual machine in a root panel, and popping open another to read the xm man page, I find:
# cd /etc/xen
# ls 
# xm create win-2000pro
 Using config file "./win-2000pro".
 Started domain win-2000pro
# virt-viewer win-2000pro
and a Windows 2000 session appears.  I let it boot to the login prompt, and then:
# cd /var/lib/xen
# xm list
# xm pause win-2000pro
# xm save win-2000pro win-2000pro-save.img
Which of course as the man page promises, terminates the running image.  Then:
# xm restore win-2000pro-save.img
# xm list
# xm unpause win-2000pro
# virt-viewer win-2000pro
And we are right were we left off, at the initial log in prompt.
Is it so hard to at least pretend to look first?

I propose that women have 28 teeth
2009-04-01T16:57:00.019+01:00
Why have men more teeth than women?
By reason of the abundance of heat and blood which is more in men than in women.
  -- "Of the Teeth.", Aristotle
One of the mysteries behind the quote above, was why Aristotle did not simply find a near-by woman, and ask her to permit him to count her teeth
How do we know what we 'know' to be true? The difference here is of course that between 'deductive' and 'inductive' analysis
Political 'debate' and flame wars on which Linux distribution (package manager, editor, MTA, and so on ad infinitum) is better, often degenerate to deductive reasoning from a firmly held (perhaps from ideological basis, perhaps from prior experience) 'Theory'. Then one is to state a testable 'Hypothesis', and actually perform field or experimental  'Observation' to validate or disprove that hypothesis, and finally, reaching a 'Conclusion' that the Theory is supported or not.  Aristotle omitted the critical stages of testing his hypothesis, and so fell into error with his assertion.  Pure reason lead him astray
It is just as easy to fall into error from the inductive reasoning side.  I have noted for many years now that in early February, I see newspaper reports that the groundhog ("Punxsutawney Phil") is reported as seeing his shadow (consider the hints from the Bill Murray movies, 'Caddyshack' and 'Groundhog Day').  That he sees his shadow seems to cause Winter to continue for six weeks or so
The cardinal birds also must read the newspaper and observe the shadow sighting report in timing their return to north of the Mason-Dixon Line.  When the timing is right, the cardinals return to my town.  It takes a week or two, but once the cardinals have reported back to the southern over-wintering havens, the robins follow them
The return of the cardinals also cause the forsythia bush out back to bloom (I suspect there is some needed chemical agent in the bird droppings).  This is important because it needs to snow on the forsythia three times before it is safe to plant the vegetable garden to avoid the seedlings being frozen and killed
My chain of 'Observation' is most careful, taken over many years. A 'Pattern' emerged that I could see, and so I formed a 'Hypothesis' as to what was occurring.  My 'Theory' seems to explain nature well.  The 'inductive' results are of course completely wrong, untestable, and confuses co-incidence (sequentially timed events) with causation
The XKCD website has this:

and if you are not reading that site regularly, you should be.  We'll be using statistics soon enough here
At the end of all the back and forth about deductive and inductive methods, we have to end up at the conclusion that pure logic is but an organized way of committing error.  Nothing can replace putting forth a testable hypothesis, and getting down and dirty in the data testing it to confirmation or refutation
Critical note. — Of a piece with the absurd pedagogical demand for so-called constructive criticism is the doctrine that an iconoclast is a hollow and evil fellow unless he can prove his case. Why, indeed, should he prove it? Is he judge, jury, prosecuting officer, hangman? He proves enough, indeed, when he proves by his blasphemy that this or that idol is defectively convincing — that at least one visitor to the shrine is left full of doubts. The fact is enormously significant; it indicates that instinct has somehow risen superior to the shallowness of logic, the refuge of fools. The pedant and the priest have always been the most expert of logicians — and the most diligent disseminators of nonsense and worse. The liberation of the human mind has never been furthered by such learned dunderheads; it has been furthered by gay fellows who heaved dead cats into sanctuaries and then went roistering down the highways of the world, proving to all men that doubt, after all, was safe — that the god in the sanctuary was finite in his power, and hence a fraud. One horse-laugh is worth ten thousand syllogisms. It is not only more effective; it is also vastly more intelligent.
  — The American Mercury. p. 75., Henry Louis Mencken (1880-1956)

But then you get a lot of angry letters, from those whose clay idol you have smashed
edit: two typo fixes

Who is in charge, here?
2009-03-31T22:37:00.011+01:00

"When I use a word," Humpty Dumpty said, in a rather scornful tone, "it means just what I choose it to mean - neither more nor less."
"The question is," said Alice, "whether you can make words mean so many different things."
"The question is," said Humpty Dumpty, "which is to be master - that's all."
   -- Through the Looking Glass
A U S Supreme Court memorandum order (called a 'Slip Opinion' here) today said:
PER CURIAM.
  The writ of certiorari is dismissed as improvidently
granted.
                                        It is so ordered.

http://www.supremecourtus.gov/opinions/08pdf/07-1216.pdfSome Latin in there.  'PER CURIAM' is: By the Court as a complete panel and entity, without specific attribution of the action to any particular Justice.  A 'writ of certiorari' is: a publicly stated intent of the Court to receive a case for presentation and argument, and possibly (usually) decision.
That a matter is characterized as: "improvidently granted" is not Latin, but is a 'term of art' -- basically:  It turns out we (as a decision making body) cannot, will not, or should decide after, and we decline to consider the particular aspect of the case we initially though we should hear for the present.
It happens -- maybe with one member of the Court ill, the Court decided it needed to lighten its load; perhaps some conflict came to light in the investment portfolio of some Justices that the remaining (non-recusing) panel members of the Court felt now that fairly they could not hear and decide the matter, to avoid the appearance of making a biased, short handed, or improper decision.
The functional effect of such a terse statement is to leave intact and in effect the next prior lower court's ruling
While a lawyer's craft is well depicted in television crime procedurals, the more cerebral parts of avoiding issues which make one interesting for television, are decidedly more valuable services a lawyer provides to the general society, when advising a client.  A lawyer's opinion can help the client see the areas to avoid; how to structure its affairs.  The Court speaks in a completely understood fashion here, to communicate just what it intends to say, and nothing else
The recent 'mob rule' in Congress of vilifying and proposing to punishingly tax the folks at AIG who had clear contracts for payment of a 'retention' and performance bonus, in exchange for staying on as the ship of AIG seemed to be sinking (and indeed staying and doing their contractual duty) show how fragile civilization is.  All the hollow words about 'fairness' and populist anger cannot mask the fact that Congress successfully pointed the finger of blame away from itself and the government, and toward a tiny minority of our society.  If a Constitution, contracts, and rule of law are so readily cast aside, no one is safe.  Paris 1789 and following, all over again.  Whom shall the mob turn to next?
The use of words is how we explain to another, and sometimes to ourselves, what we are thinking; why we believe what we believe; permit us to reflect and find weak points in our thought processes.  Structured words -- Court decisions, Constitutions, laws, opinion letters from lawyers are part of an  ongoing societal dialog
We are all diminished when cheap talk trumps reason

OMG, Round Two
2009-03-31T17:57:00.009+01:00
I wrote a bit back about a gratuitous change in Red Hat's RPM variant breaking backward SRPM readability, in a fashion which stranded users of the earlier Red Hat Enteprise release products (and rebuilds such as CentOS) away from the Raw Hide pool of developmental edge packages.
The fix I outlined: to build and freeze (in time, and against updates) a RawHide domU instance, and to use that domU and an NFS mount back into the earlier dom0 to unpack SRPMs.  This works fine for the present.
The full size screen shot is a bit large, but down that link.  It takes just a couple of seconds to set up a new unpacking destination, and to do the rebuild, once it is set up.

Promoting ignorance
2009-03-27T15:59:00.014Z
There is a good reason lawyers should not give, and are really uncomfortable having a client publicly discuss advice they have given in a public forum

This crossed a mailing list today:
Subject: fedora-d-rh] Re: question about patent

Without reading or looking at the patent at all, it is almost always really bad to discuss patents in public, especially on email.

Patents & patent trolls are so pervasive that you can help feed patent trolls by bringing up the possibility of infringement in these forums (even when they are marginal claims).

I have always been given guidance that engineers should never, ever do patent searches and never discuss the specifics of IP issues in email.
Amazing takeaway.  The poster missed the obvious extension that really NOTHING in the way of litigation awareness and preparations should be discussed
A quick Google search using: willful ignorance of a patent yields this in a pull quote:
Courts have used terms such as *intentional blindness,[15] *blind disregard of the peril it faced[16] and *willful ignorance[17] to describe the accused infringer who did not conduct a search prior to adopting a mark

[later] ... With the ease of accessing information, it is likely that courts will increasingly find that an accused infringer's failure to conduct an appropriate search before adopting its proposed mark is a clear indicator of bad faith.
The article's author 'threads the needle' nicely, between providing general information, and not giving express advice.  But he DOES assume the reader recipient will CONSIDER the implications of what is being said.  Silly lawyer
Down at the bottom of that information article, we find:
The information contained in this alert is provided for informational purposes only and does not represent legal advice. Neither the APLF nor the author intends to create an attorney client relationship by providing this information to you through this message.
Time to stretch the legs, and walk down to Stauff's for a coffee

IPv6 eats kittens (and distcc) on Debian Testing
2009-03-26T17:39:00.046Z
This can only end badly
I spent a good 5 hours this week, tracking down a problem with distcc hanging up in our Debian Testing build farm.  We use distcc to speed up compilation of the c++ sources in the development of the trading shim.  Interestingly, our end user community forced us to this decision of developing on Debian testing, as they are using later gcc versions than we were on CentOS, and it was useful to be able to see their errors, BEFORE they reported them to us
On the new compile farm, sometimes we would get a compile in, say, 44 seconds; other times it would drag out for several minutes.  This is a problem as we had just slotted a new unit into harness, and expected better results
In checking the logs in the client doing the distribution of compilation tasks, we were seeing a symptom of 'segfaults' in that client's process; other times, the client would stall, seemingly blocked waiting for a compilation result to come back from a remote buildfarm peer, that never came back. Checking on the remote build unit, one of the distccd children would die for mysterious reasons, leaving a message in the dmesg record.  Once that failed build timed out, the needed file would be built locally, and the build proceed.  Checking the log files nothing obvious jumped out
The obvious debugging technique is to get a minimal reproducer, and then to partition the problem into smaller and smaller possible causes using that reproducer tool.  the issue will manifest on one setup, but not the other, ans so one can rule out more and more issues, until the answer is left, staring you in the face
Looking at my Debian helper tool, it had rotted, and was in sorry need of removal of some constraints:  It did not use distcc when available; it did not use proper -J parallel compiles; it did not use -O3 optimization in the compiles.  My test tool was not set up to see what I needed to see
Time to pay down some 'technical debt' (If you've not read martinfowler piece, and viewed Ward Cunningham's video, stop now, and do so).  And so I made some payment there.  After testing, I got these results:
Master Clients Elapsed time (real)
 pippin   nfs2, 10.16.1.231   0m23.281s 
 nfs2   10.16.1.231, pippin, localhost   0m23.702s 
 10.16.1.231   pippin, nfs2, localhost   0m22.551s 
My first thought looking at this: Well, that pretty conclusively rules out machine specific errors, or network path issues.  It must be something different in the setup of the user provoking the issue that my tool does not duplicate.  NOTE: This is wrong-headed, of course, as: 'An absence of evidence is not evidence of absence of a problem' but was an easy trap to fall into
For every complex problem, there is a solution that is simple, neat, and wrong.

  — H. L. Mencken

For every problem there is a solution which is simple, obvious, and wrong."

  — Albert Einstein
I tossed my results at that user for their thoughts on the results, and went back to work on another issue
Later in the day, doing some thought experiments with the user, we could not pin down where to look yet.  But as a team, I had him provoke the issue with his setup, while I watched the logs on the various machines through several consoles.  And the error appeared, and then jumped out and tickled my eyeballs.  I was watching nothing in particular, until I saw the failure on process 29673, and then traced that back up.  A successful and a failed session looked like this, respectively:

distccd[29673]  (dcc_check_client) connection from :ffff:10.16.1.249:41771
distccd[29673]  (dcc_r_file_timed) 909179 bytes received in 0.078651s, rate 11289 kB/s
distccd[29627]  (dcc_collect_child)  cc times: user 1.132070s, system 0.144009s, 23039 minflt, 0 majflt
distccd[29673]  (dcc_collect_child)  cc times: user 1.092068s, system 0.104006s, 22481 minflt, 0 majflt
distccd[29673]  (dcc_check_client)  connection from ::ffff:10.16.1.249:41775
distccd[29673]  (dcc_r_file_timed)  818437 bytes received in 0.071648s, rate 11155
kB/s
distccd[31248] (dcc_check_client) connection from ::ffff:10.16.1.249:41779
distccd[31248] (dcc_r_file_timed) 886761 bytes received in 0.076688s, rate 11292 kB/s
distccd[29627] (dcc_collect_child) cc times: user 1.068066s, system 0.112007s, 23890 minflt, 0 majflt
distccd[29673] (dcc_collect_child) cc times: user 1.108069s, system 0.112007s, 22012 minflt, 0 majflt
distccd[29673] (dcc_pump_sendfile) Notice: sendfile: partial transmission of 15868 bytes; retrying 344332 @15868
distccd[1995]  (dcc_log_child_exited) ERROR: child 29673: signal 11 (no core)

A-ha!  Now we know what to look for:

dhcp-231:/var/log# grep dcc_pump_sendfile  distccd-transition-log
distccd[29673] (dcc_pump_sendfile) Notice: sendfile: partial transmission of 15868 bytes; retrying 344332 @15868
distccd[31248] (dcc_pump_sendfile) Notice: sendfile: partial transmission of 15868 bytes; retrying 586732 @15868
distccd[30262] (dcc_pump_sendfile) Notice: sendfile: partial transmission of 15868 bytes; retrying 4655916 @15868
distccd[2005] (dcc_pump_sendfile) Notice: sendfile: partial transmission of 16384 bytes; retrying 74824 @16384
distccd[2128] (dcc_pump_sendfile) Notice: sendfile: partial transmission of 16384 bytes; retrying 286560 @16384
distccd[2170] (dcc_pump_sendfile) Notice: sendfile: partial transmission of 16384 bytes; retrying 97440 @16384
distccd[2129] (dcc_pump_sendfile) Notice: sendfile: partial transmission of 16384 bytes; retrying 301000 @16384
dhcp-231:/var/log#The TCP process of shuttling code to compile, and the binary results of such compiles are failing the same way, over and over again: partial transmission of 15868 bytes is present every time.  Looking at the log entry again, the form of the connecting hosts is unusual:  ::ffff:127.0.0.1 and ::ffff:10.16.1.249.  Why that is IPv6 notation?  And I reach back to my logs as I remember I had an issue like this a year or so on a Debian box
And so, Google with the search argument: debian ipv6 distcc confirms as its first result: 1. #481951 - distcc: zeroconf support broken wrt IPv6 - Debian Bug ... ... and the bug is still open.  Killing off IPv6 is the obvious next step, and so, back to Google with: debian disable IPv6 to find: Disabling IPv6 under a 2.6 kernel.  Reading the post, there is some back and forth, and the answer seems to be, there is not an 'official Debian answer', but this is what people are doing.  Back to Google with: site:debian.org debian disable IPv6 seems to confirm that there is not a single well documented answer which has floated up in Google's searching
Compare: CentOS addresses the matter directly, and as the first Google hit with: site:centos.org disable IPv6 
7. How do I disable IPv6?

    * Edit /etc/sysconfig/network and set "NETWORKING_IPV6" to "no"
    * Add the following to /etc/modprobe.conf : 

alias ipv6 off
alias net-pf-10 off

    * Run chkconfig ip6tables off to disable the IPv6 firewall
    * Reboot the system 

Alternative (which might be easier and works on any release with /etc/modprobe.d):
echo "install ipv6 /bin/true" > /etc/modprobe.d/disable-ipv6
Sadly, there is something else on Debian testing in play as well, and it is not just an IPv6 issue (although turning off IPv6 has drastically reduced the frequency of the issue).  When I look in today to make sure the 'fix' is working

[74988.951989] distccd[8671]: segfault at 1 ip 7fdd2250e030 sp 7fff2b025da8 error 4 in libc-2.7.so[7fdd22493000+14a000]
[74989.017836] distccd[8651]: segfault at 1 ip 7fdd2250e030 sp 7fff2b025da8 error 4 in libc-2.7.so[7fdd22493000+14a000]
[74989.518050] distccd[8664]: segfault at 1 ip 7fdd2250e030 sp 7fff2b025da8 error 4 in libc-2.7.so[7fdd22493000+14a000]
[74994.152461] distccd[8659]: segfault at 1 ip 7fdd2250e030 sp 7fff2b025da8 error 4 in libc-2.7.so[7fdd22493000+14a000]

Where is that coffee cup?  I knew this would not end well

"It's different, this time"
2009-03-26T12:17:00.025Z

The British born, formerly American investment manager, Sir John Templeton, is attributed the following as to his craft:
The four most dangerous words in investing are 'This time it's different.'

I suspect the quip is over-constrained in limiting it to just investing.  But I am meditating about another Briton's work
At last night's COLUG meeting, the presenter addressed the emergence of the latest round of internet based 'social networking' applications: twitter, facebook, blogging, multi-features personal information devices (cell phones, Blackberries, iTouchs, digital cameras and the like).  I say latest round, because the assertion was made that: "Terrorists have never used photo reconnaissance" and contrarian I suggested that the people of Dresden might have a different point of view
The takeaway from the matter had to be a thoughtful person needs to be mindful of the obvious and non-obvious implications of these new technologies
The ability to build a 'mosaic' image of a person, from their public 'internet persona' is only getting easier, and more accessible to a wider audience of potential prying eyes.  What once required the resources of a government or major multi-national corporation to 'dig out' are perhaps thoughtlessly revealed with all good intention.  See, e.g., the 'Sarah' PSA: ("Online Sexual Exploitation - Everyone Knows Your Name"), which ends with the outline:  "... so think before you post"
But the information leakage is much broader than that already, and at this point not controllable by any individual.  When a member of a 'private' or 'backwater' mailing list uses GMail to subscribe, every poster suddenly is added to Google's indexing corpus; when someone at a local meeting snaps a cell phone picture and posts it publicly, it feeds the automated identification algorithms publicly known (Google's Photo), and otherwise (Think: the Tampa Bay Super Bowl photo identification effort of the crowd).  Note the date of the Register article just cited:  7th February 2001.  This was no Bush-ian crypto facist over-reaction to the 9/11 hijackings
During the presentation last night, the first advert link offered was for anti-aging patent drugs, along side the meeting photo (full of several grey haired and bald male persons; the second link was of 'Valerie Bertinelli -- Bikini Babe!'  and had a weight loss advert in the 'doubleclick' advert box on the top right; but our presenter is interested in and follows a television show 'The Biggest Loser' and is browsing weight control related sites and mailing lists.  A third, rather personal example from the presenter's prior experience completed the circle to make it clear that Google's advert engine is reading every word we read or write
The first time is an occurence; the second a co-incidence; after the third, one has to stop shaving with Occam's razor as the blade has gone dull

I took a screenshot (full-size image) of what I am offered as to Valerie, and you'll notice that the upper right panel is blank.  This is because some years ago, I amended the DNS records which computers using my DNS servers are provided, to return '127.0.0.2' for all of 'doubleclick.net'
[root@xps400 conf]# grep -i doubleclick *.conf
NULLROUTE.conf:127.0.0.2                ad.doubleclick.net.
[root@xps400 conf]# 
Adding that value (which causes the request for an advert to never reach the central advert monitoring and image feeding servers), and several more was part of a campaign for a corporate client I was consulting for at the time.  The Windows 98 desktop computers which were issued to the staff did not have effective software installation access controls, to preventing addition of random malware and time wasters.  Memos and meetings had not stopped the practice of a staffer downloading, say, Yahoo! Instant Messenger, and showing all her friends in that department how to do the same. Bandwidth exhaustion was becoming an issue; I assume that management also had some thoughts about lost productivity
As a technical fix the IS department was asked to remove it when found (done, but not persistent without effective access controls), and asked again.  I was escalated in, and went to work with tcpdump
It turns out that the software designers at Yahoo knew their craft well.  From memory, it first tried the universal Firewall Transversal Protocol (http), and then secure http and FTP
I blocked each new approach in turn.  It fell back to nntp, and as I recall ntp.  I do not recall that it tried to use dns content tunneling, but I certainly would have.  The eventual solution had both port blocking and domain blacklisting
There is nothing new, nor indeed to my thinking, wrong for the owner of an asset to seek to profit maximize with it.  But I think my thoughts and my words are my property, and on occasion on a 'think piece', I'll add the copyright reminder tag

 .-- -... ---.. ... -.- -.--
Copyright (C) 2009 R P Herrold
      herrold@owlriver.com
   My words are not deathless prose,
      but they are mine.
I also hold to the quaint notion that I am not a number, but an individual and the property of no one but my God.  Silly, I know, but there you are

edit: typo fix

People do go both ways
2009-03-25T17:21:00.029Z
There is a scene depicted in the movie: 'Battle of the Bulge' (1965) about the 1944 attempted German breakout offensive through the Ardennes, where German commandos are tasked with and shown changing road signs to confuse Allied troops
When I started this blog, it was in response to a desire to make the CentOS internals a little more transparent to interested observers.  We at the project do get the questions, and I think a thoughtful reader can pull connections from the little stories and examples I choose from the full breadth of the blog.  While I might 'tag' something specifically 'CentOS', real life has no such natural boundaries, and these are just guide markers in the channel of life. 
I added the blog into the CentOS aggregator at planet.centos.org, and set to writing.  I cribbed the configs from an example of another CentOS member.  I tried then to restrict the feed to the 'CentOS' label, but following the documentation just did not work.  I settled for the default full feed, and resolved to solve the revisit the matter later
My friend toracat gently reminded me of the need to finish the job, this morning.  Sigh ... back to wrestling markup
The example follows [there are annoying line breaks in the blog layout as rendered, and indeed in the doco upstream that need to be pasted back together, mentally].  Can you spot the error?
Full site feed:
Atom 1.0: http://blogname.blogspot.com/feeds/posts/default
RSS 2.0: http://blogname.blogspot.com/feed/post/default?alt=rss
  ...
Label-specific site feed:
Atom 1.0: http://blogname.blogspot.com/feeds/comments/default/-/labelname
RSS 2.0: http://blogname.blogspot.com/feeds/comments/default?alt=rss/-/labelname
  ...
Individual post comment feed:
Atom 1.0: http://blogname.blogspot.com/feeds/postId/comments/default
RSS 2.0: http://blogname.blogspot.com/feeds/postId/comments/default?alt=rss
There is the obvious need to s/comments/posts/g, but more is needed.  I am accustomed to 'magic CGI directories' that accept variables.  I use them myself.  See, e.g., the expanded URL to the thumbnail of Mothra which is not just an image, but the  filename, and a link to the full size one.  No express CGI script is called out, as the index file for that directory is actually a smart CGI script
Enough clues, and on to the answer.  I put bit of text around the answer so your eyes do not pick it out.  The text at the fourth bullet above is malformed ... the part following: alt=rss needed to precede the question mark marker that identified the start of variables to the CGI script.  We move before it the part: /-/labelname and add the desired label.  Now a custom subfeed chosen by label is properly specified
But there are no road signs on the Blogger provided doco page to permit easily reporting errors, so that they might have be fixed
Nuts

No relation
2009-03-23T02:20:00.010Z
Separated at birth?
We get questions, asking about the use of the 'orc' moniker, in IRC and at this blog. Tolkein inspired?
World of Warcraft?
some older mythology?
None of the above
Nothing so derivative.

When first using IRC, the Freenode 'nickserv' wanted a userid not in current use, and the 'usual suspects I prefer to use were long locked up; so:  orc_orc and the related variants I use.  The 'Blogger' software added a later constraint to the DNS character set, causing the drop of the Underscore to form a valid domain name.

This pair of latecomers pictured above may be related to one another, but I have to disclaim any connection.

Every step you take ...
2009-03-20T15:58:00.021Z
I received the above email [which I converted to a maskable image], with embedded web link, seeking market research data.  I have masked the full URL, to prevent 'ballot box stuffing' and to protect my privacy

Now in doing good statistical sampling, customarily one assures the recipient / respondent that the responses are aggregated, and that no personally identifying information is available to the researcher. This is done to foster truthfulness and frankness from people responding to the survey, by reassuring them that no information leaks, say back to the entity covered by the survey can tie particular positive or negative 'pull comments' to a specific person
Other survey research techniques use 'calibration' questions, repeated in slightly varying form a couple of times in the survey, to make sure the respondent is actually reading the questions, is answering consistently, matches the 'shaped sample' desired demographic, and similar concerns

Here, I am solemnly (or perhaps, cheerfully) assured:We will also gladly share the aggregate results of the survey with you, as it may be of interest to you.

All responses will remain anonymous and confidential.

What is does not say is that the author is not planning to use the data for selling 'individual drill down' detail by respondent

The sender is sort of aware of this, or perhaps it is just a boilerplate footer from SurveyMonkey:
This link is uniquely tied to this survey and your email address, please do not forward this message.
I think I will pass on this one.  Time for more coffee

Revised to lay better in the top table 20 march 2009

Caveats and Disclaimers
2009-03-18T15:09:00.016Z
  This is a bit of housekeeping about this blog -- the boilerplate so to speak.  I mentioned the need to do it, so here it is

I am an economist, duly trained both in academia, and in that broader school of life.  I am a 'rough around the edges' statistician.  I have been coding since before formal exposure to either of those disciplines.  I am a mathematician.  None of these pursuits carry formal certifications relevant here.

I am a lawyer, trained at a top ten school, long ago and far away, it seems.
---------------start disclaimer-------------------
I_A_AL, but not your lawyer.  I offer legal advice and formal
opinion only within the confines of a previously  established
and explicit attorney-client relationship where privilege may
be had;  and NEVER on a public list server.
----------------end disclaimers ------------------

I may own positions from time to time in entities mentioned, and while I will try to flag such, obviously times and holdings change, and I'll not be updating such enumerations.  I am NOT your investment adviser, not licensed as such, offer merely opinion which I may or may not advocate (an economist and lawyer can and should be ready to argue any side of an issue; sort of like 'high school debate club', but no holds barred) and do not render any advice or recommendation as to such matters

And this fun one:"This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold with the understanding that the publisher is not engaged in rendering legal, tax, accounting, or other professional service. If legal advice or other expert assistance is required, the services of a competent professional person should be sought."

-- from a Declaration of Principles jointly adopted by
a Committee of the American Bar Association and a
Committee of Publishers and Associations.
Generally, the appearance of trademarks or registered trademarks
within this blog are done as a nominative and factual matter, as
and for description and identification.
See, generally, 15 USC 1115(b)(4).

I am in no wise interested in any implied trademark
infringement or counterfeiting (11 USC 1114(1)); false
designation or unfair competition (15 USC 1125(a));
dilution (15 USC 1125(c); common law infringement or
unfair competition, or dilution; violation of business
practice law or regulation as to use of marks.

No patents are knowingly infringed, nor 'trade secret' or NDA matter disclosed

The photos used are Creative Commons licensed, or otherwise under a copyright I have proper access for reproduction. 

Please respect my copyright

No electrons are harmed permanently in the production of my blog content, although several get quite annoyed

I saw mommy, kissing Santa ...
2009-03-17T20:11:00.008Z
I can see her lying back in a satin dress
In a room where you do what you don't confess
  ...
I could picture every move that a man could make
Getting lost in her loving is your first mistake

   -- Sundown, Gordon Lightfoot 

It is always kind of a sad moment, watching a younger idealist encounter something that tears asunder their old mental model, and puts them on the path to being a battered, old, steel eyed mercenary.  But with that loss of innocence, new doors open

One useful paradigm to look at the consumers of Enterprise *nix software is to break them into a partition of three major types:Those that Have to have the 'Real McCoy', possibly for 'CYA' purposes, or because a upstream vendor says that they need the 'real' one as part of the 'silo' they will support without extra charge (if at all) to meet a performance SLA
Those who do not have a strong mandate, but are generally willing to pay the minimal incremental cost such a subscription adds to their bundle of functions, and
Those who will simply not pay for 'free' software: No how, no way; no, sir
The commercial enterprise Linux' have been generally successful in 'cannibal conversions' of enterprise consumers of 'olde skoole' proprietary Unix -- The morning's news has rumor that IBM is sniffing around JAVA.  We covered the topic, and Ted T'so's proto-quant thought piece on this [Ted being on leave of absence from IBM to the Linux Foundation, as I recall] some months back, in the context of the future for software freedom

All the young idealists from the BSD side of the FOSS house saw their holdings of SUNW eroded away in recent years, the progressive shifts away from hardware, away from ksh v. csh language debates, into the tangled place of license issue and re-inventions of approaches on scaling, as their firm flailed with JAVA [v. rather than use the one true type safe modern OO language, c++ // sorry, could not resist], into databases with a product that will NEVER be Oracle DB, no matter how hard it tries   

JAVA felt it had to move past Berkeley DB, and darn it, all the cool kids use SQL.  ORCL is the only credible lead player in database space (IBM and DB2 are there of course, but databases are rounding error to IBM's financial statement).  JAVA never could articulate the unique value proposition that picking up MySQL, AB, brought to the table, and let the acquisition languish, perhaps hoping that the database's engine in the 'LAMP' stack would pull in tier 2 conversion sales (see the next part, infra).  I think they have pretty well demonstrated that "hope" is not a business strategy to follow

Then there is that second tier -- FOSS *nix in through the side door, and without formal support contracts at first.  "Under the Radar", so to speak.  [Note: The linked article is a bit 'snarky' about Bob's new venture: Lulu, but I find it a wonderful and reliable service, to convert 'print pre-flighted' PDF's to bound books, for cheap, fast and reliably.  Highly recommended.] 

Just as I might choose to burn up a laser printer to print a manual, and do home-brew binding, Lulu has found a value proposition that makes me 'buy' their service, rather than 'build' it myself.  They have convinced me that outsourcing my printing to tier 1 is the 'right' decision.  He has converted me to producing wonderful documents from TeX that his business handles the ink to paper, binding and delivery parts.  It seems Bob is also 'whiteboxing' short run, 'just in time' print of conference manuals, and continuing education materials.  A nice niche, but low barriers to entry

And then there are the 'No how, no way' school in tier three.  This recent post in the CentOS forums, "leasing CentOS5 from DataCenter", caught my eye:Recently we had a customer come to us asking how much we lease out CentOS for.
I thought this was an odd question - since CentOS is ... FREE

When in dialogue with them I learned they have a number of servers with a different provider that charges them $5.00 per month for the Operating system.

I thought this was a bit strange - and wondered - Is it even legal?

How can a datacenter lease out something that is free?
I could understand perhaps charging a setup fee based upon a customers requirements - this is a service --- but
for a datacenter to live off of the backs of someone else by charging for something that is free -

it just bugs me and rubbed me the wrong way -

Any thoughts - ?

Not sure why it bugged me so much - perhaps its because we write a ton of opensource software and could not imagine someone charging for the software itself.

Support / Installation / Service yes - but the software ... i thought thats what GPL protected folks from 
This poster has missed the point of the GPL so widely, it is painful.  

The GPL is perfectly fine with charging for software which requires that it be accompanied with an offer of access to the sources it was built from.  This is what builds markets, and indeed, what makes CentOS possible in part.  CentOS is fine with a redistribution and commercialization, so long as our marks and brand are not mis-represented.  [Advert: The CentOS project would put a 'tithe' of that rental to good use -- money, machines, bandwidth, and so forth, but it is not mandatory.]  A better question might be: Is the data center that employs that poster itself providing the GPL required offer of sources access, and meeting its duty to provide, when they provide binaries under 'lease'?

Someone may well come along and undercut a person selling GPL and related FOSS licensed software for less.  I wrote a post encouraging people who 'cannot wait' for the CentOS 5.3 respin, or the updates which get stacked up, waiting for that stabilization process to end, encouraging them to 'outcompete' CentOS.  I am fine with that.  I know it won't happen generally [Scientific Linux is the closest credible 'fellow traveller' remaining on this highway; Hi, Connie and Troy] soon, as it is non-trivial to ship and support the full line product

The protection of the Four Freedoms under the GPL makes it inevitable that someone will make a run at commercializing FOSS;  this is a 'Good Thing'.  But then the trick is to provide value; that is, also provide design services, consulting, 'service after the sale', or build a support infrastructure, to make it safe to entrust one's most valuable assets to that software.  I feel CentOS meets that test in the 95% case for tier 2; others may dial that number up or down, and do according to their risk tolerance 

And with that, we are back to my post sending people with an external factor 'beating on them' about SLA's, to: Go Buy from CentOS' Upstream

Disclaimer: I hold direct positions in JAVA (minimal to get keep skin in the game, and to remind me to follow it) and ORCL, and have held IBM in my past; I regularly quote against IBM as to providing third-party *nix support services.  I probably need to write a Caveats and Disclaimers post

Embarrassingly parallel
2009-03-12T18:38:00.018Z
Bruce Schneier, in his 'Crypto-gram' summary this month, has an outlink to a story in The Register on a purported desire of the US NSA to crack Skype's call crypto 

But this misses the point -- the needed technology and infra-structure are out there already, fielded, and ready to go, pretty everywhere.  Let's take a hypothetical country -- call it 'Glassware' ("US", "China" and "Elbonia" were taken)

The country of Glassware has a population of M * 10 ^ N people

Of those M * 10 ^ N people, the average family size is three, and there are an average of two cell phones and one television (the latest -- digital)

There is a broadcast infrastructure suitable to distributing portions of a problem sample -- say, the header block -- sufficiently long that one can detect when a 'good' private key has been found, which is sufficient to decode something encoded with an asymmetric encoding public key.  

That target information is distributed over the airwaves, in the vertical blanking interval or sub-carrier side layer, itself encoded with a private key, readily decodeable with one of several 'factory included 'public keys'

The power supply switches in the television sets do not actually place the sets into a 'No power drawn' mode -- just into a lower power use 'sleep' stand-by mode.  When tickled with the right signal, and not otherwise engaged in presenting content to possessors of that unit (who might complain about glitches if the video graphics display processor did not fully paint their screen), it is possible to wake them up to do some ciphering.  Good for them -- recycles the electrons, and so forth

The television has a handy feature -- it will accept and display caller ID information from nearby affiliated cellular phones, over BlueTooth -- it can be configured to ONLY display wanted cell phones, but it will receive data and collate data from all ringing near it.  

So when Mrs Glassware has her girlfriends over, and the babysitter calls during the home sales party, the TV will pop up an alert for them of the call over the din of the fun.

The TV also sends back, over SMS messages, duly encoded and encrypted, the logfiles to series of central collation points -- Father Glassware can see when the oldest son is over at the home of the girl from the wrong side of the tracks.  The benefits are as broad as the imagination can see.  Who could be against protecting the children?

Those cell phones as it turns out are really not using very much of all that processing power they have in THEIR 'CUDA chips to draw those dinky screens, and are really off most of the time as well.  

Let's not waste their graphics processor chips as well, when they are on the charger.  This is great, as it simplifies the math.

Perhaps Glassware have an even better infrastructure -- say a national conversion to High Definition digital media signaling, and a mature broadband or cable modem backbone.  All the better for shuttling information around digitally.

A friend who deals with quants, tells me the quants are all hot and bothered to get 4 x quad head graphics cards in Dell Precision units -- 16 GPU's, because each of them can do a 10,000 (10 ^ 4) speedup over the simple general purpose processors in the underlying processors the chassis carry.  All for under $10k a unit.  They are doing the math and think they can have a huge HPC farm, just in the normal overhead which their traders and developers have to have anyway to do their day jobs.

M is 3 in the US (we'll round to 4 to make the math prettier), and perhaps 10 in China, and N is 8 (a hundred million).  Feel free to pick a value for your local Glassware

So properly harnessed, we have at least: M * (10 ^ N) * (10 ^ 4) in compute engines available to us -- we should be able to crank out at least 100,000 samples a second ... 10 ^ 5,  In cough numbers -- sufficiently accurate for our 'back of the envelope' purposes here, 10 is equal to 2 ^ 3.  2 is useful, as it is bits of key strength to solve.  There are 8.6 * 10 ^ 4 seconds in a day -- call it 2 ^ 16

so:  M * 2 ^ (3 + 3 + 3 + N + 4 + 5+ 16)

US:    2 ^ 43 key trials per day; 
China: 2 ^ 44 key trials per day.

The old DES cipher had a 2 ^ 56 bit keyspace -- worst case time to solution is 2 ^ 13 days and always getting better as build out scales in, without even beginning to bear pre-processing tricks, One time pad reuse, identifying non-perfect implementations, planting known cribs, and the rest.  

And it is Free, free, free -- or better yet, paid for by others.  What was that old saw about people living in glass houses? 

No sparrow falls, but that ....
2009-03-09T21:15:00.027Z
Letters, we get letters ... 

> I saw a blog posting recently where you  
> commented that RHEL 2.1 was based on RHL    
> 8.0.  My memory /understanding is that 
> RHEL 2.1 is based on RHL 7.1... or maybe 
> 7.2...  but not  8.0.

memory fades ... let me dig a bit

Let's see ... I was on Red Hat's 'tester-list' external beta tester (under NDA) program

The watershed email (which put my memories in the RHL 8 timeframe) was:

Date: Thu, 17 Jul 2003 14:06:07
From: Michael K. Johnson 
To: testers-list@redhat.com
Subject: Heads-up -- change coming [Red Hat Confidential]

This mail contains only Red Hat Confidential material. ...

But there had been rumblings ever since RHAS 2.1 issued, which made me want the 'insurance policy' community RPM based distribution

Which led previously led to me setting off with Greg Kurtzer to get cAos going.  Greg and I discussed an enterprise product from the earliest cAos meetings (earliest I have a record of is: 29 Apr 2003).  Greg, Rocky Mcgaugh and I pitched this product concept and extension of the 'two Linux' policy to an IBM VP at SuperComputing 03 in November 2003 at Phoenix, AZ.  I remember flying down from Las Vegas for the meeting

And it matured quickly.  Parallel efforts were underway by others -- Tao Linux, Xos, and more.  There was a mailing list: rhel-rebuild list <rhel-rebuild-l@uibk.ac.at> but I've not seen a post on that list for eons

Date: Tue, 9 Dec 2003 09:30:23 -0600 (CST)
From: Rocky McGaugh 
Reply-To: caos@caosity.org
To: caos@caosity.org
Subject: cAos]  Announce:  centos

The cAos-EL projects have been officially renamed to:
        centos (Community ENTerprise Operating System)

Sadly, Rocky has died, too young. Greg (sensibly for his needs) stayed with the cAos part.  And I am an old, tired dog, am left on the CentOS part... Does anyone not not die too young while they are still young of heart?

so thus my recollection of late 2003 ...

http://www.owlriver.com/redhat_versions.html
RHL 8 general release Sept 30 2002
RHL 9 general release Mar 31 2003
RHL 7.3 GA May 6, 2002

http://www.redhat.com/security/updates/errata/
RH AS 2.1 General Availability:         May 17, 2002

negative rebuild report under 7.2
http://www.redhat.com/archives/rpm-list/2003-April/msg00001.html
April 2003

So .. it seems true that RHAS 2.1 was on a pre RHL 8 base.  But RHL 7.3 was so good, and RHAS 2.1 ... wasn't.  Perhaps that led me to conflate the fork point

A 'tip of the hat' to the sharp eyed blog reader Scott Dowdle who noticed the matter and called it to my attention

aid to memory -- 5.3 QA archive .repo file
2009-03-09T15:56:00.011Z
I mentioned needing to add the local install archive in a test candidate, at directory: /etc/yum.repos.d/ so that updates happen properly

[herrold@centos-5 yum.repos.d]$ cat CentOS-RPH.repo
[53]
name=CentOS-$releasever - 53
baseurl=ftp://xps400/pub/mirror/centos/centos-qa/centos/5.3/os/i386
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5
[herrold@centos-5 yum.repos.d]$ 

Without it, as noted in the twitter feed, we get some security updates from 5.2 [avahi, nss, avahi-compat-libdns_sd and nss-tools], and a unsolveable update set.

CentOS 5.3 qa - fresh installs testing on i386
2009-03-09T15:31:00.014Z

KB released released a twitter note: ah! the smell of a freshly baked distro.
earlier today, reflecting the presence of updated binaries, and wire-install ready images.  Yummy, and I noted them in my twitter feed.

I completed the rsync of the new candidate archive, and have knocked out both GUI and text mode domU wire installs of the i386 candidate.  As you may recall, I had done updates testing before, without incident.

The screenshot is fullscreen of 1440x1050 and I am not about to booger the PCO feed again -- direct link is at:  http://www.herrold.com/images/blog/c53.-qa.jpg

The test running (side by side in two xen domU test candidates) is a rather hard one, which build graphviz, on the way to the trading shim -- the test script can be retrieved from:
ftp://ftp.owlriver.com/pub/local/COLUG/shim-builder.sh 
for those who want to play along.  

Something is slightly wrong in the install of the graphviz binary produced, and I'll get that fixed.  Also there is some LaTeX issue which I need to run down in the current doco build, generated from a SPOT -- a single point of truth.  The -f option to shim-builder.sh may be enough to get past it, regenerating the .dot files, and this the .ps included art.

It succeeded in the GUI based installation.  We have some slight warning level noise in today's shim in building today's shim, but it is just c++ being picky (hoo-ray for type checking).  The TUI test lags, as I had to build the TUI instance separately, and get the local install archive set up in /etc/yum.repos.d/

This is ready to go as to my needs, but I'll test media installs, and on x86_64, next.

Wrestling blog markup, Round Three
2009-03-06T18:41:00.003Z

Lovely ... the large image in the prior post boogered the CentOS aggregator.  Posting this to push it further down the page.

Where is that coffee cup?

Wrestling blog markup, Round Two
2009-03-06T17:11:00.023Z
Perhaps I should have thought through what I was saying in preparing a posting a bit more to avoid getting bitten
This should look better in the CentOS consolidator: 

   Stuck between a rock and a hard place here, it seems. Notice the ocean of whitespace to the left of the picture of the pregnant lady on the left image    

than this mess:

PCO eats markup diffferently than Blogger.  Point taken.

But, dang it: 'wrangling' through blogging software quirks by adding tables with: <td>&nbsp;<img src="http://www.herrold.com/images/blog/spacer.gif" width="0" height="0">&nbsp;</td>
 blocks littered through it sure feels nasty and a lot like fighting browser quirks in a by-gone era. This should be WYSIWYG

 Where did my coffee cup get to?  

Wrestling blog markup
2009-03-06T16:12:00.010Z
Stuck between a rock and a hard place here, it seems.  Notice the ocean of whitespace to the left of the picture of the pregnant lady on the left image

I had previously noticed that the Centos feed was honoring exposing the <hr> 'separator rules' in my markup, where the Blogger was not, on my OMG post earlier this week.  Not a big deal, and I did a quick re-edit and report, but I dislike doing so.  It exposes a problem in the Blogger preview engine as well, I guess

But darn it -- I should not have to set up a table to box image placement, at this late date.  I had to do so to get the 'clown' picture at the bottom of Nine pregnant gals to lay right at the bottom in the Blogger client.  Then I see these side-by-sides once it hits the CentOS consolidator:

Grrr ....

News recap for Rip van Winkle
2009-03-06T15:47:00.007Z

This posted, in light of a question seen today:

CentOS' upstream published an announcement, reminding users that its 2.1 line reaches its scheduled end of life soon.

By the way: Just a quick 'shout out' and 'Thank you', by the way to John Newbigin, for being its shepard for the CentOS project.

From memory, substantive bug fixes were turned off, say, a year and a half ago.  John has been there for CentOS 2.1 (based on a product itself based on the largely lame and unloved RHL 8).  RHL 8 was a sick puppy, but also a gateway to the upstream's re-entry into the Linux market with the 'Enterprise' model, and so to CentOS itself

Reviewing this page, I see: tzdata enhancement update 2008-11-05, but that scarcely counts.  Before that, libxml2 bug fix update 2008-09-02 (but that carries security overtones)

Before that, the first 'real' bugfix seems to be:  dump bug fix update 2007-02-02  -- So, two years now since a material (non-security) bug fix.  

Time to get that conversion to CentOS 3 series done.

Nine pregnant gals in a queue
2009-03-06T14:17:00.013Z

I wrote a bit ago on the fact that the CentOS 5.3 update release was proceeding apace.  I may have been insufficiently direct.
If you feel you need the facilities provided by the CentOS project sooner than it is provided, or that you need deterministic releases of support: Please go buy such from our upstream, or from a third party vendor who can sell you the expedited subset of services truly needed.

Almost all of the CentOS team have active consulting practices, or had such before their present $DAYJOB.  They had demonstrated the ability to handle the matter.  

Similarly, CentOS' upstream has a unit cost for a three year JBOSS and their enterprise distribution product of US $297 -- WITH non-metered support -- This is unbeatable to point your pointy haired boss at when she is badgering you about 'CentOS is late'.

We are not late of course; we will issue the 5.3 respin when it is right, and not before.  There was some loose talk speculating that there were insufficient resources, or that QA testing had not begun.  Neither is correct.

Consider the well known lesson as to the futility of trying to parallelize a task at a sequential constraint chokepoint. This was pointed out by Fred Brooks in The Mythical Man Month.Adding manpower to a late software project makes it later.

Brooks constrains his example to late tasks, but it turns out this is a broader rule than that, and has general applicability.

A quick example might be to consider the time it takes to produce one new human baby.  For one woman, the time is nine months.  No matter how hard they try, nine women working in parallel cannot get that one baby any quicker by adding eight more pregnant women to that queue.

CentOS has some goals in its build process which the upstream does not -- we strive to produce the packages we release on a 'self-hosting' basis, so that anyone who works at it can replicate our work freely.  Upstream has never had that goal in their RHL nor now their Enterprise product.  We have to identify failures with the tools such as the ones KB talked about recently.  

Also, build sequence matters a lot in bootstrapping into a next point release;  there are hidden build order dependencies which need to be solved -- sort of like packing a station wagon with furniture and household goods, when moving.  The big stuff HAS to go in first, and the little stuff later placed in 'found' gaps.  This cannot be well parallelized.

We have the QA team up and primed [there is a non-public webpage, and I see 29 members by a quick count]; the needed ACL's to get at the candidates are in place and tested [I pushed an update earlier this week for one member]; some QA has occurred.  I updated some results I had announced a couple weeks ago, in a coordinating mailing list on the QA's notes on this release as well.  

This is a little too wide a parallelizing fanout, in terms of coordination of testing, but it happens to be how it turned out this time.  The future with a CentOS 4.8 and 6.0 coming will probably be a bit smaller.   The QA master (and indeed I hope, each of the 29 individually) will review the participation at the end of this cycle, and some on the list will get dropped from a QA role, and slots freed up for new members to be invited in.

Experience has shown that there is no sense adding 'community' that does not put their shoulder to the wheel and work;  We on the CentOS team see the laments from people wanting to join.  It seems to me from watching, that they really want to consume and not carry the load of CentOS.  Note that 'users' of CentOS are not our target 'community'; they are welcomed, but really, how does having lots of support HELP the project;  the main IRC channel #centos is consciously limited to CentOS specific issues, and structured as a learning environment for a reason.

Want to be asked onto CentOS QA? Go for it, there are no barriers to demonstrating competence and interest in any of the following venue: file good bugs; comment bugs with reproducers or better, fixes; participate on a sustained basis in a knowledgeable fashion on any of: mailing list, wiki, forums, and IRC

The CentOS team watches all these venue all the time -- the CentOS 'community' is a meritocracy, and merit will be welcomed in -- but also know, this means there is an implicit 'Bozo filter' as well. 

Oh ... my ... goodness ...
2009-03-04T17:05:00.026Z
I've been mirroring, rebuilding, patching, filing bugs, and so on, against  bleeding edge source package RPMs (SRPMs) out of 'RawHide' for at least a decade.  It is one means I have used to push features from later Red Hat releases and hotfixes back into previous releases, to customers in our consulting (and on my own account).

We pushed post-RHL support, and commercial RPM-based support for side architectures including Netwinders [MIPS], PA-RISC, PPC, Sparc hardware, and Alphas from RawHide.  We have built FOSS-based 'latest and greatest' LTSP forks of reduced package sets for commercial applications from RawHide

A largely unheralded change to a new RPM package file format for Raw Hide SRPMs coming out to the builder, at Red Hat's rpm-4.6 breaks all that, for the first time in at least a decade ... Jeff Johnson, a former lead developer and maintainer for Red Hat preceding the current incumbent lead, bent over backward and jumped through hoops, to make rpm a lingua Franca.  Jeff was followed by a short-time incumbent, Paul Nasrat, who similarly did no harm to rpm

[herrold@centos-5 ctrlproxy]$ cp /mnt/nfs/var/ftp/pub/mirror/redhat/rawhide/SRPMS/ctrlproxy-3.0.8-2.fc11.src.rpm .
[herrold@centos-5 ctrlproxy]$ ls
ctrlproxy-3.0.8-2.fc11.src.rpm
[herrold@centos-5 ctrlproxy]$ rpmbuild --rebuild \
   ctrlproxy-3.0.8-2.fc11.src.rpm
Installing ctrlproxy-3.0.8-2.fc11.src.rpm
warning: user mockbuild does not exist - using root
warning: group mockbuild does not exist - using root
error: unpacking of archive failed on file /home/herrold/rpmbuild/SOURCES/ctrlproxy-3.0.8.tar.gz;49aeb249: cpio: MD5 sum mismatch
error: ctrlproxy-3.0.8-2.fc11.src.rpm cannot be installed
[herrold@centos-5 ctrlproxy]$ rpm -Vp ctrlproxy-3.0.8-2.fc11.src.rpm
Unsatisfied dependencies for ctrlproxy-3.0.8-2.fc11.src: rpmlib(FileDigests) <= 4.6.0-1
missing     ctrlproxy-3.0.8.tar.gz
missing     ctrlproxy.config
missing     ctrlproxy.init
missing     ctrlproxy.spec
[herrold@centos-5 ctrlproxy]$ 

Amid all that, the part which is important is that:  a new rpmlib of at level 4.6.0-1 is needed (marked in red), and that without it, it produces a rather unhelpful cpio md5sum error message (marked in blue)

For the short term, until I get the matter sorted better, I'll set up a domU Raw Hide xen instance (which has the later rpmlib, and so can manipulate the package), such domU will be upgraded just enough to handle rpmlib(FileDigests) <= 4.6.0-1, then frozen against other breakage from later other updates

Further I'll grant that domU RW access to the NFS export that contains my build tree (/home/herrold/rpmbuild/), so that I can position a SRPM into that tree, and unpack it rpm -U packagename.src.rpm

With that unpacked set in the SOURCES subdirectory -- the tarball, patches, and such; and the SPECS subdirectory .spec file, I can then (hopefully) switch into an earlier rpm variant on an older unit, and rebuild the package to write a new SRPM with the older rpmlib form

We'll see.  Change requires that the caterpillar moult and break out of the crysalis.  It does not require that an angry Mothra result, and destroy the surrounding city in concert with Godzilla.

It might be asserted that this is some sort of performance or speed optimiszation.  We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. -- Donald Knuth
rpm has had the instrumentation capabilities in place to see about where performance penalties lie, and has them for some time, so that one can easily test where load is.  This is not rocket science; similarly, we drilled in micro-second instrumentation and time-stamping in the shim, as we are accustomed to looking for code bottlenecks, and people using our software, and trading in the financial markets really care about where lags are.  I wrote a bit about this last month in the New Future Always Coming piece.

As proof of the pudding, take a moment and  run rpm 4.6.0 with --stats on a large package built under the old format, and rebuild and repeat the test with the new 4.6.0 variant.   Please feel free to get back to me at: timetrials at owlriver dot com  as to where YOU find the true performance issues are, comparing a package build with SHA2, v. MD5.  I have my preliminary stats for a later post, but welcome more data

This sure seems like a gratuitous and thoughtless format breakage to me, with no backward compatibility path announced.  RHAS 2.1 is about to go out of support, of course (which dates back to a foundation including a fix for the as shipped RHL 8 rpm database locking issues), but RHEL 3, 4 and 5 just lost the ability to use Raw Hide so far as I can currently see

I may be missing something obvious by way of a workaround, and would be glad to be corrected, but ...

I really feel that the current incumbent Panu Matilainen has NOT clearly articulated this on the mailing list and bugs at Red Hat which I read very closely

This WILL cause at a minimum confusion, and also compatibility problems down the road with bi-directional interoperability at the SRPM level between both the LSB, and Red Hat's major enterprise Linux distribution competitor, Novell and its SLES line [seemingly being branded: SUSE Linux Enterprise, presently].  Oracle's UBL and its consumers are sort of off in a 'market niche' world of its own here, and the impact will be less pronounced. It will probably cause such SRPMs to 'just NOT work' in the SuSE buildservice, but I have not tested this yet

Heaven help the users of side distributions -- cAos, Mandriva, PLD, and the non-English fluent RPM file format using -- who try to rebuild and use a random found SRPM.  It really seems that Red Hat marginalizes (or forces one to choose a camp to join) other formats with this move.  Perhaps that was their intent.  Fedora has been used by Red Hat as a 'wedge' for this purpose in the past [consider studied dis-interest in inter-archive compatability], and it may be just more of the same

Grrr ... 
Disclaimer: These days, I am aligned with the RPM5 insurgents, and served as the long time maintainer of the old RPM website which Red Hat has since re-claimed

SportsCenter for Quants
2009-03-04T14:53:00.011Z
SportsCenter is familiar, has a friendly user interface, and is indeed omni-present -- We ate dinner last night at a sport bar, feasting on pub grub with my mother, my daughter, and her one year old son (and thus, my grandson), the baby.  That young man  is just getting walking fast down pat, preparatory to learning to run. 

But he was also craning his head from side to side, sitting in the booster chair, to see past the adult's heads, for a good view of a panel television, following the fast paced clips

but ...  What's a 'quant'? 

A quant is a specialized kind of a geek, focused on studying, modeling, and seeking to understand Financial markets.  They do this with an aim of being able to know just a little bit more, just a little bit sooner than other quants and other forms of market counterparty traders against whom they compete, to be able to arbitrage a profit from fleeting asset mis-pricings in a market by buying one that is 'too cheap' and selling the one 'too dear'.  Profit !!

Part of my morning routine, is to check the overnight results of the markets in summary, and to get a preview of what is coming up as 'scheduled' events' for the upcoming trading day; of course, today's unscheduled exogenous external events are not previewed.  I still need that time machine.  

This morning, Jim Chanos of Kynikos Associates was on CNBC's Squawk Box [6 to 9 am, US ET], coming into the 8:15 release of the ADP Jobs data.  For those of you who may not follow the Financial Layer, Chanos was the fellow who early on, clearly pointed out that the Enron empire and its emperor's had no clothes, in a quarterly conference call.  He was called a rude name, but Enron could not thereafter hide from the truth of its frauds, and toppled in short order.

Chanos had the courage of his conviction (that Enron was overvalued) to sell 'short' Enron stock, and then later 'cover' and close the position for cents from the dollars he had received in the initial sale.

Today, anchor Joe Kernen and wingman Carl Quintanilla had a good ten minutes of give and take with Chanos.  Good questions, and a very thoughtful and reasoned set of replies from Chanos. 

Chanos asks the very sensible question of WHY a arbitrary minimal capitalization requirement is 'set in stone' at 2.5 percent, rather than say: 0.5 [under a convention called Basel II].  Economics 101 covers fractional reserve banking, and the multiplier effect; governmental (social) 'guarantees' and the moral hazard of 'too big to fail' in the US [compare contra, the last year in New Zealand] have removed market discipline by re-incentivizing leveraging in the last few years.  

I say: Re-incentivized.  Note that we went through all this in a smaller scale in the so-called S and L crisis only twenty years ago -- did no-one in government remember?

I was raised in a family that invests, reads and thinks about application of what we see in the WSJ and related financial press [I currently favor IBD], mailing lists, and newsletters, and historical literature about markets.  Add training and practice as an Economist, and a lawyer, and developing quant tools, as well as having high power computing readily available all my life [currently, on a CentOS platform].  And so I am comfortable with Finance issues, and can work through what the 'correct' answers should be, by and large.  

There is a large body of well written and often entertaining economic history -- I'll cover a bit in future posts -- which a geek will enjoy reading

Some geek friends who hang out in the side #centos IRC channels seem to feel helpless before market events.  Treating CNBC as SportsCenter, and following the stars (and learning to recognize the rogues) is one way to gain confidence and financial literacy

Seven layer ISO stack, plus two
2009-03-04T13:40:00.010Z
The Seven layer OSI model stack -- an ISO standard, palindromically -- characterizes how computer and data network (and thus internet) applications communicate between separate computers in a defined fashion.  From the bottom up: Physical, Data Link, Network, Transport, Session, Presentation, and Application layers

Computer networking geeks use that the seven-layer stack, to specify and to analyze matters running from what a given voltage level means at the bottom (Physical layer), to how a email client retrieves email from a mailstore at the top (Application layer). 

In preparing this piece, I find see reference some  to mnemonic forms to get the sequence of the Layers correct that I did not know before: The mnemonics "People Design Networks To Send Packets Accurately", "Please Do Not Throw Sausage Pizza Away", and "All People Seem To Need Data Processing" may help you remember the layers.
 BambooWeb article

Some wags extend the definition thus:
Of course, above those seven layers are two more: the Financial, and the Political Layers, which also need to be functional.

The dis-functionality and friction of office politics, or perhaps of being unable to get funding needed for a more reliable and functional network infrastructure rather than a Bigger, Newer, Better computer to sit unused on a pointy headed boss' desk, each come to mind as obvious examples.  In wider scopes, Financial may encompass Economics and Markets; Political expands to social interaction, policy and polity

It seems in at least one restatement, to also have added an explicit 'religious' layer as well under some restatements.  I think one can perhaps subsume Religious into Political, but I can see the genesis of the historical tension to draw a difference between those asserting dominion over a person's earthly presence, and spiritual corpus.
Network technicians will sometimes refer euphemistically to "layer-eight problems," meaning problems with an end user and not with the network.

Carl Malamud, in his book [1991] book"Stacks," defines layers 8, 9, and 10 as "Money", "Politics", and "Religion". The "Religion layer" is used to describe non-rational behavior and/or decision-making that cannot be accounted for within the lower nine levels. (For example, a manager who insists on migrating all systems to a Microsoft platform "because everyone else is doing it" is said to be operating in Layer 10.) 
BambooWeb article

I see also that people have adapted use of [pdf] the Political and Finacial Layers.  No model is so good it cannot be twisted, abused and extended, it seems

Anyway, I lay this foundation to provide a kick-off target to point back to in future discussions of the Political and Financial layers of the stack

A new future always coming
2009-02-10T15:36:00.015Z
A mailing list interested in the trading of securities that I participate in had this thread in the last couple days.  My reaction after the setup:
There are API's in big languages that encourage adding calendaring systems. Does anyone want me to think about this

I am interested in your ruminations because I think the most difficult thing in statistical trading is not finding edges or whatever. It is getting good synchronized data. This may not be an issue for people restricting themselves to one time zone, and one data provider with one server with consistent time stamps.

The issue is much more complex than this -- and in part it is a 'trading with perfect future knowledge' one;  removal of scratched trades, re-ordering away of reporting lags, and the rest are often found in 'historical' datasets.  With 'cleaner' data, different trading possibilities, EX POST, seem more obvious.

I have not yet found a broker who will take the other side of a trade against a 'replay' yesterday's data corpus -- I assume they think the temptation for me to cheat and use my perfect knowledge has something to do with it.

The data as and when you see it is all you can derive trading signals from. One can datestamp it in when it crosses your side of the communication demarcation threshold, and 'game' possible responses against a synchronized corpus;  After reduction (another delay), an order may leave your plant.

One can simulate what _should have_ happened on orders leaving your plant, but this simplifying simulation usually 'assumes' that the orderbook (of your counterparties) is not affected by your entries, and that it will stand still for your order to arrive.  The markets, speed of light, and data communications networks do not and have never worked that way.

I helped in the discussions underlying this piece some years ago, written in connection with a redesign of a major Air Carrier's on line reservation system, spread across the North American continent:
        physics of systems design
What was wanted simply could not be obtained with the then-observed speed of light, let alone processing lags.  ;)

Rule 3 -- lots of sprightly little autonomous (loosely coupled) systems can sometimes trump a (monolithic) Big Iron dinosaur is my operative belief, and a centerpiece of most of my designs.

Knowing the in-built assumptions and limits of any system is basic to developing a Godel-ian model of the formal weaknesses of a system; finding ways to get inside an OODA frontier is essential to compete effectively against an informed counterparty.  This brief John Boyd [pdf] piece on this should be on any systems analyst's periodic re-reading list. The longer, Pulitzer Prize winning Godel, Escher and Bach, by Douglas Hofstadter is a staple for periodic re-reads, slowly and for contemplation, perhaps on the nightstand.

Ironically I was selected to take a survey on RIMM support  ...
2009-02-09T18:35:00.004Z
... and seemingly looking for places where BlackBerry support services can be priced: 

Here is your answer: I'll pay almost anything for a fix of a non-functioning product that is 'almost there', and won't continue to pay for a product that cannot be fixed (I'll switch away from the vendor, if the doco is incomplete, the support bad, or the offering bogus; I'll eat my loss and move on).

The survey design NEVER asked if the support received from RIMM worked or had value; what my opinion of the existing RIMM web offerings was; how the responsiveness of support modes were; how RIMM is handling its external interactions doing support.

I assume I was contacted to take a satisfaction survey in light of interactions with RIMM support -- recent ones have been about getting sync through the  PocketMac program [not recommended] to work in an OS/X environment.  Never could get it to work.

(URL was:  http://www.blackberry.com/redirect/rdr?sap-client=110& ... etc )

header was:
**********************************
Happy with your BlackBerry solution?
Please help us serve you better.
**********************************

and what looks like the contact data I enter when using the BB website.

Executive summary:

No -- I am not happy.

I have found RIMM support for the PocketMac backup of the BB device to be incredibly frustrating and 'live' support useless [on line doco is clearly incomplete, and email requests seeking clarification have gone ignored]

Ticket handling techs using canned replies, with absolutely NO interest in follow through, or ACTUALLY FIXING and confirming a fix for the issue.  The RIMM emphasis seems to be on closing the ticket, seemingly within three days, without a confirmation that the issue is fixed.

This leaves a very bad taste behind -- ticket numbers on request.

And the product Pocket Mac so bad that I have abandoned it for third party vendors' approaches (MarkSpace, and Google Sync)

Still not right, but at least partially usable -- and what do you know -- MarkSpace is running a beta [of its upcoming update to its BB synchronization tool], and opening and following bugs I file.

ps -- The blackberry.com website very safely and skillfully hides email addresses, so that all contact can come in only through avenues (webform inserts into databases) likely to be able to 'close and ignore' issues.  I assume there is no post-close QA review, as I have seen no sign of it.

Sad -- I thought the 'killer app' that lead to the BB was 
pervasive contact capabilities.  

I sent copies of this to each of the co-CEO's seeming email addresses for the exec suite pulled from a bit of google searching -- Please share or redirect as needed.  Call if you want to talk; I'll answer any email as well.  [I received a phone call from 'customer retention' and a 'follow the call email' with a 'role account' email address, and no telephone number for me to use.  RIMM just does NOT want communication to be initiated to it except on its terms.]

For what it is worth, prior live support by RIMM, gated through T-Mobile (where I have the ability at the end of the transaction to say:  'We both see that it just does not work -- cancel the service as you cannot deliver that you have sold me') had been fine.  

There's a message there, I think.

'Every move you make, every step you take'
2009-02-09T15:38:00.005Z
A while back I commented on the fact that the computers of the world seem to have issuance of traffic citations well integrated; After this piece about Italian parking citations, I received yet another citation advice, from the Milan area through the vehicle rental firm, Europcar, asserting the vehicle I was driving was noted traveling over limit.

Maybe, but I also carry a commercial driver's license [and so drive very conservatively from habituation driving very dangerous large trucks at one time in my life].  I regularly am ridiculed by family members for never speeding, coasting up to red signals, and easing away rather than 'jack-rabbiting' off the line at a green light. As it had already been charged against my credit card, there was no sense worrying about it.

The takeaway from this article: 
Smart traffic lights rigged to trap drivers is clear.  I have no reason to suspect a more fastidious level of care to avoid error elsewhere in the electronic traffic citation system.

... typo fix: Parking, not Packing citations

'Money for nothing, and the chicks for free'
2009-02-09T14:31:00.010Z
The weekend started with a private email from a long time friend, wanting to rebuild a semi-FOSS mixed commercial and community project, and asking for some analysis of what such a project would entail.  It turns out to be partially Debian based.  We'll be talking later today on the matter to see if he needs some consulting services to get the project done.

Unrelated to that, I watch the mailing list traffic in another project, which is designed to be a short lifespan, bleeding edge 'proving grounds'. One of the perennial threads that resurfaces is a proposal to take one of the 'better' releases (under some unclear metric of 'goodness' -- time based is most often seen [consider Ubuntu's LTS every N'th release]), and for 'the community' to support it for a longer time frame.  

A poster unfamiliar to me, Marc Schwartz, noted this over the weekend:
Keating quote in C|Net about the end of 'Fedora Legacy'

"Nobody has responded to our calls for help," Keating said. "There are a good number of consumers, people who will happily consume until the project ends; however they are not willing to actually do any of the work necessary to keep the project alive."

In other words, FL had a parasitic, not a symbiotic, relationship with its users.

If Scott is willing to do the heavy lifting and he has people that will step up with him to do the heavy lifting, then this project might have a chance. On the other hand, if people just want the output, but are unwilling to step up to contribute to the input, then this project, like FL will fail. It might take months, but it will fail.

At Owl River, we designed, built and offered a commercial general market product to work in parallel with our 'community side' work with first cAos, and now CentOS. Wings really never caught on, and neither did Ian Murdoch's Progeny venture, each offering commercial SLA post RHL updates offerings.  Progeny closed its doors a couple years ago now, and the domain progeny.com looks to have been sold off to a domain linkfarmer.  

As it turns out in our consulting, we seem to need just a few packages on top of a CentOS base, and related configuration.  It is my thesis that it represents an uneconomic waste of cycles to spin yet another full blown distribution, rather than just solving the remaining ten percent of 'hard parts'.  We meet our GPL obligations and our broader sense of giving back in support of FOSS by making our solution SRPM's initially built for customers freely available, and have do so for many many years.

The random drop-in posters on the mailing list, and in the CentOS IRC channel are of course whining that their free updates are slow in coming.  How dare we have personal lives, take time to get married, etc.?

It takes discipline and a thick skin to NOT rush a poor product out, but the purpose of CentOS is to replicate its upstream, warts and all, with trademark elidement and the minimal stabilization to get the installer and update tool working properly with our updates mirror solution.  We have other checklist items on this QA round as well, and frankly, it will ship when it ships.

For those who cannot wait:  Go for it; the wiki documents a non-root build environment, and it is not a dark art to build a limited set of updates. The Source RPM's are freely available upstream. We have documented the comparison scripts long since.  

You can solve the build, verification and stabilization issues just fine with just a few months work;  if you start now, when the next point updates come out, you won't have to wait at all.

One view of Contacts, everywhere
2009-01-15T13:50:00.021Z

I've been working through cleaning up my Gmail Contacts entries the last few days.  Google released an extension of their mobile application suite which does a basic but workmanlike bi-directional sync between the webbish Gmail Contacts information, and my Blackberry's internal store from the mobile device's point of view.  Thanks, Gmail team at Google.

I had looked in to writing a 'third party control' synchronization tool, so that I could have an authoritative store safely behind the firewalls, and in a CentOS LDAP backend.  Google publishes the needed API; RIM is less open, but the API and a device simulator is available as well behind some Export Control disclaimers, identity harvesting, and such.  In the FOSS world, the fruit from the barry project is maturing nicely as well, but tackling cracking open the datastore blobs (which RIM manipulates with some Java code in their implementation and SDK) is somewhat tricky and it is not for the timid.  

I write this having 'bricked' the phone a couple nights ago, and had to fall back to restoring from a backup image.

You _do_ take and test Level 0 backups at least weekly, right?

Just as I thanked Google, RIM also deserves 'kudos' for continuing to roll out fixes, and feature set upgrades on the phone chassis; it has added video movie capture, dictation ('voicenote') from them, and as it has a sufficiently 'open' API, I have been able to easily add applications from Google mobile, Remember the Milk, and Jott all co-existing on that chassis.  Add Google Docs, and drop.io tools, for mobile productivity completeness.

High recommendation for a high end approach to their portable devices, but the trackball retention ring on my unit is cracked and the dirty trackball issue others mentioned is also there.  Apple probably had it right early on, choosing the touch screen approach which the iPhone uses.  But Apple and ATT think they can profit maximize with an exclusivity (i.e., non Free and non-discriminatory access to the platform) deal in the US; They are free to their opinion, of course, but not with my support.

Back to the matter at hand, the Blackberry and the Gmail Contacts sync works sufficiently well that I'll leave it running; I need to learn a bit about how to do edits and sync in stages, rather than doing all ACDs ('Adds, Changes, and Deletes') in a single pass, so that I don't end up with 2, then 4, then 8 slightly different records shuttling back and forth.  The proper sequence is probably:  Edit to a single record first and sync; Delete the strays and sync.  Add's seem not to be a problem.

Once I have the process down, I'll still have to face that LDAP and third party local control [CentOS from the office] issue.  I'll probably pick up an Apple Mini based on the Intel processors for some other development needs, and so I can retire my trusty PPC eMac, which will not handle the OS/X 10.5 OS level.  The later OS/X versions have markedly beter LDAP and synchronization tools, and unlike RIM, Apple is pretty clear about the need to 'keep up' with the hardware franchise in order to get later enhancements.

Only you can prevent forest fires
2008-12-05T13:31:00.039Z
You say you want a revolution
Well, you know
We all want to change the world
You tell me that it's evolution
Well, you know
We all want to change the world 
 -- John Lennon, The Beatles

Ted Tso has a blog post commenting about the market capitalization of Sun (stock ticker: JAVA) being 'underwater' relative to its cash on hand:
Sun’s current market cap: $2.40 billion USD.  Sun’s cash on hand: $2.63 billion USD
 and advancing a concern that a firm hostile to Software Freedom might purchase and kill it. [A bit ironic, thinking about the Cobalt Qube; a PFY I trained a few years ago moved to Portland to work supporting the Qube software, only to have Sun functionally kill the Cobalt product within the year ... oops]

Not surprisingly, there is a maze of intellectual property rights to navigate in considering using an asset so valuable that a company renames its Wall Street 'nickname' from 'SUNW' ("Stanford University Network Workstation") to JAVA

Hopefully, Ted Tso was just playing 'Devil's Advocate' earlier this year in a thread of debate surrounding a proposed adoption of a java (Sun's 'Java' or otherwise) into the Linux standards base upcoming ver. 4.0 release stirred concern

Reviewing the bidding;
LSB weekly conference call chair Jeff Licquia stated the non-Free (OSI or FSF wise) Java Test Conformance Kit concisely in the post call minutes
Jeff: summary, Ted: do a checkpoint two weeks from now, to see that the features are getting in. Jeff: agenda item for August 13th call?  Ted: yes.  Mats: Java is also an issue.  Ted: issues with Java?  Mats: which spec?  Required methods, classes, etc.  Test suite question is also a big deal.  Ted: should follow up on these issues before Ron gets back
LSB Committee work happens in part on the mailing list.  The thread sharpening the Sun Java issues  in advance of that call kicked off here
Alan Cox questioned the wisdom of such an inclusion of a java
I put together a rather large piece detailing some issues surrounding the present Sun licenses as to distribution of the [Sun] Java's actually in use today as well as the accessing and using the TCK
Note:  If one wants the flavor of the thread, read the posts by Tso, Cox and Herrold out of the pipermail archive for August 2008.  At one point in that thread, I was asked: Why do you care.  I care because FOSS culture matters and needs defenders to stand up and advocate for it.  I can live with the balloting results for the 4.0 release

Software Freedom issues are an 'elephant in the room' at the Linux Foundation (current 'owners' of the LSB), and they try to thread a delicate balance to draw the commercial into the FOSS world 

My personal opinion is that LF need not worry so: The 'should we have a FOSS strategy' discussion is over; 'of course,' being the outcome.  Market forces will solve adoption because the firms in an given industry NOT using FOSS properly will have higher cost structures, and go extinct

Oh, and you too can make a difference; by showing up, participating with considered intent, doing justice, loving kindness, and walking humbly, all the while remembering that "politics ain’t beanball"

Behind Blue Eyes
2008-11-13T16:47:00.014Z

Dateline: U.S. Department of Labor
The largest increases in initial claims for the week ending Nov. 1 were in Ohio (+3,885), Michigan (+2,619), Pennsylvania (+2,155), Wisconsin(+2,119) ...
 -- UNEMPLOYMENT INSURANCE WEEKLY CLAIMS REPORT (week ending Nov. 8 2008)

A friend wrote:
All this proves is that when someone crosses a state line to register to vote it is just as easy to register for unemployment while you're at it.

I think it is probably much worse than that

It is easy enough for anyone to set up a (several!) new 'employers' and then walk away from them 8 weeks later with no individual financial responsibility for the 'tail' -- after all, we encourages formation of 'small business, the engine of economic growth' and barriers to entry should be small, right?

Unemployment benefits may be had at full rate for 6 months after 6 weeks employment at a given 'employer' if one is otherwise qualified; when an 'employer' goes out of business, the employees are eligible for benefits Several telephone poles in central Ohio had signs, with differing phone numbers, for what appeared to be short term 'jobs' working to elect Obama and 'make Change'.  I snapped a picture with my mobile device, and will see if I can find it for the exact text;  I recall thinking at the time:
-- Don't the 'employee candidates' KNOW they will be let go the day after the election

Now, I think the answer is:
-- Sure -- indeed they were TOLD by the recruiter at the other end of the phone, that this was a way to get rid of a pesky 'termination for cause' {disqualifying} black mark which was keeping them from what they were 'entitled to'

As the One won, and 'We can do it!' if the system is properly 'gamed', I think there will be no investigations after Jan 20 to 'connect the dots', and the Lame One will just snooze out his term. 'No law will prevent it'

And so the Republic was lost. "Meet the new boss; same as the old boss"

rpm -import of GPG keys, revisited
2008-11-10T18:24:00.018Z

Okay, I guess I covered too much too fast last time I discussed adding a signing key to RPM.  Let's do it again with more annotation and color commentary.

The RPM package manager (see: the old RPM.ORG website, which I maintained as 'rpm.org' for several years; JBJ's 'way forward' for RPM development site; and the rather sparse, intentionally stale, and to me useless site controlled by and populated to suit the Red Hat corporate agenda -- details of the fork in RPM are out of scope here) has the capability to verify through strong cryptography that a package is intact, and is counter-signed by a person in possession both halves of an asymmetric public and private keypair.  Assuming that reasonable care (where 'reasonable' is a very large and paranoid number) is used to protect the confidential nature of the private half, the chances of a successful substitution are vanishingly small.

Anyone can examine and inventory the keys in RPM's trusted keystore. The process of additions, changes, and deletions of keys is an operation requiring root level privileges, and so assuming a machine can be trusted (both network level and local physical level attacks need to be considered)

Enumerate the keys present:

$ rpm -qa gpg\*

Examine a specific key:

$ rpm -qi gpg-pubkey-e8562897-459f07a4

If we know or can determine the 'fingerprint' of the public half of a signing key, and if that key has been placed at a public keyserver, we can retrieve it, examine it, or even directly import it.  For the sake of this example, we again consider the Raw Hide SRPM signing key (with the re-organizations over time, Red Hat presently signs Raw Hide content with key: 0x4F2A6FD2 which the MIT keyserver identifies thus)

The CGI query on the link above used the 'op=index' modifier; the next uses the 'op=get' -- one assumes 'op' is shorthand for the type of query operation made -- terse, or key-bearing.  In any event, we retrieve the key into a local file thus:

$ wget -O fedora-key "http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x4F2A6FD2"

and then may examine it with the conventional 'nix tools:

$ less fedora-key
<title>Public Key Server -- Get ``0x4F2A6FD2
''</title><p>
<h1>Public Key Server -- Get ``0x4F2A6FD2
''</h1><p>
<pre>
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: PGP Key Server 0.9.6

mQGiBD+dnTsRBACwnlz4AhctOLlVBAsq+RaU82nb5P3bD1YJJpsAce1Ckd2sBUOJ
D11NUCqH8c7EctOquOZ5zTcWxHiWWbLyKQwUw2SUvnWa5SSbi8kI8q9MTPsPvhwt
 ... snip ...
r/T7zLrJeiljDxvX+6TyawyWQngF6v1Hq6FRV0O0bOp9Npt5zqCbDGs/iE4EGBEC
AAYFAj+dnTwAEgkQtEJp0E8qb9IHZUdQRwABAf/+AJwNVicN6A0I7EOfWx50PDHD
7SHw5wCfUJkeh/XlCrGdPASe/AXZB44jl2c=
=aXEw
-----END PGP PUBLIC KEY BLOCK-----
</pre>

The important thing to notice, amid the HTML markup, is that the key is 'armoured text' well set off with start and end markers, so that GnuPG (and also RPM) may pick the key out of the chaff.

We discussed previously the chain of steps we used to decide that the key was authentic, and worthy of trust; as such we do not repeat them here.

Then, using the 'sudo' command to temporarily attain 'root' rights for the importation step, we can insert (import into the RPM database) the locally checked key:

$ sudo rpm -import fedora-key

Or, assuming that we will do a post-insertion check, we can do the import directly from the keyserver:  

$ sudo rpm -import  "http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x4F2A6FD2"

Then we can re-inventory keys, and see the new one present, and the full name under which it may be found; part of the name is, conveniently, the 'fingerprint' of that key.

$ rpm -qa gpg\*
$ rpm -qi gpg-pubkey-4f2a6fd2-3fcdf8c9

Hopefully this clears things up a bit.

Going, going, gone ...
2008-11-08T03:34:00.014Z

They paved paradise and put up a parkin' lot
With a pink hotel, a boutique, and a swingin' hot spot
Don't it always seem to go
That you don't know what you got till it's gone
They paved paradise and put up a parkin' lot

-- Joni Mitchell
I flew into Phoenix for a week's rest, and was unpleasantly confronted with America West USAir's 15 dollar baggage check flying in.  Took the SuperShuttle up to Scottsdale, and had a driver on his third day on the job for them.  We were seeing part of the Valley new to him, anyway.

He drove a route up Scottsdale past the site of the former Raw Hide Western Town and Steakhouse.  It has been gone three years now.  The old site was bull-dozed flat, but  it looks as though the developer who started condo construction ran out of money half-way through the project.

Dinner early in the week up in Carefree was to be at Crazy Ed's 'Satisfied Frog' -- its menu has the old saw about how 'it is so popular, no-one goes there anymore'.  Last time, we ate at 'The Horny Toad' (as I recall Ed lost the Toad in a divorce; his ex kept running the place).  Drove in from the west after a visit to Fry's Electronics on Thunderbird.  Gone -- an imposter with new signage in its place -- google says I missed the close by a month.

Drove up to Page AZ for a wonderful nature hike in a private 'slot canyon' with Overland Canyon Tours and photo session [highly recommended and well worth the premium price]; on the way back decided to stop in Sedona for a nice dinner, and found that the high end restaurant we dined at a year ago May had closed doors; at least there is a new bank branch in its place.

Oh yes -- and two traffic circles -- a new concept there; their City Engineer must have heard how great they are -- on State Route 89 inside the city limits, and another four or five on the way back toward I-17 south of town.  As traffic circles are a foreign concept, each cardinal entry point had illuminated, gas generator powered signage explaining their use.  The fumes and noise are only remporary, right?

Well, it's the last night in town, and so we decided to go to Pinnacle Peak Patio;  the resturant may have opened to the public in 1957 (per its website last updated 2007), but their display cabinets show postcards and envelopes from WWII simply addressed to 'John Doe; Pinacle Peak; Phoenix'.  The waitress, proud of her new leather belt, and 'new in town' still bearing a California accent, came to the table.   Once we indicated we had been eating there for nearly 20 years, she blurted out that the place is 'to close next March, or mebbe a year later as the developer "gave an extension"' before it is to be knocked down for yet another 'resort community' near Troon.  She had heard about those other places, 'though.  Seeing my reaction, she offered that 'perhaps they'll rebuild inside the new facility, but it will probably look like all the other chain restaurants'.  Yeah, probably.

Well, at least a pack of coyotes woke me up at 2 in the morning, midweek, delighting at the moon; the red rocks watched silently as they have forever, and with any luck will continue to do.

... and then there were none
2008-10-29T17:11:00.019Z

(full size)

When the Nazis came for the communists,
I remained silent;
I was not a communist.

When they locked up the social democrats,
I remained silent;
I was not a social democrat.

When they came for the trade unionists,
I did not speak out;
I was not a trade unionist.

When they came for the Jews,
I remained silent;
I was not a Jew.

When they came for me,
there was no one left to speak out.
-- Pastor Martin Niemöller (1892–1984)

We had a truck strike the power pole for the building hit last week; it took out the transformer with a most satisfying 'pop'.  It also had the secondary effect of a power surge, which caused a 'fried' monitor, so that I had occasion to need a new one to get us back up to full complement.

New monitors offer an occasion to play 'monkey move up,' it is my turn for the upgrade, and the $200 price point has a nice Westinghouse L2210NW panel display [1680 x 1050 pixels, 22" diagonal] at the moment.  I have had a Westinghouse LTV 19W3 [1440 x 1050, 19"] which I have enjoyed using since January 2006, and it seemed to make sense to stay in the brand. (I bought the 3 year service plan on that one for an extra 25% on the price, as I was unsure as to durability of this, by first panel, but that has never been needed)

One trial and tribulation (and geeky challenge) of a new resolution is the need to adjust the video card driver to support the new Modeline, and to squeeze every ounce of performance out of the monitor.  I am an old hand with the Intel Modeline tool,  810resolution, and its successor, 915resolution, for my present X desktop chassis' video card.

Over time, 'progress' has removed the tools for a 'nix admin to configure a display for the X window manager:
Xconfigurator
xf86setup
a working X -configure
kudzu
system-configure-display
manual configuration of /etc/X11/xorg.conf

I find that the new panel has consumed 6 hours of setup time at this point, and is still not working, edge to edge at full resolution.  Unpleasantly I was surprised to find kudzu erroring and dying; ddcprobe --raw returns nothing; X -configure  and system-config-display seem to know only how to turn the screen blank and lock up the keyboard so that a power cycle is needed to regain the unit (I'll write more on this later); and manual edits of xorg.conf have so far succeeded in getting only an off center, mis-sized image up.

This is not at the magnitude of the atrocities of which Niemöller wrote so well; I see the battle raging about making a gratuitous change to VT's over on the Fedora-devel mailing list with false statistics abounding, and the usual 'don't bother us with the facts, kid; our mind is made up' on knowing what you need and want.  

Dax Kelson wrote well with diagnosis and action plan, but it seems to have fallen on deaf ears; 'pearls before swine', and 'the tragedy of the commons' again.  We must fight the good fight anyway for"The punishment of wise men who refuse to take part in the affairs of government is to live under the government of unwise men"

-- Plato

Summary, for those still listening: I want fallback (and degraded but partial performance) modes when a tool is not working as determined by the person looking at it; I want diversity rather than monoculture in tools; I want a upstream community which does not 'break expectation' by 'feeping creaturism' (or 'creeping featurism').

I'll take a stroll to Stauf's (the coffee shop down the street) to lower my blood pressure.

Master	Clients	Elapsed time (real)
pippin	nfs2, 10.16.1.231	0m23.281s
nfs2	10.16.1.231, pippin, localhost	0m23.702s
10.16.1.231	pippin, nfs2, localhost	0m22.551s

distccd[29673]	(dcc_check_client)	connection from :ffff:10.16.1.249:41771
distccd[29673]	(dcc_r_file_timed)	909179 bytes received in 0.078651s, rate 11289 kB/s
distccd[29627]	(dcc_collect_child)	cc times: user 1.132070s, system 0.144009s, 23039 minflt, 0 majflt
distccd[29673]	(dcc_collect_child)	cc times: user 1.092068s, system 0.104006s, 22481 minflt, 0 majflt
distccd[29673]	(dcc_check_client)	connection from ::ffff:10.16.1.249:41775
distccd[29673]	(dcc_r_file_timed)	818437 bytes received in 0.071648s, rate 11155 kB/s
distccd[31248]	(dcc_check_client)	connection from ::ffff:10.16.1.249:41779
distccd[31248]	(dcc_r_file_timed)	886761 bytes received in 0.076688s, rate 11292 kB/s
distccd[29627]	(dcc_collect_child)	cc times: user 1.068066s, system 0.112007s, 23890 minflt, 0 majflt
distccd[29673]	(dcc_collect_child)	cc times: user 1.108069s, system 0.112007s, 22012 minflt, 0 majflt
distccd[29673]	(dcc_pump_sendfile)	Notice: sendfile: partial transmission of 15868 bytes; retrying 344332 @15868
distccd[1995]	(dcc_log_child_exited)	ERROR: child 29673: signal 11 (no core)