hitting the ground running

I've mentioned creating and injecting a root ssh key into a new instance. The images we run are deployed with ssh enabled and not TCP wrappered; we COULD wrapper them and use the 'add an exception for ssh from a single IP' tool which the PMman web control interface has, but we have found the support load fallout from people just getting started is too high. Once they have deployed and hardened a couple of boxes, they 'get it and can use the 'lock from all' web tool, and then add a single IP if their taste runs to web tools

The very next step I take as to each machine I administer, is to run a hardening script. While I have published an outline here, I use a script rather than reading and scrape and pasting from that outline. This step is done through a script, not because I think I WILL forget something, but because I know the script will NOT forget anything, and is written to perform the hardening process in an idempotent fashion -- that is, when done, finishing the same end result, time after time. One path to get to better host security is to have good processes, consciously designed, systematically applied, and continuously improved

herrold@centos-5 admin]$ ./hardening.sh hostname.pmman.net
The authenticity of host 'hostname.pmman.net (198.178.231.xyz)' can't be established.
RSA key fingerprint is 86:6e:84:e0:27:57:dd:4d:1f:88:82:fc:42:1d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hostname.pmman.net,198.178.231.xyz' (RSA) to the list of known hosts.
hosts.allow                                   100%  488     0.5KB/s   00:00
hosts.deny                                    100%  390     0.4KB/s   00:00
iptables                                      100% 1337     1.3KB/s   00:00
sshd_config                                   100% 3325     3.3KB/s   00:00
README                                        100%  897     0.9KB/s   00:00
rollup.pem                                    100%    0     0.0KB/s   00:00
openssl.cnf                                   100% 9682     9.5KB/s   00:00
arm-pmman.sh                                  100%  363     0.4KB/s   00:00
sa-update-local-NOTES                         100%  877     0.9KB/s   00:00
sa-update-local                               100%  117     0.1KB/s   00:00
logwatch.conf                                 100%   80     0.1KB/s   00:00
rollup.pem                                    100%    0     0.0KB/s   00:00
Package sendmail-8.13.8-8.el5.x86_64 already installed and latest version
Package 1:make-3.81-3.el5.x86_64 already installed and latest version
Package m4-1.4.15-2orc.x86_64 already installed and latest version
Package iputils-20020927-46.el5.x86_64 already installed and latest version
logwatch.conf                                 100%   80     0.1KB/s   00:00
Stopping crond: cannot stop crond: crond is not running.[FAILED]
Starting crond: [  OK  ]
Shutting down sendmail: [FAILED]
Starting sendmail: [  OK  ]
Starting sm-client: [  OK  ]
Flushing firewall rules: [  OK  ]
Setting chains to policy ACCEPT: filter [  OK  ]
Unloading iptables modules: [  OK  ]
Applying iptables firewall rules: [  OK  ]
Loading additional iptables modules: ip_conntrack_netbios_ns [  OK  ]
Stopping auditd: [FAILED]
Starting auditd: [  OK  ]
/etc/aliases: 76 aliases, longest 10 bytes, 765 bytes total

info: inspecting /etc/aliases  for a root email forwarder off the box
# Person who should get root's mail
#root:          marc

info: 1. do you want fail2ban -- if so, run: ./fix-fail2ban.sh hostname.pmman.net

info: 2. updates are not run by this script: consider running:
   ssh -l root hostname.pmman.net yum -y -q upgrade --enablerepo=pmman-mail

info: 3. verify that root's email is properly handled

info: 4. now:           ssh -l root hostname.pmman.net
                cd /root/hardening/

   and do some patching and service restarting ...
[herrold@centos-5 admin]$ 

Note: the IP is obscured, and the host name and ssh host key altered. The edit to add an opff-box alias entry for root's email is to centralize all the miscellaneous cron and asynchronous notifications off the box, to centrally monitored point

Then as noted before, this is a stock CentOS 5 image, and so needs some further tightening done and updates run. I have long since scripted that process:

[herrold@centos-5 admin]$ ./fix-fail2ban.sh hostname.pmman.net
local-fb-fix.sh                               100%  256     0.3KB/s   00:00
Stopping fail2ban: [FAILED]
Starting fail2ban: [  OK  ]
[herrold@centos-5 admin]$ ssh -l root hostname.pmman.net yum -y -q upgrade --enablerepo=pmman-mail
[herrold@centos-5 admin]$ 

Then, as suggested, the edits on the remote machine

[root@vm175551137 hardening]# netstat -pant
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      3641/sshd
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      2627/sendmail: acce
tcp        0    240 198.178.231.xyz:22          76.242.0.abc:41936           ESTABLISHED 3593/0
[root@vm175551137 hardening]# history
    1  cd /etc/
    2  joe aliases
    3  yum install ipsec-tools
    4  rm *~
    5  newaliases
    6  cd /root/hardening/
    7  ls
    8  joe /etc/ssh/sshd_config
    9  /sbin/service sshd restart
   10  ls
   11  cp hosts.allow hosts.deny /etc
   12  joe iptables /etc/sysconfig/iptables
   13  /sbin/service iptables restart
   14  netstat -pant
   15  history
[root@vm175551137 hardening]# reboot 

Broadcast message from root (pts/0) (Wed Dec 22 14:47:10 2010):

The system is going down for reboot NOW!
[root@vm175551137 hardening]# Connection to hostname.pmman.net closed by remote host.
Connection to hostname.pmman.net closed.
[herrold@centos-5 admin]$ 

I added the ipsec-tools in support of an application this particular unit will be performing. The reboot at the end is for good measure to apply any new kernel and libraries through a clean boot. The 'true' hostname will be picked up from DNS PTR records, once that has been done already, after a reboot, as well

two minutes on keyed ssh access

In a Linux box, in BSD or derived such as under OS/X, or under Windows in Putty, a person can generate a LOCAL keypair which is used for keyed SSH access to such Virtual Machine instances. I have completely moved away from password based external access for all new instances, as keys carefully managed are demonstrably safer

So you know: There is some heat, but not a lot of thoughtful light about permitting and using root ssh access. Some not well though out security policies have a phobic avoidance of such. I'll address the matter in a later post, discussing hardening generally, sshd config file hardening, remote syslogs, TCP wrappers, iptables, and dynamic dictionary attack response

For the time being, let's put to one side and get past that security design choice rant, and accept that at least initially, a PMman instance has already made a short term choice for setting up access and for management of such units which are running CentOS or others following the Red Hat approach for management of instances [i.e., not SuSE/SLES, Debian family, nor the BSDs]

Under such Linux, this looks like this:

$ # create a high strength passphrase
$ # I have written of gen-pw.sh before 
$ gen-pw.sh  -a
a2Wa4aSaLWkRac
$ cd ~
$ cd .ssh
$ ssh-keygen -t dsa -f pippin.pmman.net.dsa

$ # there is a passphrase prompt here and 
$ # we use that: a2Wa4aSaLWkRac   
$ # -- also make a record of it in a safe place
$ # -- if one maintains multiple keys per box, it can be a 
$ # chore to manage this -- but see: man ssh-agent

$ #       this generated ~userid/.ssh/hostname.pmman.net.dsa    
$ #       [the private part] ... and 
$ #               ~userid/.ssh/hostname.pmman.net.dsa.pub   
$ #       [the PUBLIC part

$ cat ~userid/.ssh/hostname.pmman.net.dsa.pub

and scrape and place it in your mouse pastebuffer, and proceed to the web interface. The -f file's name 'hostname.pmman.net.dsa' is arbitrary, but chosen to be mnemonic

Then add a new stanza to: ~userid/.ssh/config like this:

#
Host                    hostname32.pmman.net hostname64.pmman.net
#  optionally one can make a note of the passphrase here, but 
#  at the risk of exposing such if a local dire read compromise
#  is experienced, or a backup of such falls into untrusted hands
IdentityFile            /home/userid/.ssh/hostname.pmman.net.dsa
PasswordAuthentication  no
Protocol                2
PubkeyAuthentication    yes
#

Note here a key may be used on more than one host; that is, we can add the same public key into /root/.ssh/authorized_keys of more than one unit -- here, both a 32 bit and a 64 bit instance with similar hostnames. But I get ahead of myself ...

Using a secure means, we need to transfer taht public key to a remote instance, and to add it to the right file; Here, we use the SSL protected web interface under the PMman machine management interface for a given machine, in the [more] menu, first item. By placing the public part into the web form box, the management backend at the datacenter, will be validated as to form, and then place that public key into ROOT's /root/.ssh/authorized_keys file

At that point, one can then ssh to that remote box as root, accept the host key, and take steps for hardening, adding a working userid, and so forth

loop -de- loop

As I count it at the moment, I am building and using content from more than eight loop mounted ISOs on a principal NFS, TFTP and 'next', and FTP server in the internal network

Red Hat has updates for 4.9, and 5.6 in beta; CentOS is in QA on a initial '6' release; I am doing private builds for a Fortune 50 on some backports out of RawHide and from some local packaging; and I am working on a 'back of the envelope' design and test to try to get control of the huge bloat on Red Hat ISO space for installs, to see if I can get a trimmed minimal installer for i386, x86_64, ppc[64] and s390x down to a single piece of CD sized ISO media. Then there is my favorite of the minimal wire install image, again which I package up into an ISO

Going forward, we will see more of encrypted filsystems across loop devices, and that will also put load on here. It may be time for the kernel folks to consider bumping that limit to 16

As such I regularly crest over the stock eight provided loop devices. To address this without a reboot, one simply has to:

# shutdown all uses of loop devices, so we can remove the module
/sbin/rmmod loop
echo "options loop max_loop=255" > /etc/modprobe.d/loop.local.conf
/sbin/depmod -a
/sbin/modprobe loop 

Note: that 'depmod' may not be strictly required, but will in any event be harmless, so I do it -- heck, I still type sync ; sync before rebooting a box, and I KNOW that is not needed any more. The force of habit ...

I add the suffix .conf on that file, because I was scolded by a Debian box a couple weeks ago on the topic; it seems that they are deprecating sourcing files in /etc/modprobe.d/ lacking such. Since when did Linux starting use file name suffixes to determine a file's content? -- at least it is not required to be in 8.3 format

Another approach is doing it with hard-coded values at boot time, with sysctl.conf or such

---

Tip of the hat to Paul Howarth on the SELinux rant I went off on last week; The interaction of loop mounted ISOs, and mounts in the FTP space of a filesystem can also be addressed with options to the mount command, and in the /etc/fstab with context= choices. He writes and points out:

I use context mounts to avoid it, e.g. in fstab:

/path/to/CentOS-5.5-x86_64-bin-DVD-1of2.iso
   /path/to/dvd1 iso9660
   _netdev,ro,loop,fscontext=system_u:object_r:public_content_t:s0
   0 0

... sorry about the funky line wrapping, but there is just no good way to display really long /etc/fstab entries

later, bye

CentOS is not for some people -- I get it and there is no sense getting agitated about it

17:20 rictec> regret to informe that i will have to un-install Centos as soon as i find the un-installer
17:21 orc_emac> rictec: we do not publish one, but it is this:
   as root:
   dd if=/dev/zero of=/dev/sda bs=512 count=1
17:21 orc_emac> as it is so easy to remember, we don't bother publishing one

'later, bye'

---

10 Dec 2010: reformatted due to some rendering issue reports

ripping out the safeties

There have been the endless bull session threads on the CentOS main mailing list, nominally on the subjects of SELinux and IPv6 the last couple of weeks. I am just not of a mind to tolerate cr*ppy content on mailing lists anymore. On one such list, a 'regular' whom I identify as '...' had the misfortune of being the 'designated bad example' of the day

On Fri, 10 Dec 2010, ... wrote:

Make sure iptables is off. Is selinux on and enforcing? If so, reverse that too (disabled).

and ... followed up with yet another ready description on how to disable the safeties

you know -- this is just a poor response all around ...

I understand that the poster produces a lot of content. I know that he is an old bull in the pen there. I also see that he races to post first, and I feel that others do not answer as a result. I see that he carries lots of URLs in all his posts so that Google's spiders crawling through the mailing list archive will widely index his business because it seems to have a lot of links

But quantity is not quality

The response SHOULD have been to sharpen the issue if it was unclear, and describe a diagnostic flow so that one can BOTH have iptables security, and SElinux protections, AND a working system

The question, trimmed to its essence:

I get this error:
(13)Permission denied: make_sock: could not bind to address [::]:8091

(13)Permission denied: make_sock: could not bind to address 0.0.0.0:8091

no listening sockets available, shutting down

Unable to open log

So the first question to know the answer to is:

did the process start and persist?

Check for this thus:

# netstat -pant | grep 8091 

If that command, which looks for a listening socket returns some content, the process is running and its name is shown

If that command returns nothing, the process in question did not start, and we need to try to manually start it

IN NO CASE does iptables PREVENT a process from starting as it is running in the kernel and blocking parts of the network stack from transiting packets

I note in passing that it is a usual voodoo local folk wisdom in this project that one does not bind to ALL interfaces, [0.0.0.0] -- I do not know if this is the case or in play here, but note in passing that the usual practice ** here ** is to build to a specific interface, or more commonly a specific IP

Such constraints usually indicate a problem in the underlying application [ISO layer 7] not being sufficiently mature to reply to the IP that it received a service request from, and to let the routing tables manage routing at the proper ISO layer

Turning to how to amend iptables for socket based services:

If one needs to have 8091/tcp open (or 8091/udp), OPEN IT PROPERLY

Add to /etc/sysconfig/iptables the following entry:

[herrold@elided iptables]$ diff -u iptables-ORIG iptables
--- iptables-ORIG       2010-12-10 10:15:19.000000000 -0500
+++ iptables    2010-12-10 10:15:48.000000000 -0500
@@ -14,6 +14,8 @@
 -A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
 -A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
 -A RH-Firewall-1-INPUT -p tcp -m tcp --dport 631 -j ACCEPT
+-A RH-Firewall-1-INPUT -p udp -m udp --dport 8091 -j ACCEPT
+-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 8091 -j ACCEPT
 -A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
 -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
 -A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 37 -j
ACCEPT
[herrold@elided iptables]$ 

and then as root run:

# /sbin/service iptables restart 

The rules are of a common form across all Linux variants, and exhaustively documented. There is not any valid reason NOT to manage iptables rules generation and maintenance 'right'

---

As to SELinux, there is full logging running when the auditd and the restorecond are present. One can identify, and add rules on the fly, to progressively add 'permit rules' all SELinux based 'intercepts'

Here is a sample script of general applicability, under the GPLv3+:

[root@elided bin]# cat selinux-fixup.sh
#!/bin/sh
#
#       selinux-fixup.sh
#       Copyright (c) 2010 R P herrold <info@owlriver.com>
#       License: GPLv3+
#
#       Additively build SELinux rule sets to investigate what
#       a new application needs
#
export
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
#
#       make sure we have all our tools, which may not install
#       in a stock CentOS 5 minimal installation
rpm -q diffutils       2> /dev/null || yum -y -q install diffutils
rpm -q audit           2> /dev/null || yum -y -q install policycoreutils
rpm -q policycoreutils 2> /dev/null || yum -y -q install policycoreutils
#
/sbin/chkconfig auditd on
/sbin/service   auditd restart
/sbin/chkconfig restorecond on
/sbin/service   restorecond restart
#
cd /root/bin/
#
/bin/echo "A"
/bin/touch oldlog
/usr/bin/audit2allow -i denial-log > oldlog
#
# /bin/grep ftp /var/log/audit/audit.log* > /root/bin/ftp_audit.log
/bin/grep "avc:  denied" /var/log/audit/audit.log* > /root/bin/denial-log
#
# echo A
# audit2allow -a -M ftpmirror
#
/bin/echo "B"
/usr/bin/audit2allow -i denial-log -M deniallog
#
/bin/echo "C"
/usr/sbin/semodule -i deniallog.pp
#
/bin/echo "D"
/usr/bin/audit2allow -i denial-log > newlog
/usr/bin/diff -u oldlog newlog
#
/bin/echo "E"
/bin/cat deniallog.te
#
/bin/echo "F"
# 

which is used iteratively, running the application, and when the candidate under test stops for 'mysterious' reasons (more about 'mysterious' later), re-running it to see if some new SELinux denial has occurred --- the 'diff' is designed to make it perfectly clear what the new denial was, and to add and apply the needed allow rule

We don't KNOW IF there was a SELinux denial yet, and if so, what the denial was yet, as the sharpening question was not asked, but to wrap matters up

To make a set of local allow rules permanent, see:

$ man 8 semanage 

for the methodology for making such permanent and persistent once the full set are known

But sometimes (here in my example case !! ) the fix NEEDS to be upstreamed so that others using FOSS gain from the fix -- let's examine that next

---

If the problem occurs in in a package from an upstream, one can 'file bugs' against the 'selinux' component, and that group are quite attentive to addressing such

A run of that script looks like this [and in point of fact, at one customer's premise, there ARE some fixes needed with Red Hat's DHCP client and VSFTPD when loop mounted ISOs are used, that I need to file a couple of bugs on]

[root@elided bin]# ./selinux-fixup.sh
diffutils-2.8.1-15.2.3.el5
audit-1.7.17-3.el5
policycoreutils-1.33.12-14.8.el5
Stopping auditd:                                           [  OK  ]
Starting auditd:                                           [  OK  ]
Shutting down restorecond:                                 [  OK  ]
Starting restorecond:                                      [  OK  ]
A
B
******************** IMPORTANT ***********************
To make this policy package active, execute:

semodule -i deniallog.pp

C
D
E

module deniallog 1.0;

require {
        type iso9660_t;
        type ftpd_t;
        type iptables_t;
        type initrc_t;
        class unix_stream_socket { read write };
        class lnk_file getattr;
        class unix_dgram_socket { read write };
        class dir { read getattr search };
        class file { read lock getattr };
}

#============= ftpd_t ==============
allow ftpd_t iso9660_t:dir { read getattr search };
allow ftpd_t iso9660_t:file { read lock getattr };
allow ftpd_t iso9660_t:lnk_file getattr;

#============= iptables_t ==============
allow iptables_t initrc_t:unix_dgram_socket { read write };
allow iptables_t initrc_t:unix_stream_socket { read write };
F
[root@elided bin]# 

which rules are saying that iptables and ftp need help:

[root@elided bin]# cat /etc/sysconfig/iptables
# Firewall configuration written by system-config-securitylevel
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -i eth1 -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p 50 -j ACCEPT
-A RH-Firewall-1-INPUT -p 51 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 21 -j
ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s a.b.c.0/24 --dport 21 -j ACCEPT
#       permit the north data center
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s a.b.c.0/24 --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with  icmp-host-prohibited COMMIT
[root@elided bin]# 

pretty bog standard, but for the:

-A RH-Firewall-1-INPUT -i eth1 -j ACCEPT 

general allow rule on a backside RFC-1918 network on that interface

Note: a.b.c.0/24 is a replacement I did for privacy purposes, as the specific values do not matter

and the loop mounted ISO images:

[root@elided bin]# cat /etc/mtab
/dev/sda1 / ext3 rw 0 0
proc /proc proc rw 0 0
sysfs /sys sysfs rw 0 0
devpts /dev/pts devpts rw,gid=5,mode=620 0 0
tmpfs /dev/shm tmpfs rw 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
/var/ftp/pub/mirror/redhat/rhel/ISOS/rhel-server-6.0-i386-boot.iso
        /var/ftp/pub/mirror/redhat/rhel/loop/1 iso9660
        ro,loop=/dev/loop0 0 0
/var/ftp/pub/mirror/redhat/rhel/ISOS/rhel-server-6.0-i386-dvd.iso
        /var/ftp/pub/mirror/redhat/rhel/loop/2 iso9660
        ro,loop=/dev/loop1 0 0
/var/ftp/pub/mirror/redhat/rhel/ISOS/rhel-server-6.0-s390x-dvd.iso
        /var/ftp/pub/mirror/redhat/rhel/loop/3 iso9660
        ro,loop=/dev/loop2 0 0
/var/ftp/pub/mirror/redhat/rhel/ISOS/rhel-server-6.0-source-dvd1.iso
        /var/ftp/pub/mirror/redhat/rhel/loop/4 iso9660
        ro,loop=/dev/loop3 0 0
/var/ftp/pub/mirror/redhat/rhel/ISOS/rhel-server-6.0-source-dvd2.iso
        /var/ftp/pub/mirror/redhat/rhel/loop/5 iso9660
        ro,loop=/dev/loop4 0 0
/var/ftp/pub/mirror/redhat/rhel/ISOS/rhel-server-6.0-x86_64-boot.iso
        /var/ftp/pub/mirror/redhat/rhel/loop/6 iso9660
        ro,loop=/dev/loop5 0 0
/var/ftp/pub/mirror/redhat/rhel/ISOS/rhel-server-6.0-x86_64-dvd.iso
        /var/ftp/pub/mirror/redhat/rhel/loop/7 iso9660
        ro,loop=/dev/loop6 0 0
[root@elided bin]# 

As a side note, I do not quite understand (it is a mystery to me) how Red Hat would have been able to test 'nightlies' for anaconda based FTP wire installs of loop mounted ISOs without that rule, but perhaps that case was not in their test coverage plan

---

SELinux has been around for eight years now I am told, and iptables longer (looking back to the ancestor packet filtering approaches (ipchains !!) under the 2.4 kernel and before) -- no business would run a box all 777 on permissions or on a root account with no password

This is not Dark Arts, or Black Magic, and one does not use voodoo methods of shaking a rubber chicken at such problems to solve them. Old dogs need to learn new tricks. Simply ripping out such protections is to be irresponsible. It is NOT proper sysadmin nor proper development

I see that composing this piece took over two hours, but it is durable content and accurate on the topics of iptables and SELinux for this project. THAT is a better answer than snapping out a quick:

turn off all the safeties

reply, I submit

Those who cannot remember the past are condemned to repeat it

A member of the trade press, formerly at the Linux Foundation, has speculated at length as to the release date of a CentOS 6. I and at least one other member of the CentOS core group were approached for comment on this topic, coming into the US Thanksgiving holiday. We were discussing how to respond, but we had not issued a formal reply. That writer went to press with a piece that expresses a date not of any formal CentOS origin or estimate. His words, his choice, his opinion, and nothing more

Here is a statement which is perhaps more accurate:

CentOS really doesn't do pre-release interviews as to release dates and process, other than what anyone may read in and infer from the 'centos-devel' mailing list. Any CentOS 6 series will ship when it is ready and will be available when it is announced

CentOS is the successor in part by merger of Tao Linux ('Hi, David'). This quote comes from the Tao

Those who know, do not speak; those who speak, do not know

Coping with xz under the RPM tools in CentOS 5

So there I am, minding my own business, building a SRPM from Red Hat's 'rawhide' archive, and it fails. They are cutting over to 'xz' compression for the tarballs they ship. Their archive, and so their call, and not the end of the world

The symptom shows up when rpmbuild goes to uncompress such:

...
+ rm -rf clamav-0.96.4
+ tar -xf /home/herrold/rpmbuild/SOURCES/clamav/clamav-0.96.4-norar.tar.xz
tar: This does not look like a tar archive
tar: Skipping to next header
tar: Archive contains obsolescent base-64 headers
tar: Read 6508 bytes from /home/herrold/rpmbuild/SOURCES/clamav/clamav-0.96.4-norar.tar.xz
tar: Error exit delayed from previous errors
error: Bad exit status from /var/tmp/rpm-tmp.36477 (%prep)

RPM build errors:
Bad exit status from /var/tmp/rpm-tmp.36477 (%prep)
[herrold@centos-5


The error messages could be better, but the older compression methods that are known to the 'file' program available to rpmbuild that ships with CetnOS 5 do not contain the relevant 'magic numbers' yet. Progress is like that, and so until and unless Red Hat backports support into its RHEL sources, CentOS will not pick up the fix in its version 5 mainline

One perfectly suitable response is to use the RPM5 branch of the package manager, which DOES know. But some people cannot relax that constraint for various non-technical reasons

This issue is rather like the old cutover from md5sums to shasums which RPM did a while ago, and that I wrote aboutI wrote about

The fix is straightforward:

1. Install the compressed tarball, spec file and any patches with rpm in the usual fashion
2. Uncompress from the unknown compression format and re-compress with a known one
3. Amend the spec file; here, I use grep to look, and as there is just one edit, sed to edit
4. Rebuild using the '-ba' option from the revised .spec file with the tools of the target environment (here, CentOS 5)
5. The resulting SRPM will be portable and as a result of the second step, uses a known compression

Lets look:

[herrold@centos-5 clamav]$ unxz /home/herrold/rpmbuild/SOURCES/clamav/clamav-0.96.4-norar.tar.xz
[herrold@centos-5 clamav]$ gzip /home/herrold/rpmbuild/SOURCES/clamav/clamav-0.96.4-norar.tar
gzip: /home/herrold/rpmbuild/SOURCES/clamav/clamav-0.96.4-norar.tar.gz already exists; do you wish to overwrite (y or n)? y

[herrold@centos-5 clamav]$


That question about over-writes happened because it appears the sources from a prior build of clamav-0.96.4 were not re-rolled into a 0.96.5 tarball by the upstream packager at RawHide, but may have been patched instead. I've not expressly looked

[herrold@centos-5 clamav]$ cp ~/rpmbuild/SPECS/clamav.spec .
[herrold@centos-5 clamav]$ rpmbuild -ba clamav.spec
error: File /home/herrold/rpmbuild/SOURCES/clamav/clamav-0.96.4-norar.tar.xz: No such file or directory
...


As we have not yet fixed the .spec file, this was expected, but is shown here so the diagnosis path is clear

[herrold@centos-5 clamav]$ grep xz clamav.spec
Source0: %name-%version%{?prerelease}-norar.tar.xz
[herrold@centos-5 clamav]$ sed -i -e 's@xz@gz@g' clamav.spec


And now the .spec file is ready as well

[herrold@centos-5 clamav]$ rpmbuild -ba clamav.spec
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.47404
+ umask 022
+ cd /home/herrold/rpmbuild/BUILD
+ LANG=C
+ export LANG
+ unset DISPLAY
+ cd /home/herrold/rpmbuild/BUILD
+ rm -rf clamav-0.96.4
+ /bin/gzip -dc /home/herrold/rpmbuild/SOURCES/clamav/clamav-0.96.4-norar.tar.gz
+ tar -xf -
...
checking host system type... x86_64-redhat-linux-gnu
checking target system type... Invalid configuration `noarch-redhat-linux-gnu': machine `noarch-redhat' not recognized
configure: error: /bin/sh config/config.sub noarch-redhat-linux-gnu failed
error: Bad exit status from /var/tmp/rpm-tmp.86669 (%build)
...


The build fails for other reasons out of scope for this post, in that a new configure 'target' is emitted. This is similar to a later compression format addition, but a different problem, solved elsewhere. Such a change is another part of distribution and brand management matters at Red Hat's part. I'll note the solution for this (putting to side seriously amending the rpm build environment macros, which is the 'one way' path into later versions) in a later post

Once all the changes are done, and the 'scratch' test builds and will install cleanly, I go in with an editor, manually bump the release value by one, and add a note in the changelog stanza. Then I repeat the build 'for record', signing, and distribution. The Release 'bump' is needed so the NEVR (name, Epoch, Version, and Release comparison which librpm does, and that yum calls through librpm to do can detect the fact that a later updated version is in an updates repository in due course

All done