disable IPv6 DNS results

We had an end user appear in the main #centos IRC channel the other day with a IPv6 problem.  That person had leased a VPS somewhere, and their provider had included and enabled IPv6, at least partially.  Something was wrong in the network fabric, so that while some IPv6 services worked, others did not; DNS results returned with AAAA record results; but then the VM hoster was not transiting port 80 TCP traffic.  Very curious, and frustrating to the end user who just wanted yum to work so they could install updates and packages on their instance

The culprit is the grafting in of IPv6 readiness in man 2 getaddrinfo.  This is the way of the future, so there is no fighting it on a long term basis, but tactically having a means to be IPv4 only is appealing for people just wanting to work in the older address space.  The TL; DR uptake of that man page is that in a properly functioning system, name resolution answers under IPv6 are preferred, and only if not available, does one fall back to the older IPv4.  But this places a premium on IPv6 actually working when present.  We've shipped a full native IPv6 setup for customers at PMMan for a couple of years ago, but I assure you that we had some head-scratching as we rolled it out, and found customers using tunnels from HE or SixXs were also leaking advertisements to other VM's.  We added rules to filter out the false traffic after a bit of tracing with TCPDUMP

I have blogged about it before when IPv6 link-local address resolution (the ^FE family) was confusing distcc under Debian a couple of years ago.  There are links in the CentOS wiki for approaches on disabling IPv6 traffic, which vary between C5 and C6

That last mentioned article has an outlink to a bugzilla ticket that offers food for thought.  It mentions in passing that one can direct a nameserver to NOT deliver IPv6 results with a fairly simple tweak

Another option is to add this to /etc/sysconfig/named:

OPTIONS="-4"

... so, ... it should be possible to set up a local cacheing nameserver on the localhost, configured to NOT return IVv6 content, and so workaround the issue.  This smells sort of 'hackish', but it would have the benefit of being a single method that should work in the general case and not be tied to any particular kernel version, or other variable

about this week's GoDaddy outage

It is not clear if a cabal of Anonymous hackers, or simple network administration issues, caused the GoDaddy outage of Monday past. I guess it does not really matter

What really would have hurt is if the root domain server constellation had been compromised, to well and truly take down the internet. A Domain Registrar sends along updates to those root servers periodically, and GoDaddy's outage, from the extent of our involvement with them, simply impaired our ability to renew domains, and set new nameservers (NS records). As we had no urgent renewals pending, that is to say, not at all

We do not rely on GoDaddy for DNS services, and really, never have relied on them for production purposes. For PMman and for our ISP and COLO services, we run three geographically diverse nameservers for most of our purposes. We also run a few others for customers' needs (PTR records for a couple of datacenters we are in, testing, demonstration units)

The true 'masters' of our externally visible DNS servers are simply not accessible from the public internet. We push out updates to our public nameservers by cryptographically protected rdnc transactions. Those transactions are logged, and the information causing a given RDNC transaction are created by queries into a local database with a custom written LAMP control interface based on the FOSS tools that are in a stock CentOS install. Compared to manually editing zone files, checking variants in and out of a version control system, and so forth, this more readily provides us with scalability, traceability and auditability. Why, I caught a piece of lint in a zone file just last week, reading the overnight error report emails

We also retrieve the state of the generated zone files at the client public nameservers, and check them for consistency and coherency, essentially after each update, to prevent errors from propagating. ACLs, transaction logging and other checks provide more tracability, and we closed the mouse hole that that 'lint' crept in through in short order

As a result of the GoDaddy outage, a couple of our 'alumni' tech support folks who have moved on in their careers to other employment, gave us a call Tuesday, because they remembered how paranoid I am on making sure DNS is available. I appreciate the calls, and we've some new customers as a result

People have strong opinions about GoDaddy, sometimes for reasons of political correctness; I like them, by and large, because they provide a workmanlike product for a price that is hard to beat. They sure beat the heck out of the old Network Solutions rates. I have something like 500 domains that I administer and renew and most are there, although some are at other registrars for both historical and other reasons

And while Danica Patrick is not my cup of tea, she is not hard on the eyes, either

Right, Left, down the middle

A couple weeks ago, a 'Derecho' blew through Columbus, on its way to the metro DC area. Amazon had some failures that cascaded through to people who did not have site redundancy. People know that the East Coast was hit hard, but as we are out in 'fly-over' country they did not perhaps realize that we had several hundred thousand people around here without electricity for a couple weeks as well

I've mentioned before that the primary datacenter that we run our PMman product out of is at the Tier IV level -- multiply redundant cooling, power grid, power backup, fiber entrances, carriers. The owner, a friend, is just a fiend that he does not HAVE outages

Me, too. In our after-event review, I see that one of our secondary sites here in town fell back to its generators, but the rest were all fine. But all sites we use are well covered, all fiber, all multi-homed. Planning for failure was in our deployment planning checklist; we pay for (and we charge for) that coverage; and I consider it worth it

A national footprint customer based in Canada agrees. And their lead technical person reports that our connectivity is haster than their datacenter eighty miles from their home ofice. Not surprising, as oAltantaur main DC is on a 'main line' fiber route between Chicago, NYC, DC, and Atlanta -- financial markets and federal government presences can help, that way

If the availability of your online presence matters to you, feel free to ask for a quote

LOPSA at the PMman DC

I went up to a meeting at our North datacenter for PMMan, where local group of system admins held a meeting, starting up a local LOPSA chapter. Food and soda were provided by the DC operator, along with salad ... since when did sysadmins starting more healthy food, rather than a diet of high sugar, high caffeine junk food?

The presentation slide deck was fine, and the presenter (a 'long timer' at a local credit-card clearance operation) ran through his bullet list of what to look for in the 'build vs. buy (lease space)' decision, and then a number of siting concerns.

Now I am familiar with his firm's site from prior visits, and it is adjacent to a major highway with regular closures for accidents; adjacent to a major rail yard where chemical spills have caused evacuations; and sole serviced into the power company grid

Our North site was chosen after a survey of all offerings within a radius we were willing to drive to for 'end of the world' 'hands on' intervention; is jacked into two independent power grids along with the on-site generators, is a premier demonstration location for the former Liebert (now owned by Emerson) power and site conditioning

I happen do drink coffee daily at Staufs with Liebert's representative here in town. My evaluation team suggested the location as a finalist, and when I checked, it turned out that I already knew the owner / developer from long, long ago telephony days, and when I have time, I'll go up and 'shoot the bull' with him on Saturday mornings at the DC

We have had a grand total of ZERO power related outages, and only one network connectivity issue in the last three years, that lasting less than 15 minutes, and that, due to human error in not handling a BGP fail-across migration properly [the cut-over protocol was changes, as I noticed the drop from my monitoring and called the owner's cell phone at once ;) ]. Well suited to our 'enterprise' customers

It is 'carrier neutral' and hugely connected -- multiple entrances of up to 88 x 100 G fiber spread across six or seven principal carriers. Native IPv6 to all drops we run through multiple carriers, along with the IPv4. I helped with the IPv6 design and cut-over some 18 months ago, and it has been seamless. The facility, and our services, just do not have outages except that human error causes

It is not the cheapest in town ... but it is fairly priced for the value we have received

I had not sat down and reflected on how satisfied I was with that shift of our center of operations to the DC, but as I think about it, I am well pleased

hitting the ground running

I've mentioned creating and injecting a root ssh key into a new instance. The images we run are deployed with ssh enabled and not TCP wrappered; we COULD wrapper them and use the 'add an exception for ssh from a single IP' tool which the PMman web control interface has, but we have found the support load fallout from people just getting started is too high. Once they have deployed and hardened a couple of boxes, they 'get it and can use the 'lock from all' web tool, and then add a single IP if their taste runs to web tools

The very next step I take as to each machine I administer, is to run a hardening script. While I have published an outline here, I use a script rather than reading and scrape and pasting from that outline. This step is done through a script, not because I think I WILL forget something, but because I know the script will NOT forget anything, and is written to perform the hardening process in an idempotent fashion -- that is, when done, finishing the same end result, time after time. One path to get to better host security is to have good processes, consciously designed, systematically applied, and continuously improved

herrold@centos-5 admin]$ ./hardening.sh hostname.pmman.net
The authenticity of host 'hostname.pmman.net (198.178.231.xyz)' can't be established.
RSA key fingerprint is 86:6e:84:e0:27:57:dd:4d:1f:88:82:fc:42:1d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hostname.pmman.net,198.178.231.xyz' (RSA) to the list of known hosts.
hosts.allow                                   100%  488     0.5KB/s   00:00
hosts.deny                                    100%  390     0.4KB/s   00:00
iptables                                      100% 1337     1.3KB/s   00:00
sshd_config                                   100% 3325     3.3KB/s   00:00
README                                        100%  897     0.9KB/s   00:00
rollup.pem                                    100%    0     0.0KB/s   00:00
openssl.cnf                                   100% 9682     9.5KB/s   00:00
arm-pmman.sh                                  100%  363     0.4KB/s   00:00
sa-update-local-NOTES                         100%  877     0.9KB/s   00:00
sa-update-local                               100%  117     0.1KB/s   00:00
logwatch.conf                                 100%   80     0.1KB/s   00:00
rollup.pem                                    100%    0     0.0KB/s   00:00
Package sendmail-8.13.8-8.el5.x86_64 already installed and latest version
Package 1:make-3.81-3.el5.x86_64 already installed and latest version
Package m4-1.4.15-2orc.x86_64 already installed and latest version
Package iputils-20020927-46.el5.x86_64 already installed and latest version
logwatch.conf                                 100%   80     0.1KB/s   00:00
Stopping crond: cannot stop crond: crond is not running.[FAILED]
Starting crond: [  OK  ]
Shutting down sendmail: [FAILED]
Starting sendmail: [  OK  ]
Starting sm-client: [  OK  ]
Flushing firewall rules: [  OK  ]
Setting chains to policy ACCEPT: filter [  OK  ]
Unloading iptables modules: [  OK  ]
Applying iptables firewall rules: [  OK  ]
Loading additional iptables modules: ip_conntrack_netbios_ns [  OK  ]
Stopping auditd: [FAILED]
Starting auditd: [  OK  ]
/etc/aliases: 76 aliases, longest 10 bytes, 765 bytes total

info: inspecting /etc/aliases  for a root email forwarder off the box
# Person who should get root's mail
#root:          marc

info: 1. do you want fail2ban -- if so, run: ./fix-fail2ban.sh hostname.pmman.net

info: 2. updates are not run by this script: consider running:
   ssh -l root hostname.pmman.net yum -y -q upgrade --enablerepo=pmman-mail

info: 3. verify that root's email is properly handled

info: 4. now:           ssh -l root hostname.pmman.net
                cd /root/hardening/

   and do some patching and service restarting ...
[herrold@centos-5 admin]$ 

Note: the IP is obscured, and the host name and ssh host key altered. The edit to add an opff-box alias entry for root's email is to centralize all the miscellaneous cron and asynchronous notifications off the box, to centrally monitored point

Then as noted before, this is a stock CentOS 5 image, and so needs some further tightening done and updates run. I have long since scripted that process:

[herrold@centos-5 admin]$ ./fix-fail2ban.sh hostname.pmman.net
local-fb-fix.sh                               100%  256     0.3KB/s   00:00
Stopping fail2ban: [FAILED]
Starting fail2ban: [  OK  ]
[herrold@centos-5 admin]$ ssh -l root hostname.pmman.net yum -y -q upgrade --enablerepo=pmman-mail
[herrold@centos-5 admin]$ 

Then, as suggested, the edits on the remote machine

[root@vm175551137 hardening]# netstat -pant
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      3641/sshd
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      2627/sendmail: acce
tcp        0    240 198.178.231.xyz:22          76.242.0.abc:41936           ESTABLISHED 3593/0
[root@vm175551137 hardening]# history
    1  cd /etc/
    2  joe aliases
    3  yum install ipsec-tools
    4  rm *~
    5  newaliases
    6  cd /root/hardening/
    7  ls
    8  joe /etc/ssh/sshd_config
    9  /sbin/service sshd restart
   10  ls
   11  cp hosts.allow hosts.deny /etc
   12  joe iptables /etc/sysconfig/iptables
   13  /sbin/service iptables restart
   14  netstat -pant
   15  history
[root@vm175551137 hardening]# reboot 

Broadcast message from root (pts/0) (Wed Dec 22 14:47:10 2010):

The system is going down for reboot NOW!
[root@vm175551137 hardening]# Connection to hostname.pmman.net closed by remote host.
Connection to hostname.pmman.net closed.
[herrold@centos-5 admin]$ 

I added the ipsec-tools in support of an application this particular unit will be performing. The reboot at the end is for good measure to apply any new kernel and libraries through a clean boot. The 'true' hostname will be picked up from DNS PTR records, once that has been done already, after a reboot, as well

two minutes on keyed ssh access

In a Linux box, in BSD or derived such as under OS/X, or under Windows in Putty, a person can generate a LOCAL keypair which is used for keyed SSH access to such Virtual Machine instances. I have completely moved away from password based external access for all new instances, as keys carefully managed are demonstrably safer

So you know: There is some heat, but not a lot of thoughtful light about permitting and using root ssh access. Some not well though out security policies have a phobic avoidance of such. I'll address the matter in a later post, discussing hardening generally, sshd config file hardening, remote syslogs, TCP wrappers, iptables, and dynamic dictionary attack response

For the time being, let's put to one side and get past that security design choice rant, and accept that at least initially, a PMman instance has already made a short term choice for setting up access and for management of such units which are running CentOS or others following the Red Hat approach for management of instances [i.e., not SuSE/SLES, Debian family, nor the BSDs]

Under such Linux, this looks like this:

$ # create a high strength passphrase
$ # I have written of gen-pw.sh before 
$ gen-pw.sh  -a
a2Wa4aSaLWkRac
$ cd ~
$ cd .ssh
$ ssh-keygen -t dsa -f pippin.pmman.net.dsa

$ # there is a passphrase prompt here and 
$ # we use that: a2Wa4aSaLWkRac   
$ # -- also make a record of it in a safe place
$ # -- if one maintains multiple keys per box, it can be a 
$ # chore to manage this -- but see: man ssh-agent

$ #       this generated ~userid/.ssh/hostname.pmman.net.dsa    
$ #       [the private part] ... and 
$ #               ~userid/.ssh/hostname.pmman.net.dsa.pub   
$ #       [the PUBLIC part

$ cat ~userid/.ssh/hostname.pmman.net.dsa.pub

and scrape and place it in your mouse pastebuffer, and proceed to the web interface. The -f file's name 'hostname.pmman.net.dsa' is arbitrary, but chosen to be mnemonic

Then add a new stanza to: ~userid/.ssh/config like this:

#
Host                    hostname32.pmman.net hostname64.pmman.net
#  optionally one can make a note of the passphrase here, but 
#  at the risk of exposing such if a local dire read compromise
#  is experienced, or a backup of such falls into untrusted hands
IdentityFile            /home/userid/.ssh/hostname.pmman.net.dsa
PasswordAuthentication  no
Protocol                2
PubkeyAuthentication    yes
#

Note here a key may be used on more than one host; that is, we can add the same public key into /root/.ssh/authorized_keys of more than one unit -- here, both a 32 bit and a 64 bit instance with similar hostnames. But I get ahead of myself ...

Using a secure means, we need to transfer taht public key to a remote instance, and to add it to the right file; Here, we use the SSL protected web interface under the PMman machine management interface for a given machine, in the [more] menu, first item. By placing the public part into the web form box, the management backend at the datacenter, will be validated as to form, and then place that public key into ROOT's /root/.ssh/authorized_keys file

At that point, one can then ssh to that remote box as root, accept the host key, and take steps for hardening, adding a working userid, and so forth

loop -de- loop

As I count it at the moment, I am building and using content from more than eight loop mounted ISOs on a principal NFS, TFTP and 'next', and FTP server in the internal network

Red Hat has updates for 4.9, and 5.6 in beta; CentOS is in QA on a initial '6' release; I am doing private builds for a Fortune 50 on some backports out of RawHide and from some local packaging; and I am working on a 'back of the envelope' design and test to try to get control of the huge bloat on Red Hat ISO space for installs, to see if I can get a trimmed minimal installer for i386, x86_64, ppc[64] and s390x down to a single piece of CD sized ISO media. Then there is my favorite of the minimal wire install image, again which I package up into an ISO

Going forward, we will see more of encrypted filsystems across loop devices, and that will also put load on here. It may be time for the kernel folks to consider bumping that limit to 16

As such I regularly crest over the stock eight provided loop devices. To address this without a reboot, one simply has to:

# shutdown all uses of loop devices, so we can remove the module
/sbin/rmmod loop
echo "options loop max_loop=255" > /etc/modprobe.d/loop.local.conf
/sbin/depmod -a
/sbin/modprobe loop 

Note: that 'depmod' may not be strictly required, but will in any event be harmless, so I do it -- heck, I still type sync ; sync before rebooting a box, and I KNOW that is not needed any more. The force of habit ...

I add the suffix .conf on that file, because I was scolded by a Debian box a couple weeks ago on the topic; it seems that they are deprecating sourcing files in /etc/modprobe.d/ lacking such. Since when did Linux starting use file name suffixes to determine a file's content? -- at least it is not required to be in 8.3 format

Another approach is doing it with hard-coded values at boot time, with sysctl.conf or such

---

Tip of the hat to Paul Howarth on the SELinux rant I went off on last week; The interaction of loop mounted ISOs, and mounts in the FTP space of a filesystem can also be addressed with options to the mount command, and in the /etc/fstab with context= choices. He writes and points out:

I use context mounts to avoid it, e.g. in fstab:

/path/to/CentOS-5.5-x86_64-bin-DVD-1of2.iso
   /path/to/dvd1 iso9660
   _netdev,ro,loop,fscontext=system_u:object_r:public_content_t:s0
   0 0

... sorry about the funky line wrapping, but there is just no good way to display really long /etc/fstab entries

checklist RO rsync server

Setting up a new RO RSYNC server setup

The primary usage case is we describe is how to deploy a read-only RSYNC server with no end user accounts, to be used for distribution of content (here, to move a builder result archive that is intentionally NOT 'visible' from the internet to a more capable transfer server) From there, the content is integrated into a internal archiving server, and after that, to a publicly accessible binary archive, accessible through ftp, rpm, or yum

As before, we start with a freshly deployed, and hardened PMman instance. At all times, we will strive to follow proper sysadmin 'best practices' discipline under SElinux, wrappers and iptables

Install and enable rsync, which is the package holding the stock rsync daemon. As rsync supports wrappers, we also need the xinetd which is the package holding the stock inetd in recent Red Hat derived distributions -- Let's get started:

1. yum can do the install trivially

   yum install rsync xinetd

2. Then enable the needed services:

   /sbin/chkconfig rsync on
/sbin/chkconfig xinetd on

3. We need to do some configuration for the rsync daemon as to permissions and directories to serve:

   [root@trap64 etc]# cd /etc
[root@trap64 etc]# cat rsyncd.conf
# motd file = /etc/rsyncd.motd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock

[trap64]
path = /var/ftp/pub/local
comment = x86_64 fruit
uid = nobody
gid = nobody
read only = yes
list = yes
# auth users = username
# secrets file = /etc/rsyncd.scrt
hosts allow = 10.0.0.0/24 127.0.0.0/24
hosts deny = 0.0.0.0/0

[root@trap64 etc]#

4. Set up the iptables -- I do not recall the rsync daemon port off the top of my head, so I look it up:

   [root@trap64 etc]# grep rsync /etc/services | head -2
rsync 873/tcp # rsync
rsync 873/udp # rsync
[root@trap64 etc]#

   ... so the port is 873

   # localhost can do all ...
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
# ...
# rsync daemon
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s 10.0.0.0/8 --dport 873 -j ACCEPT
# ...

5. Open the wrappers

   #
ALL: ALL@127.0.0.1
#
# ...
#
rsync: ALL@10.0.0.0/255.0.0.0
#

6. Restart the wrappers enforcing daemon

   [root@trap64 sysconfig]# /sbin/service xinetd restart

7. Test it:

   [root@trap64 sysconfig]# rsync localhost::
trap64 x86_64 fruit
[root@trap64 sysconfig]#

8. To put it into production on a client, we can use something like this:

   #!/bin/sh
#
# this file: /root/bin/update-archive.sh
# Copyright (c) 2010 R P Herrold
# License: GPLv3+
#
# ln -s /root/bin/update-archive.sh /etc/cron.hourly/
#
export PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
umask 022
# -- non local content goes into the mirror constellation
[ ! -e /var/ftp/pub/mirror/pmman/RPMS/x86_64/ ] && \
mkdir -p /var/ftp/pub/mirror/pmman/RPMS/x86_64/
#
# export VERBOSE="-v "
export QUIET="-q "
#
/usr/bin/rsync -a ${VERBOSE} ${QUIET} --exclude=working \
trap64.darkside.lan::trap64/pmman/RPMS/x86_64/. /var/ftp/pub/mirror/pmman/RPMS/x86_64/.
chown -R root.root /var/ftp/pub/mirror/pmman/RPMS/x86_64
#

9. All done

---

Earlier in this series:

Function	Link
hardening	http://www.pmman.com/usage/hardening/
lftp	http://orcorc.blogspot.com/2010/08/mirroring-upstream-master-with-lftp-to.html
RO vsftpd	http://orcorc.blogspot.com/2010/07/checklist-ro-ftp-server-setup.html
RO NFS	http://orcorc.blogspot.com/2010/08/nfs-aide-to-memory.html

NFS aide to memory

I had occasion to add NFS Read Only exports from several servers holding several generations of source RPMs to build, to be mounted on a build client. I always have to look at an existing setup, and replicate the configuration files ( /etc/exports and /etc/fstab ). Some RPM packages are needed as well

On the server side, we need to install the package: nfs-utils as a 'keystone' that pulls in other dependencies it needs


# yum install nfs-utils

Do an edit in /etc/exports


#
/path/to exported/directory 10.85.0.0/16(ro)
#

And finally enable the services, and start them:

# /sbin/chkconfig portmap on
# /sbin/chkconfig nfs on
# /sbin/service portmap start
# /sbin/service nfs start

Turning to the client side, we needs a running portmap and netfs


# yum install nfs-utils

Make any needed mountpoints:


# mkdir -p /mnt/nfs/1
# mkdir -p /mnt/nfs/2
# mkdir -p /mnt/nfs/3
# mkdir -p /mnt/nfs/4

Add needed entries in the /etc/fstab [Note: I spread the content over two lines for each entry for presentation purposes]


#
10.85.85.232:/var/ftp /mnt/nfs/1 nfs    rsize=32768,wsize=32768,soft,nolock 0 0
10.85.85.253:/var/ftp /mnt/nfs/2 nfs    rsize=32768,wsize=32768,soft,nolock 0 0
10.85.85.154:/var/ftp /mnt/nfs/3 nfs    rsize=32768,wsize=32768,soft,nolock 0 0
10.85.85.133:/var/ftp /mnt/nfs/4 nfs    rsize=32768,wsize=32768,soft,nolock 0 0
#

And finally enable the services, and start them:

# /sbin/chkconfig portmap on
# /sbin/chkconfig netfs on
# /sbin/service portmap start
# /sbin/service netfs start

Do the mounts:

# mount -a


Test using df -h and ls down in the mounts

All done

Checklist: RO FTP server setup

Setting up a new RO FTP server setup

---

The primary usage case is we describe is how to deploy a read-only FTP server with no end user accounts, to be used for distribution of content (here, to be a 'hotfix' archive for publicly accessible binary updates, accessible through yum). We need this to work around a temporarily broken update in CentOS space. We can also use it to add additioanl packages under and under the mediation of the rpm package database

We start with a hardened PMman instance. A secondary purpose of this post is to work from first principles through adding a proper local 'forked packages' archive for CentOS users to follow as a worked example. At all times, we will strive to follow proper sysadmin 'best practices' discpline under SElinux, wrappers and iptables

1. Install and enable vsftpd which is the package holding the stock ftp daemon -- yum can do this trivially

   yum install vsftpd

   Then enable the ftp server:

   /sbin/chkconfig vsftpd on

   and create a pilot file to look for in later testing:

   mkdir -p /var/ftp/pub/mirror
echo test > /var/ftp/pub/mirror/README.txt

2. Run updates, just 'because' and as a matter of good sysadmin

   yum update
yum clean all

3. Open wrappers to permit anonymous FTP connections. We edit /etc/hosts.allow and add:

   vsftpd: ALL@ALL

4. Amend the iptables rules to allow ftp. The file /etc/services reminds us that FTP normally lives at TCP port 21

   1. Add to /etc/sysconfig/iptables-config to include 'ip_conntrack_ftp' in the list of 'IPTABLES_MODULES='

      IPTABLES_MODULES="ip_conntrack_ftp "

   2. and then, in /etc/sysconfig/iptables we add a line to pass FTP content:

      -A RH-Firewall-1-INPUT -m state --state \
   NEW -m tcp -p tcp --dport 21 -j ACCEPT

      [Note: We use the backslash convention here, but iptables does not support this in its config files]

5. Run the unit through a reboot, both to 'set' the updates by stopping use of any libraries held open through that update, and also to ensure that it works as expected after a 'hands off reboot'

6. Test from a remote host that FTP works as expected

   [herrold@centos-5 ~]$ lftp 198.49.244.190
lftp 198.49.244.190:~> cd /pub/mirror
cd ok, cwd=/pub/mirror
lftp 198.49.244.190:/pub/mirror> ls
-rw-r--r-- 1 0 0 5 Jul 20 16:56 README.txt
lftp 198.49.244.190:/pub/mirror> cat README.txt
test
5 bytes transferred
lftp 198.49.244.190:/pub/mirror> exit
[herrold@centos-5 ~]$

   ... great

At this point, we have a working RO anonymous ftp server, and can populate it with content.

SELinux sanity outline

Rusty Coker mentioned in a recent blog post that he had not found a COLO facility or VM provider that enabled SELinux in its hosts by default. People regularly whine: It's too hard, and I don't need it and disable the SELinux protections. Foo

I call: Bull on the latter As to the former I sent a private email to Rusty, and offered to 'comp' him an instance to break

If anyone knows of a virtual hosting company that runs Xen or KVM virtual machines with SE Linux support then please let me know, I'll write a blog post comparing such companies if there are some.

umm -- I would be embarrased to be a hosting provider which did NOT enable SElinux

Please feel free to set up a 'comp' account at:
http://www.pmman.com/signup/
at the green arrow. Use the [please do not repeat this] 'Offer Code' of: ...

... I repeated the offer at his blog's comment site

And the question came up today in the #centos IRC channel

13:52 Andro1d> orc_orc: how can i recompile a pp from a te ?
13:53 Andro1d> checkmodule -M -m -o vsftpd.mod vsftpd.te gives a lot of errors :-/
13:53 orc_orc> ehh?
13:53 wolfy> Andro1d:
http://wiki.centos.org/HowTos/SELinux [CentOS wiki]
13:53 orc_orc> make a working dir -- say:
mkdir -p /etc/selinux/targeted/foo
and cd into it
13:54 orc_orc> Gather all the selinux noise:
audit2allow -i /var/log/audit/audit.log* -m local > local.te
13:54 Andro1d> hm, I think I'm missing some types in my .te file
13:54 orc_orc> Note the '*' in that prior line, which reads all log files present
13:54 Andro1d> mom...
13:54 orc_orc> Install the selinux-devel package for the needed Makefile
13:54 Andro1d> don't wanna make a "huge" selinux policy :)
13:54 orc_orc> Then run:
make -f /usr/share/selinux/devel/Makefile
13:55 orc_orc> and apply it:
semodule -i local.pp
13:55 orc_orc> Test again
13:55 Andro1d> yop, mompl
13:55 orc_orc> When happy, be sure to save a versioned copy, because SELinux audit file ageing will cause you to forget what was needed in that merge
13:55 orc_orc> For extra credit, amend:
/etc/audit/auditd.conf
to retain a sensible universe of back logs
13:56 orc_orc> '4' is wayyy too small


wolfy (a channel regular who offers reliable answers), pointed to the CentOS secondary source answer in the wiki; this post will also pass into our planet as yet another piece of documention and 'cheatsheet'. You saw a self-described RHCE (and he was proud of it coming into the channel today) doing that whimpering for his mommy as I read him the 'riot act'. I don't care in the least that this is new and 'hard' -- growing and learning new tools is part of the Unix culture, always has been, and always will be. That is why I try to make #centos a learning venue rather than a drive-by 'spoon-feeding' shop

How many times do we need to bang the SELinux drum to get your attention?

Yes, you lazy slogs of alleged sysadmins who simply disable SELinux, I am talking to YOU! yep - words are hard to memorize, but this is a basic 'lather, rinse and repeat' cycle which one can solve experimentally if not predictively from knowledge of what is happening. Run a tail -f /var/log/audit/audit.log if you must to see when the rule set needs to be rebuilt

But stop disabling SELinux and stop making excuses

Reading the logs, part 3 -- Run your updates

It looks like I'll be writing these for a while as I clean up logfile noise. The earlier pieces are here and here. I say 'noise' here because they are not false positives, but neither are they material, just more a nuisance

One the things every admin who reads log files sees are automated scanners looking for exploits in 'canned' packages that were installed but have not been updated, either because the admin for a given machine has neglected to run updates, because it is not a publicly known exploit, or because the upstream has not yet addressed the matter.

A pattern that has emerged with our PMman with a data center with large contiguous swaths of IP space (and hosts scattered in assignment in that relatively compact range, said hosts reporting to me centrally) is as follows. The hostile exploit scanners are not even trying to be subtle any more -- they simply march sequentially through IP ranges, and inventory if a given weakness is present on every host to which they connect

Today, I focus on one sample report stanza:

--------------------- httpd Begin ------------------------

Requests with error response codes
400 Bad Request
HTTP/1.1: 1 Time(s)
403 Forbidden
/index.html: 1 Time(s)
404 Not Found
/cms/e107_files/e107.css: 1 Time(s)
/db/e107_files/e107.css: 1 Time(s)
/e107/e107_files/e107.css: 1 Time(s)
/e107_files/e107.css: 1 Time(s)
/forum/e107_files/e107.css: 1 Time(s)
/index.php: 1 Time(s)
/manager/html: 1 Time(s)
/portal/e107_files/e107.css: 1 Time(s)
/site/e107_files/e107.css: 1 Time(s)
/web/e107_files/e107.css: 1 Time(s)

---------------------- httpd End -------------------------

and apache can handle this trivially:

#
# file: noexploit.conf
#
# send scanners off to see the wizard
#
Redirect permanent /cms http://127.0.0.1/
Redirect permanent /db http://127.0.0.1/
Redirect permanent /e107 http://127.0.0.1/
Redirect permanent /forum http://127.0.0.1/
Redirect permanent /manager http://127.0.0.1/
Redirect permanent /mysql http://127.0.0.1/
Redirect permanent /phpmyadmin http://127.0.0.1/
Redirect permanent /phpMyAdmin http://127.0.0.1/
Redirect permanent /portal http://127.0.0.1/
Redirect permanent /site http://127.0.0.1/
Redirect permanent /user http://127.0.0.1/
Redirect permanent /users http://127.0.0.1/
Redirect permanent /web http://127.0.0.1/
#

The obvious next step is to package deployment hardenings, and add them to a local RPM repository so that simply running updates, as with yum will get the current best approaches on hardening, en masse, on all the servers