10 December 2010

ripping out the safeties

There have been the endless bull session threads on the CentOS main mailing list, nominally on the subjects of SELinux and IPv6 the last couple of weeks. I am just not of a mind to tolerate cr*ppy content on mailing lists anymore. On one such list, a 'regular' whom I identify as '...' had the misfortune of being the 'designated bad example' of the day

a bull jumping the shark

On Fri, 10 Dec 2010, ... wrote:


Make sure iptables is off. Is selinux on and enforcing? If so, reverse that too (disabled).

and ... followed up with yet another ready description on how to disable the safeties

you know -- this is just a poor response all around ...

I understand that the poster produces a lot of content. I know that he is an old bull in the pen there. I also see that he races to post first, and I feel that others do not answer as a result. I see that he carries lots of URLs in all his posts so that Google's spiders crawling through the mailing list archive will widely index his business because it seems to have a lot of links

spider in a world wide web

But quantity is not quality

The response SHOULD have been to sharpen the issue if it was unclear, and describe a diagnostic flow so that one can BOTH have iptables security, and SElinux protections, AND a working system

The question, trimmed to its essence:

I get this error:
(13)Permission denied: make_sock: could not bind to address [::]:8091

(13)Permission denied: make_sock: could not bind to address 0.0.0.0:8091

no listening sockets available, shutting down

Unable to open log

So the first question to know the answer to is:

did the process start and persist?

Check for this thus:

# netstat -pant | grep 8091 

If that command, which looks for a listening socket returns some content, the process is running and its name is shown

If that command returns nothing, the process in question did not start, and we need to try to manually start it

IN NO CASE does iptables PREVENT a process from starting as it is running in the kernel and blocking parts of the network stack from transiting packets

I note in passing that it is a usual voodoo local folk wisdom in this project that one does not bind to ALL interfaces, [0.0.0.0] -- I do not know if this is the case or in play here, but note in passing that the usual practice ** here ** is to build to a specific interface, or more commonly a specific IP

Such constraints usually indicate a problem in the underlying application [ISO layer 7] not being sufficiently mature to reply to the IP that it received a service request from, and to let the routing tables manage routing at the proper ISO layer

Turning to how to amend iptables for socket based services:

If one needs to have 8091/tcp open (or 8091/udp), OPEN IT PROPERLY

Add to /etc/sysconfig/iptables the following entry:

[herrold@elided iptables]$ diff -u iptables-ORIG iptables
--- iptables-ORIG 2010-12-10 10:15:19.000000000 -0500
+++ iptables 2010-12-10 10:15:48.000000000 -0500
@@ -14,6 +14,8 @@
-A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 631 -j ACCEPT
+-A RH-Firewall-1-INPUT -p udp -m udp --dport 8091 -j ACCEPT
+-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 8091 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 37 -j
ACCEPT
[herrold@elided iptables]$

and then as root run:

# /sbin/service iptables restart 

The rules are of a common form across all Linux variants, and exhaustively documented. There is not any valid reason NOT to manage iptables rules generation and maintenance 'right'


As to SELinux, there is full logging running when the auditd and the restorecond are present. One can identify, and add rules on the fly, to progressively add 'permit rules' all SELinux based 'intercepts'

Here is a sample script of general applicability, under the GPLv3+:

[root@elided bin]# cat selinux-fixup.sh
#!/bin/sh
#
# selinux-fixup.sh
# Copyright (c) 2010 R P herrold <info@owlriver.com>
# License: GPLv3+
#
# Additively build SELinux rule sets to investigate what
# a new application needs
#
export
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
#
# make sure we have all our tools, which may not install
# in a stock CentOS 5 minimal installation
rpm -q diffutils 2> /dev/null || yum -y -q install diffutils
rpm -q audit 2> /dev/null || yum -y -q install policycoreutils
rpm -q policycoreutils 2> /dev/null || yum -y -q install policycoreutils
#
/sbin/chkconfig auditd on
/sbin/service auditd restart
/sbin/chkconfig restorecond on
/sbin/service restorecond restart
#
cd /root/bin/
#
/bin/echo "A"
/bin/touch oldlog
/usr/bin/audit2allow -i denial-log > oldlog
#
# /bin/grep ftp /var/log/audit/audit.log* > /root/bin/ftp_audit.log
/bin/grep "avc: denied" /var/log/audit/audit.log* > /root/bin/denial-log
#
# echo A
# audit2allow -a -M ftpmirror
#
/bin/echo "B"
/usr/bin/audit2allow -i denial-log -M deniallog
#
/bin/echo "C"
/usr/sbin/semodule -i deniallog.pp
#
/bin/echo "D"
/usr/bin/audit2allow -i denial-log > newlog
/usr/bin/diff -u oldlog newlog
#
/bin/echo "E"
/bin/cat deniallog.te
#
/bin/echo "F"
#

which is used iteratively, running the application, and when the candidate under test stops for 'mysterious' reasons (more about 'mysterious' later), re-running it to see if some new SELinux denial has occurred --- the 'diff' is designed to make it perfectly clear what the new denial was, and to add and apply the needed allow rule

We don't KNOW IF there was a SELinux denial yet, and if so, what the denial was yet, as the sharpening question was not asked, but to wrap matters up

To make a set of local allow rules permanent, see:

$ man 8 semanage 

for the methodology for making such permanent and persistent once the full set are known

But sometimes (here in my example case !! ) the fix NEEDS to be upstreamed so that others using FOSS gain from the fix -- let's examine that next


If the problem occurs in in a package from an upstream, one can 'file bugs' against the 'selinux' component, and that group are quite attentive to addressing such

A run of that script looks like this [and in point of fact, at one customer's premise, there ARE some fixes needed with Red Hat's DHCP client and VSFTPD when loop mounted ISOs are used, that I need to file a couple of bugs on]

[root@elided bin]# ./selinux-fixup.sh
diffutils-2.8.1-15.2.3.el5
audit-1.7.17-3.el5
policycoreutils-1.33.12-14.8.el5
Stopping auditd: [ OK ]
Starting auditd: [ OK ]
Shutting down restorecond: [ OK ]
Starting restorecond: [ OK ]
A
B
******************** IMPORTANT ***********************
To make this policy package active, execute:

semodule -i deniallog.pp

C
D
E

module deniallog 1.0;

require {
type iso9660_t;
type ftpd_t;
type iptables_t;
type initrc_t;
class unix_stream_socket { read write };
class lnk_file getattr;
class unix_dgram_socket { read write };
class dir { read getattr search };
class file { read lock getattr };
}

#============= ftpd_t ==============
allow ftpd_t iso9660_t:dir { read getattr search };
allow ftpd_t iso9660_t:file { read lock getattr };
allow ftpd_t iso9660_t:lnk_file getattr;

#============= iptables_t ==============
allow iptables_t initrc_t:unix_dgram_socket { read write };
allow iptables_t initrc_t:unix_stream_socket { read write };
F
[root@elided bin]#

which rules are saying that iptables and ftp need help:

[root@elided bin]# cat /etc/sysconfig/iptables
# Firewall configuration written by system-config-securitylevel
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -i eth1 -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p 50 -j ACCEPT
-A RH-Firewall-1-INPUT -p 51 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 21 -j
ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s a.b.c.0/24 --dport 21 -j ACCEPT
# permit the north data center
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s a.b.c.0/24 --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited COMMIT
[root@elided bin]#

pretty bog standard, but for the:

-A RH-Firewall-1-INPUT -i eth1 -j ACCEPT 

general allow rule on a backside RFC-1918 network on that interface

Note: a.b.c.0/24 is a replacement I did for privacy purposes, as the specific values do not matter

and the loop mounted ISO images:

[root@elided bin]# cat /etc/mtab
/dev/sda1 / ext3 rw 0 0
proc /proc proc rw 0 0
sysfs /sys sysfs rw 0 0
devpts /dev/pts devpts rw,gid=5,mode=620 0 0
tmpfs /dev/shm tmpfs rw 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
/var/ftp/pub/mirror/redhat/rhel/ISOS/rhel-server-6.0-i386-boot.iso
/var/ftp/pub/mirror/redhat/rhel/loop/1 iso9660
ro,loop=/dev/loop0 0 0
/var/ftp/pub/mirror/redhat/rhel/ISOS/rhel-server-6.0-i386-dvd.iso
/var/ftp/pub/mirror/redhat/rhel/loop/2 iso9660
ro,loop=/dev/loop1 0 0
/var/ftp/pub/mirror/redhat/rhel/ISOS/rhel-server-6.0-s390x-dvd.iso
/var/ftp/pub/mirror/redhat/rhel/loop/3 iso9660
ro,loop=/dev/loop2 0 0
/var/ftp/pub/mirror/redhat/rhel/ISOS/rhel-server-6.0-source-dvd1.iso
/var/ftp/pub/mirror/redhat/rhel/loop/4 iso9660
ro,loop=/dev/loop3 0 0
/var/ftp/pub/mirror/redhat/rhel/ISOS/rhel-server-6.0-source-dvd2.iso
/var/ftp/pub/mirror/redhat/rhel/loop/5 iso9660
ro,loop=/dev/loop4 0 0
/var/ftp/pub/mirror/redhat/rhel/ISOS/rhel-server-6.0-x86_64-boot.iso
/var/ftp/pub/mirror/redhat/rhel/loop/6 iso9660
ro,loop=/dev/loop5 0 0
/var/ftp/pub/mirror/redhat/rhel/ISOS/rhel-server-6.0-x86_64-dvd.iso
/var/ftp/pub/mirror/redhat/rhel/loop/7 iso9660
ro,loop=/dev/loop6 0 0
[root@elided bin]#

As a side note, I do not quite understand (it is a mystery to me) how Red Hat would have been able to test 'nightlies' for anaconda based FTP wire installs of loop mounted ISOs without that rule, but perhaps that case was not in their test coverage plan


SELinux has been around for eight years now I am told, and iptables longer (looking back to the ancestor packet filtering approaches (ipchains !!) under the 2.4 kernel and before) -- no business would run a box all 777 on permissions or on a root account with no password

This is not Dark Arts, or Black Magic, and one does not use voodoo methods of shaking a rubber chicken at such problems to solve them. Old dogs need to learn new tricks. Simply ripping out such protections is to be irresponsible. It is NOT proper sysadmin nor proper development

old dog, old tricks

I see that composing this piece took over two hours, but it is durable content and accurate on the topics of iptables and SELinux for this project. THAT is a better answer than snapping out a quick:

turn off all the safeties

reply, I submit