SELinux sanity outline

Rusty Coker mentioned in a recent blog post that he had not found a COLO facility or VM provider that enabled SELinux in its hosts by default. People regularly whine: It's too hard, and I don't need it and disable the SELinux protections. Foo

I call: Bull on the latter As to the former I sent a private email to Rusty, and offered to 'comp' him an instance to break

If anyone knows of a virtual hosting company that runs Xen or KVM virtual machines with SE Linux support then please let me know, I'll write a blog post comparing such companies if there are some.

umm -- I would be embarrased to be a hosting provider which did NOT enable SElinux

Please feel free to set up a 'comp' account at:
http://www.pmman.com/signup/
at the green arrow. Use the [please do not repeat this] 'Offer Code' of: ...

... I repeated the offer at his blog's comment site

And the question came up today in the #centos IRC channel

13:52 Andro1d> orc_orc: how can i recompile a pp from a te ?
13:53 Andro1d> checkmodule -M -m -o vsftpd.mod vsftpd.te gives a lot of errors :-/
13:53 orc_orc> ehh?
13:53 wolfy> Andro1d:
http://wiki.centos.org/HowTos/SELinux [CentOS wiki]
13:53 orc_orc> make a working dir -- say:
mkdir -p /etc/selinux/targeted/foo
and cd into it
13:54 orc_orc> Gather all the selinux noise:
audit2allow -i /var/log/audit/audit.log* -m local > local.te
13:54 Andro1d> hm, I think I'm missing some types in my .te file
13:54 orc_orc> Note the '*' in that prior line, which reads all log files present
13:54 Andro1d> mom...
13:54 orc_orc> Install the selinux-devel package for the needed Makefile
13:54 Andro1d> don't wanna make a "huge" selinux policy :)
13:54 orc_orc> Then run:
make -f /usr/share/selinux/devel/Makefile
13:55 orc_orc> and apply it:
semodule -i local.pp
13:55 orc_orc> Test again
13:55 Andro1d> yop, mompl
13:55 orc_orc> When happy, be sure to save a versioned copy, because SELinux audit file ageing will cause you to forget what was needed in that merge
13:55 orc_orc> For extra credit, amend:
/etc/audit/auditd.conf
to retain a sensible universe of back logs
13:56 orc_orc> '4' is wayyy too small


wolfy (a channel regular who offers reliable answers), pointed to the CentOS secondary source answer in the wiki; this post will also pass into our planet as yet another piece of documention and 'cheatsheet'. You saw a self-described RHCE (and he was proud of it coming into the channel today) doing that whimpering for his mommy as I read him the 'riot act'. I don't care in the least that this is new and 'hard' -- growing and learning new tools is part of the Unix culture, always has been, and always will be. That is why I try to make #centos a learning venue rather than a drive-by 'spoon-feeding' shop

How many times do we need to bang the SELinux drum to get your attention?

Yes, you lazy slogs of alleged sysadmins who simply disable SELinux, I am talking to YOU! yep - words are hard to memorize, but this is a basic 'lather, rinse and repeat' cycle which one can solve experimentally if not predictively from knowledge of what is happening. Run a tail -f /var/log/audit/audit.log if you must to see when the rule set needs to be rebuilt

But stop disabling SELinux and stop making excuses

lost memories

From time to time, "we" 'clean house' and find the black trash bags. It is no surprise to me, of course for and earlier me have done to work of carefully tied packaging closed, and cacheing treasures up in the attic; from time to time, I am instructed to 'get rid of that clutter' as the now grown kids 'will never use those again' I am slow to act on this injunction

The Brio trains, the metal Erector set, the cast lead soldiers and molds, the Duplo blocks, the stuffed animals, Lincoln logs, the McGuffey readers, the arrow and ax heads collected in the fields, have all fallen to head of the queue for disposition over time. Stuffed animals were in the dock this past weekend. At that point, I usually nod silently, carefully re-tie the sack, and set it to one side for a moment. Then my new task is to find a new hiding place for the bag in question after her attention turns to other matters

But a grandchild's mother and the child were delighted with the animal figures from my preservation efforts, even if the spouse was not as well pleased to see 'those old things' again

A few weeks ago, the Brio train set that was set aside in a cardboard box, up in the dark to rest almost two decades ago came out. It moved in with a grandson infatuated with rolling stock and was 'new' again; The Erector set, the melting pot and molds, all gone (not to return with current day safety rules — choking hazard of the nuts and bolts, heavy metal fumes). I am on the lookout for a replacement McGuffey reader set (that friend of books that taught me to read upstairs in a quiet room as the adults 'talked' downstairs), so I can 'seed' a room for young visitors

The flints and shaped stones? I was not attuned to their disposition occurring; a 'sharpie' sweet-talked a sale for a pittance from a elderly family member when 'cleaning up' prior to closing down a house before sale. That lot of childhood treasures also carried out the door the minnie balls I dug from the earth at Gettysburg

Entropy won a round that time; I know we'll battle again.

---

[An earlier version of this appeared at Victor Niederhoffer's Daily Speculations, which aggregator I recommend]

Debian mkfs is working again

It's been a long June. I noticed early on that an update in Debian testing had moved mke2fs from one package to another without getting all the library dependencies right. As such I spent June without the ability to lay down a filesystem on a new partition with the 'proper' tool. Part of my series on logfile reading includes a task to review the 'percent full' for each partition (and to relocate or clean out fat ones) to avoid running out of room in a self-inficted denial of services attack

I tried the obvious fallback to build a new filesystem: busybox but the version found in Debian Testing was lacking a needed build time switch. I filed the bug, and considered a local patch, or perhaps whether to rebuild of part of the chain needed to fork mkfs for a bit, but my need for space to reorganize a host's files was not that great nor urgent. Just pesky each day to see

I knew from reading the bug reports that the fix had been committed and 'ageing' in the Debian fashion to its move from an Unstable 'nightly' to a mildly tested (or at least not black-balled) state and promotion into Testing

nfs2:~# apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages have been kept back:
ksysguard libdevmapper1.02.1
The following packages will be upgraded:
bsdutils e2fslibs e2fsprogs iptables iso-codes libblkid1 libcomerr2
libenchant1c2a libffcall1 libmime-tools-perl libnetpbm10 libss2 libuuid1
lockfile-progs mount mutt netpbm shared-desktop-ontologies util-linux
19 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
Need to get 9,841kB of archives.
After this operation, 115kB disk space will be freed.
Do you want to continue [Y/n]? y
...
nfs2:~#

I've been running repository data update operations daily .. the Debian approach is more measured in its pace than we use with CentOS, and I think we may have something to learn there. It is a rare package update that cannot wait for a daily repo data update, push and mirror overnight in our space, and it would avoid much confusion to casual sysadmins

Those bolded packages in that clutch of upgrades looks promising ...

nfs2:~# mkfs /dev/sda12
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
237568 inodes, 949835 blocks
47491 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=973078528
29 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736

Writing inode tables: done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 28 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
nfs2:~# date
Thu Jun 24 10:13:17 EDT 2010
nfs2:~#

Lovely; I'm back in business

Reading the logs, part 3 -- Run your updates

It looks like I'll be writing these for a while as I clean up logfile noise. The earlier pieces are here and here. I say 'noise' here because they are not false positives, but neither are they material, just more a nuisance

One the things every admin who reads log files sees are automated scanners looking for exploits in 'canned' packages that were installed but have not been updated, either because the admin for a given machine has neglected to run updates, because it is not a publicly known exploit, or because the upstream has not yet addressed the matter.

A pattern that has emerged with our PMman with a data center with large contiguous swaths of IP space (and hosts scattered in assignment in that relatively compact range, said hosts reporting to me centrally) is as follows. The hostile exploit scanners are not even trying to be subtle any more -- they simply march sequentially through IP ranges, and inventory if a given weakness is present on every host to which they connect

Today, I focus on one sample report stanza:

--------------------- httpd Begin ------------------------

Requests with error response codes
400 Bad Request
HTTP/1.1: 1 Time(s)
403 Forbidden
/index.html: 1 Time(s)
404 Not Found
/cms/e107_files/e107.css: 1 Time(s)
/db/e107_files/e107.css: 1 Time(s)
/e107/e107_files/e107.css: 1 Time(s)
/e107_files/e107.css: 1 Time(s)
/forum/e107_files/e107.css: 1 Time(s)
/index.php: 1 Time(s)
/manager/html: 1 Time(s)
/portal/e107_files/e107.css: 1 Time(s)
/site/e107_files/e107.css: 1 Time(s)
/web/e107_files/e107.css: 1 Time(s)

---------------------- httpd End -------------------------

and apache can handle this trivially:

#
# file: noexploit.conf
#
# send scanners off to see the wizard
#
Redirect permanent /cms http://127.0.0.1/
Redirect permanent /db http://127.0.0.1/
Redirect permanent /e107 http://127.0.0.1/
Redirect permanent /forum http://127.0.0.1/
Redirect permanent /manager http://127.0.0.1/
Redirect permanent /mysql http://127.0.0.1/
Redirect permanent /phpmyadmin http://127.0.0.1/
Redirect permanent /phpMyAdmin http://127.0.0.1/
Redirect permanent /portal http://127.0.0.1/
Redirect permanent /site http://127.0.0.1/
Redirect permanent /user http://127.0.0.1/
Redirect permanent /users http://127.0.0.1/
Redirect permanent /web http://127.0.0.1/
#

The obvious next step is to package deployment hardenings, and add them to a local RPM repository so that simply running updates, as with yum will get the current best approaches on hardening, en masse, on all the servers

Reading the logs ...

I see the following from logwatch in the overnight log file review:

 --------------------- httpd Begin ------------------------

 Requests with error response codes
    404 Not Found
       /crossdomain.xml: 1 Time(s)

 ---------------------- httpd End -------------------------

and so I go digging:

[root@centos-5 httpd]# cat error_log
[Sun Jun 06 04:02:04 2010] [notice] Digest: generating secret for digest authentication ...
[Sun Jun 06 04:02:04 2010] [notice] Digest: done
[Sun Jun 06 04:02:05 2010] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Mon Jun 07 14:20:39 2010] [error] [client 127.0.0.2] File does not exist: /var/www/html/crossdomain.xml

Sure enough. It looks as though some piece of Flash code is hoping to 'leverage' a cross-domain permission to include something I may not have intentionally intended to allow.

See the note at: http://kb2.adobe.com/cps/142/tn_14213.html

For the sake of argument, assume you HAD to web view as root, as say with an operating system that required you use a browser front end to access system updates. Assume also that you improvidently viewed a 'seeder' of bad things that WROTE a hostile crossdomain.xml for later use by a second piece of hostile Flash to 'reap'

Oops ... game over

Running down stray errors

Part of my daily routine is to check the logwatch summary, note and address any security matters, and then to chip away at the friction and non-working parts of the compute environments in which I can effect change

This one has been on my radar for a while, but it is on a protected interior machine, not disabling, and so not critical. From a configuration file review, and with reading of the sendmail and openssl documentation, and some 'googleing' I just could not see where the error was.

**Unmatched Entries**
STARTTLS=client, error:
SSL_CTX_use_certificate_file(/etc/mail/certs/xps400.first.owlriver.net-10.pem) failed: 173 Time(s)
STARTTLS=client, error: SSL_CTX_check_private_key
failed(/etc/mail/certs/xps400.first.owlriver.net-10.key): 0: 173 Time(s)

To test if sendmail is compiled with STARTTLS support, we can run the following command:


$ sendmail -bt -d0.8 < /dev/null

Compiled with: DNSMAP HESIOD HES_GETMAILHOST LDAPMAP LOG MAP_REGEX
MATCHGECOS MILTER MIME7TO8 MIME8TO7 NAMED_BIND NETINET NETINET6
NETUNIX NEWDB NIS PIPELINING SASLv2 SCANF STARTTLS TCPWRAPPERS
USERDB USE_LDAP_INIT

more concisely:


[root@xps400 certs]# sendmail -bt -d0.8 < /dev/null | grep -i tls
NETUNIX NEWDB NIS PIPELINING SASLv2 SCANF SOCKETMAP STARTTLS

And, yet when we connect to the mailserver to test if STARTSSL is advertised

[herrold@centos-5 ~]$ telnet xps400 25
Trying 10.16.1.112...
Connected to xps400.first.lan (10.16.1.112).
Escape character is '^]'.
220 xps400.first.owlriver.net ESMTP Sendmail 8.14.3/8.14.3; Thu, 8 Apr 2010 14:41:48 -0400
EHLO localhost
250-xps400.first.owlriver.net Hello centos-5.first.lan [10.16.1.101], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE
250-DSN
250-ETRN
250-AUTH GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN PLAIN
250-DELIVERBY
250 HELP
quit
221 2.0.0 xps400.first.owlriver.net closing connection
Connection closed by foreign host.
[herrold@centos-5 ~]$

openssl has the rather interesting sub-tool s_client 'SSL/TLS client program' which knows how to talk several protocols though a transition into a secure sockets mode as well

[root@xps400 ~]# openssl s_client -connect localhost:25 -starttls smtp
CONNECTED(00000003)
didn't found starttls in server response, try anyway...
2005:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:583:
[root@xps400 ~]#

so ... starttls is linked in sendmail as present, but is not working

Let's run this down by making sure all the needed moving parts are present:

[root@xps400 mail]# cd /etc/init.d/
[root@xps400 init.d]# ls *sasl*
saslauthd
[root@xps400 init.d]# chkconfig --list saslauthd
saslauthd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
[root@xps400 mail]# /sbin/service saslauthd restart
Stopping saslauthd: [ OK ]
Starting saslauthd: [ OK ]
[root@xps400 mail]# /sbin/chkconfig saslauthd on
[root@xps400 mail]# /sbin/service sendmail restart

and from another panel watching the log files:

# tail -f /var/log/maillog
Apr 8 11:39:30 xps400 sendmail[3536]: STARTTLS=server, error: SSL_CTX_use_certificate_file(/etc/mail/certs/xps400.first.owlriver.net-10.pem) failed
Apr 8 11:39:30 xps400 sm-msp-queue[3547]: starting daemon (8.14.3): queueing@01:00:00

.. so ... sendmail is telling us that it refuses to use: /etc/mail/certs/xps400.first.owlriver.net-10.pem Looking at the certificate countersign:

# less /etc/mail/certs/xps400.first.owlriver.net-10.pem
N CERTIFICATE-----
MIIHATCCBemgAwIBAgICFokwDQYJKoZIhvcNAQEFBQAwgYwxCzAJBgNVBAYTAklM
MRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRh
...

a HA !!

Looks like it was a bad 'scrape and paste' by me when I retrieved and installed the counter-signing of the certificate from startssl. The start of that file should look like:


-----BEGIN CERTIFICATE-----
MIIHATCCBemgAwIBAgICFokwDQYJKoZIhvcNAQEFBQAwgYwxCzAJBgNVBAYTAklM
MRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRh
bCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAy
...

As always, such mistakes are only obvious once found.

---

A side observation. A recent blog bost "Securing the Enterprise" by Eddy Nigg of Startssl pointed out the willingness of some Certificate Authorities to sign whatever they are offered, and some admins to not consider this attack vector in submitting such, in the case of hosts in an RFC-1918 IP assignment block or non-DNS mediated formal namespace. Undetected forgeries are facilitated, and I am certain such Man in the Middle compromises occur in coffee-shops with wireless access all the time. As such the host: xps400.first.lan at 10.16.1.112 also appears with a internal split DNS PTR and A record as: xps400.first.owlriver.net The certificate for it countersigned by startssl is useful here for verifications

[herrold@centos-5 ~]$ host xps400.first.owlriver.net
xps400.first.owlriver.net has address 10.16.1.112
xps400.first.owlriver.net mail is handled by 20 mailhub.owlriver.net.
xps400.first.owlriver.net mail is handled by 10 new.owlriver.com.
[herrold@centos-5 ~]$

The trick to getting the mailserver to answer as xps400.first.owlriver.net was even easier -- just some DNS work, and a quick addition of a non-customary line in the /etc/mail/sendmail.mc, a rebuild, and a restart:

[herrold@xps400 mail]$ grep xps400.first.owlriver.net sendmail.mc | head -1
define(`confDOMAIN_NAME', `xps400.first.owlriver.net')dnl
[herrold@xps400 mail]$

Are you using SSL certificates where you can and should? ... Is the namespace of network they protect thoughtfully designed? StartSSL makes it easy to do, for a person willing to be minimally 'validated' as to their identity and their right to administer a given domain. Once that identity check is done, the process is essentially free of any marginal cost to roll out as many certificates as one wishes, and to NOT 'cop out' or cut corners here

Caller ID, wiretapping, call recording, and the federal Do Not Call list

There is a witches brew of rules that people making outbound telephone calls need to thread through. Also, the recipient of a call needs to observe some as well. Let's start in reverse order:

Particularly, in the US, some states require consent from only ONE party to a telephone communication; others require TWO [or ALL, in the case of a conference call] participants to so consent. The asserted misconduct case of Linda Tripp in Maryland comes to mind. Linda got into some hot water for chatting up Monica's lovelife with some girltalk about Bill Clinton and recording it without needed consents from "that woman, Ms. Lewinsky" and then turning those recordings over to Kenneth Starr's office

Neither side of the aisle is without stain in this space, it seems; recall that back earlier in the Clinton administration that a couple in Florida recorded a conference call bridge leg, on which the cell phone conversation of Representative John Boehner (R-Ohio), was connected. They later pled out to a criminal charge concerning this. That call (said to have been intercepted within the state of Florida through a common radio scanner) also included then-Speaker of the House Newt Gingrich and other House Republican leadership folks. The tape turned up, inter alia, into the possession of Representative James McDermott (D-Wash.), who then flipped the tape to The New York Times and the Atlanta Journal-Constitution. This drew a lawsuit from Boehner against McDermott, seeking to impose to civil liability for violation of the federal [anti-]wiretap law, alleging that no effective consent existed

Stock brokers commonly record ALL calls, and I assume have paperwork in place at account opening time, that effectively and irrevocably obtain consent to such monitoring and recordation, and as I think it through, must contain some sort of representation and warranty by the customer that all parties connected from their side of the call brought in have also consented. Clearly, sometimes this turns out NOT to be the case, and yet I do not recall seeing any litigation as to improper recording of a conference bridge. Curious

And then there is the federal Do Not Call list -- seemingly a shield for the consumer to ward off unwanted solicitation calls from unknown third parties. All the phone numbers under my control have been registered with the enforcing agency, the FTC, and should be showing up on the database tapes for telephone solicitors to elide. This does not happen of course -- sadly, anonymous VOIP calls, false and forged Caller ID information, and simple omission of caller ID data prevails; the ways to dodge the requirement are well know to telemarketers, it seems

But I have been working in the caller ID adjunct industry -- if you need real time screen pop information of inbound callers, I have been a rep for TelComp -- for longer than I care to remember. Be sure to mention that Russ sent you if you call Larry directly, or contact me for a system design and suggested implementation

I was on the phone with Larry earlier today. We have provided the web and email presence since the start. The domain registration says 1995, but I know we did a trade show in LA before that with a web presence up. I was doing a bit of debugging on SMTP AUTH issues with him. Commonly we will leave an open line when we do this, and I listened to him field calls for an hour or so. Larry is endlessly patient on support calls, and I hope to be as patient when I am doing support. ;)

The call had discussed industry trends and practices, and in part the topics of this blog post were fresh in my mind, for we 'talked shop' during running down his email issue

The next call, not two minutes later, went like this:

Phone rings, and the caller ID has no name information, is from a number not known in a lookup to my real time 'whitelist' database, and is from out of the local area code --- a potential outbound solicitation call

Me: Good afternoon. May I help you?

Other party identifies himself as calling from "Merchant Services" and asks for 'the decision maker' at my business.

Me: That's me, all right; we have a practice and policy of recording all calls for quality and training purposes. May I have your consent to such recording, please?

Other party: (confused) uhh -- OK, I guess

Me: Great, and thank you. How may I help you?

Other party: Well, I am calling about your merchant services account. I was calling to make sure you were getting the best rate ...

Me: (interrupting) Sure -- thanks. What is your firm's name and address please?

Other party: ummm

Me: (interrupting) ... you see, I need that because this is a residential number that is on the Do Not Call list, and I need that information to send the lawsuit papers to ...

Other party: (click)

Much more satisfying that simply silently hanging up at my end. Feel free to "clip and save" this handy outline. A copy to crib from at each phone just may come in handy