Part of my daily routine is to check the logwatch summary, note and address any security matters, and then to chip away at the friction and non-working parts of the compute environments in which I can effect change
This one has been on my radar for a while, but it is on a protected interior machine, not disabling, and so not critical. From a configuration file review, and with reading of the sendmail and openssl documentation, and some 'googleing' I just could not see where the error was.
**Unmatched Entries**
STARTTLS=client, error:
SSL_CTX_use_certificate_file(/etc/mail/certs/xps400.first.owlriver.net-10.pem) failed: 173 Time(s)
STARTTLS=client, error: SSL_CTX_check_private_key
failed(/etc/mail/certs/xps400.first.owlriver.net-10.key): 0: 173 Time(s)To test if sendmail is compiled with STARTTLS support, we can run the following command:
$ sendmail -bt -d0.8 < /dev/null
Compiled with: DNSMAP HESIOD HES_GETMAILHOST LDAPMAP LOG MAP_REGEX
MATCHGECOS MILTER MIME7TO8 MIME8TO7 NAMED_BIND NETINET NETINET6
NETUNIX NEWDB NIS PIPELINING SASLv2 SCANF STARTTLS TCPWRAPPERS
USERDB USE_LDAP_INITmore concisely:
[root@xps400 certs]# sendmail -bt -d0.8 < /dev/null | grep -i tls
NETUNIX NEWDB NIS PIPELINING SASLv2 SCANF SOCKETMAP STARTTLSAnd, yet when we connect to the mailserver to test if STARTSSL is advertised
[herrold@centos-5 ~]$ telnet xps400 25
Trying 10.16.1.112...
Connected to xps400.first.lan (10.16.1.112).
Escape character is '^]'.
220 xps400.first.owlriver.net ESMTP Sendmail 8.14.3/8.14.3; Thu, 8 Apr 2010 14:41:48 -0400
EHLO localhost
250-xps400.first.owlriver.net Hello centos-5.first.lan [10.16.1.101], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE
250-DSN
250-ETRN
250-AUTH GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN PLAIN
250-DELIVERBY
250 HELP
quit
221 2.0.0 xps400.first.owlriver.net closing connection
Connection closed by foreign host.
[herrold@centos-5 ~]$ openssl has the rather interesting sub-tool s_client 'SSL/TLS client program' which knows how to talk several protocols though a transition into a secure sockets mode as well
[root@xps400 ~]# openssl s_client -connect localhost:25 -starttls smtp
CONNECTED(00000003)
didn't found starttls in server response, try anyway...
2005:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:583:
[root@xps400 ~]#so ... starttls is linked in sendmail as present, but is not working
Let's run this down by making sure all the needed moving parts are present:
[root@xps400 mail]# cd /etc/init.d/
[root@xps400 init.d]# ls *sasl*
saslauthd
[root@xps400 init.d]# chkconfig --list saslauthd
saslauthd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
[root@xps400 mail]# /sbin/service saslauthd restart
Stopping saslauthd: [ OK ]
Starting saslauthd: [ OK ]
[root@xps400 mail]# /sbin/chkconfig saslauthd on
[root@xps400 mail]# /sbin/service sendmail restartand from another panel watching the log files:
# tail -f /var/log/maillog
Apr 8 11:39:30 xps400 sendmail[3536]: STARTTLS=server, error: SSL_CTX_use_certificate_file(/etc/mail/certs/xps400.first.owlriver.net-10.pem) failed
Apr 8 11:39:30 xps400 sm-msp-queue[3547]: starting daemon (8.14.3): queueing@01:00:00.. so ... sendmail is telling us that it refuses to use: /etc/mail/certs/xps400.first.owlriver.net-10.pem Looking at the certificate countersign:
# less /etc/mail/certs/xps400.first.owlriver.net-10.pem
N CERTIFICATE-----
MIIHATCCBemgAwIBAgICFokwDQYJKoZIhvcNAQEFBQAwgYwxCzAJBgNVBAYTAklM
MRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRh
... 
a HA !!
Looks like it was a bad 'scrape and paste' by me when I retrieved and installed the counter-signing of the certificate from startssl. The start of that file should look like:
-----BEGIN CERTIFICATE-----
MIIHATCCBemgAwIBAgICFokwDQYJKoZIhvcNAQEFBQAwgYwxCzAJBgNVBAYTAklM
MRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRh
bCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAy
... As always, such mistakes are only obvious once found.
A side observation. A recent blog bost "Securing the Enterprise" by Eddy Nigg of Startssl pointed out the willingness of some Certificate Authorities to sign whatever they are offered, and some admins to not consider this attack vector in submitting such, in the case of hosts in an RFC-1918 IP assignment block or non-DNS mediated formal namespace. Undetected forgeries are facilitated, and I am certain such Man in the Middle compromises occur in coffee-shops with wireless access all the time. As such the host: xps400.first.lan at 10.16.1.112 also appears with a internal split DNS PTR and A record as: xps400.first.owlriver.net The certificate for it countersigned by startssl is useful here for verifications
[herrold@centos-5 ~]$ host xps400.first.owlriver.net
xps400.first.owlriver.net has address 10.16.1.112
xps400.first.owlriver.net mail is handled by 20 mailhub.owlriver.net.
xps400.first.owlriver.net mail is handled by 10 new.owlriver.com.
[herrold@centos-5 ~]$ The trick to getting the mailserver to answer as xps400.first.owlriver.net was even easier -- just some DNS work, and a quick addition of a non-customary line in the /etc/mail/sendmail.mc, a rebuild, and a restart:
[herrold@xps400 mail]$ grep xps400.first.owlriver.net sendmail.mc | head -1
define(`confDOMAIN_NAME', `xps400.first.owlriver.net')dnl
[herrold@xps400 mail]$ 
Are you using SSL certificates where you can and should? ... Is the namespace of network they protect thoughtfully designed? StartSSL makes it easy to do, for a person willing to be minimally 'validated' as to their identity and their right to administer a given domain. Once that identity check is done, the process is essentially free of any marginal cost to roll out as many certificates as one wishes, and to NOT 'cop out' or cut corners here
