16 August 2010

Mirroring a upstream master with lftp to make a local yum repository

I've been assembling parts for an automated builder, over on the ia64 (Intel Itanium architecture). It has been a while since CentOS had this in the active part of the updates rotation, and I've been working on builders again, as much for relaxation as for anything else. The old binary RPMs aged away to the archive systen under the vault.centos.org hostname, and I don't want to be repeatedly hitting and loading that link for local purposes

As such I set to establishing a local mirror to run providing ftp access to that mirrored content for the local ia64 to draw from in populating a build chroot. I deployed a fresh host, gave it some drive space, and hardened it. Then I installed and turned up vsftpd. And I installed lftp which I use for mirroring remote FTP or WWW sites. It is more lightweight than rsync, and as this is anonymous mirroring, there is no security issue

Setting up the control file to drive the mirror could not be easier. Note: we control permitted shell level access via wrappers and iptables, and use 'keyed' ssh access, and a measure of ~/.ssh/config , ssh-agent , and ssh-add and well named config files, to manage complexity, so it feels like reaching out to a host away in a data center is the same as to one inside the local trusted network. This is not a transitive trust, of course, and remote machines cannot reach in

[herrold@centos-5 localcopy]$ scp root@198.178.231.209:/root/*conf .
... snip ...
[herrold@centos-5 localcopy]$ cat c55-ia64.conf
#
mirror -c -e \
http://vault.centos.org/4.4/os/ia64/ \
/var/ftp/pub/mirror/centos/4/4.4/os/ia64
#
[herrold@centos-5 localcopy]$

and running it just as easy:

[root@ia64-c44-mirror ~]# lftp -f c55-ia64.conf

I took a look to make sure it got a full measure of content:

[root@ia64-c44-mirror ~]# du /var/ftp/pub/mirror/centos/4/4.4/os/ia64
10076 /var/ftp/pub/mirror/centos/4/4.4/os/ia64/images/pxeboot
59680 /var/ftp/pub/mirror/centos/4/4.4/os/ia64/images
2048812 /var/ftp/pub/mirror/centos/4/4.4/os/ia64/CentOS/RPMS
165296 /var/ftp/pub/mirror/centos/4/4.4/os/ia64/CentOS/base
2214116 /var/ftp/pub/mirror/centos/4/4.4/os/ia64/CentOS
7952 /var/ftp/pub/mirror/centos/4/4.4/os/ia64/repodata
25080 /var/ftp/pub/mirror/centos/4/4.4/os/ia64/headers
2307452 /var/ftp/pub/mirror/centos/4/4.4/os/ia64
[root@ia64-c44-mirror ~]#

... and then configured a custom set of yum repository files to point to that archive

[root@ia64-builder hardening]# cd /etc/yum.repos.d/
[root@ia64-builder yum.repos.d]# grep -v ^# pmman-CentOS-Base.repo | grep -v ^$
[base]
name=CentOS-$releasever - Base
baseurl=ftp://ia64-c44-mirror.pmman.net/pub/mirror/centos/4/4.4/os/$basearch/
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-centos4
priority=1
protect=1
[root@ia64-builder yum.repos.d]#

The proof of the pudding is in the eating, of course, so on that ia64, let's remove and then reinstall a leaf node package (here, the joe text editor)

[root@ia64-builder ~]# yum -d 0 -e 0 clean all
[root@ia64-builder ~]# rpm -e joe
[root@ia64-builder ~]# rpm -q joe
package joe is not installed
[root@ia64-builder ~]# yum -d 0 -e 0 -y install joe
[root@ia64-builder ~]# rpm -q joe
joe-3.1-7.rhel4
[root@ia64-builder ~]#

Seems fine