This crossed the puppet-users mailing list earlier today:
We have an engineering environment of around 200 CentOS servers, plus a production environment of roughly the same size. Currently, when we roll out a new server, we do a 'yum update' so the new server has the latest packages; however this means that just about every server has a different set of package versions - a system rolled out today will have different versions from one rolled out last month, and that will have different versions from one rolled out last year.
...
Has anybody else been faced with this problem, and if so, how did you resolve it?
Let's consider just the problem of 'package version skew' in operations, and come up with a solution for it. [The questioner is also 'starting' with a couple of deployment targets that vary over time because of a poorly considered 'start image' creation ... An obvious approach here is to have a couple of stable base deployment image, and a set of defined transforms to produce a basic engineering workstation or server, per to specification, and is largely uninteresting here]
- Set up a local mirror of the centos external mirrors, and call it 'incoming'
- Optionally, set a sub-mirror of 'incoming' called 'vault', and mirror in a fashion that does NOT delete old content no longer present on 'incoming'
- Set a third mirror called 'testing', which 'picks and chooses' selected packages to test, and their dependencies (see the package: yum-utils for some tools to permit confirming that one has 'closure' of those dependencies)
- Test on your pre-deployment 'bench' against 'testing' until you have a change-set you wish to deploy throughout the universe of your boxes under management. Obviously, several 'testing' mirrors can be set up, for differing classes of machines
- FINALLY, have a master distribution mirror called 'rtm' that has a change-set from a 'testing' mirror deployed to it. Remove the stock repository specification files from
and deploy local variants to taste, that point at 'rtm'. Again, several 'rtm' mirrors can be set up, for differing classes of machines/etc/yum.repos.d/
Something like this to ensure coherency of a enterprise wide deployment is usually mandated by a Change Control Board (explicitly, or implicitly). Obviously, other aspects of an IT policy document will attend to getting the various mirrors properly recoverable in one's backup strategy. [there, the 'testing' mirrors are often NOT covered, as they are ephemeral as to their usefulness, and recoverable out of 'vault' (top down) or from a 'rtm' (bottom up)]
