11 May 2009

Rainy Days & Mondays


Karen Carpenter made the song famous for its authors, but clearly none of them were sysadmins

The rule, long known, for sysadmins is:

Never make a major change on a Friday, nor before leaving for vacation

I've been wrestling with the fallout from a violation of the sysadmin's rule by an upstream provider -- the vendor pushed in some change on Friday in the preparation of CDR -- Call Detail Records. For four days running, my sub-processes which manage the account have been failing for want of data. Those processes retrieve and apply CDR data, to emit accounting detail for customers, and have not been working

I've filed five or six sub-issue tickets which that primary change exposed, in trying to get the matter resolved: The current Firefox cannot open tickets under the current Windows XP, current SP [no problems with CentOS and FireFox or konqueror]; my 'closed' tickets were not visible; tickets were being closed by upstream before I confirmed a fix worked, so I ended up essentially re-opening the same ticket three times as each day's CDR pull failed; I was not receiving email updates of tickets; and so on. project managerI am quite sure they consider me a 'stickler for details' and something of a pedantic pest at the moment, but dammit, I'm paying their bills. The PHB supervisor may want tickets closed quickly; but I want my issues fixed first

... as no one likes to be called into work on the weekend to revert a change, the sysadmin's rule must be faithfully applied

Getting to a x86_64 build environment

In the #centos IRC channel on freenode, today, a new user was trying to clean out the 'multi-lib' artifacts in his build environment, so that it was only generating 'x86_64' results

Tom mentioned:

10:26 Zathrus> realistically, just removing glibc.i?86 should nuke everything else...

and so I fired up a victim xen instance to test that hypothesis

sudo su -
cd /etc/xen
ls
cp centos-5-x86_64-test centos-5-x86_64-victim
joe centos-5-x86_64-victim
# the edit is to rename the instance name, and the image to be used
cd /var/lib/xen
cp centos-5-x86_64-test.img centos-5-x86_64-victim.img
xm create centos-5-x86_64-victim
virt-viewer centos-5-x86_64-victim

Then inside the instance as root, I ran:

rpm -qa --qf '%{name} \t %{arch} \n' | sort > pre-remove.txt
yum remove glibc.i?86
rpm -qa --qf '%{name} \t %{arch} \n' | sort > post-remove.txt
grep -v x86 post-remove.txt | grep -v noarch

getting the result:

gpg-pubkey (none)
libaio i386
libgcc i386
python-devel i386

That's a pretty good result for a first pass, and a quick hack. I think I'll go down for some coffee, and think about it a bit morecoffee mug

05 May 2009

Revenge of the Jedi (Part II)

I wrote last week from memory about the use by the Basel II standard of a method, additively combining non-linear and correlated risk events

I see a post on the R-SIG-Fin mailing list from conference organizer Jeff Ryan, that the presentations from R/Finance 2009 are up

Page 4 on the PDF of the Klaus Rheinberger, et al. presentation nicely states the executive summary that this is 'problematic'. The work then shows a worked example

Let's call Basel II what it is -- a top down pronouncement on meaningless rules, written in a fashion that is willfully ignorant of the lessons from the US S & L 'hot money' and actuarially un-sound deposit insurance debacle 20 years ago, of LTCM as to correlated risks and 'being the market', and of the recent Credit Default Swap insurance blowup of AIG

01 May 2009

May Day celebration

Renegade People's Movement -- our leader

I see a tweet from KB about the monthly mailman mailing list reminder emails; I took steps long ago to use procmail to watch for these, and re-mark their subject line. I then sort my mailspool by subject in alpine, and delete this noise all in one pass.

# mailing list memberships reminder
#
:0 fw
* ^Subject: \/.*mailing list memberships reminder
*!^X-Reminder
  | formail -i "Subject: mlmr] $MATCH"     \
    -A "X-Reminder:$MATCH"    \
    -A "X-Munge: moved mailing list memberships reminder"

All power to the people

27 April 2009

Revenge of the Jedi

"The pen is mightier than the sword"
I posted a bit earlier today about the forgotten religion of Monetarism in the context of my weekend at a conference in Chicago. I had not heard mention of the faith, nor seen anyone but myself doing analysis using the old tools for a long, long time

I blogged a bit back about Jim Chanos' critique on CNBC of the new Mark to Market 'requirements' and the artificiality of the Basel II reserve requirement target value, Chanos suggesting a relaxation to a transition value of say 1.5 percent to 'conform' to Basel II in the short term. Two weeks ago, I had mentioned 'A Monetary History of the United States' (Friedman, Schwartz) to a friend wanting to understand how we got where we are; yesterday evening, I was discussing the Nixonian repudiation of Bretton Woods and the need to revisit Basel II, as I saw a clear demonstration that Basel II is defective at the conference. I do not have my notes at hand, the slide decks are not up yet, but I believe it was in the mixed currency risk analysis (Austrian, Swiss, and back to Central Europe as to residential property loans), which I believe Rheinberger gave entitled: 'VEC and GVAR Models using R' which exposed quite clearly that the 'experts' are using simple additive risk summing in Basel II, seemingly oblivious of the concept of the non-linear nature of correlated risks

The afternoon's email brings a report that Anna Schwartz is still out there as well

The old craft will live so long as a single practitioner remembers them

"Hokey religions and ancient weapons are no match for a good blaster at your side, kid."
The good fight continues; I'll keep swinging with the tools I know, thanks


p.s.: I do know the regular titles for the third and sixth released Star Wars films

'R' you experienced?

" ... To confer, converse, and otherwise hob-nob with my brother wizards ..."

I spent a productive weekend up in Chicago, at the R/Finance 2009: Applied Finance with R conference, which billed itself as the "first annual R/Finance conference for applied finance using R". The conference organizers and hosts are the 'usual suspects' on the 'R sig fin' mailing list; Jeffrey Ryan, Dirk Eddelbuettel, Dale Rosenthal, Brian Peterson, Peter Carl, Gib Bassett, and John Miller, assisted by the talented and imperturbable Holly Griffin of UIC. Pretty clearly most of this group code together regularly; see the committer list on the blotter module. The venue was at the 'other school' in Chicago, the one with a more practical interest in Economics and Finance

An aside about Chicago: Long ago, and far away, I was trained as a acolyte 'monetarist' by disciples of Herb Stein's CEA and the Fed, in the [University of] Chicago school [a fad, seemingly long forgotten by recent Economics and Finance grads, so far as I can tell]. Monetarism is a forgotten religion these days; the Fed stopped formally publishing its M3 series a few years ago, in light of the rise of what Bill Gross calls the 'shadow banking' system.

Luke: [The robot] claims to be the property of an Obi-Wan Kenobi. Is he a relative of yours? Do you know what he's talking about?
Obi-Wan: Obi-Wan Kenobi. Obi-Wan... Now, that's a name I've not heard in a long time. A long time.
Luke: I think my uncle knows him. He said he was dead.
Obi-Wan: Oh, he's not dead... Not yet.
Luke: You know him?
Obi-Wan: But of course I know him

Many of the organizers were known to me from my email correspondence or from observing their packages, and I had spoken with one (Dirk) two or three years ago briefly after the trading shim was first usable

Dirk seems to be a bundle of unbounded energy. His tools had solved a lot of data storage and visualization issues for me early on in our project. He led a push a couple years back to drill in many of the R add-on modules into the main Debian archives. I still hope to emulate his example in rpm space using R2spec and some post-processing scripts (Dependency enumeration is not quite perfect yet). Dirk has already mentioned in his blog the 'after-sessions' at Jak's; we closed the place down Saturday with useful brainstorming happening long into the night

I had resolved to travel to the conference to learn, and to stay quiet as to matters of FOSS and advocacy. I was almost even able to keep to that intent, save at the interstitial times. The formal presentations were amazing in their quality of content, competence of the presenters, and challenging to my old knowledge of Statistics and Mathematics. I even understood most of what the presenters were doing, and why, on the formal finance side and will re-read the slide decks with great interest when they appear to fill in the holes. One would have probably had to be there to draw much more from the decks, as the presenters were not doing the occasional 'stand and read' presentation one finds at some conferences, but rather largely used their decks as reminders of the points they wanted to hit and elaborate on in their presentation, and to state exactly the code and formulae in play. The committee have really set a high bar to reach for next year's event to top, and I look forward to it already

The pre-conference tutorials were worthwhile. I knew Jeff Ryan's work from xts and IBrokers of course, and gained insight into his mental roadmap on where the code is going and how it will get there. I think the enhancements he is trialling in xts will pretty clearly flow back upstream into zoo in general form; I had not heard of Dale before, but his breakout and presentation of an analytic approach on addition and testing of single constraints (I have covered scientific method and epistemology here before, and will again) served as a fine warm-up to the formal sessions

During the session breaks, at meals, and into the night, I had a chance for give and take at length with several of the committee, presenters, and attendees, to bridge what Patrick Burns spoke on -- the chasm between Practice and Theory

Part of the trip and my need for listen, was to get a handle on how to match the shim and R as a pair of heavy-weight co-processes, so that the user of the shim can hook in and use the wonderful tools already in R space. We'll most likely get there, but the timing is not clear. Having said 'we', permit me to make it clear that the heavy lifting will be done, if and when done, by Bill, and not me. At the prodding of Peter, who as I understand it regularly team codes with Brian, I have started the sign-up process for an account at r-forge, and will 'cut my teeth' on a simple connector or module, to warm up my skills as a co-development tester and 'guinea pig' consumer of the major task of integrating the shim

FIX rules the roost for being the 'lingua Franca' for interchange to exchange order, position and fill data with counter-party upstream brokers or exchanges (thanks here to the CME Foundation for partially funding the event). We will not soon be adding a compressed FIX connector to the shim, and certainly not before we attain our major milestone of a formal 'complete' first release.

Finally, a couple folks asked why we were playing down in retail space with the TWS and its vendor specific API. For a researcher, and for a small proprietary trader, we still find IB's API and services the most affordable, and substantially complete. It is a gateway to enable any interested researcher to do material research (the 'Theory') and strategy development and execution (the 'Practice'). For student academics, the availability of IB's 'trading Olympiad' program and the shim, and R offer all one needs for a better than free price


Update: We see also the summary of the event at Revolution Computing, an R vendor

16 April 2009

Afraid of experimentation

The #centos IRC channel at irc.freenode.net never ceases to amaze me. We get questions that would take at least 30 seconds of reading a man page and experimention to answer, asked over and over again.

Here is one of the latest:

16:14 clueless> I installed a Windows Vista Business x64 VM on 5.2. Is it possible to get hibernate/sleep to work?
16:15 clueless> I want only want the VM up occasionally and I'd rather not wait for a full boot every time

Firing up a xen virtual machine in a root panel, and popping open another to read the xm man page, I find:

# cd /etc/xen
# ls
# xm create win-2000pro
Using config file "./win-2000pro".
Started domain win-2000pro
# virt-viewer win-2000pro

and a Windows 2000 session appears. I let it boot to the login prompt, and then:

# cd /var/lib/xen
# xm list
# xm pause win-2000pro
# xm save win-2000pro win-2000pro-save.img

Which of course as the man page promises, terminates the running image. Then:

# xm restore win-2000pro-save.img
# xm list
# xm unpause win-2000pro
# virt-viewer win-2000pro

And we are right were we left off, at the initial log in prompt.

Is it so hard to at least pretend to look first?