'R' you experienced?

" ... To confer, converse, and otherwise hob-nob with my brother wizards ..."

I spent a productive weekend up in Chicago, at the R/Finance 2009: Applied Finance with R conference, which billed itself as the "first annual R/Finance conference for applied finance using R". The conference organizers and hosts are the 'usual suspects' on the 'R sig fin' mailing list; Jeffrey Ryan, Dirk Eddelbuettel, Dale Rosenthal, Brian Peterson, Peter Carl, Gib Bassett, and John Miller, assisted by the talented and imperturbable Holly Griffin of UIC. Pretty clearly most of this group code together regularly; see the committer list on the blotter module. The venue was at the 'other school' in Chicago, the one with a more practical interest in Economics and Finance

An aside about Chicago: Long ago, and far away, I was trained as a acolyte 'monetarist' by disciples of Herb Stein's CEA and the Fed, in the [University of] Chicago school [a fad, seemingly long forgotten by recent Economics and Finance grads, so far as I can tell]. Monetarism is a forgotten religion these days; the Fed stopped formally publishing its M3 series a few years ago, in light of the rise of what Bill Gross calls the 'shadow banking' system.

Luke: [The robot] claims to be the property of an Obi-Wan Kenobi. Is he a relative of yours? Do you know what he's talking about?
Obi-Wan: Obi-Wan Kenobi. Obi-Wan... Now, that's a name I've not heard in a long time. A long time.
Luke: I think my uncle knows him. He said he was dead.
Obi-Wan: Oh, he's not dead... Not yet.
Luke: You know him?
Obi-Wan: But of course I know him

Many of the organizers were known to me from my email correspondence or from observing their packages, and I had spoken with one (Dirk) two or three years ago briefly after the trading shim was first usable

Dirk seems to be a bundle of unbounded energy. His tools had solved a lot of data storage and visualization issues for me early on in our project. He led a push a couple years back to drill in many of the R add-on modules into the main Debian archives. I still hope to emulate his example in rpm space using R2spec and some post-processing scripts (Dependency enumeration is not quite perfect yet). Dirk has already mentioned in his blog the 'after-sessions' at Jak's; we closed the place down Saturday with useful brainstorming happening long into the night

I had resolved to travel to the conference to learn, and to stay quiet as to matters of FOSS and advocacy. I was almost even able to keep to that intent, save at the interstitial times. The formal presentations were amazing in their quality of content, competence of the presenters, and challenging to my old knowledge of Statistics and Mathematics. I even understood most of what the presenters were doing, and why, on the formal finance side and will re-read the slide decks with great interest when they appear to fill in the holes. One would have probably had to be there to draw much more from the decks, as the presenters were not doing the occasional 'stand and read' presentation one finds at some conferences, but rather largely used their decks as reminders of the points they wanted to hit and elaborate on in their presentation, and to state exactly the code and formulae in play. The committee have really set a high bar to reach for next year's event to top, and I look forward to it already

The pre-conference tutorials were worthwhile. I knew Jeff Ryan's work from xts and IBrokers of course, and gained insight into his mental roadmap on where the code is going and how it will get there. I think the enhancements he is trialling in xts will pretty clearly flow back upstream into zoo in general form; I had not heard of Dale before, but his breakout and presentation of an analytic approach on addition and testing of single constraints (I have covered scientific method and epistemology here before, and will again) served as a fine warm-up to the formal sessions

During the session breaks, at meals, and into the night, I had a chance for give and take at length with several of the committee, presenters, and attendees, to bridge what Patrick Burns spoke on -- the chasm between Practice and Theory

Part of the trip and my need for listen, was to get a handle on how to match the shim and R as a pair of heavy-weight co-processes, so that the user of the shim can hook in and use the wonderful tools already in R space. We'll most likely get there, but the timing is not clear. Having said 'we', permit me to make it clear that the heavy lifting will be done, if and when done, by Bill, and not me. At the prodding of Peter, who as I understand it regularly team codes with Brian, I have started the sign-up process for an account at r-forge, and will 'cut my teeth' on a simple connector or module, to warm up my skills as a co-development tester and 'guinea pig' consumer of the major task of integrating the shim

FIX rules the roost for being the 'lingua Franca' for interchange to exchange order, position and fill data with counter-party upstream brokers or exchanges (thanks here to the CME Foundation for partially funding the event). We will not soon be adding a compressed FIX connector to the shim, and certainly not before we attain our major milestone of a formal 'complete' first release.

Finally, a couple folks asked why we were playing down in retail space with the TWS and its vendor specific API. For a researcher, and for a small proprietary trader, we still find IB's API and services the most affordable, and substantially complete. It is a gateway to enable any interested researcher to do material research (the 'Theory') and strategy development and execution (the 'Practice'). For student academics, the availability of IB's 'trading Olympiad' program and the shim, and R offer all one needs for a better than free price

---

Update: We see also the summary of the event at Revolution Computing, an R vendor

Embarrassingly parallel

Bruce Schneier, in his 'Crypto-gram' summary this month, has an outlink to a story in The Register on a purported desire of the US NSA to crack Skype's call crypto

But this misses the point -- the needed technology and infra-structure are out there already, fielded, and ready to go, pretty everywhere. Let's take a hypothetical country -- call it 'Glassware' ("US", "China" and "Elbonia" were taken)

The country of Glassware has a population of M * 10 ^ N people

Of those M * 10 ^ N people, the average family size is three, and there are an average of two cell phones and one television (the latest -- digital)

There is a broadcast infrastructure suitable to distributing portions of a problem sample -- say, the header block -- sufficiently long that one can detect when a 'good' private key has been found, which is sufficient to decode something encoded with an asymmetric encoding public key.

That target information is distributed over the airwaves, in the vertical blanking interval or sub-carrier side layer, itself encoded with a private key, readily decodeable with one of several 'factory included 'public keys'

The power supply switches in the television sets do not actually place the sets into a 'No power drawn' mode -- just into a lower power use 'sleep' stand-by mode. When tickled with the right signal, and not otherwise engaged in presenting content to possessors of that unit (who might complain about glitches if the video graphics display processor did not fully paint their screen), it is possible to wake them up to do some ciphering. Good for them -- recycles the electrons, and so forth

The television has a handy feature -- it will accept and display caller ID information from nearby affiliated cellular phones, over BlueTooth -- it can be configured to ONLY display wanted cell phones, but it will receive data and collate data from all ringing near it.

So when Mrs Glassware has her girlfriends over, and the babysitter calls during the home sales party, the TV will pop up an alert for them of the call over the din of the fun.

The TV also sends back, over SMS messages, duly encoded and encrypted, the logfiles to series of central collation points -- Father Glassware can see when the oldest son is over at the home of the girl from the wrong side of the tracks. The benefits are as broad as the imagination can see. Who could be against protecting the children?

Those cell phones as it turns out are really not using very much of all that processing power they have in THEIR 'CUDA chips to draw those dinky screens, and are really off most of the time as well.

Let's not waste their graphics processor chips as well, when they are on the charger. This is great, as it simplifies the math.

Perhaps Glassware have an even better infrastructure -- say a national conversion to High Definition digital media signaling, and a mature broadband or cable modem backbone. All the better for shuttling information around digitally.

A friend who deals with quants, tells me the quants are all hot and bothered to get 4 x quad head graphics cards in Dell Precision units -- 16 GPU's, because each of them can do a 10,000 (10 ^ 4) speedup over the simple general purpose processors in the underlying processors the chassis carry. All for under $10k a unit. They are doing the math and think they can have a huge HPC farm, just in the normal overhead which their traders and developers have to have anyway to do their day jobs.

M is 3 in the US (we'll round to 4 to make the math prettier), and perhaps 10 in China, and N is 8 (a hundred million). Feel free to pick a value for your local Glassware

So properly harnessed, we have at least: M * (10 ^ N) * (10 ^ 4) in compute engines available to us -- we should be able to crank out at least 100,000 samples a second ... 10 ^ 5, In cough numbers -- sufficiently accurate for our 'back of the envelope' purposes here, 10 is equal to 2 ^ 3. 2 is useful, as it is bits of key strength to solve. There are 8.6 * 10 ^ 4 seconds in a day -- call it 2 ^ 16

so: M * 2 ^ (3 + 3 + 3 + N + 4 + 5+ 16)

US: 2 ^ 43 key trials per day;
China: 2 ^ 44 key trials per day.

The old DES cipher had a 2 ^ 56 bit keyspace -- worst case time to solution is 2 ^ 13 days and always getting better as build out scales in, without even beginning to bear pre-processing tricks, One time pad reuse, identifying non-perfect implementations, planting known cribs, and the rest.

And it is Free, free, free -- or better yet, paid for by others. What was that old saw about people living in glass houses?