Showing posts with label trading-shim. Show all posts
Showing posts with label trading-shim. Show all posts

27 April 2009

'R' you experienced?

" ... To confer, converse, and otherwise hob-nob with my brother wizards ..."

I spent a productive weekend up in Chicago, at the R/Finance 2009: Applied Finance with R conference, which billed itself as the "first annual R/Finance conference for applied finance using R". The conference organizers and hosts are the 'usual suspects' on the 'R sig fin' mailing list; Jeffrey Ryan, Dirk Eddelbuettel, Dale Rosenthal, Brian Peterson, Peter Carl, Gib Bassett, and John Miller, assisted by the talented and imperturbable Holly Griffin of UIC. Pretty clearly most of this group code together regularly; see the committer list on the blotter module. The venue was at the 'other school' in Chicago, the one with a more practical interest in Economics and Finance

An aside about Chicago: Long ago, and far away, I was trained as a acolyte 'monetarist' by disciples of Herb Stein's CEA and the Fed, in the [University of] Chicago school [a fad, seemingly long forgotten by recent Economics and Finance grads, so far as I can tell]. Monetarism is a forgotten religion these days; the Fed stopped formally publishing its M3 series a few years ago, in light of the rise of what Bill Gross calls the 'shadow banking' system.

Luke: [The robot] claims to be the property of an Obi-Wan Kenobi. Is he a relative of yours? Do you know what he's talking about?
Obi-Wan: Obi-Wan Kenobi. Obi-Wan... Now, that's a name I've not heard in a long time. A long time.
Luke: I think my uncle knows him. He said he was dead.
Obi-Wan: Oh, he's not dead... Not yet.
Luke: You know him?
Obi-Wan: But of course I know him

Many of the organizers were known to me from my email correspondence or from observing their packages, and I had spoken with one (Dirk) two or three years ago briefly after the trading shim was first usable

Dirk seems to be a bundle of unbounded energy. His tools had solved a lot of data storage and visualization issues for me early on in our project. He led a push a couple years back to drill in many of the R add-on modules into the main Debian archives. I still hope to emulate his example in rpm space using R2spec and some post-processing scripts (Dependency enumeration is not quite perfect yet). Dirk has already mentioned in his blog the 'after-sessions' at Jak's; we closed the place down Saturday with useful brainstorming happening long into the night

I had resolved to travel to the conference to learn, and to stay quiet as to matters of FOSS and advocacy. I was almost even able to keep to that intent, save at the interstitial times. The formal presentations were amazing in their quality of content, competence of the presenters, and challenging to my old knowledge of Statistics and Mathematics. I even understood most of what the presenters were doing, and why, on the formal finance side and will re-read the slide decks with great interest when they appear to fill in the holes. One would have probably had to be there to draw much more from the decks, as the presenters were not doing the occasional 'stand and read' presentation one finds at some conferences, but rather largely used their decks as reminders of the points they wanted to hit and elaborate on in their presentation, and to state exactly the code and formulae in play. The committee have really set a high bar to reach for next year's event to top, and I look forward to it already

The pre-conference tutorials were worthwhile. I knew Jeff Ryan's work from xts and IBrokers of course, and gained insight into his mental roadmap on where the code is going and how it will get there. I think the enhancements he is trialling in xts will pretty clearly flow back upstream into zoo in general form; I had not heard of Dale before, but his breakout and presentation of an analytic approach on addition and testing of single constraints (I have covered scientific method and epistemology here before, and will again) served as a fine warm-up to the formal sessions

During the session breaks, at meals, and into the night, I had a chance for give and take at length with several of the committee, presenters, and attendees, to bridge what Patrick Burns spoke on -- the chasm between Practice and Theory

Part of the trip and my need for listen, was to get a handle on how to match the shim and R as a pair of heavy-weight co-processes, so that the user of the shim can hook in and use the wonderful tools already in R space. We'll most likely get there, but the timing is not clear. Having said 'we', permit me to make it clear that the heavy lifting will be done, if and when done, by Bill, and not me. At the prodding of Peter, who as I understand it regularly team codes with Brian, I have started the sign-up process for an account at r-forge, and will 'cut my teeth' on a simple connector or module, to warm up my skills as a co-development tester and 'guinea pig' consumer of the major task of integrating the shim

FIX rules the roost for being the 'lingua Franca' for interchange to exchange order, position and fill data with counter-party upstream brokers or exchanges (thanks here to the CME Foundation for partially funding the event). We will not soon be adding a compressed FIX connector to the shim, and certainly not before we attain our major milestone of a formal 'complete' first release.

Finally, a couple folks asked why we were playing down in retail space with the TWS and its vendor specific API. For a researcher, and for a small proprietary trader, we still find IB's API and services the most affordable, and substantially complete. It is a gateway to enable any interested researcher to do material research (the 'Theory') and strategy development and execution (the 'Practice'). For student academics, the availability of IB's 'trading Olympiad' program and the shim, and R offer all one needs for a better than free price


Update: We see also the summary of the event at Revolution Computing, an R vendor

26 March 2009

IPv6 eats kittens (and distcc) on Debian Testing

Flikr domo and kitten

This can only end badly

I spent a good 5 hours this week, tracking down a problem with distcc hanging up in our Debian Testing build farm. We use distcc to speed up compilation of the c++ sources in the development of the trading shim. Interestingly, our end user community forced us to this decision of developing on Debian testing, as they are using later gcc versions than we were on CentOS, and it was useful to be able to see their errors, BEFORE they reported them to us

On the new compile farm, sometimes we would get a compile in, say, 44 seconds; other times it would drag out for several minutes. This is a problem as we had just slotted a new unit into harness, and expected better results

In checking the logs in the client doing the distribution of compilation tasks, we were seeing a symptom of 'segfaults' in that client's process; other times, the client would stall, seemingly blocked waiting for a compilation result to come back from a remote buildfarm peer, that never came back. Checking on the remote build unit, one of the distccd children would die for mysterious reasons, leaving a message in the dmesg record. Once that failed build timed out, the needed file would be built locally, and the build proceed. Checking the log files nothing obvious jumped out

The obvious debugging technique is to get a minimal reproducer, and then to partition the problem into smaller and smaller possible causes using that reproducer tool. the issue will manifest on one setup, but not the other, ans so one can rule out more and more issues, until the answer is left, staring you in the face

Looking at my Debian helper tool, it had rotted, and was in sorry need of removal of some constraints: It did not use distcc when available; it did not use proper -J parallel compiles; it did not use -O3 optimization in the compiles. My test tool was not set up to see what I needed to see

Time to pay down some 'technical debt' (If you've not read martinfowler piece, and viewed Ward Cunningham's video, stop now, and do so). And so I made some payment there. After testing, I got these results:

MasterClientsElapsed time (real)
 pippin  nfs2, 10.16.1.231  0m23.281s 
 nfs2  10.16.1.231, pippin, localhost  0m23.702s 
 10.16.1.231  pippin, nfs2, localhost  0m22.551s 

My first thought looking at this: Well, that pretty conclusively rules out machine specific errors, or network path issues. It must be something different in the setup of the user provoking the issue that my tool does not duplicate. NOTE: This is wrong-headed, of course, as: 'An absence of evidence is not evidence of absence of a problem' but was an easy trap to fall into

For every complex problem, there is a solution that is simple, neat, and wrong.

  — H. L. Mencken

For every problem there is a solution which is simple, obvious, and wrong."

  — Albert Einstein

I tossed my results at that user for their thoughts on the results, and went back to work on another issue

Later in the day, doing some thought experiments with the user, we could not pin down where to look yet. But as a team, I had him provoke the issue with his setup, while I watched the logs on the various machines through several consoles. And the error appeared, and then jumped out and tickled my eyeballs. I was watching nothing in particular, until I saw the failure on process 29673, and then traced that back up. A successful and a failed session looked like this, respectively:


distccd[29673] (dcc_check_client)connection from :ffff:10.16.1.249:41771
distccd[29673] (dcc_r_file_timed)909179 bytes received in 0.078651s, rate 11289 kB/s
distccd[29627] (dcc_collect_child) cc times: user 1.132070s, system 0.144009s, 23039 minflt, 0 majflt
distccd[29673] (dcc_collect_child) cc times: user 1.092068s, system 0.104006s, 22481 minflt, 0 majflt
distccd[29673] (dcc_check_client) connection from ::ffff:10.16.1.249:41775
distccd[29673] (dcc_r_file_timed) 818437 bytes received in 0.071648s, rate 11155
kB/s
distccd[31248](dcc_check_client)connection from ::ffff:10.16.1.249:41779
distccd[31248](dcc_r_file_timed)886761 bytes received in 0.076688s, rate 11292 kB/s
distccd[29627](dcc_collect_child)cc times: user 1.068066s, system 0.112007s, 23890 minflt, 0 majflt
distccd[29673](dcc_collect_child) cc times: user 1.108069s, system 0.112007s, 22012 minflt, 0 majflt
distccd[29673](dcc_pump_sendfile)Notice: sendfile: partial transmission of 15868 bytes; retrying 344332 @15868
distccd[1995] (dcc_log_child_exited)ERROR: child 29673: signal 11 (no core)

A-ha! Now we know what to look for:


dhcp-231:/var/log# grep dcc_pump_sendfile distccd-transition-log
distccd[29673] (dcc_pump_sendfile) Notice: sendfile: partial transmission of 15868 bytes; retrying 344332 @15868
distccd[31248] (dcc_pump_sendfile) Notice: sendfile: partial transmission of 15868 bytes; retrying 586732 @15868
distccd[30262] (dcc_pump_sendfile) Notice: sendfile: partial transmission of 15868 bytes; retrying 4655916 @15868
distccd[2005] (dcc_pump_sendfile) Notice: sendfile: partial transmission of 16384 bytes; retrying 74824 @16384
distccd[2128] (dcc_pump_sendfile) Notice: sendfile: partial transmission of 16384 bytes; retrying 286560 @16384
distccd[2170] (dcc_pump_sendfile) Notice: sendfile: partial transmission of 16384 bytes; retrying 97440 @16384
distccd[2129] (dcc_pump_sendfile) Notice: sendfile: partial transmission of 16384 bytes; retrying 301000 @16384
dhcp-231:/var/log#

The TCP process of shuttling code to compile, and the binary results of such compiles are failing the same way, over and over again: partial transmission of 15868 bytes is present every time. Looking at the log entry again, the form of the connecting hosts is unusual: ::ffff:127.0.0.1 and ::ffff:10.16.1.249. Why that is IPv6 notation? And I reach back to my logs as I remember I had an issue like this a year or so on a Debian box

And so, Google with the search argument: debian ipv6 distcc confirms as its first result: 1. #481951 - distcc: zeroconf support broken wrt IPv6 - Debian Bug ... ... and the bug is still open. Killing off IPv6 is the obvious next step, and so, back to Google with: debian disable IPv6 to find: Disabling IPv6 under a 2.6 kernel. Reading the post, there is some back and forth, and the answer seems to be, there is not an 'official Debian answer', but this is what people are doing. Back to Google with: site:debian.org debian disable IPv6 seems to confirm that there is not a single well documented answer which has floated up in Google's searching

Compare: CentOS addresses the matter directly, and as the first Google hit with: site:centos.org disable IPv6
7. How do I disable IPv6?

* Edit /etc/sysconfig/network and set "NETWORKING_IPV6" to "no"
* Add the following to /etc/modprobe.conf :

alias ipv6 off
alias net-pf-10 off

* Run chkconfig ip6tables off to disable the IPv6 firewall
* Reboot the system

Alternative (which might be easier and works on any release with /etc/modprobe.d):
echo "install ipv6 /bin/true" > /etc/modprobe.d/disable-ipv6


Sadly, there is something else on Debian testing in play as well, and it is not just an IPv6 issue (although turning off IPv6 has drastically reduced the frequency of the issue). When I look in today to make sure the 'fix' is working


[74988.951989] distccd[8671]: segfault at 1 ip 7fdd2250e030 sp 7fff2b025da8 error 4 in libc-2.7.so[7fdd22493000+14a000]
[74989.017836] distccd[8651]: segfault at 1 ip 7fdd2250e030 sp 7fff2b025da8 error 4 in libc-2.7.so[7fdd22493000+14a000]
[74989.518050] distccd[8664]: segfault at 1 ip 7fdd2250e030 sp 7fff2b025da8 error 4 in libc-2.7.so[7fdd22493000+14a000]
[74994.152461] distccd[8659]: segfault at 1 ip 7fdd2250e030 sp 7fff2b025da8 error 4 in libc-2.7.so[7fdd22493000+14a000]

Where is that coffee cup? I knew this would not end welldomo eating a kitten

09 March 2009

CentOS 5.3 qa - fresh installs testing on i386

centos-5.3 QA fresh installs
KB released released a twitter note:
ah! the smell of a freshly baked distro.
earlier today, reflecting the presence of updated binaries, and wire-install ready images. Yummy, and I noted them in my twitter feed.

I completed the rsync of the new candidate archive, and have knocked out both GUI and text mode domU wire installs of the i386 candidate. As you may recall, I had done updates testing before, without incident.

The screenshot is fullscreen of 1440x1050 and I am not about to booger the PCO feed again -- direct link is at: http://www.herrold.com/images/blog/c53.-qa.jpg

The test running (side by side in two xen domU test candidates) is a rather hard one, which build graphviz, on the way to the trading shim -- the test script can be retrieved from:
ftp://ftp.owlriver.com/pub/local/COLUG/shim-builder.sh
for those who want to play along.

Something is slightly wrong in the install of the graphviz binary produced, and I'll get that fixed. Also there is some LaTeX issue which I need to run down in the current doco build, generated from a SPOT -- a single point of truth. The -f option to shim-builder.sh may be enough to get past it, regenerating the .dot files, and this the .ps included art.

It succeeded in the GUI based installation. We have some slight warning level noise in today's shim in building today's shim, but it is just c++ being picky (hoo-ray for type checking). The TUI test lags, as I had to build the TUI instance separately, and get the local install archive set up in /etc/yum.repos.d/

This is ready to go as to my needs, but I'll test media installs, and on x86_64, next.