Archive for the ‘Uncategorized’ Category

What can Darwin’s finches tell us about the downturn?

Thursday, November 20th, 2008

Newspaper articles paint the markets in metaphors like “difficult climate” and “harsh landscape” –but these clichéd phrases have a kernel of truth.   Thinking about markets as natural environments reveals that selective forces are at work.  But it also predicts when they work.  In the natural world, as the story of Darwin’s finches tells us, selection acts in times of crisis:  drought, famine, and disease.  For our markets, that time is now.

(Aside:  I confess that relating the economic crisis to Darwin is a symptom of an academic bad habit:  namely, mapping every phenomenon onto the intellectual giant of one’s field.  Somewhere there is a psychologist blogging about Freud and the economy).

(more…)

What I’ll be presenting at O’Reilly Money Tech 2009

Tuesday, October 21st, 2008

(April 2009 Update:  Unfortunately, The Money Tech Conference was indefinitely postponed, but fortunately I will be presenting a version of this talk in July at OSCON 2009).

I’ve been invited to speak at O’Reilly’s Money Tech conference this coming February 4-6th in New York City and thought I’d share the abstract for my talk here.  I’ll likely be in New York for several days, if you’d like to get together to chat about data drop me a line!

My talk is entitled “Open Source Analytics: Visualization and Predictive Modeling of Big Data with the R Programming Language”
(more…)

John Henry and decision making

Sunday, April 6th, 2008

Cap’n said to John Henry,
You’ve got a willin’ mind.
But you just well lay yoh hammah down,
You’ll nevah beat this drill of mine,
You’ll nevah beat this drill of mine.
(Lyrics)

John Henry

Today we take it for granted that a human is no match for a machine drill when it comes to digging through a mountain. At the time such songs were first sung, the verdict was dodgier; some believed that human ability would always be superior to machines. It seems that every time our technology advances, we must struggle a little to accept that our domain of superiority has narrowed.

Nowhere is this struggle more acute than in the contemplation and recent realization of thinking machines. Our discomfort with the idea that a machine that performs intellectual labor could ever be our equal dates at least to the roots of the industrial revolution and information revolution in the Enlightenment.

“…although such machines might execute many things with equal or perhaps greater perfection than any of us, they would, without doubt, fail in certain others from which it could be discovered that they did not act from knowledge, but solely from the disposition of their organs” –René Descartes, Discours de la méthode, 1637

When it comes to decision making, which often requires drilling through a mountain of data, we still manifest a deep skepticism of our machines. We call these creations black boxes, soulless as the machine drill, and place our faith in modern John Henrys who make decisions based on gut instinct. Human insight and judgment are still irreplaceable, but they are being increasingly augmented by thinking machines that can account for far more factors then we can in making a decision. And far from a black box, these machines are just our knowledge made scalable and executable.

Popular culture picks up on our need to differentiate ourselves; futuristic artificial intelligences are portrayed as, ultimately, too rigid to compete, unable to grasp the subtleties we humans use to drive our decision making. A scientific approach means the ability to perceive such subtleties should generate a testable hypothesis. In more and more areas, the quantitative approach is showing itself able to outperform raw human judgement, even in fairly subtle situations. Particularly famous is the work of the Oakland As in evaluating baseball players. Orley Ashenfelter’s remarkable success in predicting the quality of wines was another day the steam drill won.