John Lam on Software

Do Databases Rot the Mind?

Do databases rot the mind? Charles Petzold’s recent post about Visual Studio triggered this thought in the back of my mind.

All too often when confronted with a “business” problem I, like many other people I know, wind up reaching for my trusty database. Or, in a more common scenario, someone hands you an existing schema and says “solve my problem” using it. The reality, however, is that relational databases are a lousy solution to many problems that we commonly run across. In a recent e-commerce platform that I created there was a 7-way join that had to be executed just to retrieve a list of products to display to a customer. We did LOTS of performance tuning of that database and our queries to make it go about as fast as it could reasonably be expected to go under SQL Server.

Just this week, I decided to do an experiment. I reimplemented the core of our e-commerce platform by loading the entire database into a custom in-memory data structure. It took me about 1 hour to write the code to load the core tables into my custom data structure, and about an additional 15 hours to tune the data structure and refactor the code to my liking. Oh yeah, I wrote it using Ruby.

My app was 201 lines of Ruby code (including liberal use of blank lines to enhance readability). For sake of comparison, just one of the dynamically generated SQL queries that it replaced was in the order of 150+ lines of SQL. And that, of course, doesn’t count all of the lines of C# code required to generate, execute, and parse the results of the SQL query.

One more thing: it also ran 100x faster.

A bit of history: the original application that I inherited was on the order of 20,000 lines of SQL + C#. This was largely due to an enormous amount of duplication of code due to the (ab)use of stored procedures to handle queries. My rewrite of that application trimmed it down to 4000 lines of SQL + C#. My Ruby application, while not at feature parity with the existing application, solved essentially all of the “hard” problems. I would estimate that it duplicates >80% of the functionality of the real application.

The first version of the code was about 5x faster than the original application. I then spent some time looking for optimizations. When you only have about 100 lines of code to look through, optimizations become pretty obvious. I was able to cache the results of a (relatively) expensive O(M+N) algorithm that sat on the rate-determining-step of the computation. This netted me a 20x speedup. This optimization is simply not possible using SQL Server, but is dead-simple when you get to implement your own query engine.

I love the symmetry of my results: 1/100th the code and 100x the performance. It’s also much easier to maintain 200 lines of code than 20,000 lines of code. Testing was also much more straightforward.

Now the code that I rewrote was tailor-made for this kind of rewrite. It was a read-only database that was relatively (< 1GB) small in size. If you have such an application lying around – try rewriting it using in-memory data structures. If you’re a bit more adventurous, try using a dynamically typed language to do this (might I suggest Ruby?). You just might be surprised by the outcome and come away with a new technique in your toolbox by doing so.

Notation matters

Gerald Sussman’s talk at OOPSLA was tied to one of the themes of this year’s conference: poetry, the arts, and their relationship to software. He talked about code that is prose, and code that is poetry. He showed us some examples of prose that he wrote: some C code that controlled the servos of a large radio telescope array in Chile. He also showed us some code that is poetry: the two-page implementation of eval in Lisp.

He also talked about how writing programs can help improve the way you think. One example came from a first-year graduate level course in Classical Mechanics at MIT. In this course, he gets his students to write Scheme code to describe certain equations. The students found that they understood the equations far better when asked to reformulate them using a computer language instead of mathematical notation. There was also the added side effect that they could now perform computations using their equations.

Gerald Sussman comes across as a geeky version of Deion Sanders. Deion Sanders was great and he wanted to remind you of it when he spoke. Similarly, Sussman is great and he reminds you of that fact when he speaks as well. Some people find this kind of grand standing offensive. Others, like myself, find nothing wrong with this – if you can talk the talk and walk the walk then that’s fine by me.

Why not a Ruby.NET?

Building a native Ruby implementation on the CLR is an ambitious goal. John Gough and associates are attempting to do exactly this in their Ruby.NET project. This is a much harder problem than it looks, and kudos to John for even attempting it. The semantics of Ruby are significantly different than other CLR languages.

As a simple example, consider the fact that Ruby does not allow method overloading. How would you construct a Ruby type that must override methods in an abstract base class that differ only based on their signatures?

Another example is the case where you want to surface CLR features natively in Ruby. How, for example, will you declare a generic array in a language that does not support type declarations?

Last but certainly not least are the sophisticated features in Ruby such as continuations that make it very challenging for an implementor to implement efficiently. This feature requires a stack copying mechanism that isn’t a native feature of the CLR. Can such a feature be added to the CLR without compromising the performance of existing languages? This is not clear, and was one of the topics that came up for discussion during the dynamic languages BOF at OOPSLA.

Building bridges

I’ve been building a bridge from Ruby to the CLR in my spare time. It’s a complicated piece of software, and I plan on explaining how I created it in a series of posts on this blog. At the time of this writing, I can consume CLR objects from Ruby scripts and implement event handlers in Ruby scripts. The Ruby language can make it possible for people to create applications that would be more difficult and take more time using C# or VB.NET. The Ruby language has certainly changed the way I think while writing code; I hope to bring that power to people writing .NET code today. Here’s a teaser:

require   'RubyShim'
reference 'System.Windows.Forms'

include System::Drawing
include System::Windows::Forms

f               = Form.new
f.Text          = 'My Form'
f.AllowDrop     = true 
f.SuspendLayout

label1          = Label.new
label1.Location = Point.new(40, 32)
label1.Width    = 200
label1.Text     = 'My Label'

button1          = Button.new
button1.Name     = "MyButton" 
button1.Location = Point.new(40, 57)
button1.Text     = 'Click me'

button1.Click do |sender, args|
  p = sender.Location
  label1.Text = "clicked on #{sender.Name} at #{p.X}, #{p.Y}" 
end

f.Controls.Add(label1)
f.Controls.Add(button1)
f.ResumeLayout

Application.Run(f)

This will be an open-source project, and will be redistributed under an MIT-style license. I have not yet released the source code since it is largely proof-of-concept code at this time. If anyone really wants to take a look at it right now, just send me an email and I’l be more than happy to share.

I believe that I have solved most of the hard problems. I have Windows Forms and Avalon applications written in Ruby and running on my computer today.

A Life Changing Week

This year, I was fortunate enough to attend both RubyConf and OOPSLA. These conferences exposed me to many new ideas and lots of new people. If you’re used to living in the Microsoft (or Java, or PHP, or Linux, or Apple …) world, it’s a really good idea to break out of your monoculture and experience something different.

For the Microsoft folks, instead of attending a PDC, Tech Ed, VS Live!, or DevConnections conference every year, choose to attend a different conference with your conference dollars. Attend a RubyConf or a JAOO or an OSCON. Rails is hot this year, but something else will be hot next year.

For me personally, I’ve spent way too long in the statically typed languages world. Last year at the Middleware conference in Toronto, Steve Vinoski and I had a great bonding moment when we recounted our recent experiences with Python. At OOPSLA, I had a chance to chat with some old-school Smalltalk programmers (hello Jim!) who were more than happy to point out that most “new” language features in Python/Ruby were in existence long ago in Smalltalk. The same, of course, can be said for languages like C# as well :)

That said, it’s still important to use the right tool for the job. The libraries in the .NET world are simply too useful to be ignored. Bridging this gap is an itch I have to scratch. More on that soon.

Best Talk Of All Time

This year at OOPSLA, a group took a look at the last 20 years of OOPSLA. When a man asked the group “what was the best talk of all time at OOPSLA?”, they all said Guy Steele’s talk from OOPSLA 1998.

They showed a film of his talk in the hall of books. I thought that I had seen great talks in my life, but I can’t think of one that tops this. If you can, watch the film (I think you will need to buy it from the ACM). You may read the talk on the web, but if you do read it out loud. Yes – this may seem like a strange thing to do, but you may miss the best part of this talk if you do not.

New blog engine

Wow. It’s been busy. I finally got some time recently to restart this blog, and my first order of business was migrating to a new blog engine – Typo. Typo is a Ruby and Rails powered blog engine.

I’m going to be posting much more regularly to this new blog. I plan on adopting the Raymond Chen like approach to publishing at least one new post each day. One of my first things to do is hack an article queue into the Typo engine.