Taking a step back, here is a simple case that demonstrates the difference between Rails version 2 and Rails version 3 with respect to << (aliased as ‘concat’ and ‘push’ in the ActiveRecord framework) when adding to a collection established using has_many :through.  Given three models: User, List, and UserListLink (which has foreign keys for user and list):

class User < ActiveRecord::Base

has_many :user_list_links, :dependent => :destroy

has_many :lists, :through => :user_list_links

end

class List < ActiveRecord::Base

has_many :user_list_links, :dependent => :destroy

has_many :users, :through => :user_list_links

end

class UserListLink < ActiveRecord::Base

belongs_to :user

belongs_to :list

end

In both Rails 2 and 3, the following works:

u = User.create

l = List.create

u.lists << l

However, if you add attr_accessible [] to UserListLink, under Rails 3 the UserListLink that is automatically generated will have nil/null values for both list_id and user_id. You’ll see the expected message in the Rails log, “WARNING: Can’t mass-assign protected attributes: list_id, user_id”, which indicates that Rails uses mass assignment to create this intermediary object. In my case, removing the attr_accessible line in my code allowed ‘<<’ to work as it did before upgrading to Rails 3.

I’m not sure how I feel about the change in behavior in ActiveRecord 3 where mass assignment protection is triggered by the ‘concat’ method on a has_many :through relationship.  I suppose whether or not this is ‘correct’ behavior depends on whether there is a chance for user-generated data to be passed into the attribute hash used to construct the intermediary object.  If so, this is probably a nice fix in ActiveRecord 3.    Otherwise, though, it may help to bypass the mass assignment check when saving intermediary objects using concat.

First, my goal in creating Clean.ly was to add a layer of information on top of the NYC restaurant data to make them more interesting and useful.  Geocoding the restaurant addresses was a first step in this process, which was easily accomplished using the Google API.  Next, I grabbed some other information about the restaurants, including type of food, from other freely available sources on the web.

For these data mining tasks, I needed to serialize a bunch of operations to make sure I didn’t go over API quotas.  I also wanted to create a system that could accept data from different inputs without much trouble later.  To meet these needs, I used a Kestrel queue to serialize HTTP retrievals from various providers.

Once I had the necessary data in place, Sphinx was an obvious choice to bring these data in front of the user on demand.

For those of you who have used the fulltext searching engine in Sphinx, it’s easy to envision how restaurant searching based on name or “tag” (such as type of food) would work.  A lesser known fact about Sphinx is that in the newest releases, simple geodistance calculations are supported.  The only real trick in implementing this part of the Clean.ly was that the latitude and longitude obtained from Google had to be converted to radians before piping them to the Sphinx indexer.  After that, I had near-instant results with distance calculations from an input address, making it possible for users to search for restaurants close to where they want to eat.

Once I had geodist calculations in Sphinx working, I wanted to make sure that pages loaded quickly.  Since Clean.ly doesn’t really have user-modifiable content, I immediately thought of page caching.  In other work that I’ve done scaling high-performance web sites, I had implemented Varnish, a reverse page caching daemon, and had great success.  It’s an extremely elegant and easy-to-use program which I’d highly recommend checking out for anyone interested in scaling high volume web sites.

Implementing Varnish for Clean.ly turned out to be a fairly straight-forward task. Clean.ly only has one server, a small instance on EC2 that I also use to host a variety of other small Rails projects.  Varnish proxies port 80 on this instance to another port where Apache is listening, and it short-circuits and returns results for pages stored in the page cache.

In another post, I’d like to go into more detail on using Varnish with Rails applications.  For now, though, I hope that you enjoy using Clean.ly.  Be sure to check it out, as well as the other great submissions to the NYC BigApps contest.

SubwayDelay is not affiliated with the MTA.  Unfortunately, the MTA has yet to release easy access to data on service alerts.  This application rests on MTA Feeds (another site that I created), which provides free access to the MTA data in an easy-to-consume format for transit software developers and subway riders.

In other words, software engineering best-practices often go out the window as Rake tasks are built, leading to missed opportunities for software reusability and increased maintenance costs.  In my experience, it’s only a bit more effort to turn the effort of writing a one-off Rake task into an opportunity to expand on the reusable components already in your system.

When I’ve coded maintenance scripts in Rake, two of the signs that a task needs refactoring are length of the task itself and the number methods that have to be defined in the namespace to support the method.  A task, just like any other method, should do just one thing.  If the Rake task is so long that it scrolls off the screen, opportunities for error are high and the next person working on the code is much more likely to start from scratch rather than sort things out in the method body.

The same applies to Rake tasks that require the presence of lots of other methods defined loosely inside of the namespace.  Defining methods in the Rake namespace for munging data or doing other task setup usually means that these methods aren’t formally tested anywhere.  In many cases, it also indicates that opportunities to hide complexity by using an Object-Oriented interface are being lost.

I’ve found that Rake tasks often descend into a mass of procedural code to manage data.  Part of the strength of Ruby is the set of tools it provides for defining objects that we can use to hide complexity, leading to more maintainable code.  It’s a shame to not use these tools when it only takes a short amount of time to think about how your task could be decomposed into a set of methods to be invoked on new or existing objects.

It takes some practice to think about writing good object-oriented code in short Rake tasks.  One of the hurdles is thinking of your task in terms of the existing objects in your codebase.  These may be existing models, in which case you have a pre-existing place to put code that would be more cohesive than just including it in the namespace of your Rakefile.  In other cases, you may have to define new classes (often under the “lib” folder of your project) to best encapsulate the data that you have to act upon.

Writing short Rake tasks that operate on objects in your codebase rather than defining long task bodies with supporting procedural methods often requires a shift in thinking about your Rake tasks and how it relates to the objects in your codebase.  The effort is worthwhile, however, as you add new classes and more precisely define the interfaces of existing classes in your system in order to achieve a more maintainable system.

Sinatra + Tokyo Cabinet

SubwayDelay requires a server to communicate with the iPhone application.  Instead of going with the popular Ruby on Rails stack, I decided to use a lighter-weight alternative called Sinatra.  Sinatra is great when you don’t need the overhead of Rails… it’s easy to figure out what’s going on in the framework since the codebase is small, and I’ve successfully used it in several cases where a web service component is needed.  For the user-visible web pages, I used Haml and Sass instead of the usual ERB since I find those DSL implementations a bit more succinct.

More importantly, in this application I was faced with data that would steadily increase in size.  I wanted to make sure that application performance remained constant, without becoming steadily worse as the size of data increased.  With AT&T’s current network woes, the last thing I wanted was to cause users to wait extra time for their app to load because of slow delivery speed of my web stack!

For readers interested in things like computational complexity, this meant that I was looking to make sure that the algorithms would be able to complete in constant time.  In order to achieve this, I realized that a Hash-type data store would come in handy.  I used Tokyo Cabinet with Tokyo Tyrant, which serializes and deserializes small Ruby Hash objects as my application needs them.

The result was an application that was extremely quick to develop, and which should have great scaling properties.

Core Data on iPhone

When I was approximately halfway through developing my iPhone application, which stores user preferences, Apple released iPhone OS 3.0.  Among many improvements to the toolkit was Core Data, a feature which had only been available previously on Mac OS X for desktop computers.  Essentially, Core Data is an Object-Relational Mapping system.  It enables the user to set up a backend data store using sqlite, and helps with flushing changes, selecting objects, and instantiating the correct models for your application.

I realized immediately that it was exactly what I was looking for to handle storing the basic data that I needed to maintain.  I took a step back, realizing that it would probably delay the release by a few days, and dug into the Core Data learning materials on the Apple Developers web site and the sample applications.  In the end, the slight detour was well worth the effort.  I have an application that adheres to the data store that Apple suggests, and the fact that Core Data integrates well into the strict MVC framework on the iPhone was icing on the cake.  I’d recommend a similar approach to anyone currently working on an iPhone application.

Apple Developer Docs and App Submission

As a Ruby developer, I’m used to being able to Google just about any problem that I come across and find a solution on the web.  With the iPhone, this isn’t necessarily the case.  The NDA that Apple had in place during early App development stifled some important discussion.  Now that this has been lifted, the Internet is starting to fill up with more useful information for the iPhone developer.  In particular I’ve been impressed with StackOverflow.com for relevant and helpful exchanges among developers.

There are still substantial deficits in information on the web, though, especially in things like the provisioning process.  It also complicates matters a bit because I believe that Apple has made some very slight changes to their tools, rendering some public web pages obsolete.  What I found in the end is that if you follow the yellow brick road laid out by Apple, things tend to fall into place.  Having trouble getting your keys in place to release the application?  Read the HowTo guide in iTunes Connect a couple of times and nine out of ten times the suggested approach will just work.  Afterwards, maybe there will be some useful information to be gleaned by general Googling, but try to restrain yourself and you may end up fixing things faster by just reading the Apple docs closely.  They really are very well written and comprehensive.

Conclusion

The availability of elegant Web service frameworks like Sinatra, key-value storage systems such as Tokyo Cabinet, and tools such as Core Data in the iPhone development kit can be composed to create flexible and scalable web services that extend to mobile devices, as I found in my recent development of SubwayDelay.  Any other tools that have come in handy for you in building apps using a similar stack?  Let me know in the comments.

I’m still not confident that the semantics of the ruby memcache client are what one would want for use with persistent key-value stores.  In particular, I haven’t delved into the code enough to figure out if the cases where failure occurs in the memcache client should be different based on the type of key-value storage system being used.  Since the ruby memcache client is so widely installed, it seems like being able to specify different behavior for failure cases would be a great help for people using different forms of key-value storage systems with the memcached protocol.

Although there are other libraries out there that do some of the same things, I decided to write this library after some careful thought because nothing else really had the right balance of features that I needed in terms of abstraction of the details of new feed detection and conditional GET.  Feedtosis is also lightweight in that it delegates the heavy lifting of actually making HTTP requests and normalizing feeds to the very capable ruby libraries taf2-curb and FeedNormalizer respectively.  In Feedtosis I tried to follow a Unix-like philosophy of building utilities that only solve one particular problem, while allowing composition with other minimal utilities in order to create robust systems.  I hope that you find it useful!  Feel free to forward me any comments, and please post any issues that you have with the library on its GitHub page.

In case you’d like to see the material from the presentation, check out the attached Presentation PDF or Powerpoint.