Blog Moved!

Please visit the new location of this blog:

http://tektastic.com

 

RailsConf 2011 Best of Sessions

Below I put together my personal list of best RailsConf 2011 presentations that have online slides or PDFs. I put my comments next to each. Click on the link, then click on the “slideshow” or “PDF” links for each talk.

• ActiveSupport 3: What We Should Know About What We Don’t Know – how to use ActiveSupport in your ruby apps and get lots of behaviors and useful stuff for free
• Building Bulletproof Views really decent presentation on state of views in general, and best practices. Recommended reading.
• Building Pageless Apps with Rails and Backbone JS is a great presentation on a new up and coming JS client side framework for building reach apps on the client using pure MVC.
• David Heinemeier Hansson — keynote, worth watching
• From 1,000 Transactions a Month to 1 million in a Day: Lessons in Credit Card Processing from LivingSocial – cool presentation on how BrainTree API is used to encrypt card data in JavaScript before submitting to BT for processing. Lots of other useful tidbits on credit card processing.
• How To Handle 1,000,000 Daily Users Without Using A Cache most interesting presentation for performance junkies, on how they outgrew MySQL and explored various other solutions to scale to 1M daily users. Highly recommended.
• KnowSQL: Database Tricks To Make Your Life Easier this is a PostgreSQL heavy presentation on how to get the most out of your database engine. Recommended to anyone who wants to learn more about using a real database =)
• OmniAuth from the Ground Up a really good presentation on OmniAuth – a generic authentication gem that supports a lot of options, including OpenID, LDAP, etc.
• Open Source E-Commerce With Spree VERY interesting presentation on this open source e-commerce platform. Great examples on using engines, customizable CSS, and many more. URL: https://github.com/spree/spree
• Rails Performance Tools – system tools such as lsof, strace, ltrace, gdb, perftools, memprof, etc to identify bottlenecks in your ruby apps that NewRelic can’t, by Aman Gupta.
• Sass: The Future of Stylesheets a great overview of SASS and compass, with examples of @extend and @include.
• Stateful, Scalable Servers with EventMachine and Rails a talk by Aman Gupta (again) on EventMachine and writing non-blocking high concurrency but single-threaded TCP/IP servers. Highly recommended.
• The Holy Grail (of Databases) Overview of SQL, key-value stores and other NoSQL storage options. Highly recommended.

And if you are feeling sentimental, and added bonus:

• 50 in 50 is a really zany presentation on computer science achievements in the last century. Great meditative presentation, and goes way over 1 hour. No relationship to ruby/rails, but everyone will get something out of it

 

mms-mime: MM7/MMS MIME parsing gem

https://github.com/kigster/mms-mime

This gem was written with a simple aim to parse MM7 wrapped binary and base64 encoded MMS messages received via MM7/XML HTTP post from an MM7 compatible gateway connection (such as OpenWave, OpenMarket, etc).

The gem provides a simple way to parse and access MMS message contents, such as from, to, subject and content parts (including image and text parts).

 

>Amazing Map of World of Warcraft World Routes and Travel Paths post Cataclysm

>If you are playing Cataclysm, you may be baffled by the new travelling routes between continents. Well, baffle no more! My amazing artistic wifey had created this fantastic map of routes, so you can figure out how to get from anywhere to everywhere without a handy mage 🙂

Enjoy, and happy playing 🙂

 

>GoGaRuCo 2010 – San Francisco Ruby Conference

>Had an awesome time here for two days, listening to talks, hacking on some code, learning, networking, even managed to sign up for the UCSF gym and to take a swim.

Very nice organization, excellent venue, and fantastic talks. Definitely coming back next year. For the price it’s well worth it.

Here are some highlights:

• Super useful resource for lookup up shell commands: http://shellhaters.heroku.com/posix
• Terminator plugin: start your dev environment as you like it. Gem install terminator
• http://github.com/rdy/fixture_builder Factory to Fixtures converter to speed up your tests.
• pprof profile Ruby interpreter. Rack-Profiler project, great profiling tool.
• Coffee Script – wrapper (ruby-esque) for javascript; rails 3.1 supports coffee script templates
• Machine Learning – great talk, and O’Reiley book, http://twitter.com/igrigorik
• minitest fastest testing framework, many mentions, very fast, supports RSpec and Test::Unit syntax
• Caching: using fresh_when(:last_modified => …) to enable proper HTTP caching in Rails 3.1
• Arel: enables fragment caching that does not run SQL if the fragment is cached
• ruby 1.9: require ‘objspace’ allows inspection of object counts and memory usage in VM

 

>Twitter is the slowest social network, Facebook the fastest

>

Interesting statistics just came out on AlertSite blog post about performance and response time of various highly trafficked social networks.

Unsurprisingly, Facebook is the fastest, and Twitter is the slowest. Facebook has so much more data to deal with, so much media, and so many more users, that it really is a shame for Twitter with it’s 140 character data set to be in this unglorious last place.

After continuous reports about migrating to Cassandra, Twitter still serves their tweets out of MemCaches that sit in front of MySQL. When those caches die, it takes a long time to refill them and the site yet again drops the very familiar Fail Whale.

Perhaps Twitter’s infrastructure group could use some new blood to work on scaling and up-time. I am guessing that adding experienced people who scaled Facebook, or other high traffic sites could really help in the long run. The problem is that hiring experienced contributors to work on this pain point may feel threatening to the existing crew who continually tried and ultimately failed to fix Twitter’s performance problems. Don’t fall into this trap Twitter: people who are having hard time re-engineering current architecture will continue to have a hard time without an influx of fresh energy and new ideas. Consider making the necessary changes, and then perhaps you could pull out of that miserly last place.

 

>Rails3, Rack and "Where did my Metal go?"

>

Our Rails3 (beta4) application had one route mapped in config/routes to a Sinatra app, by means of the following route:

   match '/foo', :to => EndPointApp, :as => :endpoint

The route was being defined to run as a Sinatra Application

require 'sinatra'
class EndPointApp < Sinatra::Application
  post '/foo' do
    ...
  end
end

This was working mostly fine, but it was returning Set-Cookie header with the standard Rails sessions cookie, which in this case was preventing the client of this endpoint from successfully interpreting the result. As I could do nothing about the client side, I had to remove Set-Cookie from the headers, but only for this end-point and obviously not from the entire app. This proved to be somewhat more complicated than I had hoped, so let me share the solution here in hopes it might save someone else an hour or two.

First, I ran “rake middleware” and observed the following Rack stack:

use ActionDispatch::Static
use Rack::Lock
use ActiveSupport::Cache::Strategy::LocalCache
use Rack::Runtime
use Rails::Rack::Logger
use ActionDispatch::ShowExceptions
use ActionDispatch::RemoteIp
use Rack::Sendfile
use ActionDispatch::Callbacks
use ActiveRecord::ConnectionAdapters::ConnectionManagement
use ActiveRecord::QueryCache
use ActionDispatch::Cookies
use ActionDispatch::Session::CookieStore
use ActionDispatch::Flash
use ActionDispatch::ParamsParser
use Rack::MethodOverride
use ActionDispatch::Head
run Kigster::Application.routes

As can be immediately seen from here, the routes execute very last after Session::CookieStore already wrapped the request. OK, so looks like I need to bypass the routes somehow, and so I started to look at Rails::Metal, which is supposed to run before all other processing.

Once I started to look for Rails::Metal, I realized pretty quickly that I am missing metal generator:

> rails g metal
Could not find generator metal.

After a few more rounds of digging around, it turns out that in Rails3 Beta4 Rails::Metal has been completely removed, because it is no longer needed in a Rack environment.

So I had convert my Sinatra module to a Rack module, and insert it into the Rack middleware stack before the Cookie/Sessions:

require File.expand_path('../../../config/environment',  __FILE__) unless defined?(Rails)

module Kigster
  class EndPoint
    def initialize(app)
      @app = app
    end
    def call(env)
      if env["PATH_INFO"] =~ /^foo/
        process_request(env)
      else
        @app.call(env)
      end
    end

    private
    def process_request(env) 
       req = Rack::Request.new(env)
       params = req.params
       # do stuff
       [ 200, 
         { "Content-Type"   => "text/html", 
           "Content-Length" => "0" }, 
         [""]   
       ]
    end
  end
end

I had add the following to my config/application.rb to enable this Rack module, and have it run before the ActionDispatch::Session::CookieStore:

# config/application.rb
# require the file directly
require File.join(File.dirname(__FILE__), '../app/metal/kigster_endpoint')
....
module Kigster
  class Application < Rails::Application
    config.middleware.insert_after Rails::Rack::Logger, 
                                   Kigster::EndPoint
  end
end

Now my handler executes before the session, and the result does not include Set-Cookie header.

Any other suggestions on how to make this any simpler, or more correct are as always welcome!

 

>Rails3 and The Inevitable Fragmentation

>

I remember one of the early talks at the Canada On Rails in 2006 had a slide showing how many books one needs to read to be able to write web applications in Rails, versus Java.

Of course Java side had about 10 books: Java SE, Java EE, Hibernate, Struts/Tiles/JSF, WebServices, Ant, Maven, Eclipse, JUnit, etc, etc.

The Rails slide proudly showed the now hopelessly outdated “Agile Web Development With Ruby on Rails”, 1st edition.

Those were the times. Back then, during my work for Blurb.com myself and three other engineers managed to learn ruby, rails and build a fully functional e-commerce site in about 3 months. I was blown away by the productivity gains compared to Java, where months could be spent laying out the project, and creating all necessary infrastructure for builds, deployment, automated testing with or without the database, etc.

Fast-forward to 2010. We are on a brink of Rails3 release, and oh boy, has the landscape changed since back then. I would argue that in some ways, you may be better off with Java than with Rails today for a couple of reasons:

• Rails moves very fast. Books published in 2009 are already outdated. That pretty much leaves blog posts and online documentation as the main source of information.
• Blog posts are not written as well as books are, they are not edited and are often heavily biased. They are also often outdated, and they are also often hard to find.
• Rails now supports swapping out view renderers (from Erb, to Haml for example), Javascript libraries (from Prototype to jQuery, etc), ORM layers.
• Testing frameworks are now (I think) approaching on a dozen. Forget Test::Unit, now it’s all about RSpec, RSpec-Rails, Shoulda, Cucumber, WebRat, Selenium, TDD, BDD, ShmoDD. Hot Damn.
• Finally, to make the matters “worse”, it’s become very trendy to replace RDBMS with a document-based databases such as MongoDB, CouchDB, or even a step further — with distributed data stores like Cassandra or Riak. As with any trend, people are doing this regardless of whether or not they actually need it. Although this part clearly has nothing to do with Rails.

As someone who is trying to “quickly” catch up on all the recent changes that happened in the last 18 months during which I did not do any major rails development, it feels like I have to learn a lot of things from scratch, and unlearn a lot of things too. Even though I had three-four years of production Rails experience building Blurb.com and Infectious.com. That makes me wonder, how the newbies feel now joining the “new” Rails ecosystem. I am very curious to find out. I am guessing there is a lot of frustration and a steep learning curve required to master all the frameworks in isolation, and then bring them all together into the project. And that brings us back to where Java was exactly four years ago, and still is today.

I am not at all advocating for you all to go back to coding Java or COBOL :=) And I am absolutely not against Rails. But it’s about time we pull the curtain of naiveté and realize that RoR has become the complex multifaceted enterprise framework it was claiming to replace back in 2006, and to become proficient in it may take just as long, if not longer, then with Java EE.

Ironic? I’ll let you decide.

 

>Learning Git, And Should You Switch from SVN

>If you are just learning to use git, or you have been using it for a while without too much thinking, this introduction to Git principles provides a fantastic overview into the concepts behind git, using a very simple and natural examples.

Do you need to switch to git from svn? My personal take on this is as follows:

• If you have several developers far away with bad internet connection, then YES.
• If you have more than several developers (say hundreds) then YES.
• If your developers often work on long multi-day features, where they want to commit often, but commits may result in instability of their branch, then YES. Git allows much easier branching than SVN.
• If you want to leverage GitHub’s infrastructure for hosting your project privately or publicly, then YES.

To balance this and not to appear as I am advocating everyone to switch, here is the reverse:

• If you have a small team who works locally and uses a local SVN server then NO.
• If your team does not need branching, or prefers to check-in complete features instead of incremental check-ins then NO.
• If your team is used to SVN and there are no major issues, then NO.
• If your team is using SVN authorization module to create groups and grant them special access per subdirectory then NO. I am unaware of Git providing this level of access control.

The actual tutorial is here: http://www.eecs.harvard.edu/~cduan/technical/git/

 

>Integer Division with Modulus in Ruby, using Linear and Binary Search

>

I was recently chatting with someone about algorithms, and we were talking about efficient algorithm for implementing integer division with modulus, and how to make it efficient for large integers.

The following code snippet shows a class that implements two division methods, linear and binary. I wonder if there is a more elegant way to implement binary, please feel free to post to comments if there are. Also, any other faster methods are welcome.

# divide.rb
# Integer Divider Class that implements two algorithms for finding the 
# division result and modulus of two integers.
class IntegerDivider
  def self.linear n, d
     raise 'Cant divide by zero!' if (d == 0)
     multiplier = n * d = n)
     i -= 1 if (i*d > n)
     return multiplier * i, n - i*d, i
  end

  def self.binary n, d
     raise 'Cant divide by zero!' if (d == 0) 
     multiplier = n * d = n) 
     return multiplier * i, 0 if (i*d == n) 
     i /= 2; j = i; cnt = 0
     begin
       j /= 2; cnt += 1
       sign = ((i + j)*d > n) ? 0 : 1
       i = i + sign * j 
     end until (i*d  n) 
     return multiplier * i, n - i*d, cnt
  end
  
  def self.divide(how, numerator, denominator)
    before = Time.now.to_f
    (result, remainder, iterations) = self.send(how, numerator, denominator)
    after = Time.now.to_f
    puts "#{sprintf('%8.3f',(after - before)*1000)}ms #{how}" + 
         " (#{sprintf '%10d', iterations} iterations):  #{numerator} / " +
         " #{denominator} = #{result}, mod #{remainder}"
    return [result, remainder]
  end
end

[:linear, :binary].each do |method|
  ARGV.each do |numerator|
    IntegerDivider.divide(method, numerator.to_i, 3)    
  end
end

And some results of running it. Of course for large number like 100000000, binary search takes 24 iterations to get the answer, while linear … well 100000000. The total speed difference at such large number is big: 1524ms/0.012ms = 127,000.

> ruby divide.rb 10 100 1000 20000 100000000
   0.006ms linear (         3 iterations):  10 /  3 = 3, mod 1
   0.003ms binary (         1 iterations):  10 /  3 = 3, mod 1
   0.003ms linear (        33 iterations):  100 /  3 = 33, mod 1
   0.003ms binary (         5 iterations):  100 /  3 = 33, mod 1
   0.017ms linear (       333 iterations):  1000 /  3 = 333, mod 1
   0.004ms binary (         8 iterations):  1000 /  3 = 333, mod 1
   0.411ms linear (      6666 iterations):  20000 /  3 = 6666, mod 2
   0.006ms binary (        11 iterations):  20000 /  3 = 6666, mod 2
1524.712ms linear (  33333333 iterations):  100000000 /  3 = 33333333, mod 1
   0.012ms binary (        24 iterations):  100000000 /  3 = 33333333, mod 1