RSS

Category Archives: Technology

>Twitter is the slowest social network, Facebook the fastest

>

Interesting statistics just came out on AlertSite blog post about performance and response time of various highly trafficked social networks.

Unsurprisingly, Facebook is the fastest, and Twitter is the slowest. Facebook has so much more data to deal with, so much media, and so many more users, that it really is a shame for Twitter with it’s 140 character data set to be in this unglorious last place.

After continuous reports about migrating to Cassandra, Twitter still serves their tweets out of MemCaches that sit in front of MySQL. When those caches die, it takes a long time to refill them and the site yet again drops the very familiar Fail Whale.

Perhaps Twitter’s infrastructure group could use some new blood to work on scaling and up-time. I am guessing that adding experienced people who scaled Facebook, or other high traffic sites could really help in the long run. The problem is that hiring experienced contributors to work on this pain point may feel threatening to the existing crew who continually tried and ultimately failed to fix Twitter’s performance problems. Don’t fall into this trap Twitter: people who are having hard time re-engineering current architecture will continue to have a hard time without an influx of fresh energy and new ideas. Consider making the necessary changes, and then perhaps you could pull out of that miserly last place.

Advertisements
 
1 Comment

Posted by on August 3, 2010 in Technology

 

>Rails3, Rack and "Where did my Metal go?"

>

Our Rails3 (beta4) application had one route mapped in config/routes to a Sinatra app, by means of the following route:

   match '/foo', :to => EndPointApp, :as => :endpoint

The route was being defined to run as a Sinatra Application

require 'sinatra'
class EndPointApp < Sinatra::Application
  post '/foo' do
    ...
  end
end

This was working mostly fine, but it was returning Set-Cookie header with the standard Rails sessions cookie, which in this case was preventing the client of this endpoint from successfully interpreting the result. As I could do nothing about the client side, I had to remove Set-Cookie from the headers, but only for this end-point and obviously not from the entire app. This proved to be somewhat more complicated than I had hoped, so let me share the solution here in hopes it might save someone else an hour or two.

First, I ran “rake middleware” and observed the following Rack stack:

use ActionDispatch::Static
use Rack::Lock
use ActiveSupport::Cache::Strategy::LocalCache
use Rack::Runtime
use Rails::Rack::Logger
use ActionDispatch::ShowExceptions
use ActionDispatch::RemoteIp
use Rack::Sendfile
use ActionDispatch::Callbacks
use ActiveRecord::ConnectionAdapters::ConnectionManagement
use ActiveRecord::QueryCache
use ActionDispatch::Cookies
use ActionDispatch::Session::CookieStore
use ActionDispatch::Flash
use ActionDispatch::ParamsParser
use Rack::MethodOverride
use ActionDispatch::Head
run Kigster::Application.routes

As can be immediately seen from here, the routes execute very last after Session::CookieStore already wrapped the request. OK, so looks like I need to bypass the routes somehow, and so I started to look at Rails::Metal, which is supposed to run before all other processing.

Once I started to look for Rails::Metal, I realized pretty quickly that I am missing metal generator:

> rails g metal
Could not find generator metal.

After a few more rounds of digging around, it turns out that in Rails3 Beta4 Rails::Metal has been completely removed, because it is no longer needed in a Rack environment.

So I had convert my Sinatra module to a Rack module, and insert it into the Rack middleware stack before the Cookie/Sessions:

require File.expand_path('../../../config/environment',  __FILE__) unless defined?(Rails)

module Kigster
  class EndPoint
    def initialize(app)
      @app = app
    end
    def call(env)
      if env["PATH_INFO"] =~ /^foo/
        process_request(env)
      else
        @app.call(env)
      end
    end

    private
    def process_request(env) 
       req = Rack::Request.new(env)
       params = req.params
       # do stuff
       [ 200, 
         { "Content-Type"   => "text/html", 
           "Content-Length" => "0" }, 
         [""]   
       ]
    end
  end
end

I had add the following to my config/application.rb to enable this Rack module, and have it run before the ActionDispatch::Session::CookieStore:

# config/application.rb
# require the file directly
require File.join(File.dirname(__FILE__), '../app/metal/kigster_endpoint')
....
module Kigster
  class Application < Rails::Application
    config.middleware.insert_after Rails::Rack::Logger, 
                                   Kigster::EndPoint
  end
end

Now my handler executes before the session, and the result does not include Set-Cookie header.

Any other suggestions on how to make this any simpler, or more correct are as always welcome!

 
1 Comment

Posted by on July 26, 2010 in Ruby on Rails, Technology

 

>Rails3 and The Inevitable Fragmentation

>

I remember one of the early talks at the Canada On Rails in 2006 had a slide showing how many books one needs to read to be able to write web applications in Rails, versus Java.

Of course Java side had about 10 books: Java SE, Java EE, Hibernate, Struts/Tiles/JSF, WebServices, Ant, Maven, Eclipse, JUnit, etc, etc.

The Rails slide proudly showed the now hopelessly outdated “Agile Web Development With Ruby on Rails”, 1st edition.

Those were the times. Back then, during my work for Blurb.com myself and three other engineers managed to learn ruby, rails and build a fully functional e-commerce site in about 3 months. I was blown away by the productivity gains compared to Java, where months could be spent laying out the project, and creating all necessary infrastructure for builds, deployment, automated testing with or without the database, etc.

Fast-forward to 2010. We are on a brink of Rails3 release, and oh boy, has the landscape changed since back then. I would argue that in some ways, you may be better off with Java than with Rails today for a couple of reasons:

  • Rails moves very fast. Books published in 2009 are already outdated. That pretty much leaves blog posts and online documentation as the main source of information.
  • Blog posts are not written as well as books are, they are not edited and are often heavily biased. They are also often outdated, and they are also often hard to find.
  • Rails now supports swapping out view renderers (from Erb, to Haml for example), Javascript libraries (from Prototype to jQuery, etc), ORM layers.
  • Testing frameworks are now (I think) approaching on a dozen. Forget Test::Unit, now it’s all about RSpec, RSpec-Rails, Shoulda, Cucumber, WebRat, Selenium, TDD, BDD, ShmoDD. Hot Damn.
  • Finally, to make the matters “worse”, it’s become very trendy to replace RDBMS with a document-based databases such as MongoDB, CouchDB, or even a step further — with distributed data stores like Cassandra or Riak. As with any trend, people are doing this regardless of whether or not they actually need it. Although this part clearly has nothing to do with Rails.

As someone who is trying to “quickly” catch up on all the recent changes that happened in the last 18 months during which I did not do any major rails development, it feels like I have to learn a lot of things from scratch, and unlearn a lot of things too. Even though I had three-four years of production Rails experience building Blurb.com and Infectious.com. That makes me wonder, how the newbies feel now joining the “new” Rails ecosystem. I am very curious to find out. I am guessing there is a lot of frustration and a steep learning curve required to master all the frameworks in isolation, and then bring them all together into the project. And that brings us back to where Java was exactly four years ago, and still is today.

I am not at all advocating for you all to go back to coding Java or COBOL :=) And I am absolutely not against Rails. But it’s about time we pull the curtain of naiveté and realize that RoR has become the complex multifaceted enterprise framework it was claiming to replace back in 2006, and to become proficient in it may take just as long, if not longer, then with Java EE.

Ironic? I’ll let you decide.

 
6 Comments

Posted by on July 8, 2010 in java, Ruby on Rails, Technology

 

>Integer Division with Modulus in Ruby, using Linear and Binary Search

>

I was recently chatting with someone about algorithms, and we were talking about efficient algorithm for implementing integer division with modulus, and how to make it efficient for large integers.

The following code snippet shows a class that implements two division methods, linear and binary. I wonder if there is a more elegant way to implement binary, please feel free to post to comments if there are. Also, any other faster methods are welcome.

# divide.rb
# Integer Divider Class that implements two algorithms for finding the 
# division result and modulus of two integers.
class IntegerDivider
  def self.linear n, d
     raise 'Cant divide by zero!' if (d == 0)
     multiplier = n * d = n)
     i -= 1 if (i*d > n)
     return multiplier * i, n - i*d, i
  end

  def self.binary n, d
     raise 'Cant divide by zero!' if (d == 0) 
     multiplier = n * d = n) 
     return multiplier * i, 0 if (i*d == n) 
     i /= 2; j = i; cnt = 0
     begin
       j /= 2; cnt += 1
       sign = ((i + j)*d > n) ? 0 : 1
       i = i + sign * j 
     end until (i*d  n) 
     return multiplier * i, n - i*d, cnt
  end
  
  def self.divide(how, numerator, denominator)
    before = Time.now.to_f
    (result, remainder, iterations) = self.send(how, numerator, denominator)
    after = Time.now.to_f
    puts "#{sprintf('%8.3f',(after - before)*1000)}ms #{how}" + 
         " (#{sprintf '%10d', iterations} iterations):  #{numerator} / " +
         " #{denominator} = #{result}, mod #{remainder}"
    return [result, remainder]
  end
end

[:linear, :binary].each do |method|
  ARGV.each do |numerator|
    IntegerDivider.divide(method, numerator.to_i, 3)    
  end
end

And some results of running it. Of course for large number like 100000000, binary search takes 24 iterations to get the answer, while linear … well 100000000. The total speed difference at such large number is big: 1524ms/0.012ms = 127,000.

> ruby divide.rb 10 100 1000 20000 100000000
   0.006ms linear (         3 iterations):  10 /  3 = 3, mod 1
   0.003ms binary (         1 iterations):  10 /  3 = 3, mod 1
   0.003ms linear (        33 iterations):  100 /  3 = 33, mod 1
   0.003ms binary (         5 iterations):  100 /  3 = 33, mod 1
   0.017ms linear (       333 iterations):  1000 /  3 = 333, mod 1
   0.004ms binary (         8 iterations):  1000 /  3 = 333, mod 1
   0.411ms linear (      6666 iterations):  20000 /  3 = 6666, mod 2
   0.006ms binary (        11 iterations):  20000 /  3 = 6666, mod 2
1524.712ms linear (  33333333 iterations):  100000000 /  3 = 33333333, mod 1
   0.012ms binary (        24 iterations):  100000000 /  3 = 33333333, mod 1
 
Leave a comment

Posted by on June 26, 2010 in Ruby on Rails, Technology

 

>Techtarget’s Java ServerSide Symposium 2009 Impressions

>

Just got back from Vegas where between Wednesday and Friday a small-ish java conference was taking place. This was my second one, the first being five years ago in 2004.

Of course it’s hard not to compare, and I find myself doing it anyway… I can’t help but feel that the excitement has left java server software. Remembering one of the very first java conferences in Sydney 1995, when it was still almost academic and less of an industry conference, I feel like we’ve come to an end of an era.

The tedious reality of maintaining enormous enterprise systems built over the the last decade has sinked in, and unfortunately there is nothing exciting about it, but everything difficult and complex. Those less fortune whose systems took advantage of EJB1 and EJB2s will suffer the most, as those are dying technologies and are rapidly being replaced by simpler more component friendly software.

The first set of simplification came with frameworks over the last five years – Spring and Hibernate being the most notable. Now the simplifications are being pushed out across language boundaries, because of the need to stay competitive against exploding Ruby on Rails community, and even smaller but still disproportionally more productive frameworks like Django, when compared to java.

Several talks focused on jRuby, and I think this is where the future power lies. Combining the two platforms, Ruby on Rails for web development, and server side java for multi-threaded, asynchronous, transactional behavior seems like a fantastic opportunity.

However nobody is yet talking about how this all works together in production. What does it mean to run a multi-app server distributed java application, with a RoR application hosted inside JVM and interacting with the service layer? What if service layer mixes ActiveRecord and Hibernate? Questions like this, while entirely logical in early evaluations of emerging technologies, were simply beyond the depth of TSSJS09.

But for what it’s worth, there was a decent overview of a multitude of scripting languages: javascript, ruby and scala, and a very impressive talk on Ruby meta programming.

There were several exciting news for me personally, the first having to do with Flex and BlazeDS integration in Spring 3.0. Basically what this means is that it’s now possible to build a typically modular and well understood server-side backend architecture in Spring, the client side in Flex, and have them directly converse over binary AMF protocol, which is highly compressed and optimized to send data back and force. I believe the entire stack is free, except Flex Builder Eclipse-based tool that’s not free. My exposure to Flex is minimal, but I love the ability to build beautiful application UIs that run consistently within the browser, and expose your backend in a way of clearly defined REST/XML api. And the load time in demos appeared significantly faster compared to java RMI applications I’ve had experience working with.

I’ll add part II in a couple of days.

 
1 Comment

Posted by on March 22, 2009 in Ruby on Rails, Technology

 

>Tell Your Mana About Obama, and Obama Car Art

>

Obama’s Potential Presidency

There is plenty of excitement around Barack Obama’s potential presidency, and I think there’s a lot to be excited about. There’s is an overwhelming feeling among people I talked to in San Francsisco, how Obama is “Our President” — not a poster boy for old money or a powerful family. The fact that the Internet fund-raising helped Obama to compete and win against the Clinton clan is a testament to the truly new era, a more democratic one, where if someone can fire up a lot of people they can rally up quckly and effectively. This efficiency is something to look forward to, if and when Obama becomes a President.

Whether or not all the people behind Obama will continue to support him, is another question. People tend to idealize their heroes way to quickly, and that’s also dangerous. The recent backlash of liberal bloggers about Obama’s compromise vote — is a great example. Politics is still politics, and nobody ever gets their way all the time. Everyone must compromise at some point.

But enough said, I think that overall Obama is a great candidate, and will hopefully bring a new fresh perspective into the Washington. So, how do we get there? Well, donating money is, for once, a very easy thing to do thanks to http://barackobama.com. But now there are other ways.

Car Art Contest

Infectious.com (the company I work for) is running an open contest for Car Art Submissions around Obama. There are some restrictions – some specific words can not be used, but it’s an amazing opportunity for artists around United States to create Obama Car Art that can be seen by thousands of people commuting daily in the next few months before the election. This is as grass roots as it gets.

If you know any artists that may be interested please send them to Infectious. Or you can grab the banner above and put it up on your blog. Either way – thank you!

 
Leave a comment

Posted by on July 17, 2008 in obama, politics, Technology

 

>Aptana – Please Don’t Suck Any Harder

><!–

I rarely rant about software. Reliable software is hard to make, and I know this because that’s what I do for living myself. But once in a while I come across such ignorance that a rant is really all I have left.

I’ve been coding in Ruby on Rails for over two years now, and have been using RadRails standalone, Eclipse with RadRails plugin, TextMate, IntelliJ IDEA and just good old plain Vim. But for the most part, I found myself very comfortable with RadRails because I liked the integrated environment for automated testing, outline views, project views, and all the goodies that are so nice in Eclipse, or IntelliJ IDEA if you work with Java.

Well, sometime back RadRails was abandoned by it’s young developers, who received a comfortable package from IBM, and the whole RadRails thingy somehow was inherited by Aptana, of which at that point nobody I talked to ever heard of.

What the heck is Aptana trying to do now, I still don’t quite truly understand, but I went along with the plan and switched. Instead of Eclipse, I had Aptana Studio, and for some time (while I assume Aptana’s contribution was merely the new name) things went well, and we all got along.

Well not anymore.

Aptana stopped being useful altogether with their most recent release, but things have been slowly creeping in even before that. The software now routinely spins at 100% cpu, the log file shows nothing useful, and a couple of versions back they broke Apple-Shift-R (although a later update fixed it), the command to pull up a resource by name after selecting a resource the whole thing would just crash.

So today I was like, “heck, let me reinstall the whole thing, maybe it’s my problem”? Huh. Not so fast.

Sure, I can download and install Aptana OK, but none of their bundled sweet plugins install at all. And what meaningful error message do I get when I try to install SubClipse? “Problem with your configuration”. And this is on a brand spanking new downloaded DMG. Sweet! Useful as a brick.

I am sorry, but the ignorance of Aptana Corp, who are now a proper company in San Mateo, trying to sell “Pro Version” as a worthwhile add-on while they can’t get their s**t together with the free community edition, just stinks to me of the worst kind of amateurism.

Thanks for ruining RadRails for me, Aptana Corp. At least it took a while to add enough of YOUR code to make things really break.
–>

I wrote this post after hours of frustration, trying to fix my broken environment, and losing valuable time on a project that was falling behind.

I now realize that the post was overly emotional, and probably not entirely fair to the Aptana team. I’ll post updates here as I work through the issues to get my RadRails working again.

 
3 Comments

Posted by on April 22, 2008 in Ruby on Rails, Technology