Accidental Technologist

Musings about Entrepreneurship, Technology and Software Development

  • Home
  • About
  • Still River Software
  • Privacy Policy

Powered by Genesis

Delete Large Numbers of Amazon S3 Files using Ruby

September 28, 2010 by Rob Bazinet 3 Comments

Tweet

I recently found I problem I needed to solve; remove hundreds of thousands of files from Amazon S3.  I mean, it had to be a common problem, right?  Well, it certainly be a common problem but the solution was less than common.

I tried a few tools available, both the tool from the Amazon S3 site but it keep erroring out and I was never sure why.  I then went to third-party tools used for managing S3 buckets but their either errored-out or behaved as if they worked but later determined did nothing.

I posted my need on Twitter and was pointed to a solution (thanks @Kishfy) I had not thought of, use Ruby.  There is a great open-source project named S3Nukem which its sole purpose is to remove Amazon S3 buckets.

S3Nukem

This is an open source project hosted on Github.   Installation and setup is pretty simple (from the Github repo readme), install required gems.

For Ruby >= 1.9:

sudo gem install dmarkow-right_aws --source http://gems.github.com

The docs don?t mention it but I needed to install the right_http_connection gem, the above command fails unless it is installed.

For Ruby < 1.9:

sudo gem install right_aws

Install S3Nukum:

curl -O http://github.com/lathanh/s3nukem/raw/master/s3nukem

Make it executable:

chmod 755 s3nukem

This is done in the directory where the above curl command was executed from.

Usage:

Usage: ./s3nukem [options] buckets...

Options:
    -a, --access ACCESS              Amazon Access Key (required)
    -s, --secret SECRET              Amazon Secret Key (required)
    -t, --threads COUNT              Number of simultaneous threads (default 10)
    -h, --help                       Show this message

Running the application in a terminal window shows large numbers of files being deleted:

s3nukem

This script is fast.  I tried running this under both Ruby 1.8.7 and 1.9.2 with 1.9.2 quite a bit faster.  I didn?t run any benchmarks but it was noticeably faster, my goal was really to just delete large amounts of files.  Ruby 1.9.2 thread handling really shines here and with the ability to control the number threads from the command line, is really nice.

The nice thing about this script version is the cap on the number of items to be deleted each time, 1000 * thread_count, which defaults to 10.  With this limit in place the script won?t chew up all your system memory. 

This script is designed to delete an entire bucket but could be modified to just remove the content or a directory tree within the bucket.  I may do this for a project I am working which has a need for such functionality.

Share this:

  • LinkedIn
  • Twitter
  • Facebook
  • Email
  • More
  • Pinterest
  • Tumblr
  • Pocket
  • Reddit

Filed Under: Ruby Tagged With: Amazon S3, Ruby, S3Nukem

Interviewed about IronRuby

March 29, 2010 by Rob Bazinet 6 Comments

Tweet

I was recently interviewed by Mike Pontacooloni of SearchWinDevelopment.com for an article about IronRuby and how it may be used by developers.  I was asked about IronRuby since I use both .NET, IronRuby and Ruby on Rails in my day-to-day work and Mike thought I might have some insight into how .NET developers would likely use IronRuby to supplement their .NET toolbox.

I think the article gives some good perspectives by a number of .NET and Ruby developers.

Mike and I had a discussion a few weeks ago and the article was published today.  Please give it a read.

Technorati Tags: Microsoft,IronRuby

Share this:

  • LinkedIn
  • Twitter
  • Facebook
  • Email
  • More
  • Pinterest
  • Tumblr
  • Pocket
  • Reddit

Filed Under: Ruby

Running Rails 2.3.x on IronRuby

February 19, 2010 by Rob Bazinet 6 Comments

Tweet

There have been some interesting discussions recently on Twitter about IronRuby and Rails.  The IronRuby team has been plodding along and with IronRuby 1.0 RC2 out I wanted to see if it could truly run a Rails application.

In case anyone does not know what IronRuby is, it is a project from Microsoft to bring Ruby into the .NET platform.

Installing IronRuby

Installation of IronRuby is pretty straightforward using the MSI installer from Microsoft available on Codeplex.  The installation is just like any Windows installer, follow the prompts and when complete IronRuby binaries should be in your path, something like C:\Program Files\IronRuby 0.9.4.0\bin.  Notice the version, 0.9.4, odd for a 1.0 RC2.  The installation says the same:

IronRubyRC2Install

I installed IronRuby and looking at the directory of the IronRuby binaries, I see all commands are preceded by an ?i?.  So for those familiar with Ruby-related commands we get igem, irake, iirb and etc, not a big deal but something that needs to be remembered.  There is not a Ruby.exe either, but rather an ir.exe which we use to execute commands.  Also note that RubyGems comes bundled with IronRuby, no need to install this ourselves, which is nice.

Installing Rails

Installing Rails is not hard but make sure IronRuby is in your path and install with the following command:

igem install rails

This installs all dependencies and documentation.  It is interesting to note that we aren?t left with the normal (if you are familiar with Rails on OS X or Linux) rails command but rather irails.  In order to find the installed Rails version we type:

irails ?v

We should see Rails 2.3.5 as a result.  I find it interesting Microsoft trying to create a Ruby which developers want to use, yet deviate the way command are run.  Maybe it will change in the official release.

Creating a Rails Application

Creating a Rails application under IronRuby is pretty much the same as on any platform with the exception of the ?i? in front of the command.  I decided to just create a throwaway application called ?testapp?, just to prove Rails worked.   The command:

irails testapp

Runs and throws up a bunch of files created.  No magic here, but found some interesting things when trying to run the application.

Running the Rails Application

The default database these days used by Rails 2.3.x is SQLite3 and use this most of the time when building applications since it just works and I don?t need to deal with MySQL or PostgreSQL.

I grabbed the latest sqlite3 binary and DLL?s from the SQLite web site where you can download precompiled binaries for Windows.  I put the contents of the two zip files in the IronRuby bin directory.  Any location in your path should work.

Normally we need to install a gem named ?sqlite3-ruby? which is our Ruby driver for the sqlite3 database.  Doing so under IronRuby results in a fine installation but a failure when accessing our development.sqlite3 database, creating in particular.  Running the command to create our development database:

irake db:create

Provides a nice error ?no driver for sqlite3 found?.  Well, this is an interesting problem I was happy to not have to search very long to resolve.  It seems some has forked the original sqlite3-ruby project on GitHub and created sqlite3-ironruby.  A quick:

igem install sqlite3-ironruby

Does the trick by retrying the irake db:create command and we are in business.  Our last step to prove Rails works with IronRuby is running our WEBrick server from within our ?testapp? directory:

ir script/server

The server starts and loads up our little test application.  Those of use used to running Ruby on most platforms are used to just using script/server without the ?ir? in front of it or maybe using the ?ruby? command with it, ?ir? is your ?ruby? and you need to precede your commands with it to ensure you have the the right Ruby binary.

If all things go well, popping open a browser and putting http://localhost:3000 in your address bar should give you a running Rails application:

IronRubyApp

Granted, it doesn?t do anything but it is a start and proves it loads.  I created some models and controllers and the ir script/generate commands worked like a charm.  All commands seemed to run as fast as, if not faster, than those in MRI 1.8.6. 

IronRuby has come a long way and we may see some production Rails applications running in IIS7 before too long.

Technorati Tags: IronRuby,Ruby,Ruby on Rails,Microsoft

Share this:

  • LinkedIn
  • Twitter
  • Facebook
  • Email
  • More
  • Pinterest
  • Tumblr
  • Pocket
  • Reddit

Filed Under: Ruby

Ruby on Rails 3 Beta Setup Hurdle

February 9, 2010 by Rob Bazinet 8 Comments

Tweet

The Ruby on Rails 3 beta was released recently and has been in development for the past year, longer if you take into account Merb.

I decided I wanted to start playing around with the release for a few reasons:

  • In order to see what issues developers may face moving to Rails 3 with existing applications.
  • I wanted to see the new architecture changes from Rails 2.x
  • I also wanted to see first-hand creating a simple application and all that is involved.

There are plenty of tutorials about how to setup Rails 3 and I won’t go into details here but wanted to share the method I used and one particular error I received and how I resolved it. Jeremy McAnally has some great coverage of using Rails 3, both new projects and upgrades. His rails_upgrade gem is a great place to start when beginning to look at moving an existing Rails project to Rails 3.

Setup

I decided to use the awesome Ruby Version Manager (RVM) to manage testing Rails 3 on different flavors of Ruby. The first thing to know is Rails 3 requires Ruby 1.8.7 and above, so no 1.8.6.

Ryan Bates has a great intro to setting up Rails 3 with RVM in his latest Railscast titled Rails 3 Beta and RVM.

I decided to use Ruby 1.8.7 for initial testing. There are a bunch of gem dependencies when working with Rails 3, as you might expect. There is a gem called rails3b which installs the necessary dependencies in one swoop. These two command should do the trick:

gem install rails3b
gem install arel --pre

After these commands finished and I closed and reopened my terminal prompt, a quick rails command gave me the following error:

/usr/local/lib/ruby/site_ruby/1.8/rubygems.rb:827:in `report_activate_error': Could not find RubyGem rails (>= 0) (Gem::LoadError)
from /usr/local/lib/ruby/site_ruby/1.8/rubygems.rb:261:in `activate'
from /usr/local/lib/ruby/site_ruby/1.8/rubygems.rb:68:in `gem'
from /usr/local/bin/rails:18

Some Googling did not return any usable results from anyone with this error. I really didn’t expect anything, things are just too new. Thinking back to another way to install Rails 3, I decided to try this instead:

gem install tzinfo builder memcache-client rack rack-test rack-mount erubis mail text-format thor bundler i18n
gem install rails --pre

After running these two gem commands the new rails command works as expected. I’m not really clear what the difference was which fixed the rails command, but I don’t know if I need to.

Do keep in mind the default database provider is Sqlite3 so you need to install that gem (sqlite3-ruby). Probably one of the biggest tips is to make sure you don’t use sudo when installing gems under RVM managed Ruby installation, just use gem install <gem name>.

UPDATE: Jeremy McAnally points out the rails3b gem installs the dependencies for the Rails 3 beta, not Rails itself.  This would certainly explain why my initial attempt failed and only worked after INSTALLING RAILS.

Technorati Tags: Rails 3,Ruby

Share this:

  • LinkedIn
  • Twitter
  • Facebook
  • Email
  • More
  • Pinterest
  • Tumblr
  • Pocket
  • Reddit

Filed Under: Ruby

Patching Ruby on Rails Refresher

September 4, 2009 by Rob Bazinet 4 Comments

Tweet

UPDATE: As one comment points out, it is not a good idea to directly patch production unless it was an emergency.  The normal state of patching would be to patch the source and make sure your tests still pass and likely to test on a staging environment.  The process to patch would still be the same as I describe below.

The recent XSS Vulnerability in Ruby on Rails discussed on the Ruby on Rails blog and discovered by Brian Mastenbrook, reminded me about patching my Rails applications which are running vendored.  If not doing this all the time, one can forget how it is done.

All of my production Rails applications keep a copy of Rails in the <application_root>/vendor/rails directory.  This just keeps me in check that I don’t upgrade Rails on a server and possibly break a production application.   The only minor drawback to this approach is when a patch is released, as with this XSS Vulnerability, you have to manually update the Rails installation for each application by hand.  If you don’t do it all the time, one can forget how it’s done.

Patching Rails

Getting and applying a patch is pretty simple.  The list of patches are listed as:

  • 2-0-CVE-2009-3009.patch – Patch for 2.0 series
  • 2-1-CVE-2009-3009.patch – Patch for 2.1 series
  • 2-2-CVE-2009-3009.patch – Patch for 2.2 series
  • 2-3-CVE-2009-3009.patch – Patch for 2.3 series

Follow 3 easy steps:

  1. cd <your_application_root>/vendor/rails
  2. wget http://weblog.rubyonrails.org/assets/2009/9/4/2-2-CVE-2009-3009.patch
  3. patch -p1 < 2-2-CVE-2009-3009.patch

You need to know the Rails version you are running vendored before you can determine which patch file to apply.  The simple way for me is to run script/about from <your_application_root>.  This displays the Rails version which you can then choose from the URL links above and replace the URL you need for the #2 wget command.

Depending on the security settings where the application is installed, the wget and patch may need to have a sudo inserted before the command.  I needed to do this my production servers but it would not be needed if patching a local project which would be later committed to source control.

Technorati Tags: Ruby on Rails,XSS Vulnerability,Patching

Share this:

  • LinkedIn
  • Twitter
  • Facebook
  • Email
  • More
  • Pinterest
  • Tumblr
  • Pocket
  • Reddit

Filed Under: Ruby

JRuby to live another day, moves to Engine Yard

July 28, 2009 by Rob Bazinet 2 Comments

Tweet

Update: Charles Nutter posts about the move on the Engine Yard blog.

Some interesting news hit my stream this morning, Sun’s JRuby team is leaving the company and moving to Engine Yard.  The announcement came from Computerworld:

Sun Microsystems’ JRuby team is leaving the company to work for application hosting company Engine Yard, citing the uncertainty surrounding Sun’s planned acquisition by Oracle.

Sun hired Charles Nutter and Thomas Enebo, often called "the JRuby Guys," about two-and-a-half years ago so they could work full time on JRuby, an implementation of the Ruby programming language for the Java Virtual Machine. Some months later Sun hired Nick Sieger, another key JRuby developer.

All three will start work at Engine Yard next week. Nutter said they decided to leave Sun largely because of the uncertainty resulting from its acquisition by Oracle, a deal that’s expected to close later this summer pending regulatory approvals.

This is such great news for JRuby, an alternative to the original MRI version of Ruby.  JRuby, in my opinion, is an important Ruby implementation which helps enterprise developers ease Ruby into companies heavily vested in Java technologies.

Although the announcement has not made it to the Engine Yard blog as of the time I am writing this, I am sure it will be there soon.  Engine Yard recently announced support for JRuby in the Engine Yard Cloud.

I personally was disappointed to learn that Oracle was buying Sun, my first thoughts were about future development of JRuby and MySQL.  At least this should help clear up the future of one of those open source projects.

Technorati Tags: JRuby,Engine Yard,Sun

Share this:

  • LinkedIn
  • Twitter
  • Facebook
  • Email
  • More
  • Pinterest
  • Tumblr
  • Pocket
  • Reddit

Filed Under: Ruby

Programmatically Creating Website Thumbnails

July 27, 2009 by Rob Bazinet 44 Comments

Tweet

I have been working recently on a project which I am capturing links to web pages and need to display a thumbnail-size screen shot of the web page being linked.  For example, if I have a link to the Ruby on Rails web site, the thumbnail I need would look like this:

rubyonrails_thumb

This particular one is 202×152 pixels, just the size I need.

I was really surprised by the level of difficulty I had trying to find such a solution and searching the web led to very little.  It appears Amazon had such a solution at one time under the Alexa umbrella called Alex Site Thumbnail but they shut the service down a while ago.  There happened to be a very nice Ruby interface to the Alexa service as well.

I ended up coming across a couple viable and workable solutions, Websnapr and Bluga.net.

Websnapr

It is a web service which is pretty simple to implement and in its simplest form can just be referenced from an img tag from a web page.  My implementation uses Ruby on Rails but it’s not needed to see a very simple example.  Taking the URL I want a screen image of from above:

<img src=”http://images.websnapr.com/?size=S&key=mywebsnaprkey&url=rubyonrails.org” />

The web service will allow for snapshots to be made without the API key but limits to 80 requests per hour, which is reasonable.  The service also seems to cache previously requested snapshots taken from other users.   The normal process when requesting a snapshot is a temporary image is returned while the request is in the queue, but if there is already one available it is returned immediately. 

My tests have shown the resulting image being returned in less than a minute.

Bluga.net WebThumb

The other thumbnail service under consideration uses a different approach to the submission and retrieval of images, both thumbnail and full-size website snapshots.  Their approach is much more flexible but can be bit more complex. 

WebThumb offers more features and quicker response times then any other service.

  • Real-time thumbnails
  • Flash 10 Support
  • PDF Support
  • Quick response times
  • REST API
  • API clients for PHP, Ruby, Python
  • Cache the thumbnails on your server or Webthumbs
  • Browser windows from 75×75 to 1280×2048

Bluga offers a gem hosted on Github for Ruby developers to interface to their web service, simply wrapping the API calls in some nice objects.

One problem I saw with simply using the gem was when requesting a snapshot that it might not be ready right away and would through an exception, which we need to code.  Not a huge problem but to be aware of.  Peter Cooper posted some code at DZone which shows his implementation using the Bluga service:

require 'net/http'
require 'rubygems'
require 'xmlsimple'
class Nailer
@@api_baseurl = 'http://webthumb.bluga.net/api.php'
@@api_key = 'PUT YOUR API KEY HERE'
attr_accessor :collection_time, :job_id, :ok
def initialize(url, width = 1024, height = 768)
api_request = %Q{<webthumb><apikey>#{@@api_key}</apikey><request><url>#{url}</url><width>#{width}</width><height>#{height}</height></request></webthumb>}
result = do_request(api_request)
if result.class == Net::HTTPOK
result_data = XmlSimple.xml_in(result.body)
@job_id = result_data['jobs'].first['job'].first['content']
@collection_time = Time.now.to_i + result_data['jobs'].first['job'].first['estimate'].to_i
@ok = true
else
@ok = false
end
end
def retrieve(size = :small)
api_request = %Q{<webthumb><apikey>#{@@api_key}</apikey><fetch><job>#{@job_id}</job><size>#{size.to_s}</size></fetch></webthumb>}
result = do_request(api_request)
result.body
end
def retrieve_to_file(filename, size = :small)
File.new(filename, 'w+').write(retrieve(size.to_s))
end
def ready?
return unless Time.now.to_i >= @collection_time
api_request = %Q{<webthumb><apikey>#{@@api_key}</apikey><status><job>#{@job_id}</job></status></webthumb>}
result = do_request(api_request)
if result.class == Net::HTTPOK
@ok = true
result_data = XmlSimple.xml_in(result.body)
begin
@result_url = result_data['jobStatus'].first['status'].first['pickup']
@completion_time = result_data['jobStatus'].first['status'].first['completionTime']
rescue
@collection_time += 60
return false
end
else
@ok = false
end
true
end
def ok?
@ok == true
end
def wait_until_ready
sleep 1 until ready?
end
private
def do_request(body)
api_url = URI.parse(@@api_baseurl)
request = Net::HTTP::Post.new(api_url.path)
request.body = body
Net::HTTP.new(api_url.host, api_url.port).start {|h| h.request(request) }
end
end
url = 'http://www.rubyinside.com/'
t = Nailer.new(url)
if t.ok?
t.wait_until_ready
t.retrieve_to_file('out1.jpg', :small)
t.retrieve_to_file('out2.jpg', :medium)
t.retrieve_to_file('out3.jpg', :medium2)
t.retrieve_to_file('out4.jpg', :large)
puts "Thumbnails saved"
else
puts "Error"
end

 

You can see from the example there are some things to take into consideration which might lead taking longer to implement.   Bluga does offer an Easythumb API which looks very similar to Websnapr but gives the option of cached and non-cached snapshots, which may likely be important.

As I indicated Bluga has some good support for developers including a well-documented API, code samples in multiple languages and article references so you can see how others are using the service to solve their problems.  Bluga is also hosted on EC2 so the uptime should be really good.

Finally

The solution I am currently using for this project works well but relies on a third-party service, which I would like to avoid.  If someone has a solution, preferably in Ruby natively or Ruby interfacing to some Linux libraries or tools, I would really like to hear about it.

For now the solution with Websnapr works very well, although not instantaneous, is pretty fast.  I good part of this approach is not having to store images and the drawback being I am not storing images.  If Websnapr goes down, no images are displayed.

After finding these two solutions I came across some others that might do the job but haven’t had a reason to investigate; Shrink the Web, Girafa, PageGlimpse and scURLr.

Most of the services have a component you need to pay for beyond an initial free hits to the service.   In the long-term Bluga seems to be a really good solution because of the flexibility and being host on Amazon EC2 but for a short-term solution it is hard to be Websnapr.

Technorati Tags: Ruby,Thumbnail,websnapr,bluga

Share this:

  • LinkedIn
  • Twitter
  • Facebook
  • Email
  • More
  • Pinterest
  • Tumblr
  • Pocket
  • Reddit

Filed Under: Ruby

Reflections on Ruby for Projects

June 22, 2009 by Rob Bazinet 2 Comments

Tweet

Martin Fowler recently published an article recently looking back at projects which ThoughtWorks chose to use Ruby as the programming language.

The article reminded me of path I have traveled, coming from a background in C/C++ but moving to using ASP.NET/C# and Ruby on Rails for various projects.  Running a small software consultancy I don’t always have the opportunity to choose the platform or language to use, the decision is often made by the technology the company has standardized on, which could be Windows or Linux.

Many of Martin’s points ring true in what I have found working on projects in Ruby versus other languages.

It’s often the case that projects we take on have very small teams, sometimes 2-3 developers, sometimes 1.  When teams are small it can be hard to feel like huge leaps in progress are being made.  The tools and frameworks we use at Still River Software, often are what makes the difference.

Over the years of creating websites for clients we initially used ASP.NET and C# as our tool of choice because of its broad support and my personal experience.  This has changed over the past couple years with us moving projects to Ruby on Rails.  As Martin puts it:

When people are asked about why Ruby should be used on a project, the most common answer is for increased productivity. One early indicator was an assessment of a project that suggested that Ruby would have yielded an order of magnitude improvement in productivity.

As a result it seemed obvious to survey the project technical leads and ask them about productivity – had ruby increased productivity and if so, by how much. I asked them to compare this to a mainstream (Java or .NET) project done in the most productive way they knew how.

rubyProductivityBar

You should take these result with some salt. After all there is no way we can objectively measure software productivity. These are just the subjective, qualitative assessments from the technical lead of each project. (I didn’t get a response from all projects.) However they are still suggestive that there’s a real productivity boost going on.

The results, on average, were 2X improvements in productivity.  As Martin points out these results are subjective and are hardly scientific evidence but the results are interesting all the same.

Personal Results

The most interesting result to me from Martin’s summation is the productivity of the teams.  We all know the arguments of Ruby’s speed and other such topics, I won’t get into those.

We do find development using Ruby and in particular, Ruby on Rails, to be more productive but I won’t try to put the gains into a number.  Comparing development using Ruby on Rails vs. ASP.NET some particular aspects stand out in a big way.  We can go from idea to prototype in a matter of days using Rails compared to a much longer time using ASP.NET.  A complete change in direction is much easier and a lot less wasted effort.

Decision Making – the Ruby on Rails (RoR) framework is referred to “opinionated” and for good reason.  Many of the decisions that need to be made when using ASP.NET (including ASP.NET MVC) are made for us in RoR.  When a new Rails application is created there is almost nothing that has to be decided in order to start creating an application right now.  Using ASP.NET, decisions are aplenty, from how the application will be architected to how the database will be accessed (i.e. ADO.NET, LINQ, NHibernate).  Using Rails for the most part assumes what the architecture and data store will look like over the life of the application.

This is not a knock on ASP.NET.  ASP.NET allows the developers and architects wide open flexibility as to how their applications will be structured.  I have spent many meetings discussing how this new ASP.NET application will be organized and structured.  Even though I have had a lot of experience in this aspect of system building, it seem the same topics are addressed with every project.  These same issues are not talked about using Rails and this saves a considerable amount of time not only in planning but in implementation.

Plugin and Gem Support – There is a huge amount of Ruby Gems and Rails Plugins available for just about any aspect of a Rails project that we may need to build, such as authentication or integration with a credit card processing facility.  All are open source and most actively developed and supported.

ASP.NET has little support for this, even though the community is growing the ability to easily grab third-party open tools is very limited.

Deployment – the deployment story on Rails is pretty much one-sided using Capistrano.  Capistrano is a great tool to allow developers to create scripts in Ruby to manage all deployment aspects with a simple command.  The script takes a bit to create but a wide range of examples are easily available.

Deploying an ASP.NET application has always been painful and way too manual for my taste, maybe I am doing it wrong.  If there existed a tool like Capistrano I would have used it.

Finally

The parts of Rails which make us more productive starts from the decisions made for us to the parts available to us to build our systems.  These neatly package components bundle functionality so we can simply drop in place. 

This not a Ruby on Rails vs. ASP.NET argument and I don’t want to turn it into one, so please..no comments saying I am wrong and ASP.NET is the only thing that matters.  We still have clients using ASP.NET and we enhance their software and support them.  They will not be moving to Rails, ever.  The point of the little post here was just reflecting on some of the areas we have been more productive with Ruby and why I feel this is the case.  Your mileage may vary.

Martin’s article raises some very good and healthy points as it stands today.  The conversation may change in 5 years, who knows, but today the story is very clear to us, using Ruby on Rails we can get results to our clients faster.  Faster to convey intent, faster to iterate and faster to deploy, all for less money because we are spending less time on things that shouldn’t matter.

Technorati Tags: Ruby,Ruby on Rails,ThoughtWorks

Share this:

  • LinkedIn
  • Twitter
  • Facebook
  • Email
  • More
  • Pinterest
  • Tumblr
  • Pocket
  • Reddit

Filed Under: Ruby

Ruby Denial of Service (DoS) Vulnerability Options

June 10, 2009 by Rob Bazinet 2 Comments

Tweet

A potentially harmful vulnerability was found in all pre-Ruby 1.9.1 version of Ruby.  The issue was reported on the Ruby-lang web site and says it effects:

1.8 series

  • 1.8.6-p368 and all prior versions
  • 1.8.7-p160 and all prior versions

Ruby 1.9.1 does not suffer from this problem but seems apparent JRuby does have this issue and has been tested to prove it.

The Problem

The issue is pretty simple to reproduce.  BigDecimal, when asked to parse an overly large number, causes segmentation faults.  The following will reveal the problem:

BigDecimal("9E69999999").to_s("F")

Solutions

There are quite a few solutions to this problem and most involve patching an existing installation of Ruby.  The Ruby web site offers links to download Ruby source for both 1.8.6 and 1.8.7:

Please upgrade to 1.8.6-p369 or ruby-1.8.7-p173.

  • ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.6-p369.tar.gz
  • ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.7-p173.tar.gz

There are various things to consider when doing an update like this and building from source.  I am an Ubuntu user and installed Ruby with the aptitude package manager so I could easily update to a later version of Ruby when it was released.  The problem here is the package sources won?t be updated right away and I will be stuck with the vulnerability until new packages are created and put out.  I probably won?t be effected by this issue as I have not yet but I would rather be safe, especially for my customers.

Having used a package manager, Ruby is installed in:

/usr/bin

The options at this point is to download and build Ruby from source.  Building Ruby from the Ruby website will by default but Ruby in:

/usr/local/bin

I won?t get into installing Ruby from source here, there are many good tutorials including one from FiveRuns, Compiling Ruby, RubyGems and Rails on Ubuntu.  You can use this tutorial for either of your flavors of Ruby.  Keep in mind you may need to adjust the PATH setting in your bash script to point to the right Ruby instance.  The post above states Ubuntu already has this set already, a quick check of my PATH and it indeed has this set.

One additional option that seems particularly attractive is Ruby Enterprise Edition (REE) from Phusion, the Passenger folks.  This group keeps on top of Ruby updates and is constantly updating their code.  They posted the DoS vulnerability fix almost right away and offer the source download or an Ubuntu package for 32 and 64-bit.

This is the option I am taking.  I have been wanting to move to REE for some time and I think it now the right time.  REE does by default install to the:

/opt/ruby_enterprise

directory, so a change to the bash environment is necessary.  The Phusion site has some good docs on how to install and configure REE:

Then follow the instructions that the installer gives you.

1.6. Configuring REE as the default Ruby interpreter

It is possible to configure REE as the default Ruby interpreter, so that when you type ruby, gem, irb, rake or other Ruby commands, REE?s version is invoked instead of the system Ruby?s version.

To do this, you must add REE?s bin directory to the beginning of the PATH environment variable. This environment variable specifies the command shell?s command search path. For example, you can do this on the command-line:

$ ruby some_program.rb    # <--- some_program.rb is being run
#      in the system Ruby interpreter.
$ export PATH=/opt/ruby-enterprise-X.X.X/bin:$PATH
$ ruby some_program.rb    # <--- some_program.rb will now be run in REE!

Invoking export PATH=... on the command-line has no permanent effect: its effects disappear as soon as you exit the shell. To make the effect permanent, add an entry to the file/etc/environment instead. On Ubuntu Linux, /etc/environment looks like this:

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games"
LANG="en_US.UTF-8"
LANGUAGE="en_US:en"

Add REE?s bin directory to the PATH environment variable, like this:

PATH="/opt/ruby-enterprise-x.x.x/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games"
LANG="en_US.UTF-8"
LANGUAGE="en_US:en"

This will give me both better memory usage and the added security of fixing BigDecimal.

This does effect Ruby on Rails and there is a patch to Rails as a quick workaround.

Conclusion

It is great to have options and open source gives us the ability to have many.  If building from source is the way to go then keep in mind all of the RubyGems already installed in the system(s) need to be installed again for that new version of Ruby.  This only applies if the Ruby path is changed to which Ruby is being used.

Technorati Tags: Ruby,Ruby Enterprise Edition,REE,JRuby,Rails,DoS Vulnerability

Share this:

  • LinkedIn
  • Twitter
  • Facebook
  • Email
  • More
  • Pinterest
  • Tumblr
  • Pocket
  • Reddit

Filed Under: Ruby

Strange Restful_Authentication Plugin Issue

May 15, 2009 by Rob Bazinet 12 Comments

Tweet

I was working late last night trying to get a new user interface design implemented on one of my Ruby on Rails applications while at the same time getting an initial authentication bit done using Rick Olson?s awesome restful_authentication plugin.

I had some migration updates so I can the usual:

cap deploy:migrations

During the stream of text output by Capistrano I received an error:

uninitialized constant User::Authentication

My initial thought was that I must have missed some configuration setting on my server, but after looking around for a bit I could not find what I might have left off. The error indicated with was something to do with the restful_authentication plugin but testing in both development and production worked fine (of course).

Some searching around the web revealed a few suggestions ranging from making sure the restful_authentication directory contained an underscore and not a hyphen to various Rails versioning issues.  This application was working just fine on Rails 2.2.2 in development so I was confident it was not the issue.

I finally came upon a post by Brent Collier of Intridea who had the same issue as I experienced.  I had installed this plugin as I do with other plugins hosted on Github with the command script/plugin install git://github.com/technoweenie/restful-authentication.git.  The plugin installed this way is a local git repository from the one stored on Github.  Taken from Brent?s blog, the way the restful_authentication plugin appears in the Github repository was the same as was appearing in mine:

github-brent

The solution I needed to do was a bit different than the one described by Brent but close.

  1. Started off by deleting the vendor/plugin/restful_authentication directory manually.
  2. Did a commit/push of my project to Github and verified the folder was gone in the Github repo.
  3. I downloaded the tarball of the restful_authentication plugin from Github. 
  4. Once extracted, I copied the folder to vendor/plugins and named the folder restful_authentication.
  5. Did a commit/push again to Github and looked at the folder in the repo to see if it looked like all the others.  Indeed it did.
  6. Finally, ran cap deploy:migrations and all was good.

This was a very strange exercise and I would really like to know why this happened and the proper procedure to manage plugins which are a git repo themselves.  If anyone has an explanation and fix, I will update this post to reflect it.

Technorati Tags: restful_authentication,Rails,Plugin,GitHub

Share this:

  • LinkedIn
  • Twitter
  • Facebook
  • Email
  • More
  • Pinterest
  • Tumblr
  • Pocket
  • Reddit

Filed Under: Ruby

« Previous Page
Next Page »

Recent Posts

  • How to Fix Rails Flash Rendering When Using Hotwire
  • Hotwire Fix for CORS Error when using Omniauth
  • Fix Installation of Ruby using rbenv on macOS Big Sur
  • RailsConf 2021 and the Future of Conferences
  • Fixing Out of Diskspace Errors on Amazon EC2

Categories

Services I Love

HatchBox - Easy Rails Deploys Fathom Analytics
Follow @rbazinet

Rob Bazinet
@rbazinet

  • Exactly this…. https://t.co/yWj7fZ01HR
    about 18 hours ago
  • https://t.co/EmagdpLoNv "Introducing GitHub Copilot X · GitHub"
    about 1 day ago
  • RIP Gordon Moore: https://t.co/c5J9LaHrj8
    about 1 day ago
  • Our daughter works as a teaching assistant at a local K-8 public school. It’s become apparent that students today r… https://t.co/p2t912GVyc
    about 2 days ago
  • Working to wrap up my current consulting gig by mid-April. I will be looking for the next thing soon. If anyone has… https://t.co/sg3cSV9yqM
    about 3 days ago
  • RSS - Posts
  • RSS - Comments
Find me on Mastodon