The results and ramblings of research

phewww!

Improving Disk IO on Linux

with 2 comments

For the past few weeks I have been working on setting up a hadoop cluster on virtualized hardware (Not my first choice but we gotta work with what we get). After doing all the crappy backend end stuff (permission, partitions, etc.. etc..), I finally ran some benchmarks to test the cluster. Specifically, I used the TestDSIO function and got the following result:

—– TestDFSIO —– : write
Number of files: 10
Total MBytes processed: 5000
Throughput mb/sec: 0.8534698156744169
Average IO rate mb/sec: 0.9792065620422363
IO rate std deviation: 0.4777011765859518
Test exec time sec: 806.09

SHOOT… No way.. 1MBps IO write rate. 806 secs to write 5GB (ok it is 5GB * 3 and over the network and stuff) but seriously 1MBps sucks. The first step was to optimize the Linux OS to improve performance using a couple of things:

  1. Mount partitions using noatime attribute. Linux allows the ext2 file system where the last access time of individual files is not recorded. This effectively removes a extra write operation to disk.
  2. Edit /etc/fstab  and make changes similar to

    /dev/sda1    /    ext2    defaults,noatime  1  1
    
    Remount the partition
    $ mount -oremount /
    Check the partitions attributes by
    $ mount
  3. Edit default linux open file descriptor limit ( default is set to 1024) and make this larger:
  4. edit /etc/sysctl.conf and add the following lines
    fs.file-max = 64000
    fs.epoll.max_user_instances = 4096
    sysctl -p

  5. Edit /etc/security/limits.conf  and add the following lines
  6. * soft nofile 10000
    * hard nofile 64000

Now let us try and see what difference all these changes made:

—– TestDFSIO —– : write
Date & time: Thu Apr 07 19:16:59 UTC 2011
Number of files: 10
Total MBytes processed: 5000
Throughput mb/sec: 0.9220143653526179
Average IO rate mb/sec: 0.9504798650741577
IO rate std deviation: 0.17142934706209748
Test exec time sec: 705.386

Well, it doesnt seem like much but we removed 100secs overall. Second a slight improvement in throughput, a slight reduction in Average IO rates but a significant improvement in standard deviation for the IO rate. However, we are still way off. Back to the drawing board :(…

Advertisements

Written by anujjaiswal

April 7, 2011 at 8:44 pm

Posted in Hadoop, Linux

2 Responses

Subscribe to comments with RSS.

  1. fs.epoll.max_user_instances has been changed to fs.epoll.max_user_watches

    Sum DumGi

    February 27, 2013 at 9:20 pm


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s

%d bloggers like this: