Tim Dysinger

Importing Enron into CouchDB

2009-02-10T06:02:44-10:00

I have been goofing around with couchdb for about a year now. In order to do anything fun or interesting with it, you first must have some data to play with. To solve this I imported the enron email dataset into couchdb so we can have a couple hundred thousand documents.

How? First I downloaded all enron data from the Carnagie Melon University’s Enron Email Data. Then I used the ‘mail trends’ project’s enron.py code to convert the loose files into a unix mbox format so it’s easily understood by code. Once we have the enron data in a format we like, we can use a ruby script below to take the email and push it into couchdb. (Make sure your couchdb installed and running.) The code is as follows:

cat >;rakefile.rb <<\THEEND
%w(time tmail find restclient json).each {|l| require l}

file_create(‘enron_mail_030204.tar.gz’) do
  `curl -O http://download.srv.cs.cmu.edu/~enron/enron_mail_030204.tar.gz`
end

file_create(‘maildir’ => ‘enron_mail_030204.tar.gz’) do
  `tar xzof enron_mail_030204.tar.gz`
end

desc(‘import the email to localhost couchdb’)
task(:import => ‘maildir’) do
  RestClient.put(‘http://localhost:5984/enron’, ”) rescue nil
  Find.find(‘maildir’) do |path|
    next if FileTest.directory?(path)
    begin
      txt = IO.read(path)
      msg = TMail::Mail.parse(txt)
      next if msg.date < @t = Time.parse("1999-01-01")
      attrs = msg.header.merge(‘to’ => msg.to_addrs,
                               ‘cc’ => msg.cc_addrs,
                               ‘bcc’ => msg.bcc_addrs,
                               ‘body’ => msg.body).reject {k,v v.to_s.empty?}
      RestClient.post(‘http://localhost:5984/enron’,
                      attrs.to_json,
                      :content_type => ‘application/json’)
    rescue Interrupt
      exit(1)
    rescue Exception => ex
      puts "#{path} #{ex.inspect}"
    end
  end
end
THEEND

sudo gem install rake rest-client json tmail
rake -T
rake import
# …..wait for it…..
rake irb

This will take a while. Not long after the script starts you will see documents showing up in your couchdb. You will see a couple dozen emails are not properly formatted or that wont convert to json but you’ll still end up with most of the emails in your couchdb. Navigate to Couchdb’s Futon and start mappin’ and reducin’ :)

Using Amazon EC2 Metadata as a Simple DNS

2008-10-13T11:42:25-10:00

I use the amazon metadata for creating /etc/hosts and do this on a cron schedule. This does everything I need. Instead of fancy DynDNS tricks or having to run and manage an internal DNS server I just have a ruby script that looks at the metadata ec2 to build /etc/hosts. It’s easy. To set it up yourself and try it all you need are 3 easy steps.

Start each of your instances with unique named key that matches what you want their internal hostname to be. Such as “onion” or “potato” or whatever you want to call them.

Make sure you have ruby, rubygems and amazon-ec2 (rubygem) installed. Then create a ruby script in /usr/local/sbin/hosts that has the following:

#!/usr/bin/env ruby
%w(optparse rubygems EC2 resolv pp).each {|l| require l}
options = {}
parser = OptionParser.new do |p|
  p.banner = "Usage: hosts [options]"
  p.on("-a", "–access-key USER", "The user’s AWS access key ID.") do |aki|
    options[:access_key_id] = aki
  end
  p.on("-s",
       "–secret-key PASSWORD",
       "The user’s AWS secret access key.") do |sak|
    options[:secret_access_key] = sak
  end
  p.on_tail("-h", "–help", "Show this message") {
    puts(p)
    exit
  }
  p.parse!(ARGV) rescue puts(p)
end
if options.key?(:access_key_id) and options.key?(:secret_access_key)
  puts "127.0.0.1 localhost"
  EC2::Base.new(options).describe_instances.reservationSet.item.each do |r|
    r.instancesSet.item.each do |i|
      if i.instanceState.name =~ /running/
        puts(Resolv::DNS.new.getaddress(i.privateDnsName).to_s +
             " #{i.keyName}.ec2 #{i.keyName}")
      end
    end
  end
else
  puts(parser)
  exit(1)
end

Setup a cron job to update /etc/hosts as often as you like. I do it once per hour on all my machines

0 * * * * /usr/local/sbin/hosts -a myaccess -s mysecret >/etc/hosts

All my machines have this ec2 security key + script + cron approach. I do not have to run dyndns or any private dns servers to keep track of all my internal server ip addresses. My /etc/hosts looks like the following on the three machines in the test cluster:

127.0.0.1 localhost
10.252.202.221 oahu.ec2 oahu
10.253.115.175 maui.ec2 maui
10.253.114.190 hawaii.ec2 hawaii

Rack: An API for Web Servers and Ruby Frameworks

2008-04-25T14:11:58-10:00

In today’s ruby web application landscape, every framework developer is writing his/her own handlers for every server he/she wants to support. This results in semi-duplicate code, if not for the web-server developer then for the framework-developer. This is the pain-point that Rack aims to solve. Rack proposes “why not have some common ground?” Java did this with the Servlet API 10 years ago. Python did this with WSGI 5 years ago.

By leveraging Rack, framework developers and web-server developers gain access to one another without having to write special adapters. Today that’s WEBrick, Mongrel, CGI, Ebb, Fuzed & Thin for web-servers and Rails, Camping, Coset, Halcyon, Maveric, Merb, Racktools::SimpleApplication, Ramaze, Sinatra & Vintage for web-frameworks (this list will undoubtably be outdated soon). Tomorrow every new web-server and web-framework that supports Rack can be used together. You’ll be able to pick and choose the best web-server for you without changing your favorite web-framework and vice-versa.

“What do Rails developers really stand to gain today by leveraging Rack?” It might be the ability to run several “rackable” applications side by side inside a single web-server instance. It might be the possibility to leverage or stack applications. You can intercept requests, modify them and pass them through to other handlers. You can also have multiple rackable applications sitting next to each other that comprise one user-facing application. Don’t like file uploads with Rails? Use another web framework or the Rack API directly to write it and place it along side your Rails app in the same application. Want to use single-sign-on for 3 Rails apps? No problem. Rack makes it easy to tie apps together.

Rack Hello World

( gem install rack & mongrel & first and then after firing up the example visit localhost )

%w(rubygems rack).each {|l| require l}
Rack::Handler::Mongrel.run(
  lambda {|x| [301, {‘Location’ => ‘http://rubyurl.com/g6L’},”]},
  :Port => 3000
)

Karma Yoga in Software Engineering

2008-04-22T17:22:48-10:00

I work in software development and it is a very competitive business. At times I have to catch myself, when I feel an emotion, and ask myself “Why?”. Why am I being competitive? Why am I seeking recognition? Why am I wanting control? Is my argument on the design the best for the team? Are my motivations the best for the project?

In reading about Karma Yoga, I realize that this is exactly what software developers need to do when writing software. Karma Yoga means “discipline of action” and is based on the teachings of the Bhagwat Geeta, a sacred Sanskrit scripture of Hinduism.

Karma Yoga is described as a way of acting, thinking and willing by which one does one’s duty without consideration of personal selfish desires, likes or dislikes. Acting without being attached to the fruits of one’s deeds. In software this is doing what needs to be done for the betterment of the project and team without attaching your ego and self-worth to the code you write or the contribution you make.

When this mindset is taken on by software developers, collaboration, camaraderie and team-work increases while tensions, egos, stress, competition and caustic attitudes decrease. It has to be consciously chosen, but this is something to be strived for on teams.

Creating Blank Git Branches

2008-03-31T16:43:29-10:00

Most of the time in git you will be creating branches of your main project and working on them. What if you wanted to create a git headless branch called ‘documentation’? It doesn’t really deserve it’s own repository because it’s so closely related. The git project itself does this with documentation. The git project repository has separate branches for master, docs and man pages etc too. Here’s how you do it.

Go into your git project and type

git symbolic-ref HEAD refs/heads/empty
touch .gitignore
git add .gitignore
git commit -m ‘Initial headless branch commit’

That’s it - now you have a new branch ‘empty’.

Creating the Perfect Gentoo Amazon EC2 AMI (image)

2008-03-04T07:40:09-10:00

Update: I need to upgrade this for amazon ec2 2008-02-01 api.

I been playing with Gentoo again. I hadn’t been an active Gentoo user since it pissed me off in a emerge -u world snafu in 2004. I created some Gentoo EC2 images and thought I would share with you all.

I have recently stopped using Xen to create new images and started using Amazon EC2 AMIs to create new AMIs directly – “dog food”-style. The script below is an example of this. There is no need to have 32 & 64-bit Xen Dom0 machines around the house to get started creating custom AMIs. All you need is an Amazon EC2 account. Just fire up someone else’s Linux image and go to work creating a new AMI. I have been using Amazon’s Fedora 4 “developer” 32-bit “small” image to create a nice lean Gentoo image. Here is my script.

# Boot a developer image at EC2 && Login as root on the instance

# Move the /tmp dir to the big drive
mv /tmp /mnt && ln -sf /mnt/tmp /

# Bootstrap
mkdir /mnt/gentoo
wget -O - \
  http://gentoo.osuosl.org/releases/x86/current/stages/stage3-i686-2007.0.tar.bz2  \
  tar xjC /mnt/gentoo
wget -O - http://gentoo.osuosl.org/snapshots/portage-latest.tar.bz2  \
  tar xjC /mnt/gentoo/usr
wget -O - http://s3.amazonaws.com/ec2-downloads/linux-2.6.16-ec2.tgz  \
  tar xzC /mnt/gentoo/usr/src
zcat /proc/config >/mnt/gentoo/usr/src/linux-`uname -r`/.config

# FUSE module (has to be compiled with the same gcc as ec2’s kernel)
cd /tmp
wget -O - \
  http://superb-west.dl.sourceforge.net/sourceforge/fuse/fuse-2.7.3.tar.gz  \
  tar xz
cd fuse-2.7.3
./configure –enable-kernel-module \
  –with-kernel=/mnt/gentoo/usr/src/linux-`uname -r`
cd kernel
make && make install
mkdir -p /mnt/gentoo/lib/modules/`uname -r`
cp -r /lib/modules/`uname -r` /mnt/gentoo/lib/modules/`uname -r`

# Setup
cat /proc/mounts >/mnt/gentoo/etc/mtab
mount -o rbind /proc /mnt/gentoo/proc
mount -o rbind /dev /mnt/gentoo/dev
mount -o rbind /sys /mnt/gentoo/sys
cp /etc/resolv.conf /mnt/gentoo/etc

# Chroot
chroot /mnt/gentoo /bin/bash
env-update
source /etc/profile
export PS1="(image) $PS1"

# Modules / Kernel
depmod -a
modprobe loop
echo ‘loop’ >>/etc/modules.autoload.d/kernel-2.6
echo ‘fuse’ >>/etc/modules.autoload.d/kernel-2.6
cd /usr/src && ln -sf linux-`uname -r` linux

# Cleanup
cd /
rm -rf tmp && ln -sf var/tmp tmp
rm -rf opt && ln -sf usr/local opt
rm -rf boot

# Root
usermod -p \
  `dd if=/dev/urandom count=50 2> /dev/null  md5sum  cut -d " " -f1-1` \
  root

# Rebuild
cat >/etc/make.conf <<\EOF
CFLAGS="-O2 -march=i686 -pipe -mno-tls-direct-seg-refs"
CXXFLAGS="${CFLAGS}"
CHOST="i686-pc-linux-gnu"
MAKEOPTS="-j2"
EOF
emerge –sync
emerge -e world
emerge –update –newuse –deep world ; # are these both needed ^ <-
etc-update
emerge eix gentoolkit
emerge –depclean
revdep-rebuild

# Locale
cat >/etc/locale.gen <<\EOF
en_US ISO-8859-1
en_US.UTF-8 UTF-8
EOF
locale-gen

# Timezone
cp /usr/share/zoneinfo/GMT /etc/localtime
cat >>/etc/conf.d/clock <<\EOF
TIMEZONE="GMT"
EOF

# Mounts
cat >/etc/fstab <<\EOF
/dev/sda1 /        ext3  user_xattr          0 1
/dev/sda2 /mnt     ext3  user_xattr          0 2
/dev/sda3 swap     swap  sw                  0 0
shm       /dev/shm tmpfs nodev,nosuid,noexec 0 0
EOF

# TTY
perl -p -i -e ‘s/^c([^1])/\#c$1/g’ /etc/inittab

# Network
emerge dhcpcd ddclient net-misc/ntp
rc-update add net.eth0 default
rc-update add sshd default
rc-update add ntpd default
cat >/etc/ssh/sshd_config <<\EOF
Protocol 2
StrictModes yes
MaxStartups 10:30:60
Ciphers aes256-cbc,aes256-ctr
PasswordAuthentication no
ChallengeResponseAuthentication no
Subsystem sftp /usr/lib/misc/sftp-server
UseDNS no
EOF

# Boot
cat >/etc/conf.d/local.start <<\EOF
# /etc/conf.d/local.start
# Root SSH Public Key
[ ! -e /root ] && cp -r /etc/skel /root
wget –timeout 15 -q -O - \
  http://169.254.169.254/2007-12-15/meta-data/public-keys/0/openssh-key >\
  /root/.ssh/authorized_keys
chmod -R go-rwsx /root
# Userdata Shell Script
wget –timeout 15 -q -O - http://169.254.169.254/2007-12-15/user-data  sh
EOF

# EC2 tools
emerge ruby curl unzip symlinks
cd /tmp
wget http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.zip
cd /usr/local
unzip /tmp/ec2-ami-tools.zip
ln -sf ec2* ec2-ami-tools
chmod -R go-rwsx ec2*
rm -rf /tmp/ec2*
# Recompile rsync (lutimes doesn’t work with old ec2 kernel)
cd /tmp
wget -O - http://www.samba.org/ftp/rsync/src/rsync-2.6.9.tar.gz \
  tar xz
cd rsync-2.6.9
perl -pi.bak -e ‘s/\blutimes\b//’ ./configure
./configure –prefix=/usr/local/ec2-ami-tools
make
make install
cd ..
rm -rf rsync*

# Bundle
export AMAZON_USER_ID=’FIXME put your user id here’
export AMAZON_ACCESS_KEY_ID=’FIXME put your access key here’
export AMAZON_SECRET_ACCESS_KEY=’FIXME put your secret access key here’
cat >/mnt/pk.pem <<\EOF
—–BEGIN PRIVATE KEY—–
FIXME: put your cert here
—–END PRIVATE KEY—–
EOF
cat >/mnt/cert.pem <<\EOF
—–BEGIN CERTIFICATE—–
FIXME: put your cert here
—–END CERTIFICATE—–
EOF
export EC2_PRIVATE_KEY=/mnt/pk.pem
export EC2_CERT=/mnt/cert.pem

cat >/usr/local/sbin/image <<\EOF
#!/bin/bash
export EC2_AMITOOL_HOME=/usr/local/ec2-ami-tools
PATH=$EC2_AMITOOL_HOME/bin:$PATH
BUNDLE=`date ‘+%y%m%d%H%M%S’`
ec2-bundle-vol -r i386 -u $AMAZON_USER_ID \
  -k $EC2_PRIVATE_KEY -c $EC2_CERT \
  -b -d /mnt -s 10000 –fstab /etc/fstab \
  -e /root -p $BUNDLE
ec2-upload-bundle -b $HOSTNAME -m /mnt/$BUNDLE.manifest.xml \
  -a $AMAZON_ACCESS_KEY_ID -s $AMAZON_SECRET_ACCESS_KEY
rm -rf /mnt/$BUNDLE* /mnt/img-mnt
EOF
chmod 700 /usr/local/sbin/image

export HOSTNAME=gentoo-i686
rm -rf /var/tmp/* /usr/portage/distfiles /usr/portage/packages
symlinks -crsdv /
image

# Register & make the ami public (on another machine)
ec2-register $HOSTNAME/$BUNDLE.manifest.xml
ec2-modify-image-attribute ami-xxxxxx –launch-permission -a all

# Below is an example of a boot script that you might pass in as
# "userdata" You would configure the hostname and dyndns and/or
# maybe puppet or cfengine

#!/bin/bash
# Hostname
echo ‘HOSTNAME="fqdn.example.com"’ >/etc/conf.d/hostname
/etc/init.d/hostname restart
echo ‘127.0.0.1 ‘`hostname -f`’ ‘`hostname -s`’ localhost’ >/etc/hosts
echo ‘search ‘`hostname -d` >/etc/resolv.conf
echo ‘nameserver 172.16.0.23’ >>/etc/resolv.conf
echo ‘dhcp_eth0="release nodns nontp nonis"’ >/etc/conf.d/net
/etc/init.d/net.eth0 restart
# DynDNS
cat >/etc/ddclient/ddclient.conf <<\EOF
daemon=300
syslog=yes
mail=root
mail-failure=root
ssl=yes
use=web, web=169.254.169.254/2007-12-15/meta-data/public-ipv4
protocol=dyndns2, server=members.dyndns.org, custom=yes, \
login=FIXME, password=FIXME \
EOF
hostname >>/etc/ddclient/ddclient.conf
/etc/init.d/ddclient start
rc-update add ddclient default
fi

# After the new instance is booted, you may want to login and
# configure some basic tools or whatever

# Extras Tools
cat >>/etc/portage/package.keywords <<\EOF
dev-util/git
sys-fs/encfs
sys-fs/fuse
sys-fs/sshfs-fuse
EOF
emerge dev-util/git
emerge sys-fs/fuse sys-fs/encfs sys-fs/sshfs-fuse

Using Ruby to Control Lego Mindstorms NXT

2007-12-29T08:38:28-10:00

Playing with my son on Lego NXT requires me to get Ruby in the mix just for fun. Here is the install notes that I used to get everything going.

cd /tmp

# Ruby Serial
svn export http://ruby-serialport.rubyforge.org/svn/trunk ruby-serial
cd ruby-serial
ruby extconf.rb
make
sudo make install
cd ..

# Install libusb
svn export https://libusb.svn.sourceforge.net/svnroot/libusb/trunk/libusb libusb
cd libusb
sh autogen.sh
./configure
make
sudo make install
cd ..

# Ruby USB
svn export svn://svn@svn.a-k-r.org/akr/ruby-usb/trunk ruby-usb
cd ruby-usb
ruby extconf.rb
make
sudo make install
cd ..

# Ruby NXT
gem install ruby-nxt

# Try it in IRB
require ‘rubygems’
require ‘nxt_comm’
comm = NXTComm.new
comm.connected?
comm.get_device_info
comm.get_firmware_version

Mounting Remote Servers as a Drive on OS X with Mac FUSE and SSHFS

2007-12-15T14:11:53-10:00

A handy tip for all you Mac OS X users out there: Have servers to deal with over SSH? You can download and install MacFUSE and SSHFS. Once you have installed both, you can fire up the SSHFS app. SSHFS can mount a remote server as a local drive on your mac. Then you can edit files in place and drag and drop files to transfer securely to the remote server.

It’s pretty cool. Thanks to the FUSE team for writing it. Thanks to Google for porting it to the Mac MacFUSE

Happy 60th Birthday, Dad!

2007-12-02T20:01:15-10:00

Just wanted to acknowledge my father as he turns 60 today. I really admire you and all of your accomplishments. You have always taken the time to talk with me and be a mad-scientist-mentor to me. My first gunpowder-bombs, tool-boxes, fishing-poles, camping-gear, snow-sleds, mini-bikes, tree-forts, computers and other fun stuffs came from you, dad. Thanks for teaching me that moving forward requires persistence and perspiration. Thanks for encouraging me on my way with many a computer-gift and a pat on the back for encouragement.

I love you, dad.

Ruby Lasagna

2007-10-20T10:36:35-10:00

You have heard of Spaghetti code if you’ve done procedural or scripting-based programming. A similar thing can happen in Ruby with it’s polymorphism and the super-dynamic behavior. Ruby can become a frustrating “Lasagna” sometimes with all these dynamic classes, dynamic instances and dynamic behavior. All this can be hard to debug and follow. It’s almost like too many lisp macros.

Chad Fowler recently wrote “I love the tricks you can do with Ruby. method_missing, const_missing, autoloading, and their friends make really powerful things possible. But they do so at a price. When something goes wrong in a piece of code that relies heavily on one of these tricks, it can be much much harder to track down. So the decision to use such a tool shouldn’t be taken lightly. These are power tools. Used effectively, really cool things can happen. Used incorrectly, you can easily find yourself limb-less and bloody. So when you decide to use one of these power tools, you have to ask yourself: is it worth the risk?”

Try and Buy

2007-10-16T11:34:30-10:00

In working with my new hires, I have found the try-and-buy works really well for the company. 50% of developers don’t make it during the trial period. It’s really important to try out new developers on a paid trial for a while. There is no interview long enough to make a decision to hire. Have them submit patches to your code base, don’t give them direct source access for the first week of the trial period. Review their patches. See how they respond to criticism. Watch how they interact with the team to solve problems.

You may think 50%!? That’s high. Not really. Think about it: 80-90% of all businesses fail in the first 5 years. The start-up failure rate is most likely higher on “the internets.” Just as much as it’s important to try out your employees before you hire them, it’s important for potential employees or consultants to try out potential employers or clients before making any commitments.

Lesson: You can’t change people for the most part. “They are who they are” starts the day you meet them. Pay close attention to warnings and your gut. If it’s not a fit, let them go in the first 2 weeks. Don’t wait a month or a year. Life is too short to work with unqualified people.

Turn off your mac’s disk hibernation in OS X

2007-09-07T09:52:50-10:00

OS X has this deep-sleep mode that will save all the contents of Ram to disk. They do this just in-case you forget to charge it and the battery drains. However this causes a long hibernation time (the time measure from the time you close the lid to the time you see the “breath” light and it’s safe to move the laptop). During this long hibernation time, you MUST NOT move or especially jerk the laptop (like putting it in your bag). If you do jerk you laptop around at any time the hard disk is running, you can damage the hard disk. It is a moving-at-high-velocity part of your computer.

If you are like me, there isn’t but a few hours that go buy before I am using my laptop again. The battery-draining-all-the-way situation is almost never going to happen to me. I am always backed up and charged. The laptop is my life-blood and my primary tool for my income.

So, rather than have the thing take for ever to sleep for a feature I won’t use, I decide to discover a way to turn it off. The result is the script. Open a terminal and type the following:

sudo sh -c "\
  pmset -a hibernatemode 0 ;\
  nvram use-nvramrc?=false ;\
  rm /var/vm/sleepimage >/dev/null 2>&1 ;\
"

Sending email messages via SMS to mobile phones

2007-08-30T09:16:23-10:00

I find it incredibly simple and handy to send application alerts (and sometimes friendly notes) via email to an SMS-enabled phone. Most of the big carriers have an email to SMS gateway. Here are the email addresses of the top 6 carriers.

T-Mobile

phonenumber@tmomail.net

Virgin Mobile

phonenumber@vmobl.com

AT&T

phonenumber@mmode.com

Sprint

phonenumber@messaging.sprintpcs.com

Verizon

phonenumber@vtext.com

Nextel

phonenumber@messaging.nextel.com

Compiling Fuse kernel module for Debian 4.0 (Etch) on EC2

2007-07-28T10:08:07-10:00

I run Debian 4.0 (Etch) images at EC2 that I created on Xen and bundled to EC2. I discovered while trying to compile a kernel module for FUSE that Debian has GCC 4.1 while all of EC2’s kernels were built with GCC 4.0. This is a problem in that the kernel module, even though it compiles fine, will not insert into the EC2 kernel.

This was a bit of a road block for me as Debian Sarge has GCC 3.3/4 and Etch only has 4.1. I had no access to 4.0. Luckily Amazon provides their stock Fedora images with developer tools. I personally hate RPM-based Linux distributions and Amazon’s Fedora images are broken and won’t ‘yum update’. However, Amazon’s ‘developer’ image at least has GCC 4.0 on it along with the kernel source. With that you can compile a binary kernel module and tar it up for use elsewhere (on your nice clean Debian images).

Here are the steps I took to compile Fuse Kernel module on Amazon’s Developer Fedora image and move it to Debian 4.0. Although this is tailored for compiling the Fuse kernel module the same steps will work for any other kernel module.

# DESKTOP: Fire up an EC2 fedora "developer" instance
ec2-run-instances -g sandbox -k sandbox ami-26b6534f

# DESKTOP: Login
ssh -i ~/.amazon/id_rsa-sandbox \
  root@ec2-72-44-51-242.z-1.compute-1.amazonaws.com

# EC2: Remove old fuse
rpm -e `rpm -qa  grep fuse`

# EC2: Install FUSE module
cd /usr/local/src
curl -O http://superb-west.dl.sourceforge.net/sourceforge/fuse/fuse-2.6.5.tar.gz
tar xzf fuse-2.6.5.tar.gz
cd fuse-2.6.5
./configure –enable-kernel-module –with-kernel=/usr/src/linux-`uname -r`
cd kernel
make && make install

# EC2: Package up the modules
tar -czf modules-`uname -r`.tgz /lib/modules/`uname -r`

# DESKTOP: Pull the modules off and stash them somewhere to use with your ‘real’ linux distro
cd ~/Projects/sysadmin/files
scp -i ~/.amazon/id_rsa-sandbox
  root@ec2-72-44-51-242.z-1.compute-1.amazonaws.com:/modules-2.6.16-xenU.tgz .
svn commit . -m ‘Updated kernel modules for ec2’</pre></code>

Migrating your virtual Debian 4.0 (Etch) from Xen to EC2

2007-07-28T09:55:11-10:00

If you have a linux server with Xen installed, you can develop and test on Xen and migrate your images to EC2 easily when you are ready.

First off you need to have Linux running in a Xen image. This was covered earlier in a post about installing Xen on Debian Hosts & Guests Xen Virtulization with Debian Linux 4 (It’s easy)

Once you have a running image on Xen, migrating to EC2 is fairly easy. There are just a few steps that you need to follow. I’ve outlined them below using Debian 4.0 as the distribution but you can choose your own flavors.

# HOST: (Re)Install & Correct (for Debian) Amazon’s EC2 AMI tools & Ruby 1.8.6
sudo su -
apt-get install alien curl
curl -O http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.noarch.rpm
alien -i ec2-ami-tools.noarch.rpm
rm ec2-ami-tools.noarch.rpm
# IF YOU DO _NOT_ HAVE RUBY 1.8.6 installed from source
apt-get install ruby libopenssl-ruby1.8
ln -s /usr/lib/site_ruby/aes /usr/lib/ruby/1.8/
# IF YOU DO HAVE RUBY 1.8.6 installed from source
ln -s /usr/lib/site_ruby/aes /usr/local/lib/ruby/1.8/
# THEN
vi /usr/lib/site_ruby/aes/amiutil/image.rb
# exec( ‘for i in console null zero ; do /sbin/MAKEDEV -d ’ + dev_dir + ’ -x $
#       i ; done’ )
# ……Should be……
# exec("cd #{dev_dir} && /sbin/MAKEDEV console && /sbin/MAKEDEV std && /sbin/MAKEDEV generic")
exit

# HOST: (Re)Install Amazon’s EC2 API tools
sudo apt-get install sun-java5-jre unzip
cd /usr/local
sudo rm -rf ec2*
sudo symlinks -crsd .
curl -O http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
unzip ec2-api-tools.zip
rm ec2*/bin/*.cmd
chmod -R og-w ec2*
sudo chown -R debian:staff ec2*
rm ec2-api-tools.zip
ln -sf ec2-api-tools* ec2-api-tools
cd /usr/local/bin
ln -sf ../ec2-api-tools/bin/* .

# Download your ec2 certificate and put it in ~/.amazon/cert.pem
# Download your ec2 private key and put it in ~/.amazon/pk.pem
# Put your AWS Access key in ~/.amazon/access.txt
# Put your AWS Secret key in ~/.amazon/secret.txt
# Put your AWS Account # in ~/.amazon/user.txt
cat >> ~/.profile << EOF
export JAVA_HOME=/usr
export EC2_HOME=/usr/local/ec2-api-tools
export EC2_PRIVATE_KEY=$HOME/.amazon/pk.pem
export EC2_CERT=$HOME/.amazon/cert.pem
export AMAZON_ACCESS_KEY_ID=`cat ~/.amazon/access.txt`
export AMAZON_SECRET_ACCESS_KEY=`cat ~/.amazon/secret.txt`
export PATH=$EC2_HOME/bin:$PATH
EOF
source ~/.profile
ec2-add-keypair sandbox # put key in ~/.amazon/id_rsa-sandbox
ec2-add-keypair development # put key in ~/.amazon/id_rsa-development
ec2-add-keypair test # put key in ~/.amazon/id_rsa-test
ec2-add-keypair production # put key in ~/.amazon/id_rsa-production
chomod -R go-rwsx ~/.amazon

#
# When are ready to deploy the image
#

# GUEST: Create an Amazon EC2 fstab
sudo su -
cd /etc
mv fstab fstab.xen
ln -sf fstab.xen fstab
cat > /etc/fstab.ec2 << EOF
/dev/sda1 /     ext3  defaults             1 1
/dev/sda2 /mnt  ext3  defaults,user_xattr  1 2
/dev/sda3 swap  swap  defaults             0 0
none      /proc proc  defaults             0 0
none      /sys  sysfs defaults             0 0
EOF
exit

# GUEST: Install Amazon’s Linux Kernel modules on your image
sudo su -
cd /tmp
wget http://s3.amazonaws.com/ec2-downloads/modules-2.6.16-ec2.tgz
tar -xzf /tmp/files/modules-2.6.16-xenU.tgz
rm -rf /tmp/files
exit

# HOST: Stop the Xen image && mount the disk && switch to EC2 fstab
sudo xm shutdown image; # Wait for it to stop
sudo mount -t ext3 -o loop /xen/domains/image/disk.img /mnt
cd /mnt/etc
sudo ln -sf fstab.ec2 fstab
cd
sudo umount /mnt

#
# Setup EC2
#

# Add Groups
ec2-add-group sandbox  -d ‘Sandbox’
ec2-add-group website  -d ‘Website’
ec2-add-group database -d ‘Database’
ec2-add-group backend  -d ‘Backend’

# Open Ports
ec2-authorize sandbox -p 22
ec2-authorize website -p 80
ec2-authorize website -p 443
ec2-authorize default -p 54321

#
# Deploy to EC2
#

# Bundle Image
ec2-bundle-image -p image -i /xen/domains/image/disk.img
  -u `cat ~/.amazon/user.txt`

# Upload Image
ec2-upload-bundle -b ami.soniannetworks.com -m /tmp/image.manifest.xml
  -a `cat ~/.amazon/access.txt` -s `cat ~/.amazon/secret.txt`

# Register Image
ec2-register ami.soniannetworks.com/image.manifest.xml

# List Images
ec2-describe-images
ec2-describe-images -x all

# Run Image
ec2-run-instances -g sandbox -k sandbox ami-XXXXXXX

# List
ec2-describe-instances

# Login
ssh -i ~/.amazon/id_rsa-sandbox root@myhost.amazonaws.com

# Shutdown
ec2-terminate-instances i-XXXXXXX

# Unregister
ec2-deregister ami-XXXXXXX

#
# Post-Deploy Config @ EC2
#

# EC2: SSH in (as debian user)
ssh -p 54321 debian@my.host.at.ec2.amazonws.com
# EC2: Change password (as debian user)
# EC2: Reconfigure the new guests hostname
sudo su -
hostname myhost.ec2.mydomain.com
echo `hostname` > /etc/hostname
exit
# GUEST: Reconfigure mail (see xen.txt doc)
# GUEST: Configure services as needed (see xen.txt doc)

#
# Switching the Xen image back to work with Xen
#

# HOST: Mount the disk && switch to Xen fstab && start the Xen image again
sudo mount -t ext3 -o loop /xen/domains/image/disk.img /mnt
cd /mnt/etc
sudo ln -sf fstab.xen fstab
cd
sudo umount /mnt
sudo xm create /etc/xen/image.cfg
sudo xm console image

Xen Virtulization with Debian Linux 4 (its easy)

2007-06-14T07:54:17-10:00

Install Debian 4 just like you normally would. Choose a minimal install with no tasksel features (No Desktop - No Standard). Your finished install OS from this point forward will be referred to as the Host while virtual images will be referred to as Guests (below). I’m just going to tear into the steps that follow a basic Debian 4 install. They are as follows:

# Installing Debian 4 & Xen 3
#
# NOTE: if your HOST is on qemu add ‘-redir tcp:2222::22’ to the parameters
#
# HOST  = The base Debian Xen install OS that’s hosts guests
# GUEST = The VMs you are within the Debian Xen HOST
#
# During install add a user called ‘debian’ for non-root access
#

# HOST: Update
cat > /etc/apt/sources.list << \EOF
deb http://debian.osuosl.org/debian/ etch main contrib non-free
deb http://security.debian.org/ etch/updates main contrib non-free
EOF
apt-get update && apt-get upgrade

# HOST: Tools
apt-get install pwgen deborphan symlinks less

# HOST: Mail
apt-get install exim4 mailx
echo root: tim@dysinger.net > /etc/aliases && newaliases
dpkg-reconfigure exim4-config ; # Choose "Internet Site"
date  mail -s test root@localhost ; # should see email @ tim@dysinger.net

# HOST: SSH
apt-get install openssh-server
mv /etc/ssh/sshd_config /etc/ssh/sshd_config.bak
cat > /etc/ssh/sshd_config << \EOF
Protocol 2
Port 54321
PermitRootLogin no
PasswordAuthentication no
UseDNS no
Subsystem sftp /usr/lib/openssh/sftp-server
EOF
rm /etc/ssh/*key*
dpkg-reconfigure openssh-server
# DESKTOP: Create SSH keys (if you don’t have them already)
ssh-keygen -t dsa ; # just [Return] at all prompts
# HOST: Then paste the content of the DESKTOP ~/.ssh/id_dsa.pub
# into HOST /home/debian/.ssh/authorized_keys
su - debian
mkdir ~/.ssh
cat >> ~/.ssh/authorized_keys << \EOF
}}}
EOF
chmod -R go-rwsx ~/.ssh
exit
# DESKTOP: Try the ssh login before you trust everything’s OK
ssh -p 54321 debian@my.servers.host.name.com ; # should require no password

# HOST: Xen
apt-get install linux-image-2.6-xen-686 linux-headers-2.6-xen-686 \
xen-hypervisor-3.0.3-1-i386-pae xen-linux-system-2.6.18-4-xen-686 \
xen-ioemu-3.0.3-1 xen-tools libc6-xen bridge-utils
cat >> /etc/modules << \EOF
loop max_loop=64
EOF
cat > /etc/xen-tools/xen-tools.conf << \EOF
dir         = /xen
mirror      = http://ftp.us.debian.org/debian/
kernel      = /boot/vmlinuz-2.6.18-4-xen-686
initrd      = /boot/initrd.img-2.6.18-4-xen-686
debootstrap = 1
dist        = etch
image       = full
size        = 4Gb
memory      = 512Mb
swap        = 512Mb
dhcp        = 1
EOF
cat > /etc/xen/xend-config.sxp << \EOF
(network-script network-bridge)
(network-script network-dummy)
(vif-script vif-bridge)
(dom0-min-mem 196)
(dom0-cpus 0)
EOF
mkdir /xen

# HOST: Dhcp
apt-get install dhcp3-server
cat > /etc/dhcp3/dhcpd.conf << \EOF
default-lease-time 600;
max-lease-time 7200;
option subnet-mask 255.255.255.248;
option broadcast-address 66.199.242.207;
option routers 66.199.242.201;
option domain-name-servers 4.2.2.1, 72.29.96.250, 207.210.212.202;
option domain-name "mysite.com";
subnet 66.199.242.200 netmask 255.255.255.248 {
  range 66.199.242.203 66.199.242.206;
}
host myserver {
  option host-name "myserver.mysite.com";
  hardware ethernet 00:16:3E:18:CF:B3;
  fixed-address 66.199.242.206;
}
EOF

# HOST: Sudo
apt-get install sudo
visudo ; # make sure the bottom line has ‘debian  ALL=(ALL) ALL’

# HOST: Cleanup
reboot ; # after reboot - login as ‘debian’ (NOT ROOT EVER AGAIN ON THIS HOST!)
sudo uname -r ; # running xen kernel ?
sudo dpkg –purge linux-image-2.6-686 linux-image-2.6.18-4-686

# HOST: Harden
# Read http://www.debian.org/doc/manuals/securing-debian-howto/
sudo apt-get install nmap
sudo nmap -p 1-65535 -T4 -sS `hostname` ; # have a look at your open ports
sudo apt-get install harden harden-clients harden-tools tiger chkrootkit lsof

# HOST: Firewall ( leave open eth0 port 54321 & add xenbr0 w/o NAT )
sudo apt-get install arno-iptables-firewall

# HOST: Guest
sudo xen-create-image –hostname image
sudo xm create /etc/xen/image.cfg
sudo xm list
sudo xm console image ; # Ctrl-] is exit

# GUEST: Prepare
passwd root ; # Change the password!!!
echo 127.0.0.1 localhost localhost.localdomian > /etc/hosts
adduser debian
usermod -G staff,src debian

# GUEST: Setup
#   1. Update (same as above)
#   2. Tools (same as above)
#   3. Mail (same as above)
#   4. Ssh (same as above)
#   5. Sudo (same as above)
#   6. Harden (same as above)
#   7. Firewall ( same as above but leave open eth0 ports 80 443 54321 )

# HOST: Cloning GUESTS

# HOST: Create a new Xen image (xen-create above)
# HOST: Copy the ‘image’ disk to the clone
#   ‘image’ must be shut down then
#     (cp /xen/domians/image/disk.img /xen/domains/mynewdomain/ )
# HOST: Boot your new xen guest (xm create above)
# GUEST: Change password (as debian user)
# GUEST: Reconfigure the new guests hostname
sudo su -
hostname dev.ec2.sonianarchive.com
echo `hostname` > /etc/hostname
exit
# GUEST: Reconfigure mail (above)
# GUEST: Configure services as needed (as debian user)
sudo invoke-rc.d nginx stop && sudo update-rc.d -f nginx remove ; # example
sudo invoke-rc.d mongrel stop && sudo update-rc.d -f mongrel remove ; # example
sudo invoke-rc.d mysql stop && sudo update-rc.d -f mysql remove ; # example

Mac OS X Postfix SMTP Mail Server Configuration

2007-06-04T22:20:21-10:00

I configured my macbook to accept all local mail and then forward it to my personal email account. This is super useful for testing your local code that sends email. You can send to any made up name like herp@localhost or derp@my.machine.name and it will end up in your mail box. Here’s the configuration changes I made.

Add the following to the bottom of your /etc/postfix/main.cf:

luser_relay = me@myemail.net
local_recipient_maps =

Reload postfix & test :

sudo postfix reload
date | mail -s test lolkatz@localhost

You should have some mail delivery @ your me@myemail.net address.