Tim Dysinger

Amazon released their new EC2 API today.

The big news: You can have static IP addresses now. You can pick the data center where your machine instance is launched. Cool!

The minor news: Lots of little things were updated. They added tmpfs to the list of LOCAL_FS_TYPES so ubuntu volume images work without patching ec2-ami-tools. You no longer have to worry about having rsync lutime problems when creating a volume image. You can now use an upgraded Linux Kernel which comes with Fuse.

I was busy today upgrading all my images and notes. Gentoo and Ubuntu images were pretty easy to reconfigure. I have created a new public Ubuntu Hardy image that’s in my ubuntu-i686 bucket. Just list all public images to see it.

Posted on March 28th, 2008 by dysinger
Filed under: @work | No Comments »

Ever since I read Martin Fowler’s article on rollerskating and shared it with my partner in crime, we have been using the term and practice liberally. Fowler describes a customer service web form that a guy he knew put together. This customer service form wasn’t quite ready for the internets yet but they needed it yesterday. He talks about delivering iterative hacks and at one point they actually had someone on rollerskates going from one side of the building to the other to take information from a fax machine and input it into a database.

I have long been of the school of thought that you didn’t release anything until it was ready, tested in battle, accepted by the user and proven to supply the oxygen for a 911-call-center. This may have been a by-product of a decade and a half with static languages, consulting and writing code on projects for dead-serious large-businesses that would not stand for mediocrity. I don’t know. Now that I have been on Ruby for quite a while and experienced just how flexible dynamic languages are, I am sold on rollerskatin’….. for some applications.

I still think that a little planning and prefactoring, domain-driven design and behavior-driven development goes a long way towards crafting perfection. However, for systems that are behind the scenes at a start up and needed to be finished yesterday, roller-skating is smart business. It’s about writing the least amount of software possible to get the job done as soon as possible. Later when the jets cool, you’ll know more about what works and what doesn’t and you be in even a better place to refactor or rewrite.

This is not a license for a sloppy user experience whatsoever. The customer using your shiny new webapp had better be oblivious to your rollerskating behind the the curtain. Their experience with your service and with your business must be squeeky clean and enforce the brand of excellence you are striving to portray and deliver.

Posted on March 24th, 2008 by dysinger
Filed under: @work | No Comments »

Update: I need to upgrade this for amazon ec2 2008-02-01 api.

I been playing with Gentoo again. I hadn’t been an active Gentoo user since it pissed me off in a emerge -u world snafu in 2004. I created some Gentoo EC2 images and thought I would share with you all.

I have recently stopped using Xen to create new images and started using Amazon EC2 AMIs to create new AMIs directly — “dog food”-style. The script below is an example of this. There is no need to have 32 & 64-bit Xen Dom0 machines around the house to get started creating custom AMIs. All you need is an Amazon EC2 account. Just fire up someone else’s Linux image and go to work creating a new AMI. I have been using Amazon’s Fedora 4 “developer” 32-bit “small” image to create a nice lean Gentoo image. Here is my script.

# Boot a developer image at EC2 && Login as root on the instance

# Move the /tmp dir to the big drive
mv /tmp /mnt && ln -sf /mnt/tmp /

# Bootstrap
mkdir /mnt/gentoo
wget -O - \
  http://gentoo.osuosl.org/releases/x86/current/stages/stage3-i686-2007.0.tar.bz2 | \
  tar xjC /mnt/gentoo
wget -O - http://gentoo.osuosl.org/snapshots/portage-latest.tar.bz2 | \
  tar xjC /mnt/gentoo/usr
wget -O - http://s3.amazonaws.com/ec2-downloads/linux-2.6.16-ec2.tgz | \
  tar xzC /mnt/gentoo/usr/src
zcat /proc/config >/mnt/gentoo/usr/src/linux-`uname -r`/.config

# FUSE module (has to be compiled with the same gcc as ec2's kernel)
cd /tmp
wget -O - \
  http://superb-west.dl.sourceforge.net/sourceforge/fuse/fuse-2.7.3.tar.gz | \
  tar xz
cd fuse-2.7.3
./configure --enable-kernel-module \
  --with-kernel=/mnt/gentoo/usr/src/linux-`uname -r`
cd kernel
make && make install
mkdir -p /mnt/gentoo/lib/modules/`uname -r`
cp -r /lib/modules/`uname -r` /mnt/gentoo/lib/modules/`uname -r`

# Setup
cat /proc/mounts >/mnt/gentoo/etc/mtab
mount -o rbind /proc /mnt/gentoo/proc
mount -o rbind /dev /mnt/gentoo/dev
mount -o rbind /sys /mnt/gentoo/sys
cp /etc/resolv.conf /mnt/gentoo/etc

# Chroot
chroot /mnt/gentoo /bin/bash
env-update
source /etc/profile
export PS1="(image) $PS1"

# Modules / Kernel
depmod -a
modprobe loop
echo 'loop' >>/etc/modules.autoload.d/kernel-2.6
echo 'fuse' >>/etc/modules.autoload.d/kernel-2.6
cd /usr/src && ln -sf linux-`uname -r` linux

# Cleanup
cd /
rm -rf tmp && ln -sf var/tmp tmp
rm -rf opt && ln -sf usr/local opt
rm -rf boot

# Root
usermod -p \
  `dd if=/dev/urandom count=50 2> /dev/null | md5sum | cut -d " " -f1-1` \
  root

# Rebuild
cat >/etc/make.conf <<\EOF
CFLAGS="-O2 -march=i686 -pipe -mno-tls-direct-seg-refs"
CXXFLAGS="${CFLAGS}"
CHOST="i686-pc-linux-gnu"
MAKEOPTS="-j2"
EOF
emerge --sync
emerge -e world
emerge --update --newuse --deep world ; # are these both needed ^ <-
etc-update
emerge eix gentoolkit
emerge --depclean
revdep-rebuild

# Locale
cat >/etc/locale.gen <<\EOF
en_US ISO-8859-1
en_US.UTF-8 UTF-8
EOF
locale-gen

# Timezone
cp /usr/share/zoneinfo/GMT /etc/localtime
cat >>/etc/conf.d/clock <<\EOF
TIMEZONE="GMT"
EOF

# Mounts
cat >/etc/fstab <<\EOF
/dev/sda1 /        ext3  user_xattr          0 1
/dev/sda2 /mnt     ext3  user_xattr          0 2
/dev/sda3 swap     swap  sw                  0 0
shm       /dev/shm tmpfs nodev,nosuid,noexec 0 0
EOF

# TTY
perl -p -i -e 's/^c([^1])/\#c$1/g' /etc/inittab

# Network
emerge dhcpcd ddclient net-misc/ntp
rc-update add net.eth0 default
rc-update add sshd default
rc-update add ntpd default
cat >/etc/ssh/sshd_config <<\EOF
Protocol 2
StrictModes yes
MaxStartups 10:30:60
Ciphers aes256-cbc,aes256-ctr
PasswordAuthentication no
ChallengeResponseAuthentication no
Subsystem sftp /usr/lib/misc/sftp-server
UseDNS no
EOF

# Boot
cat >/etc/conf.d/local.start <<\EOF
# /etc/conf.d/local.start
# Root SSH Public Key
[ ! -e /root ] && cp -r /etc/skel /root
wget --timeout 15 -q -O - \
  http://169.254.169.254/2007-12-15/meta-data/public-keys/0/openssh-key > \
  /root/.ssh/authorized_keys
chmod -R go-rwsx /root
# Userdata Shell Script
wget --timeout 15 -q -O - http://169.254.169.254/2007-12-15/user-data | sh
EOF

# EC2 tools
emerge ruby curl unzip symlinks
cd /tmp
wget http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.zip
cd /usr/local
unzip /tmp/ec2-ami-tools.zip
ln -sf ec2* ec2-ami-tools
chmod -R go-rwsx ec2*
rm -rf /tmp/ec2*
# Recompile rsync (lutimes doesn't work with old ec2 kernel)
cd /tmp
wget -O - http://www.samba.org/ftp/rsync/src/rsync-2.6.9.tar.gz | \
  tar xz
cd rsync-2.6.9
perl -pi.bak -e 's/\blutimes\b//' ./configure
./configure --prefix=/usr/local/ec2-ami-tools
make
make install
cd ..
rm -rf rsync*

# Bundle
export AMAZON_USER_ID='FIXME'
export AMAZON_ACCESS_KEY_ID='FIXME'
export AMAZON_SECRET_ACCESS_KEY='FIXME'
cat >/mnt/pk.pem <<\EOF
-----BEGIN PRIVATE KEY-----
FIXME
-----END PRIVATE KEY-----
EOF
cat >/mnt/cert.pem <<\EOF
-----BEGIN CERTIFICATE-----
FIXME
-----END CERTIFICATE-----
EOF
export EC2_PRIVATE_KEY=/mnt/pk.pem
export EC2_CERT=/mnt/cert.pem

cat >/usr/local/sbin/image <<\EOF
#!/bin/bash
export EC2_AMITOOL_HOME=/usr/local/ec2-ami-tools
PATH=$EC2_AMITOOL_HOME/bin:$PATH
BUNDLE=`date '+%y%m%d%H%M%S'`
ec2-bundle-vol -r i386 -u $AMAZON_USER_ID \
  -k $EC2_PRIVATE_KEY -c $EC2_CERT \
  -b -d /mnt -s 10000 --fstab /etc/fstab \
  -e /root -p $BUNDLE
ec2-upload-bundle -b $HOSTNAME -m /mnt/$BUNDLE.manifest.xml \
  -a $AMAZON_ACCESS_KEY_ID -s $AMAZON_SECRET_ACCESS_KEY
rm -rf /mnt/$BUNDLE* /mnt/img-mnt
EOF
chmod 700 /usr/local/sbin/image

export HOSTNAME=gentoo-i686
rm -rf /var/tmp/* /usr/portage/distfiles /usr/portage/packages
symlinks -crsdv /
image

# Register & make the ami public (on another machine)
ec2-register $HOSTNAME/$BUNDLE.manifest.xml
ec2-modify-image-attribute ami-xxxxxx --launch-permission -a all

#
# Below is an example of a boot script that you might pass in as "userdata"
# You would configure the hostname and dyndns and/or maybe puppet or cfengine
#

#!/bin/bash
# Hostname
echo 'HOSTNAME="fqdn.example.com"' >/etc/conf.d/hostname
/etc/init.d/hostname restart
echo '127.0.0.1 '`hostname -f`' '`hostname -s`' localhost' >/etc/hosts
echo 'search '`hostname -d` >/etc/resolv.conf
echo 'nameserver 172.16.0.23' >>/etc/resolv.conf
echo 'dhcp_eth0="release nodns nontp nonis"' >/etc/conf.d/net
/etc/init.d/net.eth0 restart
# DynDNS
cat >/etc/ddclient/ddclient.conf <<\EOF
daemon=300
syslog=yes
mail=root
mail-failure=root
ssl=yes
use=web, web=169.254.169.254/2007-12-15/meta-data/public-ipv4
protocol=dyndns2, server=members.dyndns.org, custom=yes, \
login=FIXME, password=FIXME \
EOF
hostname >>/etc/ddclient/ddclient.conf
/etc/init.d/ddclient start
rc-update add ddclient default
fi

#
# After the new instance is booted, you may want to login and configure some
# basic tools or whatever
#

# Extras Tools
cat >>/etc/portage/package.keywords <<\EOF
dev-util/git
sys-fs/encfs
sys-fs/fuse
sys-fs/sshfs-fuse
EOF
emerge dev-util/git
emerge sys-fs/fuse sys-fs/encfs sys-fs/sshfs-fuse

Posted on March 4th, 2008 by dysinger
Filed under: @work | 4 Comments »

I struggled but ultimately succeeded in installing Bazaar with bzr-svn (subversion support) on Leopard. I added the script to the bzr-svn wiki page. The following is how I got it to work:

# Install the latest stable Bazaar (if you didn't already)
sudo easy_install -U paramiko pycrypto bzr

# Compile Subversion 1.5 (for better python support)
cd /tmp
rm -rf svn
svn export http://svn.collab.net/repos/svn/trunk svn
cd svn
curl http://www.webdav.org/neon/neon-0.27.2.tar.gz | tar xz
ln -sf neon-0.27.2 neon
./autogen.sh
./configure --without-apxs --with-ssl --prefix=$HOME/.bazaar/svn
make install install-swig-py
cd ..

# Install the bzr_svn plugin
mkdir ~/.bazaar ~/.bazaar/plugins
cd ~/.bazaar/plugins
rm -rf svn
bzr checkout --lightweight \
  http://people.samba.org/bzr/jelmer/bzr-svn/stable svn
# The following is a temporary patch - it will core-dump without it
perl -p -i -e 's/(providers \+\= auth\.Subversion)/\#$1/g' svn/transport.py
perl -p -i -e 's/(providers \+\= \[auth\.get)/\#$1/g' svn/transport.py

# Setup your profile to recognize subversion 1.5
echo >> ~/.profile << \EOF
alias bzr="DYLD_LIBRARY_PATH=$HOME/.bazaar/svn/lib \
PYTHONPATH=$HOME/.bazaar/svn/lib/svn-python \
bzr"
EOF
. ~/.profile

# Test bzr + svn w/ ssl
cd /tmp
rm -rf test
bzr branch https://svn.collab.net/repos/svn/trunk test

Posted on January 26th, 2008 by dysinger
Filed under: @work | 4 Comments »

Use svnsync to make copies of your repository. You can set up a cron job to do the last part ’svnsync sync’ to keep it up to date each day too.

export LOCAL=~/some/new/svnrepo
svnadmin create $LOCAL
echo '#!/bin/sh' > $LOCAL/hooks/pre-revprop-change
chmod +x $LOCAL/hooks/pre-revprop-change
svnsync init file://$LOCAL svn://some/remote/repo
svnsync sync file://$LOCAL

Posted on December 30th, 2007 by dysinger
Filed under: @work | No Comments »