In working with my new hires, I have found the try-and-buy works really well for the company. 50% of developers don’t make it during the trial period. It’s really important to try out new developers on a paid trial for a while. There is no interview long enough to make a decision to hire. Have them submit patches to your code base, don’t give them direct source access for the first week of the trial period. Review their patches. See how they respond to criticism. Watch how they interact with the team to solve problems.

You may think 50%!? That’s high. Not really. Think about it: 80-90% of all businesses fail in the first 5 years. The start-up failure rate is most likely higher on “the internets.” Just as much as it’s important to try out your employees before you hire them, it’s important for potential employees or consultants to try out potential employers or clients before making any commitments.

Lesson: You can’t change people for the most part. “They are who they are” starts the day you meet them. Pay close attention to warnings and your gut. If it’s not a fit, let them go in the first 2 weeks. Don’t wait a month or a year. Life is too short to work with unqualified people.

OS X has this deep-sleep mode that will save all the contents of Ram to disk. They do this just in-case you forget to charge it and the battery drains. However this causes a long hibernation time (the time measure from the time you close the lid to the time you see the “breath” light and it’s safe to move the laptop). During this long hibernation time, you MUST NOT move or especially jerk the laptop (like putting it in your bag). If you do jerk you laptop around at any time the hard disk is running, you can damage the hard disk. It is a moving-at-high-velocity part of your computer.

If you are like me, there isn’t but a few hours that go buy before I am using my laptop again. The battery-draining-all-the-way situation is almost never going to happen to me. I am always backed up and charged. The laptop is my life-blood and my primary tool for my income.

So, rather than have the thing take for ever to sleep for a feature I won’t use, I decide to discover a way to turn it off. The result is the script. Open a terminal and type the following:

sudo sh -c "\
  pmset -a hibernatemode 0 ;\
  nvram use-nvramrc?=false ;\
  rm /var/vm/sleepimage >/dev/null 2>&1 ;\
"

I find it incredibly simple and handy to send application alerts (and sometimes friendly notes) via email to an SMS-enabled phone. Most of the big carriers have an email to SMS gateway. Here are the email addresses of the top 6 carriers.

T-Mobile

phonenumber@tmomail.net

Virgin Mobile

phonenumber@vmobl.com

AT&T

phonenumber@mmode.com

Sprint

phonenumber@messaging.sprintpcs.com

Verizon

phonenumber@vtext.com

Nextel

phonenumber@messaging.nextel.com

I run Debian 4.0 (Etch) images at EC2 that I created on Xen and bundled to EC2. I discovered while trying to compile a kernel module for FUSE that Debian has GCC 4.1 while all of EC2’s kernels were built with GCC 4.0. This is a problem in that the kernel module, even though it compiles fine, will not insert into the EC2 kernel.

This was a bit of a road block for me as Debian Sarge has GCC 3.3/4 and Etch only has 4.1. I had no access to 4.0. Luckily Amazon provides their stock Fedora images with developer tools. I personally hate RPM-based Linux distributions and Amazon’s Fedora images are broken and won’t ‘yum update’. However, Amazon’s ‘developer’ image at least has GCC 4.0 on it along with the kernel source. With that you can compile a binary kernel module and tar it up for use elsewhere (on your nice clean Debian images).

Here are the steps I took to compile Fuse Kernel module on Amazon’s Developer Fedora image and move it to Debian 4.0. Although this is tailored for compiling the Fuse kernel module the same steps will work for any other kernel module.

# DESKTOP: Fire up an EC2 fedora "developer" instance
ec2-run-instances -g sandbox -k sandbox ami-26b6534f

# DESKTOP: Login
ssh -i ~/.amazon/id_rsa-sandbox \\
  root@ec2-72-44-51-242.z-1.compute-1.amazonaws.com

# EC2: Remove old fuse
rpm -e `rpm -qa  grep fuse`

# EC2: Install FUSE module
cd /usr/local/src
curl -O http://superb-west.dl.sourceforge.net/sourceforge/fuse/fuse-2.6.5.tar.gz
tar xzf fuse-2.6.5.tar.gz
cd fuse-2.6.5
./configure --enable-kernel-module --with-kernel=/usr/src/linux-`uname -r`
cd kernel
make && make install

# EC2: Package up the modules
tar -czf modules-`uname -r`.tgz /lib/modules/`uname -r`

# DESKTOP: Pull the modules off and stash them somewhere to use with your 'real' linux distro
cd ~/Projects/sysadmin/files
scp -i ~/.amazon/id_rsa-sandbox
  root@ec2-72-44-51-242.z-1.compute-1.amazonaws.com:/modules-2.6.16-xenU.tgz .
svn commit . -m 'Updated kernel modules for ec2'</pre></code>

If you have a linux server with Xen installed, you can develop and test on Xen and migrate your images to EC2 easily when you are ready.

First off you need to have Linux running in a Xen image. This was covered earlier in a post about installing Xen on Debian Hosts & Guests Xen Virtulization with Debian Linux 4 (It’s easy)

Once you have a running image on Xen, migrating to EC2 is fairly easy. There are just a few steps that you need to follow. I’ve outlined them below using Debian 4.0 as the distribution but you can choose your own flavors.

# HOST: (Re)Install & Correct (for Debian) Amazon's EC2 AMI tools & Ruby 1.8.6
sudo su -
apt-get install alien curl
curl -O http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.noarch.rpm
alien -i ec2-ami-tools.noarch.rpm
rm ec2-ami-tools.noarch.rpm
# IF YOU DO _NOT_ HAVE RUBY 1.8.6 installed from source
apt-get install ruby libopenssl-ruby1.8
ln -s /usr/lib/site_ruby/aes /usr/lib/ruby/1.8/
# IF YOU DO HAVE RUBY 1.8.6 installed from source
ln -s /usr/lib/site_ruby/aes /usr/local/lib/ruby/1.8/
# THEN
vi /usr/lib/site_ruby/aes/amiutil/image.rb
# exec( 'for i in console null zero ; do /sbin/MAKEDEV -d ' + dev_dir + ' -x $
#       i ; done' )
# ......Should be......
# exec("cd #{dev_dir} && /sbin/MAKEDEV console && /sbin/MAKEDEV std && /sbin/MAKEDEV generic")
exit

# HOST: (Re)Install Amazon's EC2 API tools
sudo apt-get install sun-java5-jre unzip
cd /usr/local
sudo rm -rf ec2*
sudo symlinks -crsd .
curl -O http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
unzip ec2-api-tools.zip
rm ec2*/bin/*.cmd
chmod -R og-w ec2*
sudo chown -R debian:staff ec2*
rm ec2-api-tools.zip
ln -sf ec2-api-tools* ec2-api-tools
cd /usr/local/bin
ln -sf ../ec2-api-tools/bin/* .

# Download your ec2 certificate and put it in ~/.amazon/cert.pem
# Download your ec2 private key and put it in ~/.amazon/pk.pem
# Put your AWS Access key in ~/.amazon/access.txt
# Put your AWS Secret key in ~/.amazon/secret.txt
# Put your AWS Account # in ~/.amazon/user.txt
cat >> ~/.profile << EOF
export JAVA_HOME=/usr
export EC2_HOME=/usr/local/ec2-api-tools
export EC2_PRIVATE_KEY=$HOME/.amazon/pk.pem
export EC2_CERT=$HOME/.amazon/cert.pem
export AMAZON_ACCESS_KEY_ID=`cat ~/.amazon/access.txt`
export AMAZON_SECRET_ACCESS_KEY=`cat ~/.amazon/secret.txt`
export PATH=$EC2_HOME/bin:$PATH
EOF
source ~/.profile
ec2-add-keypair sandbox # put key in ~/.amazon/id_rsa-sandbox
ec2-add-keypair development # put key in ~/.amazon/id_rsa-development
ec2-add-keypair test # put key in ~/.amazon/id_rsa-test
ec2-add-keypair production # put key in ~/.amazon/id_rsa-production
chomod -R go-rwsx ~/.amazon

#
# When are ready to deploy the image
#

# GUEST: Create an Amazon EC2 fstab
sudo su -
cd /etc
mv fstab fstab.xen
ln -sf fstab.xen fstab
cat > /etc/fstab.ec2 << EOF
/dev/sda1 /     ext3  defaults             1 1
/dev/sda2 /mnt  ext3  defaults,user_xattr  1 2
/dev/sda3 swap  swap  defaults             0 0
none      /proc proc  defaults             0 0
none      /sys  sysfs defaults             0 0
EOF
exit

# GUEST: Install Amazon's Linux Kernel modules on your image
sudo su -
cd /tmp
wget http://s3.amazonaws.com/ec2-downloads/modules-2.6.16-ec2.tgz
tar -xzf /tmp/files/modules-2.6.16-xenU.tgz
rm -rf /tmp/files
exit

# HOST: Stop the Xen image && mount the disk && switch to EC2 fstab
sudo xm shutdown image; # Wait for it to stop
sudo mount -t ext3 -o loop /xen/domains/image/disk.img /mnt
cd /mnt/etc
sudo ln -sf fstab.ec2 fstab
cd
sudo umount /mnt

#
# Setup EC2
#

# Add Groups
ec2-add-group sandbox  -d 'Sandbox'
ec2-add-group website  -d 'Website'
ec2-add-group database -d 'Database'
ec2-add-group backend  -d 'Backend'

# Open Ports
ec2-authorize sandbox -p 22
ec2-authorize website -p 80
ec2-authorize website -p 443
ec2-authorize default -p 54321

#
# Deploy to EC2
#

# Bundle Image
ec2-bundle-image -p image -i /xen/domains/image/disk.img
  -u `cat ~/.amazon/user.txt`

# Upload Image
ec2-upload-bundle -b ami.soniannetworks.com -m /tmp/image.manifest.xml
  -a `cat ~/.amazon/access.txt` -s `cat ~/.amazon/secret.txt`

# Register Image
ec2-register ami.soniannetworks.com/image.manifest.xml

# List Images
ec2-describe-images
ec2-describe-images -x all

# Run Image
ec2-run-instances -g sandbox -k sandbox ami-XXXXXXX

# List
ec2-describe-instances

# Login
ssh -i ~/.amazon/id_rsa-sandbox root@myhost.amazonaws.com

# Shutdown
ec2-terminate-instances i-XXXXXXX

# Unregister
ec2-deregister ami-XXXXXXX

#
# Post-Deploy Config @ EC2
#

# EC2: SSH in (as debian user)
ssh -p 54321 debian@my.host.at.ec2.amazonws.com
# EC2: Change password (as debian user)
# EC2: Reconfigure the new guests hostname
sudo su -
hostname myhost.ec2.mydomain.com
echo `hostname` > /etc/hostname
exit
# GUEST: Reconfigure mail (see xen.txt doc)
# GUEST: Configure services as needed (see xen.txt doc)

#
# Switching the Xen image back to work with Xen
#

# HOST: Mount the disk && switch to Xen fstab && start the Xen image again
sudo mount -t ext3 -o loop /xen/domains/image/disk.img /mnt
cd /mnt/etc
sudo ln -sf fstab.xen fstab
cd
sudo umount /mnt
sudo xm create /etc/xen/image.cfg
sudo xm console image

Install Debian 4 just like you normally would. Choose a minimal install with no tasksel features (No Desktop - No Standard). Your finished install OS from this point forward will be referred to as the Host while virtual images will be referred to as Guests (below). I’m just going to tear into the steps that follow a basic Debian 4 install. They are as follows:

# Installing Debian 4 & Xen 3
#
# NOTE: if your HOST is on qemu add '-redir tcp:2222::22' to the parameters
#
# HOST  = The base Debian Xen install OS that's hosts guests
# GUEST = The VMs you are within the Debian Xen HOST
#
# During install add a user called 'debian' for non-root access
#

# HOST: Update
cat > /etc/apt/sources.list << \EOF
deb http://debian.osuosl.org/debian/ etch main contrib non-free
deb http://security.debian.org/ etch/updates main contrib non-free
EOF
apt-get update && apt-get upgrade

# HOST: Tools
apt-get install pwgen deborphan symlinks less

# HOST: Mail
apt-get install exim4 mailx
echo root: tim@dysinger.net > /etc/aliases && newaliases
dpkg-reconfigure exim4-config ; # Choose "Internet Site"
date  mail -s test root@localhost ; # should see email @ tim@dysinger.net

# HOST: SSH
apt-get install openssh-server
mv /etc/ssh/sshd_config /etc/ssh/sshd_config.bak
cat > /etc/ssh/sshd_config << \EOF
Protocol 2
Port 54321
PermitRootLogin no
PasswordAuthentication no
UseDNS no
Subsystem sftp /usr/lib/openssh/sftp-server
EOF
rm /etc/ssh/*key*
dpkg-reconfigure openssh-server
# DESKTOP: Create SSH keys (if you don't have them already)
ssh-keygen -t dsa ; # just [Return] at all prompts
# HOST: Then paste the content of the DESKTOP ~/.ssh/id_dsa.pub
# into HOST /home/debian/.ssh/authorized_keys
su - debian
mkdir ~/.ssh
cat >> ~/.ssh/authorized_keys << \EOF
}}}
EOF
chmod -R go-rwsx ~/.ssh
exit
# DESKTOP: Try the ssh login before you trust everything's OK
ssh -p 54321 debian@my.servers.host.name.com ; # should require no password

# HOST: Xen
apt-get install linux-image-2.6-xen-686 linux-headers-2.6-xen-686 \
xen-hypervisor-3.0.3-1-i386-pae xen-linux-system-2.6.18-4-xen-686 \
xen-ioemu-3.0.3-1 xen-tools libc6-xen bridge-utils
cat >> /etc/modules << \EOF
loop max_loop=64
EOF
cat > /etc/xen-tools/xen-tools.conf << \EOF
dir         = /xen
mirror      = http://ftp.us.debian.org/debian/
kernel      = /boot/vmlinuz-2.6.18-4-xen-686
initrd      = /boot/initrd.img-2.6.18-4-xen-686
debootstrap = 1
dist        = etch
image       = full
size        = 4Gb
memory      = 512Mb
swap        = 512Mb
dhcp        = 1
EOF
cat > /etc/xen/xend-config.sxp << \EOF
(network-script network-bridge)
(network-script network-dummy)
(vif-script vif-bridge)
(dom0-min-mem 196)
(dom0-cpus 0)
EOF
mkdir /xen

# HOST: Dhcp
apt-get install dhcp3-server
cat > /etc/dhcp3/dhcpd.conf << \EOF
default-lease-time 600;
max-lease-time 7200;
option subnet-mask 255.255.255.248;
option broadcast-address 66.199.242.207;
option routers 66.199.242.201;
option domain-name-servers 4.2.2.1, 72.29.96.250, 207.210.212.202;
option domain-name "mysite.com";
subnet 66.199.242.200 netmask 255.255.255.248 {
  range 66.199.242.203 66.199.242.206;
}
host myserver {
  option host-name "myserver.mysite.com";
  hardware ethernet 00:16:3E:18:CF:B3;
  fixed-address 66.199.242.206;
}
EOF

# HOST: Sudo
apt-get install sudo
visudo ; # make sure the bottom line has 'debian  ALL=(ALL) ALL'

# HOST: Cleanup
reboot ; # after reboot - login as 'debian' (NOT ROOT EVER AGAIN ON THIS HOST!)
sudo uname -r ; # running xen kernel ?
sudo dpkg --purge linux-image-2.6-686 linux-image-2.6.18-4-686

# HOST: Harden
# Read http://www.debian.org/doc/manuals/securing-debian-howto/
sudo apt-get install nmap
sudo nmap -p 1-65535 -T4 -sS `hostname` ; # have a look at your open ports
sudo apt-get install harden harden-clients harden-tools tiger chkrootkit lsof

# HOST: Firewall ( leave open eth0 port 54321 & add xenbr0 w/o NAT )
sudo apt-get install arno-iptables-firewall

# HOST: Guest
sudo xen-create-image --hostname image
sudo xm create /etc/xen/image.cfg
sudo xm list
sudo xm console image ; # Ctrl-] is exit

# GUEST: Prepare
passwd root ; # Change the password!!!
echo 127.0.0.1 localhost localhost.localdomian > /etc/hosts
adduser debian
usermod -G staff,src debian

# GUEST: Setup
#   1. Update (same as above)
#   2. Tools (same as above)
#   3. Mail (same as above)
#   4. Ssh (same as above)
#   5. Sudo (same as above)
#   6. Harden (same as above)
#   7. Firewall ( same as above but leave open eth0 ports 80 443 54321 )

# HOST: Cloning GUESTS

# HOST: Create a new Xen image (xen-create above)
# HOST: Copy the 'image' disk to the clone
#   'image' must be shut down then
#     (cp /xen/domians/image/disk.img /xen/domains/mynewdomain/ )
# HOST: Boot your new xen guest (xm create above)
# GUEST: Change password (as debian user)
# GUEST: Reconfigure the new guests hostname
sudo su -
hostname dev.ec2.sonianarchive.com
echo `hostname` > /etc/hostname
exit
# GUEST: Reconfigure mail (above)
# GUEST: Configure services as needed (as debian user)
sudo invoke-rc.d nginx stop && sudo update-rc.d -f nginx remove ; # example
sudo invoke-rc.d mongrel stop && sudo update-rc.d -f mongrel remove ; # example
sudo invoke-rc.d mysql stop && sudo update-rc.d -f mysql remove ; # example

I configured my macbook to accept all local mail and then forward it to my personal email account. This is super useful for testing your local code that sends email. You can send to any made up name like herp@localhost or derp@my.machine.name and it will end up in your mail box. Here’s the configuration changes I made.

Add the following to the bottom of your /etc/postfix/main.cf:

luser_relay = me@myemail.net
local_recipient_maps =

Reload postfix & test :

sudo postfix reload
date | mail -s test lolkatz@localhost

You should have some mail delivery @ your me@myemail.net address.