See you in Berlin this September. I am speaking on using Puppet to automate whole clusters of Linux servers in a DRY fashion. Here is the abstract:
Title: Using Puppet to DRY up your Rails Servers
Description:
Puppet is a system administration tool written in Ruby with a DSL for describing system resources. We can directly leverage it to effortlessly DRY up the management of our rack of servers. We will go through puppet's installation, features and a demo where we watch brand new servers automatically configure themselves as a complete multi-machine Rails stack, ready for Capistrano deployment.
Abstract:
Puppet is a tool written in Ruby for system management. It has been reported that Google uses Puppet to manage 6000+ OS X desktop machines. However, it is most commonly used to manage a rack of servers. Puppet has a DSL with a Ruby-like syntax for describing system resources and actions to configure and keep servers up to date. It is the perfect companion to Capistrano. Capistrano is good at deploying versioned applications to a remote rack of servers. Puppet is good at configuring those servers so that they are up to date, secure and running the right services.
We often find ourselves talking, at RailsConf, about the issues involved with developing a Rails app and deploying with Capistrano. However most non-trivial Rails apps require multiple servers and clusters of rails apps fronted by a proxy webserver. Ask yourself: "How exactly did I get to the point where I _could_ just 'cap deploy'?" Most likely you are deploying to either a hosted machine or you hand configured a group of servers on your own. If you are in the later category then you know how much time it takes to keep a rack of servers in shape.
Puppet is cross platform. With it we can manage Gentoo machines in the same abstract DSL as we would manage Solaris machines. For example, we could use Puppet to describe that on all webserver machines we need NGINX installed and running with a specific nginx.conf file, even if our webservers are a mix of Solaris and Debian.
We can use Puppet to insure that our system administrator user and our rails deployment user are on all machines with the proper ssh keys. We can use it to make sure each of our machines has the right cron jobs lined up. We can use it to insure that each machine has exactly RMagick vX.XX.X installed. The list is endless. If you can do it on the command line, it can be written in Puppet's DSL and you will never have to repeat yourself again.
Join us for an in-depth tour of Puppet. We will see a demonstration on brand new empty EC2 linux instances. This demo will show a rack of servers boot up and configure themselves from scratch and join a cluster of rails servers, ready to serve your Rails application.
I have been keeping all my projects in git repositories regardless if they came from git or svn. I probably have about 3 dozen projects at any one time in my ~/Projects dir. Keeping them all up to date is a PITA. So I wrote a little sake task to find them all and update them. Here it is:
desc "Recursively update your git projects"
task "git:update" do
`find . -name .git -type d`.each do prj
Dir.chdir(File.dirname(prj)) do d
puts(d)
`git config --get svn-remote.svn.url`.empty? ?
`git remote update` : `git svn fetch`
end
end
end
I deploy Rails and Ruby projects using capistrano and I manage my source control with git. Git is most typically accessed with a ssh key and not username-password. There is a way to proxy your key through your deploy server making it possible for capistrano to retrieve the code from a 3rd party (gitub or gitorious or ssh-enabled git-server). It's called ssh agent. Not many people use it but it's super handy and comes built in to most flavs of Linux and OS X Leopard.
Assuming you have already put your desktop public key in the git server's authorized keys or on github/gitorious you have put your ssh public key in your profile. What you do is this:
echo >$HOME/.ssh/config <<\EOF
Host *.mydeployservers.com
ForwardAgent yes
Host *.myotherdeployservers.com
ForwardAgent yes
Host *
ForwardAgent no
EOF
chmod -R go-rwsx $HOME/.ssh
Now SSH-Agent is setup to forward your keys through the deploy server and and you are ready to do some deploying. You may need to login to the deploy server once and try to login to the git server (github or the like) one time to accept the servers ssh key and stash in the deploy server/user's local "known_hosts" file.
ssh myuser@mydelpoyserver.com
# (and from there)
ssh git@github.com
# the login will fail but the important part is accepting the server's ssh key
exit
Now back on your desktop you can now deploy from github or the like without "deploy keys".
ssh-add ; #only need to do this once per login to your desktop
cap deploy
:)
I was playing around with Braid and Piston for Git and didn't like either of them. I didn't feel like Piston was ready for Git andBraid seemed to complex. I just wanted git subtrees. Braid is doing the same thing but with more code around it. I like simple. It actually turns out that Git has all the tools built in to do sub-tree merging and updating. It's really easy with a few sake (or rake) tasks. I am using this on a rails project with more than a dozen plugins and vendor/rails.
Update 6/7: I added a diff task and corrected a typo
# Git Sub-Trees Sake Tasks
# http://rubyurl.com/BwnY < - Read about the technique vs submodules
#
# This works super good with Rails plugins or when you are actively working
# on two related projects.
#
# Add these tasks with:
# sake -i http://pastie.caboo.se/213492.rb
#
# Example Usage:
#
# sake git:subtree:remote
# Git Repo Url? git://github.com/mislav/will_paginate.git
# Remote Name? vendor/plugins/will_paginate
# sake git:subtree:merge
# Branch? will_paginate/tags/2.2.2
# Destination? vendor/plugins/will_paginate
# git commit
#
# From time to time update your remotes:
# git remote update
# -or-
# git fetch vendor/plugins/will_paginate
#
# View diffs:
#
# sake git:subtree:diff
# Branch? vendor/plugins/will_paginate/tags/2.2.3
#
# Merge in updates:
#
# sake git:subtree:update
# Branch? vendor/plugins/will_paginate/tags/2.2.3
# git commit
desc 'Add an external project as a remote'
task 'git:subtree:remote' do
require "readline"
url = begin
print "Git Repo Url? "
Readline.readline.chomp
end
name = begin
print "Remote Name? "
Readline.readline.chomp
end
`git remote add #{name} #{url}`
`git config remote.#{name}.fetch refs/heads/*:refs/remotes/#{name}/*`
`git config --add remote.#{name}.fetch refs/tags/*:refs/remotes/#{name}/tags/*`
`git config remote.#{name}.tagopt --no-tags`
`git fetch #{name}`
end
desc 'Merge an external project as a sub-tree'
task 'git:subtree:merge' do
require "readline"
branch = begin
print "Branch? "
Readline.readline.chomp
end
dest = begin
print "Destination? "
Readline.readline.chomp
end
`git merge --squash -s ours --no-commit #{branch}`
`git read-tree --prefix=#{dest} -u #{branch}`
end
desc 'Update an existing subtree project'
task 'git:subtree:update' do
require "readline"
branch = begin
print "Branch? "
Readline.readline.chomp
end
`git merge --squash -s subtree --no-commit #{branch}`
end
desc 'Show subtree diff against remote'
task 'git:subtree:diff' do
require "readline"
branch = begin
print "Branch? "
Readline.readline.chomp
end
puts `git-diff-tree -p #{branch}`
end
