Robert Harker <harker@harker.com>
BayLisa, Mountain View
March 20, 2014
The slides are available at: http://harker.com/Talks/gitfp/index.xhtml
What has changed: Using Git To Document System Configurations by
Robert Harker
is licensed under a
Creative Commons Attribution 3.0 Unported License.
Based on a work at
http://harker.com/Talks/gitfp/index.xhtml
Hit the space bar or swipe left for next slide
This talk is about using git as a version control system for:
The talk is also git 101 for systems administrators
The goal of this talk is to give you some useful tools you can use
Robert Harker:
"The sendmail guy": not for a long time
Interested in OpenStack. Built my own stack: cloud5
Interested in DevOps. Worked in Yahoo Sports as they automated
Cattle servers are multiple servers doing the same thing
If a cattle server dies, you spin up a new one
Automation does everything
Pet servers are the "one off" servers that every network has:
dhcp, dns, smtp relay, yum repo, git repo, monitoring, database
These servers quite often:
How do you manage and document your pet server configuration?
Most typically the answer is a guilty:
Is there a better way?
Why git?
Git commit management model:
Note: in this talk github refers to a local private git repository server
To start using git you need to create your base git repository
Immediately after installing the system:
Shortcomings:
Git can not track configuration changes stored in databases
I don't think git tracks user and group ownership
System files do not tell the full story of the system configuration
What to track:
This is not part of git but can be tracked as part of git
I have a S3 (Stupid Shell Script) that does this:
http://harker.com/Talks/gitfp/Files/Fingerprint.sh
What to track
What not to track
Make a user git useradd -m -r -p '!!' git
git config --global user.name "First Last" git config --global user.email "username@example.com" git config --global core.autocrlf input git config --global core.safecrlf true git config --global push.default simple
# GIT aliases alias gita='git add' alias gitb='git branch' alias gitc='git commit' alias gitd='git diff' alias gito='git checkout' alias gitpl='git pull origin' alias gitpsh='git push origin' alias gits='git status' alias gitv='git status > /tmp/git; vi /tmp/git' alias gitk='gitk --all&'
cd / git init git remote add origin git@github.example.com:reponame.git git config branch.master.remote origin git config branch.master.merge refs/heads/master git symbolic-ref -m "linking HEAD to master" HEAD refs/heads/master
/bin /boot /cgroup /dev /etc/selinux/targeted/policy /etc/selinux/targeted/modules/active /etc/shadow /etc/shadow- /etc/ssh/ssh_host_dsa_key /etc/ssh/ssh_host_key /etc/ssh/ssh_host_rsa_key /home /lib [. . .] /tmp /usr/*/* !/usr/share/wordpress /var/* !/var/lib /var/lib/dhclient/dhclient* /var/lib/logrotate.status /var/lib/mlocate/mlocate.db /var/lib/mysql /var/lib/ntp/drift /var/lib/random-seed /var/lib/rpm /var/lib/yum
/etc/trusted-key.key -crlf -diff *.xls -crlf -diff *.qif -crlf -diff
cd ~git mkdir repo.name.git cd repo.name.git git init --bareThat's all
Note: icon will expand to instructions by clicking on it
I use git to track changes in the system as changes are made
I recommend individual changes be wrapped in a git commit cycle:
Advantages:
Before you start a change/update cycle:
Note: icon will expand to instructions by clicking on it
When new packages are installed new configuration files may be created
Capturing there default configuration is useful for upgrades
Files that change due to normal system operation may be added
Add these files to your /.gitignore file
System configuration may change:
Make running your fingerprinting tool part of your commit cycle
Topic branches are a standard method used in many OpenSource projects
The bug fix branch model makes a separate branch for each fix:
Topic branches are not permanent
Periodically purge old topic branches by deleting them
These branches are still available in git history.
Many sites use a hybrid pet model:
A configuration management system is used to configure site wide defaults
Configuration files specific to the local service are edited by hand
Git branches can be used to track differences between these servers
Installing a git host branch on a server:
Configuration management changes:
These changes are pushed to all servers
Git updates:
Update master branch
Merge changes with each host branch
Conflicts indicate clashes between automatic and manual changes
Merging configuration management changes with the host branches probably can be automated
Problem: you have been asked to upgrade a critical server
Nobody recalls exactly how it was configured
"What was changed?"
Create a generic git repo for the server:
Use git to find and review differences
Build new server adding missing software packages
Merge changes by hand
In a cattle herd we do not need to track changes because:
The view is that each time I deploy a release I create a new "gold" image
The truth is that while each image is new, its overall configuration is mostly static
Tracking system changes in your herds can be problematical:
From a production/operations point of view:
It is much simpler to look at
cumulative changes to the configuration files.
Each production push gets a git commit of changes labeled with the push ID
The push ID becomes the link between:
All of this can be automated
gitk is a really cool way to view configuration changes through time
"Dependency hell"
It is hard to keep all of your systems synced with the upstream OS:
How do you pick and choose?
Solution: make patching the OS part of the DevOps cycle:
Benefits:
Exceptions:
The release engineer / branch manger is responsible for updating OS repos
All devel Linux boxes should also update