whoami
Robert Harker
Currently a Sr. Service Engineer at Yahoo! Sports
On a team of 4 SEs managing ~3,000 hosts
- Heavily involved with Yahoo! Sports move to a DevOps team
- and continuous integration with Jenkins
- Working on migrating Yahoo! Sports to new hardware
- into two consolidated data centers
Active in the OpenStack and cloud computing community
Yahoo! Sports Uses Jenkins For Agile Development
Yahoo! Sports has adopted an agile development environment:
Sports bases our work around a scrum team that includes Devel, QE and SE
Sports works on weekly sprints which include new features and bug fixes
Sports pushes a new release to production every week
- Sports does daily pushes for minor bug fixes
- and product driven changes like ad campaigns
Jenkins has been key for build, test and deploy automation
What is Jenkins
- Jenkins is an automated continuous integration system
- Similar to CruiseControl and DamageControl
Jenkins can automatically:
Trigger builds on events like SVN check-ins
Trigger software tests when build completes
Update system configuration information
Automatically deploy to integration and test environments
Monitor external jobs
Notify via email, IRC or IM job status and results
Manually deploy to production environments
Jenkins Features
- Jenkins Supports:
- Change sets: a list of changes made by the build
- and checked into a version control system
Tight integration with SVN/git for traditional builds
Tight integration with Mavin for Java builds
JUnit/TestNG test reporting: tabulated, summarized, and historical
Distributed builds: build slaves to speed up builds
Plugins for different build, deploy, reporting and 3rd party tools
Jenkins Is A Java Based Application
- Can be run with:
- java -jar jenkins.war
Can be deployed in a servlet container like Tomcat
No database or external applications required
Can be configured:
Entirely using its web GUI with error checking
By editing its XML config files
Via its RESTfull API
Typical Jenkins Build Job Flow
Configure a Jenkins Job
Configure Jenkins Upstream/Downstream jobs
Jenkins Deploy
Build Automation
- Sports builds new packages and deploys to develoment integration hosts
- within 10 minutes of a SVN check in
- Each time a change is made, Jenkins builds and deploys changes
- For all classes of hosts: FE, API, cache
Because of this Sports has a 3 step build process:
- A Jenkins job to discover if changes have been made,
- Recomple the code and build the new packages
A Jenkins job to install the packages in generic chroot environments
A Jenkins job to update the "roles" and create a deployment tag
Test Automation
Sports uses Jenkins to automate our test suite:
Daily regression tests
Periodic and on demand smoke tests
Jenkins automatically runs smoke tests as part of automated deploy jobs
Smoke tests allow Ops to validate basic functionality after a deploy
- Jenkins keeps track of the number of test errors
- Provides a web based interface to drill down into the test failures
Deploy Automation
- Sports uses Jenkins to deploy to all of our environments:
- Devel integration, QA, stage and production
Jenkins uses the same deploy process for:
A full install to reimaged host
Updates of farms of hosts
Jenkins applies a "tag" to all the hosts being upgraded
Jenkins then does a generic update of that tag on all the hosts
Yahoo! Sports Deploy Procedure
- Our update procedure is based on a Yahoo! tool called Pogo
- Soon to be released as OpenSource
Jenkins uses the same deploy procedure across all of our environments
Avoids problems with differences between the environments
Jenkins uses the same deploy procedure for bare metal restores and daily pushes
Jenkins can restore a host or an environment quickly and consistently
Questions?
Robert Harker
Sr. Service Engineer
Yahoo! Sports
harker@harker.com