Hi Guys
I recently attended my first ever PuppetConf’13 at SF last week. It was a Great Experience, gained a lot of exposure to new Tools, observed how many are solving complex problems in the areas of High Availability, Scalability of Infrastructure, and New Technologies, the Big Enterprises are Investing on. I thought, I would share some of the Info with all of you:
To sum up what I saw in the Conf in one sentence:
Infrastructure = Code (CFEngine, Ansible, Salt, Puppet/Chef etc) + Data (Hiera, MongoDB, PostGres SQL etc)Thats exactly, how the whole Industry want to envision the Future of Infrastructure
There will be Tools for Config Mgmt, Event Orchestration and Provisioning that pull Data (for example, username-passwords, network config, application metadata etc) in real time to setup Server Clusters on the Fly/configure/manage them.
Many Case Studies presented there were using a combination of Infrastructure Orchestration Tools like Ansible, Salt, CFEngine etc and Puppet for Configuration Management
There was a lot of informal chat about Puppet Vs Chef:
Puppet & Chef were both written in Ruby/Rails, the big difference in their Arch is in the fact that Chef chose a Internal DSL, so people can write Ruby code for recipes(pleasing for programmers) while Puppet chose an external DSL, which makes writing manifests/modules for less programming savvy Sys Admins for years easy. In fact, I felt, Puppet DSL is pretty easy to learn, I have installed and configured puppet on a few Servers, wrote a simple module and pushed it from a master to clients etc.
Both Puppet and Chef seem to have problems scaling above a few thousand nodes/master, resulting often in, in-consistent behaviour, slow operation etc, much of it, promised to improve with ruby 1.9 and above.
Puppet does not promise to solve the Big Problems by itself, but it being used in combination with other Tools like Facter, Mcollective, Heira etc to achieve the objectives.
Many Puppet Admins, seemed to have multiple puppet masters for each functional server group, location, geo etc. They rely on other tools like mcollective or cfengine to keep the puppet masters in sync. There are many mentions about Headless/Master less Puppet, wherein, they use other tools to push the newest manifests etc to all the peppet clients running on these machines and puppet would figure out the HOW and Maintain the machines in desired state.
Puppet now supports, limited functionality without root as well.
Some mature Orgazinations are doing CI for Puppet, testing their Puppet Configurations in the Dev/Int/QA/Prod Env’s along with their App Lifecycle and Code Promotion process. There was an interesting demo on Continous Delivery for Puppet (that product still in POC stage)
I went to the Conf to know answers to specific problems:
How can we do some IT Asset Management and Monitoring based on the Dell Service Tag number
Ans: Puppet Facter
We can write our own modules that would query the puppet servers for facter operations and do the needful operation. An example of Facter Command on existing server:
[root@server ~]# facter
architecture => x86_64
augeasversion => 0.9.0
boardmanufacturer => Dell Inc.
boardproductname => 0VHRN7
boardserialnumber => .3PC78X1.XXXXXXXXXX12.
domain => modeln.com
facterversion => 1.6.18
fqdn => server.modeln.com
hardwareisa => x86_64
hardwaremodel => x86_64
hostname => server
id => root
interfaces => em1,em2,lo
ipaddress => XXXXXXXXXXXXXX
ipaddress_em1 => XXXXXXXXXX
ipaddress_lo => 127.0.0.1
is_virtual => false
kernel => Linux
kernelmajversion => 2.6
kernelrelease => 2.6.32-358.el6.x86_64
kernelversion => 2.6.32
macaddress =>
macaddress_em1 =>
macaddress_em2 =>
manufacturer => Dell Inc.
memoryfree => 119.98 GB
memorysize => 125.99 GB
memorytotal => 125.99 GB
mtu_em1 => 1500
mtu_em2 => 1500
mtu_lo => 16436
netmask => 255.255.252.0
netmask_em1 => 255.255.252.0
netmask_lo => 255.0.0.0
network_em1 => 10.1.4.0
network_lo => 127.0.0.0
operatingsystem => CentOS
operatingsystemrelease => 6.4
osfamily => RedHat
path => /usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/db/oracle/11.2.0/bin:/root/bin
physicalprocessorcount => 2
processor0 => Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
processor1 => Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
processor10 => Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
…..
….
processorcount => 32
productname => PowerEdge M620
ps => ps -ef
puppetversion => 2.7.20
rubysitedir => /usr/lib/ruby/site_ruby/1.8
rubyversion => 1.8.7
selinux => false
serialnumber => 3PC78X1
sshdsakey => AAAAB3NzaC1kc3MAAACBAOA8SJayJ+lej0/
swapfree => 4.00 GB
swapsize => 4.00 GB
timezone => PDT
type => Multi-system
uniqueid => 010a9505
uptime => 93 days
uptime_days => 93
uptime_hours => 2237
uptime_seconds => 8055648
virtual => physical
How do Search a bunch of puppet servers, to know, which servers are running
particular version of an Application, a particular Oracle Version etc.
Answer: Writing Custom Puppet Facts, using other Tools like Hiera or a CMDB type of DB that is deeply inegrated with Puppet and which is updated after Deployment etc.
The above can also be solved by configuring PuppetDB and querying the nodes.
There is an Interesting Demo on Aggregating Data with Puppet Dashboard and Puppet Faces
Looks like, Puppet Faces can be used to create custom commands for deployment of apps etc.
There are some modules avilable for Zabbix and Puppet Integration here and here.
Puppet is now shipping with an IDE, that would do all this like syntax-highlight and correction etc, Geppetto Plugin
RSpec seems to be a popular choice for testing Puppet manifests.
Other Interesting Tools I came across gaining pupularity:
Fabric is a Python library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.
This is a new Open Source Project, that PuppetLabs took over from EMC, and intends to sell as a Enterprise Product one day. I attended of the new provisioning tool and found it fascinating, but, still too early a version to jump on.
It is compatible with BOTH Puppet/Chef.
The tool is written in Ruby, follows more of a Client-Server model. There is a central Razor server that continually listens to new requests from clients, it is also the location, where we update some flat files with Tags/models/policies etc. We need to tag our Inventory(HW,) for example, dell_620blade and associate that tag to CentOS policy; then, Razor would automatically install Centos and configure the machine as per the policy, anytime, it discovers a new 620 blade.
The tool also ships with a Razor Micro Kernel which is a minimal OS image that boots the HW, install/run facter to know the inventory of the new Hardware and pass it to server and keep waiting, until it hears back from Razor Server for next steps.
Once the OS is installed, the tool goes the extra mile in handing-off the node to a broker(puppet/chef) which then configure the machine to the desired state, installing/configuring packages etc. This is their Key Selling Point.
This is not a new Tool, but something, that we can definitely try and implement. I attended a talk from a Director at RedHat who mentioned, the future versions of OpenStack would even support cloning Live Real-time Clusters in one go on the fly.
This is an awesome tool that would aggregate/monitor logs(any) from all Servers, with easy to create tags(regex) to mine the logs, it is built with RabbitMQ & Elastic Search, Fast, Free, works in Centralized/standalone modes. Great Puppet Integration available as well.
There is someone from CERN who presented this tool. This seemed to be a very interesting tool too It has a powerful Web based UI/ API for all the hosts, where in, we can login to manage ALL servers, power cycle them if necessary, monitor their performance, nice graphs are rendered built-in too. There is good integration with Puppet too.
Monitoring Tools
Icinga and Other Tools in combination with Graphite
There are some intersting talks from VMWare about their Hybrid Cloud Product, Software defined Data Center and Project Zombie.
There are also talks from folks at Open Shift
The slides of All the Talks are available here: