March Recap

This post is a few days late but I wanted to put together a recap of everything that has been happening in March. To say March was a busy month was an understatement!  I'm not sure how much content I'll be able to post here in April as I have two speaking engagements to prepare for and I have decided I'm going to transition this blog away from Blogger and Feedburner to a WordPress hosted site.  Look for the new site probably sometime in late May based on my schedule right now.

March was our busiest month in recent memory at The Cloudcast (.net).  We published seven podcasts in March including the beginning of our expansion plans with our first podcast branch, the Mobilecast, as well as our first in a series of guest hosts, the always awesome Amy Lewis at Cisco.  Our goal for 2013 is to extend our reach into areas people have told us they want as well as some new faces to the podcast.  Please tell us what you think!

The Cloudcast #76 - Bringing Depth to PaaS for Real World Deployments
The Cloudcast #77 - OpenStack, PaaS APIs, Platform Tools, Automation & News
The Cloudcast #78 - Open Source Software 101
The Cloudcast #79 - DevOps Evolution and the Phoenix Project
The Cloudcast #80 - Regional Cloud Madness

The Mobilecast #1 - A Year of Going Mobile
The Mobilecast #2 - Health, Fitness and Wearable Computing

In addition to this blog I have also been asked to blog about Cloud Computing over at Tech Target.  I have a pretty extensive consulting and operations background so I have been asked to think about cloud computing from an operations standpoint.  I'm aiming for at least one blog a week over there.  Please head on over and subscribe to the blog!  I met my goal in March, here are links:

What Happens When Your Cloud Goes Away?
Cloud Applications and Vanishing Software Generations
Will Clouds Ever Be Open?
Impacts of Cloud Workload Consolidation

Last (but not least!) on this site I published two articles, one on the NYC Cloud Computing Meetup I attended and a new semi-regular news link round up I plan to do.

NYC Cloud Computing Meetup Recap
In Case You Missed It #1

As always, thanks to everyone for coming by and look for big changes coming "soon"!

NYC Cloud Computing Meetup Recap

Last week I was able to attend the New York City Cloud Computing Meetup.  It was a very cool event and Joe Brockmeier presented Deploying Apache CloudStack from API to UI.

Deploying Apache CloudStack from API to UI from Joe Brockmeier

Joe did an awesome job (as always) and the meetup was nicely attended, I would estimate about 40-50 people were in the room.  Here are a few random thoughts and impressions in no particular order:

• The session was very interactive. It took the crowd a little bit to come out of their shell but once they did the discussion was very free form and constructive
• The level of questions were very good.  Many were about how to implement and architecture related questions about specific features. Snapshots in particular generated a lot of discussion on slide 26. It appears we are starting to move beyond the basic cloud definitions and into the nitty gritty of implementations
• There were customers in the room and they greatly helped with the discussions (Thanks Jeff @ DataPipe!). It was great to hear how their real world experiences were put to use and how they were able to tackle some of the issues and concerns brought up
• I like how Joe started with some features of the NIST definition and then added an additional point (see slide 4). I agree with Joe that API access is crucial going forward
• Slide 15 (the architecture overview) generated a lot of real world discussion in the room that I believe was very helpful to everyone

All in all a great event!

In Case You Missed It #1

I'm going to try something new and see how this works. I read a LOT of Cloud Computing news.  When I was speaking on a panel recently I was asked afterwards why I don't share a lot of the news I find interesting and thought provoking.  Great question and here is an initial attempt to do just that.

Below is a list of articles I found interesting over the last two weeks and some commentary on what I see going on in the industry.  I'm still not 100% on the format so let me know what you like and want to see changed.

Events & Misc. Links

• Portability: No Snowflakes For You - David Nalley put together an awesome post on how thinking about portability and how you may be doing it wrong. Best article I've read in awhile and addresses a concern I see over and over again.
• AWS Summit 2013 Announced - I'll be attending the San Francisco event as well as the Boot Camps and hope to blog while I'm there
• Monkigras 2013 Videos Posted - If you want to see some awesome presentations, go check this out. A must attend event to say the least
• The first Triangle (as in the Raleigh & RTP area) OpenStack Meetup is this week
• Simon's On Structure - Another great and thought provoking article by Simon. Are you a Pioneer, Settler, or City Planner?
• Bay Area CloudStack Meetup Recap
• Apache CloudStack Weekly Update - I subscribe to the AWS, OpenStack, and CloudStack Weekly Updates so I can follow as much as possible.  The CloudStack Weekly Update is new so I wanted to highlight it here
• RightScale first to sell Google Compute - GigaOm & DataCenter Dynamics Links

Amazon News
Amazon continues to steam ahead but the last few announcements have been very interesting.  In their quest to add more value (and lock-in) to their ecosystem, a bunch of small companies with products built around their cloud were put on notice.  How does a small startup compete with AWS when they decide to move into that space?  Time will tell...

• Amazon OpsWorks - Build "stacks" and deploy apps easily
• AWS Diagnostics for Windows Servers - AWS pushes into the Enterprise by showing Windows more love
• AWS Trusted Advisor & GigaOm Article - Free trial announced, I see a lot of "little guys" impacted by this one

Open Compute
One big OpenStack story to focus on from yesterday, IBM going "all in" with OpenStack.  I saw this one coming a mile away.  Even though I'm now employed by one of the vendors I posted about I still contend that it depends on which vendors show up to the OpenStack Party. As an outsider looking in it appears HP is "phoning it in" (and a lot of people are leaving), while IBM and RedHat are getting serious.

• IBM Goes All-In on OpenStack - Wired and All Things D Links

VMware

Beating up VMware has become the cool thing to do.  I joked about it on Twitter but I believe the VMware's message from PEX (VMware Partner Exchange) last week sent the wrong message the same way I felt AWS sent out some bad mojo at their conference late last year.  The big guys tend to approach this as all or nothing and everyone else is the enemy (it's their job, don't blame them) but most customers I talk to don't see it this way at all.

• James Staten on VMware's Innovator's Dilemma
• Randy Bias' Analysis of VMware's situation - Randy is Cloudcast Alumni and love him or hate him, he will always tell you his opinion

ApacheCon LiveBlog: Software Defined Networking (SDN) in CloudStack

This is a live blog from ApacheCon that I'm attending this week.  This session is with Chiradeep Vittal.

Usual Live Blog Disclaimer: This is more of a brain dump typing as fast as I can, please excuse typos, format, and coherent thought process in general.

• Introduction is about how does Amazon built a cloud (see his previous session for this part)
• SDN Definition - Separation of Control Plane from the hardware performing the forwarding hardware - Also centralized control
• Central control eases configuration, troubleshooting, maintain over time
• Eliminates the tedious "log into every box" idea of network maintenance, log into controller
• OpenFlow is that SDN? - NO, it is a protocol for the control plane to talk to the forwarding elements
• Control is on the "top" and forwarding is on the "bottom"
• flexibility example, different route based on direction. Box A and Box B, different flow from A to B and B to A if needed
• IaaS and SDN go hand in hand - Agility, API configuration, Scalability,  Elasticity (all the ity's!)
• SDN enables virtual networking - the illusion of isolated networks on a physical wire
• SDN does have issues - Discovery of virtual addresses -> physical address mapping for instance
• He is now going over a multi-tenant topology example:

• CloudStack model - map virtual networks to physical network - define and provision networks and manage elasticity and scale
• CloudStack Network Model is very robust (see pic, too much to type, things in box tend to be SDN functions)

• How de we put this together?
• CloudStack Service Catalog - Cloud users don't see the "guts" of the configuration, the cloud admin or operator designs the service catalog and presents this to the users
• example - Gold Network - LB + FW + VPN using virtual appliances
• Platinum - LB + FW + VPN but using hardware devices
• Now going over topology example of the Gold offering & Platinum (uses Juniper firewall and Netscaler to Load Balance:

• In both examples the users has no idea if they are on the Gold or Platinum network
• Multi-Tier virtual networking - can define application tiers and isolate based on need as well, who is connected where
• Orchestration - He went through the Multi-Tier example and demonstrated all the steps that would have to be down manually (too many to list) and this will all be done through orchestration
• CloudStack Orchestration Architecture (see picture) - plugin Framework allows this to happen

• SDN works with CloudStack through the plugin model, the SDN controller talks to the plugin, today there is integration with Nicira NVP, BigSwitch, Midokura, and CloudStack Native (requires XenServer)

• CloudStack Native Controller uses GRE and and talks to Open vSwitch on the XenServer
• All isolation happens through the concept of a tenant key over the GRE tunnels. Each tenant has a unique key
• What makes the CloudStack controller different? 
• It is purpose built for IaaS and is not a general purpose SDN solution
• Proactive model - Deny all flows except ones programmed by the end-user API - others send to central controller and may have problems at scale
• Use the CloudStack virtual router to provide L3-L7 services (mainly because most hardware doesn't understand GRE today)

ApacheCon LiveBlog: Powering CloudStack w/ Ceph RBD

This is a live blog from ApacheCon that I'm attending this week.  This session is with Patrick McGarry.

Usual Live Blog Disclaimer: This is more of a brain dump typing as fast as I can, please excuse typos, format, and coherent thought process in general.

(No title slide picture this time - missed it)

• What is Ceph - storage that does object, block, and file all in one; block is thin provision, snapshots, cloning - object has REST API
• RADOS (Google it) object store at the lowest level
• Why Object at the lowest level - more useful than blocks, single namespace, scales better, simple API, workload is easily parallel
• Because of this: define a pool (1 to 100's), independent namespaces and object collections
• (Topic change) - Architecture
• aggregate a bunch of different machines so that you can have a "large enough" front end to handle large number of requests in
• In this "pile" you will have monitors. Monitors provide consensus for decisions, always an odd number, do not store data (traffic controllers) to the storage nodes (OSD nodes)
• On an OSD node -> physical disk -> file system -> OSD layer
• CRUSH - pseudo-random placement algorithm for data placement, CEPH "secret sauce", allows for stable mapping and uniform distribution with additional ruled configuration (can apply weights, topology rules)
• How does it work, take an object, talk to monitors, CRUSH breaks it up, places it around according to the rules
• What happens when something breaks? If an OSD node is lost, the ones with the copy of the data replicates the blocks somewhere else according to CRUSH rules and moves on
• How to talk to it? LIBRADOS - library for RADOS, support for C, C++, Java, Python, Ruby, PHP
• Also RADOSGW - Rest gateway compatible with S3 & Swift
• CEPH FS - A POSIX-compliant distributed file system with a Linux kernel
• RBD - reliable and fully-distributed block device sitting on top of the object store
• RADOS Block Device (RBD) - storage of disk images in RADOS, allows decouple of VM from the host, images stripped across the pool, snapshots, copy-on-write clones
• What does this look like? vm's are now split across the cluster, great for large capacity as well as high I/O instances of vm's
• same model as Amazon EBS
• it is a shared environment, so you can migrate running instances across cluster
• Copy-On-Write Cloning (he gets lots of question on this) - think of a Golden Image Master vm and you want 100 copies - You spin the 100 instantly and it takes up additional storage as needed and the vm's grow.
• Question: Is there a performance impact to this? A: No, but as usual it depends on the architecture (how many devices are hitting it)
• CloudStack 4.0 and RBD? via KVM, no Xen or VMW support today
• Live migrations are supported
• No snapshots yet
• NFS still required for system vm's
• Can be added easily as RBD Primary storage in CloudStack
• snapshot and backup support should be coming in version 4.2, cloning is coming, support for secondary storage in 4.2 (backup storage is coing in 4.2)

ApacheCon LiveBlog: DevCloud - A CloudStack SandBox

This is a live blog from ApacheCon that I'm attending this week.  This session is with Sebastien Goasguen.

Usual Live Blog Disclaimer: This is more of a brain dump typing as fast as I can, please excuse typos, format, and coherent thought process in general.

• Today's talk will focus on DevCloud and CloudMonkey
• Sebastien giving overview of IaaS market in general. He was actually an OpenNebula guy prior to CloudStack
• With IaaS setting up a virtual sand box can be tricky since there are a larger number of moving parts: hypervisors, storage, networking
• DevOps - quick introduction to DevOps to help everyone understand why this is such a big movement in the industry right now (bringing development coser to the operations)
• This helps us set up an environment to enable a software defined datacenter that allows for automation at all levels
• Now talking about ASF (Apache Software Foundation) and CloudStack. He has a LOT of analysis around the community. The growth once joining as an incubation project shows a HUGE spike (CloudStack is now the #1 Apache project when it comes to commits)
• On to the internals of CloudStack, goal is to be as agnostic as possible (multi-hypervisor, both block and object storage) 

• Network tends to be the most challenging for new folks (firewall, load balancing, basic networking vs. advanced networking, VPN, etc.) - See the bottom line on the picture above
• Apache 4.0 was released in November, 4.01 was just released, 4.1 set for March. Goal is new release every six months
• Architecture -> Zone(datacenter) -> Pods(rack) -> Cluster (hosts) -> primary & secondary storage -> Instance (virtual machines)
• Centralized management server - can be multiple management servers behind a load balancer and replicated MySQL for large scale

• system vm's are used to communicate from the management server to some features (firewall, secondary storage, etc.)
• (Topic change) - What is DevCloud - CloudStack in a box, aimed at developers but can be a local EC2/S3 "cloud in a box"
• self contained - cloudstack management server, ttyllinux (to stay small), system vm's, MySQL, interface all on one laptop - on a beefy laptop expect a good number of instances
• What is CloudMonkey - cloudstack CLI - great for auto-completion of features, tabular output, help, scriptable, shell interaction, etc.
• Intro - Launch CloudMonkey, you now have a shell to talk to your cloud, need to do a key exchange, then ready to access your devCloud instance
• Demo Time - He is running VirtualBox on a Mac Book Air, he is using a NAT interface, forwarding a few ports needed (8080, 2222, 8443, 5901, 7080) - The vm uses nested vm's to launch inside the virtual machine on the laptop
• 2nd Demo - He is running the 4.01 release on his laptop directly from the sourcecode instead of the devcloud vm as well.
• Back to DevCloud - He shows the system vm's up and running and an instance that is halted.
• Went into Web UI - Gave an overview of the Infrastructure, you will have a zone and pod that is defined (named devlcoud), from there a single host as well
• Secondary storage - NFS storage is built in and emulated, primary storage is "local". No need to stand up an external NFS service
• templates - the system vm's and the small linux template are already included.
• Sebastien went through creation of a new instance using the included tiny instance and shows everything spinning up.
• You can take snapshots (saves to secondary storage)
• The first time a template is used it is pulled from secondary storage and copied down to primary storage
• Global Settings - EC2 API feature turn on if you want to run EC2 commands against it
• Now going over CloudMonkey features
• First thing, set the API key (get this from the UI)
• Now you can do common tasks (list virtual machines, start/stop virtual machines, etc.)
• Another way to use DevCloud: different network type, 2 vNICS, one host only and one NAT
• Build it from source (need Maven dependencies), deploy the database, basically build it yourself. Because you build it this way, there are no zones, pods, etc.  You build everything yourself.
• One thing you can do with this is build your entire infrastructure from scripts. This allows you to test build process of CloudStack for replication.  This is a very powerful use case!

Really great presentation and great overview to those new to CloudStack and DevCloud!

Links to Everything CloudStack Collaboration Conference

Lions and Tigers and CloudStack Monkeys Oh My!

This weekend the first ever Apache CloudStack Collaboration Conference was held in Las Vegas!  A HUGE thanks to David Nalley, Karen Vuong & Joe Brockmeier for pulling off such an incredible event!  If you thought the cloud wasn't "real world", take a look below. The content was amazing!

This document is intended to be a living archive of the content from #ccc12.  This is by no means an inclusive list.  If you see a link that I haven't included, please leave me a comment and I will update this article as quickly as I can.  I believe the sessions were recorded and as soon as I get a link I will add it as well.  Thank you!

Keynote Presentations:

• "The Most Interesting Man in the Cloud" Peder Ulander opened the event. I will post his slides shortly.
• Aneel Lakhani from Gartner. Enterprise + Cloud + Open
• Jim Jagielski - Apache Code, Community, and Open Source
• Tom Raftery from GreenMonk - Can We Hack Open Source Cloud Platforms to Reduce Emissions?
• John Willis (@botchagalupe) from EnStratus - Analyzing a Complex Cloud Outage

Networking Session Presentations:

• Rohit Yadav - DevCloud & CloudMonkey - CloudStack Demos; Link to the code.
• Hugo Trippaers - CloudStack & Nicira NVP Integration
• Floyd Strimling - Monitoring CloudStack Deep Dive
• Michael Ducy - DudeOps - Why the Big Lebowski is About Building a Cloud
• Somik Behera - Networking Considerations for CloudStack
• Dan Bode - Integrating CloudStack with Puppet & code link
• Jason Hancock - Running Puppet on CloudStack Instances & YouTube Video
• Ed Laczynski - Solving the Cloud Puzzle - The Complete Stack Explored
• Chiradeep Vittal - The Evolution of CloudStack Architecture
• Andrew Bayer - CloudStack and jclouds
• John Griffith & Dave Cahill - CloudStack and Cloud Storage
• Roeland Kuipers - Mission Critical Cloud Computing
• Uri Cohen - Building a Carrier Grade PaaS with CloudStack
• Geoff Higginbottom - Introduction to CloudStack Networking
• Chip Childers - 6 Months In: What I've Learned about Apache Projects
• Diane Mueller - Adventures in Deploying a Private PaaS on CloudStack
• Andy Gross - Riak Cloud Storage
• John Kinsella - CloudStack Secured
• Prasanna Santhanam - Continuous Test Infrastructure
• Dominic Curran - Under the Hood: Open vSwitch & OpenFlow in XCP & XenServer
• Jessica Tomechak - Open Writing! Collaborative Authoring for CloudStack Documentation
• Rajesh Ramchandani - High Value Cloud Service
• Kelcey Damage & Clayton Weise - CloudStack in Production - Considerations & Design
• Duncan Johnston-Watt - Quality Control in a Cloudy World
• Chiradeep Vittal - Apache CloudStack: A Not So Cloudy Future
• Alex Huang - Apache CloudStack Evolution Proposal
• Funs Kessen - Storage in a Mission Critical Cloud(stack)
• Marcus Sorensen - Implementing CloudStack's VPC Feature
• Marcus Sorensen - Using CloudStack with Clustered LVM
• Donal Lafferty - Supporting Hyper-V 3.0 on Apache CloudStack
• Ben Cherian - Making a Case for Distributed Overlay-Based Network Virtualization
• UPDATED: Mice Xia - Integrate 3rd Party Security with CloudStack

Announcements:

• CloudCat - Manage & Report on CloudStack - Site & Code Links
• Basho Riak CS supports Multi-DataCenter Replication

My apologies ahead of time for any typos and/or botched names. It is very late on a Sunday but felt it was important to get this out as quickly as possible.  Just about everyone listed here is on Twitter so feel free to look them up!  Thank you again for everyone who attended the conference!!