AWS Re:Invent Day 2 Keynote Live Blog

Got here early enough to get coffee and get a good seat! Crap, seat not so good, still can’t see slides well… *sigh*

Werner Vogels on stage - Quick recap of yesterday’s announcements. Says the party artist will be announced at the end of the keynote

Talking about building applications on cloud - Harder, Better, Faster, Stronger than they have ever been. (hint hint)

Services delivered in a broad ecosystem make the difference (trying really to differentiate on services it would seem) vs. just an IaaS platform

Splunk on stage - All core products run on AWS, Splunk cloud (they run it for you), Splunk Enterprise, etc.

What has changed over the last year? Customers are moving from just dev/test and peak apps and moving true production workloads en masse to AWS. Splunk can help with visibility between on-prem and AWS.

Mentions customers - Coca-Cola, Nike and their use cases. often POC on Amazon and then moving production to AWS. Saved time and money using AWS.

Mention of Finra - stock trading security regulator - no more standing up hardware, they moved all applications and Splunk to AWS to focus on what matters, not management of infrastructure. Mention of multiple regions and APIs for scalability 

(I notice almost all guests on stages mention that, must be in the speaker notes for everyone. AWS is hitting scalability, API’s, and services as differentiators)

Werner back on stage - Slide -> AWS is Secure, Adaptive, Resilient, and Global. talking about “pushing a button” to make infrastructure appear

The Application Extends the Platform - talking about importance of API’s and extension of the platform in infrastructure as code and fitting tis into emerging application development models

(As an aside, the Splunk dude that just spoke sat down next to me… awkward)

OmniFone on stage - online music platform, talking about the music industry and the complications of music as an industry. They started with a 15 million platform, it didn’t hold up to the load. They could’t iterate fast enough. They had to start over and started over on AWS. “AWS was the only choice” (Also noticing that as a common theme of the guests, they are all saying it).

Now has a geographically scalable, redundant services across the globe on AWS. Building this platform has allowed the music industry to build what matters. They have delivered more audio/video faster than ever before.

Talking about high res quality sound and the challenges (about 150 times the file size of typical mobile file delivery). How do you deliver the large files in a large uninterrupted stream? talking about Podio (Neil Young’s company?) and what they are doing there

Werner back on stage - Broad Services drive the speed of development, talking about “agility as the Holy Grail” of application delivery. Increasing consumer choice is driving the market to a new model that needs to be agile and fast. Dev & Test is the Core to Agility

Says today budget’s of most CIO’s for Dev & Test are between 40%-60%. How do you optimize that and make that portion of the budget faster.

The Weather Company (The Weather Channel) on stage - talking about weather as a science and data platform. How do you great services based on information you can’t control but potentially affects both business and lives all around the world.

They have built a platform on AWS to feed others (Apple, Google, Yahoo, etc.) to move beyond cable. Also feed data to all major airplanes to help with traffic control. Provide data to local broadcasting companies all over the world. They want to be the “data warehouse” for all things weather in the world.

They didn’t start this way, had a traditional model of physical data centers with physical hardware. They had to change both the infrastructure as well as the culture. 

(I like they brought the human aspect into this, not just technology, so often overlooked)

They choose AWS for scale (scalability point hit again), as well as confidence in the services. Platform has provided close to 100% uptime and weather forecast is less than a second by analyzing over 800 sources around the world. The platform allowed them to “go faster” and constantly improve the accuracy of forecasting over time.

Over 1 billion devices served from the platform between IOS8, Android, and downloads of apps on Mac/PC

Werner back on stage - Development is changing to support agility

Pristine (Google Glass specific company with a focus on healthcare) on stage - They are using AWS and…. drumroll please…. Docker!! (You knew it was coming!)

Slide - Containers are the key to our growth, this allows them to develop once and run everywhere. Rollback are simple, etc.

Talking about the combination of AMI’s for the base image and the layering of containers on top is the “perfect match” for them and allows them to go as fast as possible and scale beyond anything else that is out there.

Werner back on stage - Why do developers love containers? Going into to all the usual containers value proposition. talking about containers do present some overhead challenges set up.

Announcement - AWS Container Service - deploy environment to make containers easy. (huge applause). All with an API, integrates with Docker repositories, also integrates with Mesos

Demo of Docker containers into the system on stage now (I can’t see the screen well sadly). Instances (AMI’s set up), register the cluster with the service. Name the Docker image that will be used, start running the task. Single instance, deploy and scale to 5 instances, deploy front end.

Scale up to 30 instances (different instance types as well)

(Got a call.. had to step out… I’m sure it was awesome… sorry about that)

Docker CEO, Ben on stage - Where isn’t Ben these days?! Good for him and good for Docker….

Developers are content creators - Docker removes the “crap work (my words)” from development and allows developers to go faster.

5 steps to containers - 
1. isolation of process in an OS
2. good API’s to run anywhere
3. create an ecosystem (Docker Hub)
4. create a new container based app model
5. create a platform for managing it all

Talking about Gilt.com - joint AWS and Docker customer, before docker 7 apps and hard to deploy, 300 micro services and 100+ deploys per day

Just passed over 50 million downloads of Docker!

Werner back on stage - Simplification drives reliability and performance

What are the primitives of cloud in an execution environment?

talking about data, triggers, and actions of applications. A data change triggers an action to update other portions of data.

Why don’t we architect that way? need to create a full, complex stack to “run a function and modify data”

Announcement - AWS Lamba - An event driven computing service for dynamic applications. You just write code and no underlying infrastructure (it’s always there somewhere, they are just taking it away so you don’t have to worry about it)

Basically state changes and events drive the system (new pricing model?) - write code without infrastructure.  - (Another PaaS without calling it a PaaS?)

code only runs when needed - cost efficient and efficient

Really interesting concept - Talking about IoT (Internet of Things) and triggers as the new currency

Netflix on stage - talking about micro-derives and Lamba, they can replace inefficient existing services with trigger based serves.

Encoding Media Files is an example - get file from studio, chunk it up, process it, ship out to CDN’s
Backup for Disaster Recovery - they can now do backups based on triggers and events vs. time
Security - when an instance is spun up, trigger security check to make sure it is configured correctly

Werner back on stage - Units of cost for Lamba - number of requests, execution time - there is a free tier for each customers each month - today it is available as a preview.

Announcement - New Instance offering C4 (based on Haswell processor), up to 36 vCPU’s, EBS optimized by default and included in the price.

Announcement - New EBS - SSD backed EBS up to 10,000 IOPS (up to 160Mps) and 20,000 IOPS (up to 320Mbps)

Intel on stage now - talking about C4 instance… speeds… feeds… The processor is actually an AWS exclusive

My take: It would appear they have hit on a few key differentiators to move forward beyond iaaS. Scalability (to differentiate from on-prem), API’s for developers (to differentiate from other public clouds), and services across the broad ecosystem. They want to be the developers model of choice and seem to get the only way to get “next generation applications” is to enable the developers and start down the micro-services and containers path.

Well played AWS… well played…

Over all, super impressed with year vs the keynotes of past years.

AWS Re:Invent - ARC307 - Infrastructure as Code Session

Live Blog of the AWS Re:Invent Infrastructure as Code Session (ARC307)

Packed house - This session is offered in one of the large ballrooms. At least 1,000 people in the session and this session on the live stream as well. David Winter & Alex Corley from AWS as well as Tom Wanielista from Simple.com presenting

David up first - his background is a very traditional datacenter hardware centric background. He had a project to build on AWS.

Started simple with manual spin-up of instances, it wasn’t fast enough one person using a console. He needed to go faster. API was the next step, he then built a bash script. His first steps…

Hired somebody else, they then wrote the same in python. This was the beginning of using this as a “cookie cutter” repository for test/dev. Then one day something bad happened… (Security related event)

Production went down… hard. (Security groups were removed by beta product they were testing), all networking went “deny all” in the security groups, locking everyone in the world out

Had to rebuild them all by hand… (ouch)  How do you prevent this from ever happening again.

AWS Cloud Formation was now the basis for “Infrastructure as Code”. Too much configuration that was done by hand needed to be automated to recover quickly. Also, this allows iteration of new development cycles very fast as a side benefit to go forward.

Alex Corley is up - version control to wrap complex systems and provide a template for roll out. Cloud Formation uses a model methodology to define the infrastructure. You create models in Cloud Formation (CF from now on), JSON structure

CF supports just about all AWS services today (security groups, compute offerings, network services, etc.)

version control is built in CF. Store the intended stated (next rev) in CF and do a code review before it is published. Can use many different repositories (GitHub for example)

Create a template, check it in, code review, deeply worldwide across AWS regions. All automation handled through CF.

Tom from Simple.com - (customer testimony) - simple is a bank. SOA architecture on AWS from Day One. 

They started at the console just like most everyone. As they developed features and grew, this got out of control. They didn’t know who changed what and what happened.

And then along came PCI compliance… No way to audit and report on the current infrastructure. Had to start over from scratch.

Goals: Security / Insight / Growth / Speed - these were the 4 pillars of the new infrastructure.

The rebuilt using AWS Cloud Formation, then they wrote cloudbank in Python. Middleware between simple.com and AWS Cloud Formation from an operations stand point. Everything stored in github. They modify cloud bank, it talks to CF. Jenkins cluster in the mix

cloud bank applies the AWS standards under the covers (security groups, network settings, etc.)

What are the benefits of this automation? They write code every day, they simple added the ability to spin up infrastructure and moved this into the code flow greatly increasing efficiency and agility of the organization. This also makes the infrastructure programmable. For example, there is a chance in PCI compliance, simply push out the change in code. The developers can now handle the infrastructure.

They have evolved from 20-200 people and are still using this method.

David back up - demonstration time (Alex doing demo)

Alex - demo application (web application) running on AWS. Cluster of 5 machines. talking to a git repo, made a chance in the code to increase the size of a graphical fix. commit the change, refresh and it was fixed.

Supposed to be 5 machines, only one is talking. Modified Cloud Formation instance to talk to 5 hosts, commit, refresh the app, now more instances talking to the front end.

Last issue, throughput is insufficient now.  Double infrastructure now from 5 machines to 10 machines to get more bandwidth to the front end. This spins up 5 new AMI’s, some custom configuration and insertion into the cluster, all done with a commit

Application problem, security problem, infrastructure problem - all three fixed through the same process and change management model

Wrap Up:

Good for startups - Agile, developers ramp quickly

Good for Enterprises - Template driven, compliance oriented infrastructure

AWS Re:Invent Day 1 Keynote Live Blog

Place is PACKED. Over 13,000 here. I’m way in the back, can’t see the slides on the screen well but can see the big screens showing the current speaker. I should have been here earlier but needed coffee. Priorities…

TL;DR - See My take at the bottom

Jassy up first - over 1 million active customers, lots (and lots) of logos slides (public sector, Enterprise, SI’s, etc.)

AWS Market Place - huge growth, 2000 offerings, 7 mil in downloads

Slide about Enterprise IT Vendors and how most large multi-billion “Enterprise IT Vendors” are all shrinking while AWS is growing (yes, they included themselves in that category)

Lydia Leong (Gartner) quote thrown up on the screen - he is really trying to embrace the Enterprise vs. just telling them they are doing it wrong in past years keynotes

Moves on to the “Old Way” of doing things, Enterprises spend millions for slow, inflexible infrastructure and software

AWS is the “The New Normal” - multiple data centers (mentioned fault tolerance), 11 regions, 28 availability zones - went on to mention all the features that are built into every region (backup, identify management, monitoring, analytics), complete offering of services and all offered on demand as needed and can spin up as needed (This has to be the longest list of features I’ve ever heard, he has been going on for about 3 minutes and I’m not sure he has taken a breath)

Still going…. 

Now talking about the features in the service. Many others offer a basic service, AWS goes deep on most offerings (another list of offerings, he is going into compute and how they are differentiated i.e. GPU specific, small compute, large compute, etc.)

Still going on list of services… Jassy is the Energizer Bunny of feature lists

First Customer is up - MLB (Major League Baseball) - CTO of MLB.com, started from scratch, now a six billion business for MLB. They built a PaaS they share with other providers (ESPN, etc.). Want to be on any screen at anytime for events. StatCast is hosted on AWS, new system to go really deep and apply big data and prediction to baseball stats and players.

How do they capture the data? Radar sampling that tracks ball over 2000 a second, can “see” the baseball rotation it is that accurate. 17 Petabytes of data per season. AWS was the only one with scale and bursting capability (what do you do in offseason when you don’t need it). Keep adding to data warehouse over time to provide historical stats.

How does it work - collect data locally, use Amazon Direct Connect to export into AWS. From there MLB’s real time PaaS delivers StatCast to devices

Example - Breakdown of play during the World Series, shows how runner started slow (because he thought is was easy) and then sped up at the end. He was out by .2 second. If he ran the whole time, he would have been safe by over a foot.

Jassy is back - talking about transformation to Cloud Native Applications. You don’t have the option to move slow anymore.

Second Customer is up - CEO, Healthcare Company (sorry, didn’t catch the name, Phillips maybe) - going through a real world customer use case who had cancer and how they determined this (took blood that indicated it, found the cancer, showed patient how to adjust lifestyle and live with it vs. radiation treatment). This was real time data and fitting a treatment to the customer vs. other traditional alternatives using big data.

How do we turn a mountain of data into Actionable Items? This is where real time data comes into play. They are adding a PetaByte a month to the system right now (common theme here of scale and how no one else can scale like AWS). No one can support the large amounts of data.

Jassy back - Slide - Is there hope for a new normal in the area of relational databases? Old world DB’s are expensive, locked in. Many Enterprises are looking to MySQL and PostGres as an alternative. The OSS DB’s are hard right now….

(Announcement) - Amazon Aurora - Commercial Grade Database Engine - in development for 3 years, MySQL compatible but at 5x performance, same or better availability than Enterprise versions at 1/10 of the cost of the leading solutions in the market.

Product dude brought out for Aurora (didn’t catch his name) - Biggest Enterprise pain today is world class databases. They started with a blank slate and knew they wanted MySQL compatibility.

Compatibility with MySQL 5.6… 6 million inserts per minute, 30 million selects (I heard some folks around me saw wow to that one, I guess that is a big deal), data automatically backed up to S3 and highly available, crash recovery in seconds, database cache survives restart (no warming). Most features available only in Ent. class offerings.

Offered at .29 per hour (audience clapped at that)

Jassy back on stage - Talking about Software Deployment now. Pushed 50 million deployments in last 12 months using “Apollo” (codename for their internal project… I sense an announcement coming)

(Announcement) - AWS Code Deploy - Central monitoring and control, works with “virtually any” language and tool chain set, available today, free to use. Performs roll backs of code as well as commits. 

Talking about CI/CD now. Develop, Build&Test, Deploy, Monitor & Analyze

(Announcement) - AWS Code Pipeline - Integrates with existing tools, used internally in Amazon

(Announcement) - AWS Code Commit - code repository without size limits.

All exist together and work with external partners. (wonder who they will play nice with)

Now talking about compliance - They are now ISO-9001 compliant. They have been working with healthcare customers to achieve this level of certification.

Security up next - talking about encryption

(Announcement) - AWS Key Management Service - Encryption, IAM and policies all in one place (sorry for lack of details here, had to take a call)

OK, back…

Talking about Service Catalog (coming in 2015) - AWS Service Catalog, create a grouping of resources, create an offering, serve it out in a service catalog… They say Enterprises want this

(This *COULD* be interesting. I talked to Ent folks about this years ago and it never took off because it was too hard or costly to create the offerings and serve out the catalogs to multiple clients. If they make this easy to consume and usable, it could take off IMO. Enterprises want it but never really adopted it at scale. This was the original Enterprise vision of “cloud”, a portal of services)

Talking New Applications vs. Old Applications (here comes the Jassy we know and love… bring on the part where he tells everyone they are doing it wrong and need to do it the AWS way)

Dev/Test - Many Enterprises are using Dev/Test as a starting point for AWS. 
Mobile - The future of applications and architecture
Talking about companies migrating fully over to AWS. Feels like the days of virtualization (we want to be a 100% virtualized environment!). I doubt that will ever happen. Some workloads might go AWS…

CTO of Intuit on stage - They are moving all their applications to AWS. As Intuit evolves into a majority SaaS company. Over 8,000 employees, 3,000 engineers. Multi-billion online and mobile services. Had lease on datacenter up and migrated over to AWS. 6x cost savings, 1/5 of the time for buildout, developers were able to move faster. Over time this trend increased, starve the old, build new in AWS. Many acquisitions were built on AWS so that made absorbing them into Intuit very easy.

Jassy back - Talking about Hybrid Infrastructure (not Hybrid Cloud according to AWS). Jassy talking about a lot of Enterprises that still have on-prem resources because they aren’t ready to move to cloud. Talking about all the Hybrid features (VPC, Direct Connect, vCenter Integration, Access Control, Directory Service).

CTO of Johnson & Johnson  - 270 operating companies in 60 countries, 100,000+ employees, more stats,,, blah blah blah…

Thousands of Servers, Complex IT Ops - new strategy, less servers, automated IT, greater business efficiency

120 applications running in AWS now, plan to triple that in the next 12 months (they have to have THOUSANDS of apps, so I wonder what the percentage actually would be)

They want to move to Amazon Workspaces for Desktops

Jassy back - Slide - Partnering is the new normal (Announcement coming?)

Talking about culture of AWS - Customer focus comes first, AWS is pioneering (first to market), long term orientation

They will never call you at the end of a quarter to close a deal to make numbers (difference between am OPEX subscription model vs. a CAPEX purchase model)

AWS as a trusted advisor, Cost Optimized Service and Advice - over a 350Mil in cost reductions on behalf of customers

My take: Keynote felt very different from past years, company has moved from announcing more offerings (look, new compute offerings!) to announcing services to expand the ecosystem. Makes sense as the growth has slowed and they need to pick it up. Felt like a VMworld keynote from 5-7 years ago. A company that is starting to branch out and may very well start eating their own ecosystem so they can continue to grow. Also thought it was weird the pre-announced a few things this year. Not sure if they didn’t get them out in time but pretty sure they haven't done that before. AWS has gone from the “stealth IT little guy” poking the Enterprise in the eye and telling them they are doing it wrong and is now embracing the idea that they need the Enterprise and they now need to be nice to them. The fact that Jassy didn’t crap all over “Hybrid Infrastructure” and actually talked about it at the end helps prove this point.

I believe the Aurora and CI/CD announcements will move the needle and look really awesome. The security announcements were needed to fill out the Enterprise portfolio. The Service Catalog could be interesting when it releases.

OpenStack Summit Session Voting - Please Vote!

Time to dust of the blog and beg some folks for votes on OpenStack Summit sessions...

First off, here are some great sessions I would love to see and I encourage you to vote for! There are so many submissions picking a few is difficult:

Scott & Ken's great session on VMware & OpenStack: OpenStack for VMware Operators

Getting Started with OpenStack

OpenStack Performance Tuning

Multitenancy with Cinder: How Volume Types Enable It

Lastly, I have three sessions up for consideration, please vote if you are interested and I hope to see everyone in Paris!

Predictable Cinder Performance with SolidFire Storage

Building a Cloud Career in OpenStack

Ask the Experts: Challenges for OpenStack Storage

April Recap

My trend of posting monthly recaps a few days late continues...  Sorry about that, hopefully the May recap will be on-time.  I was traveling most of April so the blogs this month tend to reflect that.

I'll start with the Cloudcast (.net) for the month of April.  We published a record number of episodes. A HUGE thanks to both Amy Lewis and Brian Katz for their amazing contributions!  Amy did a fantastic job as roving reporter and Brian's Mobilecast is really taking off!  As always, please send us any show feedback, we love to hear from you!

• The Cloudcast #81 - Data Gravity Meets Lean Analytics - w/ Dave McCrory & Alistair Croll
• The Cloudcast #82 - SDN, Big Data, Internet of Things and What's Next - w/ Lew Tucker & Dave McCrory
• The Cloudcast #83 - Accelerating Hybrid Cloud Options - w/ Rajeev Chawla
• The Cloudcast #84 - Red Hat OpenShift - Are We There Yet? - w/ Diane Mueller, Ryan Jarvinen, Krishna Ramen 

• The Mobilecast #3 - Identity and Access Management -  w/ Paul Madsen
• The Mobilecast #4 - Data Categorization and Security - w/ Bill Pelletier
• The Mobilecast #5 - App Development and Lessons Learned at Festo - w/ Steve Damadeo
• The Mobilecast #6 - WiFi, Small Cells and Mobile - w/ Art King

Next up is my new TechTarget Blog, you have subscribed with your latest Google Reader replacement, right??  I'm really having a good time writing over there.  This site (aarondelp.com) has always been more hands on and live blogs from events but the interest in the latest trends around Open Clouds and the operational aspects of cloud computing has been both great and humbling.  Thank you to everyone who has taken the time to read the articles and provide feedback!

• Will "THE CLOUD" Always Be Just Out of Reach? - My thoughts on why there is no silver bullet in the cloud computing world and why it all comes down to one thing: workload
• Is PaaS the Gateway Tool for the Enterprise Cloud? - Will PaaS be the nudge the Enterprise needs to accelerate cloud adoption?
• Impressions From The OpenStack Summit - Couldn't make the OpenStack Summit? Here are my thoughts
• Expectations For the AWS Summit - Here is what I was looking for going into the AWS Summit
• AWS Summit Wrap Up - My Summary of the AWS Summit event in San Francisco

The only blogging I was able to do on my site this month is Live Blogs from the AWS event.  Here are all of them.

• AWS Summit Keynote Live Blog
• AWS Summit Live Blog: Introducing AWS OpsWorks Session
• AWS Summit Live Blog: Cloud Backup and DR Session
• AWS Summit Live Blog: Hybrid IT Design with RightScale

Thanks again for coming by and I'm looking forward to May!

AWS Summit Liveblog: RightScale - Hybrid IT Design

Usual Liveblog disclaimer: typing this as I go in the session, please excuse typos and formatting issues

Title: Hybrid IT - Steps to Building a Successful Model - presented by RightScale
Presenter: Brian Adler, Sr. Services Architect, RightScale & Ryan Geyer, Cloud Solutions Engineer

Brian is services, this won't be a product pitch ;)

RightScale is a CMP (Cloud Management Platform) - provides configuration management, an automation engine, as well as governance controls and does both public and on-premise clouds (I think the word private cloud must be on the naughty list at the show, all pitches do NOT use the dirty "p word")

RightScale allows management & automation across cloud resource pools

basic overview of terminology and where we have come in IaaS to Cloud Computing today

On-Premise Key Considerations

1. Workload and Infrastructure Interaction - what are the resource needs? Does this make sense in the cloud and which size instance would be best?  Instance type is very important
2. Compliance - data may be contained on-prem for compliance
3. Latency - does the consumer require low latency for a good user experience
4. Cost - the faster it has to go (latency) the more expensive it will be in the cloud
5. Cost - What is the CAPEX vs. OPEX and does it make sense

Use Cases

1. Self-Service IT Portal (The IT Vending Machine) - Users select from fixed menu, for example, pre-configured and isolated test/dev

Demo Time - Showing off an example of a portal using the RightScale API's, basically push a big button, enter a few options, let it spin up an an environment, in this example they provisioned five servers and a php environment in a few minutes

2. Scalable Applications with Uncertain Demand - This is the typical web scale use case, fail or succeed very fast in the public cloud. "See if it stucks", once it sticks, maybe pull it in house if cost reduction can be achieved when the application is at steady state

3. Disaster Recovery - Production is typically on-premise and DR environment is in the cloud, this is often considered a "warm DR" scenario - replication in real time database from production to DR, all other servers are "down".  You then spin up the other servers and the DB is already up and running, then flip the DNS entries over when DR is up and running.  You can achieve an great RTO & RPO in this example.  You can also do this from on AWS region to another.

Demo Time - Showing RightScale Dashboard with a web app demo + DR.  Demo had 2 databases, master and slave replicating and in different regions (side discussions about WAN optimization and encryption here as well), Production in the example was in US-East AWS and DR was US-West AWS.  The front end of the app was down in West.  When you launch the West DR site, it will go and configure everything and automated as part of the server template.  All DR happens just by turning up the front end in West

Design Considerations

Location of Physical Hardware- again speed vs. latency vs. location

Availability and Redundancy Configuration - This can be easy to hard depending on your needs

Workloads, Workloads, Workloads - Does the application require HA of the infrastructure? Will it tolerate an interruption? Can it go down?  Will users be impacted?

Hardware Considerations - Do you need specialty? commodity?

(Sorry, he had others listed, I zoned out for a slide or two..)

On to Hybrid IT - Most customers start out wanting "cloud bursting" but most often an application is used in one location or the other.  Check out the slide for the reasons.

Common practice is a workload is all on-premise or public. Burting isn't a common use case.  If they do use bursting, they set up a VPC between private and public to maintain a connection.

Demo Time - What would a hybrid bursting scenario look like in the RightScale dashboard?  Customer has a local cloud that is VPC connected to AWS.  Load Balancers, one is private, one is in AWS.  They are using Apache running on top of a virtual machine to maintain compatibility between private and public.  DNS is using Route 53 (AWS DNS).  RightScale uses the concept on an Array.  As RightScale monitors the performance, additional instances are fired up and "bursted" or scaled out to AWS above and beyond the local already running resources.

You do not need the same LB's on the front end like the example above.  For example could be in a local CloudStack/OpenStack environment with a hardware firewall in front but also include AWS and AWS ELB in the rules as well

Take Away - It is very possible to use both public and private and there isn't a need for a "one size fits all approach"

Great session, probably the best session of the day so far for me today.

AWS Summit Liveblog: Cloud Backup and DR

Usual Liveblog Disclaimer: This is type as fast as I can, blog may contain typing and formatting errors, sorry about that

Session: Technical Lessons on how to do Backup and Disaster Recovery in the Cloud (whew, long title)

Presenter: Simone Brunozzi, Technology Evangelist

Simone presented in the morning keynote on the Enterprise demo, good presenter

3 parts = HA -> Backup -> Disaster Recovery

HA = Keeping Services Alive

Backup = Process of keeping a copy

DR = Recover using a backup

(Simone has is using great examples using churches and monasteries but too long to type all of that out here.)

5 Concepts of DR

1. My backup should be accessible - AWS uses API's, Direct Connect, customer owns the data, redundancy is built it, AWS has import/export capabilities

AWS Storage Gateway as an example, using a gateway cache volume on-premise that will replicate to a volume in AWS public cloud, S3, snapshots, etc.  Can be a GW-cached or GW-stored (one is a cache, the other is a full offline copy). Secure tunnel for transport over AWS Direct Connect or Internet

2. My backup should be able to scale - "Infinite scale" with S3 and Glacier, scale to multiple regions, seamless, no need to provision, cost tiers (cheaper options and at scale are available)

3. My backup should be safe - SSL Endpoints, signed API calls, stored encrypted files, server-side encryption, durability: multiple copies across different data centers, local/cloud with AWS Storage Gateway

4. My backup should work with my DR policy (I don't want to wait 10 years to recover) - easy to integrate within AWS or Hybrid, AWS Storage Gateway: Run services on Amazon EC2 for DR, cleat costs, reduced costs, You decide the redundancy/availability in relation to costs.

5. Someone should care about it - Need clear ownership, permission can be set in IAM with roles, monitor logs

Now a customer story:

Shaw Media - Canadian Media Company, before AWS - multiple datacenters, lot of equipment, downtime, different technologies across datacenters - they were told to change everything and become more agile and cost effective in the next 9 months to better serve the business

Solved the issue with AWS, fast deployment of servers, network rules, and ELB on AWS, first site in only 4 weeks, after that a full migration of 29 sites from a physical DC in 9 months - This was Phase one (This was main websites)

Phase Two - Other web services migration was next (check out the picture for the details), impressive stats.  Typical web servers, apps servers, database servers, etc.

Lessons Learned - went to fast, didn't catch it... damnit

DR - Learn from your outages (test your policy on a regular basis and refine the document)

(Sorry, he's going to fast to type or even take pictures of the slides.... Really wish he would he gone slower in this section, the content was really good grrrrrrr)

Lessons to learn from DR

1. You NEED a DR plan in place - how will you recover?  Can your business survive without it?  For AWS, across Availability Zones (AZ's) or App DR with Standby (see pictures).  The second option is cheaper to implement but will take a little longer to recover from.

 

Perform a business analysis of RTO & RPO (if you don't know what that is, Google it, you need to know what it is)  In a nutshell, RTO, how long to get it back, RPO, how much data can I lose?  This is the typical cost vs. performance trade off.  Take the various AWS services as an example:

2. Test your DR - Many may say Duh! to this one but I'm always surprised how little customers actually do this.  The ability to spin up capacity just for DR testing helps to minimize cost and the ability to not have a DR site to manage is pretty cool. Data Transfer speeds (Data Gravity) could be an issue in this kind of scenario

3. Reducing Costs - Took a screenshot, it was easier

Overall - great presentation although I wish he would have spent more time on the customer slides as there was some good technical content there...

AWS Summit Keynote Live Blog

This is a live blog from the AWS Summit Keynote by Andy Jassy.  The usual disclaimer applies, I'll be typing fast and furious so expect misspellings and some formatting errors.  Also, no Internet in the keynote (MiFi or conference) so I'll be moving this over to the blog after the keynote.

There are a TON of people at the event (I'll see if they announce numbers but easily in the thousands), impressive

Intro videos going on now…

Andy Jassy in on stage - starts with the age of AWS, 7 years old, March 2006

Now digging into the breadth of the services - they are very proud of the pace of innovation (see pictures attached)

With the exception of 2010, they have doubled the number of services every year, up to almost 160 services available today

71 new features so far in 2013

9 regions, 25 availability zones, 39 edge locations - also talked about the GovCloud and the requirements on it to support Public Sector workloads

Amazon S3 - Over 2 Trillion objects, 1,100,000 peak requests/sec

He's firing facts and figures now so fast I can't keep up. Nothing but speeds and feeds and stats to impress. He's talking very fast

Talking about customers and user base

 

Use cases - talking about the use case is really abut building blocks and letting the developers decide how to stitch together the blocks, AWS was not going to dictate the use cases

Talking about security - security is number one priority at AWS, talking about features access control from the edge, dedicated instances, encryption, etc.

Certifications are more important than security - They are HIPPA, ISO, SOX, FISMA, etc.

Now moving on to pricing (he's talking really fast, no transition in between topics)

They plan to remove cost from process and pass on to customers, 31 price drops to date, the more customers they have, the better economy of scale, they consider this a "wheel" more customers drives price drops which brings in more customers

AWS Trusted Advisor - checks for cost optimizations, security and availability checks, performance recommendations (running on demand vs. reserved instances for instance), pretty cool stuff.  I remember hearing about this but never dug into it.  It appears they are trying to change the mindset about steady state apps, they have brought this up a few times that you can run steady state in cloud, but need to do it on a reserved instance.

Now on to partners (again, no real transition) - The usual impressive list of both consulting and technology partners

AWS Marketplace - Their "App Store", 25 categories, 778 product listings - applications already configured and certified on the AWS ecosystem

Why are customers adopting cloud computing? (finally, a real transition)

1. Trade Capital Expense (CAPEX) for Operating Expense (OPEX) - $0 to get started and can fail fast if needed

2. Lower Variable Expense than most companies can do in house - they mention again how large they are and the economies of scale to pace on t customers (seems to be their new message) - They appear to be positioning themselves as the "Walmart of the Cloud" - Low Price Leader and pass savings on to you

3. You Don't Need to Guess Capacity - Talking about the typical predict up front model, what happens if you build it and nobody comes? What happens if too may people come?  If the infrastructure is elastic no need for this planning and predictive step

4. Dramatically Increase Speed and Agility - Old World server request, usually takes weeks to get servers for development, AWS takes minutes and is all self service - compares development to invention, need to perform a lot experiments, need to experiment and fail with little to no cost or collateral damage, speeds up development

5. Stop Spending Money on the Undifferentiated Heavy Lifting - They do all the "infrastructure stuff" for you, talking about how the infrastructure typically doesn't differentiate your business in anyway but it also consumes a lot of resources in operations.

6. Go Global in minutes - Because of Regions and Availability Zones the ability to scale and go grow to a different region is much easier. No need to set up operations in another area of the world

Message is very Enterprise centric (no surprise there)

Sean Beausoleil is on stage now - lead engineer for Mailbox - 2 years ago - talking about their first product, it worked but wasn't "sticky" enough, the reason was because email still held most user's data. How to tackle the mailbox as a better tool and task management

Now a video about Mailbox uses - In case you haven't tried it, Mailbox basically turns your mail into a to-do list. They were overwhelmed with the response to the initial movie that was release as a preview. They needed a massively scalable back end to support. The product pulls from IMAP -> Cloud -> to device (see picture)

They knew they would need a massive backend on AWS, they copied their existing system to AWS, they found a lot of bottlenecks in the app as they scaled up in testing.  They were able to test AHEAD of production.  Some components of the app were rewritten.  That is why the introduced the reservation system some of you that got the app may have seen.  (I was on that list)

The created the reservation system so they could scale over time until they were sure they could scale.  Even all this preparation didn't prepare them for the growth.  They were handling 100 Mil emails a day in 2 months from launch.  They are able to re-architect on the fly, comment was "you can't predict what production will look like until you are in production". I couldn't agree more based on past experience

AWS allowed them to optimize and scale and perform swaps of hardware instance sizes on the fly to balance the usage against the costs.  They would model the workload and perform swaps of hardware seamlessly in the background with no downtime.  I have to admit, that is pretty frckin cool.

Andy is back - AWS adoption into the Enterprise is the topic now

Andy is now talking about how most "old guard" are pushing for private cloud. He states none of the 6 points above are available in private cloud. He says old guard is high margin business that isn't the same as AWS. He is now talking about a balance of "old" on premise resources and new cloud era workloads - talking about Amazon Direct Connect, LDAP integration, VPC, etc. Says these tools to move from on-premise enterprises are the focus going forward. Mentions BMC and CA as partners in the future for single plane of glass management

How are Enterprises using AWS?

Strategy 1: Cloud for Development and Test - first and most common use case

Strategy 2: Build New Apps for the Cloud - this is the next generation of applications. Retire the old and create new apps, faster to build, less expensive to run, easier to manage, etc

Strategy 3: Use Cloud to Make Existing On-Prem Apps Better - Take in house apps and outsource the analytics for example for processing in the cloud. They mentioned a few enterprises including Nasdaq that do this today

Strategy 4: New Cloud Apps the Integrate Back to On-prem systems - AWS serves up the front end and the processing is on the back end on-prem

Strategy 5: Migrate Existing Apps to Cloud - he admits this is emerging and often requires consulting services, taking that very traditional workload and move it to the cloud

Strategy 6: All in - NETFLIX!  No keynote is complete with out them…

Now up - Demo of Enterprise and cloud by Simone (need his name)

They want to show you how AWS is relevant in the Enterpise

3 parts - Authentication, Integration, Migration

Authentication - Talking about Okta, an AD integration partner, brings AD into the AWS, Created an AWS Admins group in AD and it will talk to AWS IAM and preform the changes to needed to access AWS - AWS admin rights

Integration - Storage Gateway for Backup and Recovery Volumes - volume on premise - replicates to S3, replication of data happens, stand up an EC2 instance and attach to the volume on AWS if needed - talked about iSCSI targets and how to attach them (that brings back memories). Once this is done you could map back to on-premise (little fuzzy on the details)

Migration - Talking about moving export an image from VMware vCenter on-premse, transfer to AWS as an image (AMI). From there you can copy to another region. the example here is move to USA first and then transfer to Singapore.  I admit the use case of moving region to region is really cool.

Talking again about the perception of AWS and the Enterprise. The is obviously a focus.

What ar ether working on next? Amazon VPC is a focus (to continue to build the Enterprise), Direct Connect, Amazon Route 53 (DNS Services)

I'm actually gonna bail on the rest of this so i can go get a seat in the labs before they fill up. (Scratch that, line is so long for the labs they are useless)

They appear to be positioning themselves as the "Walmart of the Cloud" - Low Price Leader and pass savings on to you.  Key message also was to recognize that Enterprise will continue to use on-premise

Summary - Good stuff, it is good to hear them focus on the Enterprise and do it an a way that isn't as in your face as it was at the AWS:ReInvent conference.

March Recap

This post is a few days late but I wanted to put together a recap of everything that has been happening in March. To say March was a busy month was an understatement!  I'm not sure how much content I'll be able to post here in April as I have two speaking engagements to prepare for and I have decided I'm going to transition this blog away from Blogger and Feedburner to a WordPress hosted site.  Look for the new site probably sometime in late May based on my schedule right now.

March was our busiest month in recent memory at The Cloudcast (.net).  We published seven podcasts in March including the beginning of our expansion plans with our first podcast branch, the Mobilecast, as well as our first in a series of guest hosts, the always awesome Amy Lewis at Cisco.  Our goal for 2013 is to extend our reach into areas people have told us they want as well as some new faces to the podcast.  Please tell us what you think!

The Cloudcast #76 - Bringing Depth to PaaS for Real World Deployments
The Cloudcast #77 - OpenStack, PaaS APIs, Platform Tools, Automation & News
The Cloudcast #78 - Open Source Software 101
The Cloudcast #79 - DevOps Evolution and the Phoenix Project
The Cloudcast #80 - Regional Cloud Madness

The Mobilecast #1 - A Year of Going Mobile
The Mobilecast #2 - Health, Fitness and Wearable Computing

In addition to this blog I have also been asked to blog about Cloud Computing over at Tech Target.  I have a pretty extensive consulting and operations background so I have been asked to think about cloud computing from an operations standpoint.  I'm aiming for at least one blog a week over there.  Please head on over and subscribe to the blog!  I met my goal in March, here are links:

What Happens When Your Cloud Goes Away?
Cloud Applications and Vanishing Software Generations
Will Clouds Ever Be Open?
Impacts of Cloud Workload Consolidation

Last (but not least!) on this site I published two articles, one on the NYC Cloud Computing Meetup I attended and a new semi-regular news link round up I plan to do.

NYC Cloud Computing Meetup Recap
In Case You Missed It #1

As always, thanks to everyone for coming by and look for big changes coming "soon"!

Links to Everything CloudStack Collaboration Conference

Lions and Tigers and CloudStack Monkeys Oh My!

This weekend the first ever Apache CloudStack Collaboration Conference was held in Las Vegas!  A HUGE thanks to David Nalley, Karen Vuong & Joe Brockmeier for pulling off such an incredible event!  If you thought the cloud wasn't "real world", take a look below. The content was amazing!

This document is intended to be a living archive of the content from #ccc12.  This is by no means an inclusive list.  If you see a link that I haven't included, please leave me a comment and I will update this article as quickly as I can.  I believe the sessions were recorded and as soon as I get a link I will add it as well.  Thank you!

Keynote Presentations:

• "The Most Interesting Man in the Cloud" Peder Ulander opened the event. I will post his slides shortly.
• Aneel Lakhani from Gartner. Enterprise + Cloud + Open
• Jim Jagielski - Apache Code, Community, and Open Source
• Tom Raftery from GreenMonk - Can We Hack Open Source Cloud Platforms to Reduce Emissions?
• John Willis (@botchagalupe) from EnStratus - Analyzing a Complex Cloud Outage

Networking Session Presentations:

• Rohit Yadav - DevCloud & CloudMonkey - CloudStack Demos; Link to the code.
• Hugo Trippaers - CloudStack & Nicira NVP Integration
• Floyd Strimling - Monitoring CloudStack Deep Dive
• Michael Ducy - DudeOps - Why the Big Lebowski is About Building a Cloud
• Somik Behera - Networking Considerations for CloudStack
• Dan Bode - Integrating CloudStack with Puppet & code link
• Jason Hancock - Running Puppet on CloudStack Instances & YouTube Video
• Ed Laczynski - Solving the Cloud Puzzle - The Complete Stack Explored
• Chiradeep Vittal - The Evolution of CloudStack Architecture
• Andrew Bayer - CloudStack and jclouds
• John Griffith & Dave Cahill - CloudStack and Cloud Storage
• Roeland Kuipers - Mission Critical Cloud Computing
• Uri Cohen - Building a Carrier Grade PaaS with CloudStack
• Geoff Higginbottom - Introduction to CloudStack Networking
• Chip Childers - 6 Months In: What I've Learned about Apache Projects
• Diane Mueller - Adventures in Deploying a Private PaaS on CloudStack
• Andy Gross - Riak Cloud Storage
• John Kinsella - CloudStack Secured
• Prasanna Santhanam - Continuous Test Infrastructure
• Dominic Curran - Under the Hood: Open vSwitch & OpenFlow in XCP & XenServer
• Jessica Tomechak - Open Writing! Collaborative Authoring for CloudStack Documentation
• Rajesh Ramchandani - High Value Cloud Service
• Kelcey Damage & Clayton Weise - CloudStack in Production - Considerations & Design
• Duncan Johnston-Watt - Quality Control in a Cloudy World
• Chiradeep Vittal - Apache CloudStack: A Not So Cloudy Future
• Alex Huang - Apache CloudStack Evolution Proposal
• Funs Kessen - Storage in a Mission Critical Cloud(stack)
• Marcus Sorensen - Implementing CloudStack's VPC Feature
• Marcus Sorensen - Using CloudStack with Clustered LVM
• Donal Lafferty - Supporting Hyper-V 3.0 on Apache CloudStack
• Ben Cherian - Making a Case for Distributed Overlay-Based Network Virtualization
• UPDATED: Mice Xia - Integrate 3rd Party Security with CloudStack

Announcements:

• CloudCat - Manage & Report on CloudStack - Site & Code Links
• Basho Riak CS supports Multi-DataCenter Replication

My apologies ahead of time for any typos and/or botched names. It is very late on a Sunday but felt it was important to get this out as quickly as possible.  Just about everyone listed here is on Twitter so feel free to look them up!  Thank you again for everyone who attended the conference!!