Showing posts with label DevOps. Show all posts
Showing posts with label DevOps. Show all posts

Thursday, November 13, 2014

AWS Re:Invent Day 2 Keynote Live Blog

Got here early enough to get coffee and get a good seat! Crap, seat not so good, still can’t see slides well… *sigh*

Werner Vogels on stage - Quick recap of yesterday’s announcements. Says the party artist will be announced at the end of the keynote

Talking about building applications on cloud - Harder, Better, Faster, Stronger than they have ever been. (hint hint)

Services delivered in a broad ecosystem make the difference (trying really to differentiate on services it would seem) vs. just an IaaS platform

Splunk on stage - All core products run on AWS, Splunk cloud (they run it for you), Splunk Enterprise, etc.

What has changed over the last year? Customers are moving from just dev/test and peak apps and moving true production workloads en masse to AWS. Splunk can help with visibility between on-prem and AWS.

Mentions customers - Coca-Cola, Nike and their use cases. often POC on Amazon and then moving production to AWS. Saved time and money using AWS.

Mention of Finra - stock trading security regulator - no more standing up hardware, they moved all applications and Splunk to AWS to focus on what matters, not management of infrastructure. Mention of multiple regions and APIs for scalability 

(I notice almost all guests on stages mention that, must be in the speaker notes for everyone. AWS is hitting scalability, API’s, and services as differentiators)

Werner back on stage - Slide -> AWS is Secure, Adaptive, Resilient, and Global. talking about “pushing a button” to make infrastructure appear

The Application Extends the Platform - talking about importance of API’s and extension of the platform in infrastructure as code and fitting tis into emerging application development models

(As an aside, the Splunk dude that just spoke sat down next to me… awkward)

OmniFone on stage - online music platform, talking about the music industry and the complications of music as an industry. They started with a 15 million platform, it didn’t hold up to the load. They could’t iterate fast enough. They had to start over and started over on AWS. “AWS was the only choice” (Also noticing that as a common theme of the guests, they are all saying it).

Now has a geographically scalable, redundant services across the globe on AWS. Building this platform has allowed the music industry to build what matters. They have delivered more audio/video faster than ever before.

Talking about high res quality sound and the challenges (about 150 times the file size of typical mobile file delivery). How do you deliver the large files in a large uninterrupted stream? talking about Podio (Neil Young’s company?) and what they are doing there

Werner back on stage - Broad Services drive the speed of development, talking about “agility as the Holy Grail” of application delivery. Increasing consumer choice is driving the market to a new model that needs to be agile and fast. Dev & Test is the Core to Agility

Says today budget’s of most CIO’s for Dev & Test are between 40%-60%. How do you optimize that and make that portion of the budget faster.

The Weather Company (The Weather Channel) on stage - talking about weather as a science and data platform. How do you great services based on information you can’t control but potentially affects both business and lives all around the world.

They have built a platform on AWS to feed others (Apple, Google, Yahoo, etc.) to move beyond cable. Also feed data to all major airplanes to help with traffic control. Provide data to local broadcasting companies all over the world. They want to be the “data warehouse” for all things weather in the world.

They didn’t start this way, had a traditional model of physical data centers with physical hardware. They had to change both the infrastructure as well as the culture. 

(I like they brought the human aspect into this, not just technology, so often overlooked)

They choose AWS for scale (scalability point hit again), as well as confidence in the services. Platform has provided close to 100% uptime and weather forecast is less than a second by analyzing over 800 sources around the world. The platform allowed them to “go faster” and constantly improve the accuracy of forecasting over time.

Over 1 billion devices served from the platform between IOS8, Android, and downloads of apps on Mac/PC

Werner back on stage - Development is changing to support agility

Pristine (Google Glass specific company with a focus on healthcare) on stage - They are using AWS and…. drumroll please…. Docker!! (You knew it was coming!)

Slide - Containers are the key to our growth, this allows them to develop once and run everywhere. Rollback are simple, etc.

Talking about the combination of AMI’s for the base image and the layering of containers on top is the “perfect match” for them and allows them to go as fast as possible and scale beyond anything else that is out there.

Werner back on stage - Why do developers love containers? Going into to all the usual containers value proposition. talking about containers do present some overhead challenges set up.

Announcement - AWS Container Service - deploy environment to make containers easy. (huge applause). All with an API, integrates with Docker repositories, also integrates with Mesos

Demo of Docker containers into the system on stage now (I can’t see the screen well sadly). Instances (AMI’s set up), register the cluster with the service. Name the Docker image that will be used, start running the task. Single instance, deploy and scale to 5 instances, deploy front end.

Scale up to 30 instances (different instance types as well)

(Got a call.. had to step out… I’m sure it was awesome… sorry about that)

Docker CEO, Ben on stage - Where isn’t Ben these days?! Good for him and good for Docker….

Developers are content creators - Docker removes the “crap work (my words)” from development and allows developers to go faster.

5 steps to containers - 
1. isolation of process in an OS
2. good API’s to run anywhere
3. create an ecosystem (Docker Hub)
4. create a new container based app model
5. create a platform for managing it all

Talking about Gilt.com - joint AWS and Docker customer, before docker 7 apps and hard to deploy, 300 micro services and 100+ deploys per day

Just passed over 50 million downloads of Docker!

Werner back on stage - Simplification drives reliability and performance

What are the primitives of cloud in an execution environment?

talking about data, triggers, and actions of applications. A data change triggers an action to update other portions of data.

Why don’t we architect that way? need to create a full, complex stack to “run a function and modify data”

Announcement - AWS Lamba - An event driven computing service for dynamic applications. You just write code and no underlying infrastructure (it’s always there somewhere, they are just taking it away so you don’t have to worry about it)

Basically state changes and events drive the system (new pricing model?) - write code without infrastructure.  - (Another PaaS without calling it a PaaS?)

code only runs when needed - cost efficient and efficient

Really interesting concept - Talking about IoT (Internet of Things) and triggers as the new currency

Netflix on stage - talking about micro-derives and Lamba, they can replace inefficient existing services with trigger based serves.

Encoding Media Files is an example - get file from studio, chunk it up, process it, ship out to CDN’s
Backup for Disaster Recovery - they can now do backups based on triggers and events vs. time
Security - when an instance is spun up, trigger security check to make sure it is configured correctly

Werner back on stage - Units of cost for Lamba - number of requests, execution time - there is a free tier for each customers each month - today it is available as a preview.

Announcement - New Instance offering C4 (based on Haswell processor), up to 36 vCPU’s, EBS optimized by default and included in the price.

Announcement - New EBS - SSD backed EBS up to 10,000 IOPS (up to 160Mps) and 20,000 IOPS (up to 320Mbps)

Intel on stage now - talking about C4 instance… speeds… feeds… The processor is actually an AWS exclusive

My take: It would appear they have hit on a few key differentiators to move forward beyond iaaS. Scalability (to differentiate from on-prem), API’s for developers (to differentiate from other public clouds), and services across the broad ecosystem. They want to be the developers model of choice and seem to get the only way to get “next generation applications” is to enable the developers and start down the micro-services and containers path.

Well played AWS… well played…

Over all, super impressed with year vs the keynotes of past years.

Wednesday, November 12, 2014

AWS Re:Invent - ARC307 - Infrastructure as Code Session

Live Blog of the AWS Re:Invent Infrastructure as Code Session (ARC307)

Packed house - This session is offered in one of the large ballrooms. At least 1,000 people in the session and this session on the live stream as well. David Winter & Alex Corley from AWS as well as Tom Wanielista from Simple.com presenting

David up first - his background is a very traditional datacenter hardware centric background. He had a project to build on AWS.

Started simple with manual spin-up of instances, it wasn’t fast enough one person using a console. He needed to go faster. API was the next step, he then built a bash script. His first steps…

Hired somebody else, they then wrote the same in python. This was the beginning of using this as a “cookie cutter” repository for test/dev. Then one day something bad happened… (Security related event)

Production went down… hard. (Security groups were removed by beta product they were testing), all networking went “deny all” in the security groups, locking everyone in the world out

Had to rebuild them all by hand… (ouch)  How do you prevent this from ever happening again.

AWS Cloud Formation was now the basis for “Infrastructure as Code”. Too much configuration that was done by hand needed to be automated to recover quickly. Also, this allows iteration of new development cycles very fast as a side benefit to go forward.

Alex Corley is up - version control to wrap complex systems and provide a template for roll out. Cloud Formation uses a model methodology to define the infrastructure. You create models in Cloud Formation (CF from now on), JSON structure

CF supports just about all AWS services today (security groups, compute offerings, network services, etc.)

version control is built in CF. Store the intended stated (next rev) in CF and do a code review before it is published. Can use many different repositories (GitHub for example)

Create a template, check it in, code review, deeply worldwide across AWS regions. All automation handled through CF.

Tom from Simple.com - (customer testimony) - simple is a bank. SOA architecture on AWS from Day One. 

They started at the console just like most everyone. As they developed features and grew, this got out of control. They didn’t know who changed what and what happened.

And then along came PCI compliance… No way to audit and report on the current infrastructure. Had to start over from scratch.

Goals: Security / Insight / Growth / Speed - these were the 4 pillars of the new infrastructure.

The rebuilt using AWS Cloud Formation, then they wrote cloudbank in Python. Middleware between simple.com and AWS Cloud Formation from an operations stand point. Everything stored in github. They modify cloud bank, it talks to CF. Jenkins cluster in the mix

cloud bank applies the AWS standards under the covers (security groups, network settings, etc.)

What are the benefits of this automation? They write code every day, they simple added the ability to spin up infrastructure and moved this into the code flow greatly increasing efficiency and agility of the organization. This also makes the infrastructure programmable. For example, there is a chance in PCI compliance, simply push out the change in code. The developers can now handle the infrastructure.

They have evolved from 20-200 people and are still using this method.

David back up - demonstration time (Alex doing demo)

Alex - demo application (web application) running on AWS. Cluster of 5 machines. talking to a git repo, made a chance in the code to increase the size of a graphical fix. commit the change, refresh and it was fixed.

Supposed to be 5 machines, only one is talking. Modified Cloud Formation instance to talk to 5 hosts, commit, refresh the app, now more instances talking to the front end.

Last issue, throughput is insufficient now.  Double infrastructure now from 5 machines to 10 machines to get more bandwidth to the front end. This spins up 5 new AMI’s, some custom configuration and insertion into the cluster, all done with a commit

Application problem, security problem, infrastructure problem - all three fixed through the same process and change management model

Wrap Up:

Good for startups - Agile, developers ramp quickly

Good for Enterprises - Template driven, compliance oriented infrastructure

AWS Re:Invent Day 1 Keynote Live Blog

Place is PACKED. Over 13,000 here. I’m way in the back, can’t see the slides on the screen well but can see the big screens showing the current speaker. I should have been here earlier but needed coffee. Priorities…

TL;DR - See My take at the bottom

Jassy up first - over 1 million active customers, lots (and lots) of logos slides (public sector, Enterprise, SI’s, etc.)

AWS Market Place - huge growth, 2000 offerings, 7 mil in downloads

Slide about Enterprise IT Vendors and how most large multi-billion “Enterprise IT Vendors” are all shrinking while AWS is growing (yes, they included themselves in that category)

Lydia Leong (Gartner) quote thrown up on the screen - he is really trying to embrace the Enterprise vs. just telling them they are doing it wrong in past years keynotes

Moves on to the “Old Way” of doing things, Enterprises spend millions for slow, inflexible infrastructure and software

AWS is the “The New Normal” - multiple data centers (mentioned fault tolerance), 11 regions, 28 availability zones - went on to mention all the features that are built into every region (backup, identify management, monitoring, analytics), complete offering of services and all offered on demand as needed and can spin up as needed (This has to be the longest list of features I’ve ever heard, he has been going on for about 3 minutes and I’m not sure he has taken a breath)

Still going…. 

Now talking about the features in the service. Many others offer a basic service, AWS goes deep on most offerings (another list of offerings, he is going into compute and how they are differentiated i.e. GPU specific, small compute, large compute, etc.)

Still going on list of services… Jassy is the Energizer Bunny of feature lists

First Customer is up - MLB (Major League Baseball) - CTO of MLB.com, started from scratch, now a six billion business for MLB. They built a PaaS they share with other providers (ESPN, etc.). Want to be on any screen at anytime for events. StatCast is hosted on AWS, new system to go really deep and apply big data and prediction to baseball stats and players.

How do they capture the data? Radar sampling that tracks ball over 2000 a second, can “see” the baseball rotation it is that accurate. 17 Petabytes of data per season. AWS was the only one with scale and bursting capability (what do you do in offseason when you don’t need it). Keep adding to data warehouse over time to provide historical stats.

How does it work - collect data locally, use Amazon Direct Connect to export into AWS. From there MLB’s real time PaaS delivers StatCast to devices

Example - Breakdown of play during the World Series, shows how runner started slow (because he thought is was easy) and then sped up at the end. He was out by .2 second. If he ran the whole time, he would have been safe by over a foot.

Jassy is back - talking about transformation to Cloud Native Applications. You don’t have the option to move slow anymore.

Second Customer is up - CEO, Healthcare Company (sorry, didn’t catch the name, Phillips maybe) - going through a real world customer use case who had cancer and how they determined this (took blood that indicated it, found the cancer, showed patient how to adjust lifestyle and live with it vs. radiation treatment). This was real time data and fitting a treatment to the customer vs. other traditional alternatives using big data.

How do we turn a mountain of data into Actionable Items? This is where real time data comes into play. They are adding a PetaByte a month to the system right now (common theme here of scale and how no one else can scale like AWS). No one can support the large amounts of data.

Jassy back - Slide - Is there hope for a new normal in the area of relational databases? Old world DB’s are expensive, locked in. Many Enterprises are looking to MySQL and PostGres as an alternative. The OSS DB’s are hard right now….

(Announcement) - Amazon Aurora - Commercial Grade Database Engine - in development for 3 years, MySQL compatible but at 5x performance, same or better availability than Enterprise versions at 1/10 of the cost of the leading solutions in the market.

Product dude brought out for Aurora (didn’t catch his name) - Biggest Enterprise pain today is world class databases. They started with a blank slate and knew they wanted MySQL compatibility.

Compatibility with MySQL 5.6… 6 million inserts per minute, 30 million selects (I heard some folks around me saw wow to that one, I guess that is a big deal), data automatically backed up to S3 and highly available, crash recovery in seconds, database cache survives restart (no warming). Most features available only in Ent. class offerings.

Offered at .29 per hour (audience clapped at that)

Jassy back on stage - Talking about Software Deployment now. Pushed 50 million deployments in last 12 months using “Apollo” (codename for their internal project… I sense an announcement coming)

(Announcement) - AWS Code Deploy - Central monitoring and control, works with “virtually any” language and tool chain set, available today, free to use. Performs roll backs of code as well as commits. 

Talking about CI/CD now. Develop, Build&Test, Deploy, Monitor & Analyze

(Announcement) - AWS Code Pipeline - Integrates with existing tools, used internally in Amazon

(Announcement) - AWS Code Commit - code repository without size limits.

All exist together and work with external partners. (wonder who they will play nice with)

Now talking about compliance - They are now ISO-9001 compliant. They have been working with healthcare customers to achieve this level of certification.

Security up next - talking about encryption

(Announcement) - AWS Key Management Service - Encryption, IAM and policies all in one place (sorry for lack of details here, had to take a call)

OK, back…

Talking about Service Catalog (coming in 2015) - AWS Service Catalog, create a grouping of resources, create an offering, serve it out in a service catalog… They say Enterprises want this

(This *COULD* be interesting. I talked to Ent folks about this years ago and it never took off because it was too hard or costly to create the offerings and serve out the catalogs to multiple clients. If they make this easy to consume and usable, it could take off IMO. Enterprises want it but never really adopted it at scale. This was the original Enterprise vision of “cloud”, a portal of services)

Talking New Applications vs. Old Applications (here comes the Jassy we know and love… bring on the part where he tells everyone they are doing it wrong and need to do it the AWS way)

Dev/Test - Many Enterprises are using Dev/Test as a starting point for AWS. 
Mobile - The future of applications and architecture
Talking about companies migrating fully over to AWS. Feels like the days of virtualization (we want to be a 100% virtualized environment!). I doubt that will ever happen. Some workloads might go AWS…

CTO of Intuit on stage - They are moving all their applications to AWS. As Intuit evolves into a majority SaaS company. Over 8,000 employees, 3,000 engineers. Multi-billion online and mobile services. Had lease on datacenter up and migrated over to AWS. 6x cost savings, 1/5 of the time for buildout, developers were able to move faster. Over time this trend increased, starve the old, build new in AWS. Many acquisitions were built on AWS so that made absorbing them into Intuit very easy.

Jassy back - Talking about Hybrid Infrastructure (not Hybrid Cloud according to AWS). Jassy talking about a lot of Enterprises that still have on-prem resources because they aren’t ready to move to cloud. Talking about all the Hybrid features (VPC, Direct Connect, vCenter Integration, Access Control, Directory Service).

CTO of Johnson & Johnson  - 270 operating companies in 60 countries, 100,000+ employees, more stats,,, blah blah blah…

Thousands of Servers, Complex IT Ops - new strategy, less servers, automated IT, greater business efficiency

120 applications running in AWS now, plan to triple that in the next 12 months (they have to have THOUSANDS of apps, so I wonder what the percentage actually would be)

They want to move to Amazon Workspaces for Desktops

Jassy back - Slide - Partnering is the new normal (Announcement coming?)

Talking about culture of AWS - Customer focus comes first, AWS is pioneering (first to market), long term orientation

They will never call you at the end of a quarter to close a deal to make numbers (difference between am OPEX subscription model vs. a CAPEX purchase model)

AWS as a trusted advisor, Cost Optimized Service and Advice - over a 350Mil in cost reductions on behalf of customers

My take: Keynote felt very different from past years, company has moved from announcing more offerings (look, new compute offerings!) to announcing services to expand the ecosystem. Makes sense as the growth has slowed and they need to pick it up. Felt like a VMworld keynote from 5-7 years ago. A company that is starting to branch out and may very well start eating their own ecosystem so they can continue to grow. Also thought it was weird the pre-announced a few things this year. Not sure if they didn’t get them out in time but pretty sure they haven't done that before. AWS has gone from the “stealth IT little guy” poking the Enterprise in the eye and telling them they are doing it wrong and is now embracing the idea that they need the Enterprise and they now need to be nice to them. The fact that Jassy didn’t crap all over “Hybrid Infrastructure” and actually talked about it at the end helps prove this point.

I believe the Aurora and CI/CD announcements will move the needle and look really awesome. The security announcements were needed to fill out the Enterprise portfolio. The Service Catalog could be interesting when it releases.

Wednesday, May 23, 2012

EMC & Puppet Labs Announce Project Razor

Something really cool happened this morning!  EMC and Puppet Labs jointly announced a next generation provisioning system called Project Razor.  Brian and I had a chance to sit down with Puppet Labs and EMC to get some exclusive information on the project for a Cloudcast that we released this morning.  If you are at EMC World, be sure to check out Chad's World featuring Razor tonight at 5:30

Rather than tell you all about it, here are a bunch of links hot off the press (but go listen to the podcast!!):
UPDATE: Since I posted this here are a few links to actually get it and posts on how to get started as well!