All of Amazon Web Services (AWS) Data Analytics Services – Table of Contents

Extracting Insights and Actionable Information from Data

Extracting Insights and Actionable Information from Data

Amazon Web Services (AWS) Data Analytics Processing

Learn the comprehensive set of data and analytical services to handle every step of the analytics process chain, ideal usage patterns and anti-patterns on AWS.

PURPOSE OF THIS BLOG SERIES:
Data is not only getting bigger (in “Volume”) and in ever-increasing different formats (the “Variety”) faster (the “Velocity”), but the need to derive “Value” quickly through analytics to provide actionable insights for businesses is indeed a differentiating factor between successful businesses that can mitigate risk and respond to customer actions in near real-time vs. other businesses that will fall behind in the day and age of data deluge.

In addition to the metaphorical “V’s” mentioned above to describe big data, there is one more: “Veracity” – being sure your data is clean prior to any analytics performed whatsoever. Garbage in, garbage out. There’s no time to waste making improper, mal-informed decisions based on dirty data. This is paramount. Using solutions in the AWS Marketplace make this crucial and difficult step easy.

Analyzing extensive data sets require noteworthy compute capacity that normally exceeds processing capabilities available on-premises. In addition, that compute capacity can fluctuate in size based on the data inputs and the type of analytics being performed. This characteristic of scaling workloads is perfectly suited to AWS and the AWS Marketplace’s pay-as-you-go cloud model, where applications can scale up and down (horizontally and vertically) based on demand. Extracting insights and actionable information from data can be done within minutes rather than months, and you only pay for what you use.

The vast majority of big data use cases deployed in the cloud today run on AWS, with unique customer references for big data analytics, of which 67 are enterprise, household names. You can build virtually any big data analytics application and support any workload regardless of the volume, velocity, and variety of data. With 50+ services and hundreds of features added every year, AWS provides everything you need to collect, store, process, analyze, and visualize big data on the cloud.

Let’s dive into these services and learn the ideal usage patterns of each so you can improve the way you build business analytics solutions and run your business.

Without Data Analytics, You're Guessing at Business Decisions!

Without Data Analytics, You’re Guessing at Business Decisions!

Each item in the Table of Contents below will become a link when live.

TABLE OF CONTENTS:

  • HOW ANALYTICS WILL CHANGE YOUR BUSINESS, FOR THE BETTER, FOREVER
    1. We’re Living in a Metrics-Driven World
    2. Key Enablers  to Big Data Analytics for Everyone
    3. Faster Time to Actionable Insights & ROI
    4. Data Mixology and the End of Data Silos
  • THE CURRENT STATE OF DATA & ANALYTICS
    1. The “old” buzz word: “Big Data”
    2. The “new” buzz word: “Digital Transformation”
    3. 2017: The Year of Intelligence
  • OVERCOMING KEY CHALLENGES IN DATA ANALYTICS
    1. Capacity Guessing
    2. Scaling Workloads
    3. Processing Power
    4. Cloud & Data Security
    5. Traditional Data Warehouses
    6. Traditional Relational Database Management Systems
    7. Discover, Migrate & Deploy Pre-Configured Big Data BI & Advanced Analytic Solutions in Minutes – and Pay Only for What You Use by the Hour!!!
  • WHY USE AWS FOR DATA ANALYTICS?
    1. AWS Big Data Analytics Advantages
    2. Some Specific Advantages of AWS Big Data Analytics
    3. AWS Marketplace for Big Data Analytics Advantages
    4. Some Specific Advantages of AWS Marketplace in Big Data Analytics
    5. Self-Managed Big Data Analytics Solutions on AWS Marketplace Examples
    6. AWS Marketplace “Hidden” Helpful Sites to Find Solutions
  • DATA COLLECTION ON AWS
    1. AWS Direct Connect
    2. AWS Snowball
    3. AWS Storage Gateway
    4. Amazon Kinesis Firehose
    5. Amazon S3 Transfer Acceleration
    6. AWS Database Migration Service
    7. AWS Marketplace Solutions for Data Collection
  • OVERVIEW OF DATA ORCHESTRATION ON AWS
    1. Amazon SWF
    2. AWS Data Pipeline
    3. AWS Lambda
  • OVERVIEW OF COMMONLY USED DATA STORES FOR ANALYTICAL WORKFLOWS ON AWS
    1. Amazon S3
    2. Amazon DynamoDB
    3. Amazon Aurora
  • OTHER IMPORTANT KNOWLEDGE NECESSITIES FOR AWS ANALYTICS
    1. Understanding the Need for Cloud Tools
    2. Where to Discover Pricing for Individual and Bundled AWS Services
    3. Where to Price Out a Production Cloud Solution vs On-Prem
    4. Monitoring Analytical Processes with Amazon CloudWatch
    5. Ensuring Clean Data Prior to Analyzing
    6. Batch Processing vs. Real-Time Processing
    7. Where to Find AWS Sample Data for Practice
  • AMAZON REDSHIFT DATA WAREHOUSE
    1. Amazon Redshift Overview
    2. Amazon Redshift vs. Traditional Data Warehouses
    3. Amazon Redshift Benefits
    4. Amazon Redshift Interfaces
    5. Amazon Redshift Integration with Other AWS Services
    6. Amazon Redshift Data Integration, Loading & Cluster Configuration
    7. Amazon Redshift Ideal Usage Patterns and Anti-Patterns
    8. AWS Marketplace Solution Matillion ETL for Redshift
  • THE AMAZON KINESIS FAMILY
    1. Amazon Kinesis Streams
    2. Amazon Kinesis Applications Using Amazon Kinesis Client Library
    3. Real-Time Streaming Data Producers
    4. Amazon Kinesis Data Stores
    5. Amazon Kinesis Integration with Other AWS Services
    6. Amazon Kinesis Stream Pre-Requisites: Identity & Access Management (IAM) Policies and Users
    7. Creating & Implementing Amazon Kinesis Streams
    8. The Role of Amazon Kinesis Firehose & Amazon Kinesis Analytics
    9. Amazon Kinesis Streams Ideal Usage Patterns & Anti-Patterns
  • AMAZON ELASTIC MAP REDUCE (EMR)
    1. Amazon EMR Overview & Benefits vs. Traditional Hadoop MapReduce
    2. Open Source Frameworks: EMR Interfaces
    3. Amazon EC2 Instance Configurations for Use with Amazon EMR
    4. Amazon EMR Data Stores
    5. Amazon EMR Cluster Pre-Requisites
    6. Launching an Amazon EMR Cluster
    7. Processing Data with Amazon EMR
    8. Amazon EMR Integration with Other AWS Services
    9. A Custom Framework of AWS Services for Data Lake Ingestion
    10. Amazon EMR Ideal Usage Patterns & Anti-Patterns
  • AMAZON ATHENA
    1. Amazon Athena Overview
    2. Amazon Athena Data Formats, Data Types & Compression Formats
    3. Defining Queries, Schemas & SerDe
    4. Amazon Athena Interfaces
    5. Amazon Athena Pre-Requisites
    6. Creating a Database & Table in Amazon Athena & Querying Data
    7. Amazon Athena Integration with Other AWS Services
    8. Amazon Athena Use Case Samples
    9. Analysis & Visualization Integrations with Amazon Athena
  • AMAZON ELASTICSEARCH SERVICE (ESS)
    1. Amazon Elasticsearch Service Overview vs. Open Source Elasticsearch
    2. Amazon ESS Benefits
    3. Amazon ESS Integration with Logstash & Kibana
    4. ESS ELK Stack on AWS Marketplace
    5. ESS Integration with Other AWS Services
    6. ESS Interfaces
    7. ESS Ideal Usage Patterns & Anti-Patterns
  • AMAZON MACHINE LEARNING (ML)
    1. Amazon ML Overview
    2. Amazon ML Benefits
    3. Amazon ML Key Features
    4. Generating Predictions with Amazon ML
    5. Amazon ML Integration with Other AWS Services
    6. Amazon ML Interfaces and SDKs
    7. Amazon ML Ideal Usage Patterns & Anti-Patterns
  • AMAZON ARTIFICIAL INTELLIGENCE (AI)
    1. Amazon AI Overview
    2. Amazon Lex
    3. Amazon Rekognition
    4. Amazon Polly
    5. AWS Deep Learning AMI’s
    6. Amazon Deep Learning CloudFormation Template
  • AMAZON QUICKSIGHT
    1. Amazon QuickSight Overview
    2. Amazon QuickSight Benefits
    3. What is SPICE?
    4. Amazon QuickSight Data Sources
    5. Amazon QuickSight Data Cleansing
    6. Amazon QuickSight Data Interfaces
    7. Amazon QuickSight Sample Data
    8. Amazon QuickSight Supported Visualization Types
    9. Building a Data Visualization with Amazon QuickSight
  • AWS GLUE
    1. AWS Glue Overview
    2. Integration with Other AWS Services
    3. Register Data Sources with AWS Glue
    4. Select a Data Source & Target: Watch the Magic!
    5. Schedule & Run AWS Glue Jobs
    6. Running AWS Glue: A Fully Managed ETL Service to Prepare Data for Analytics
  • AMAZON MOBILE ANALYTICS
    1. Amazon Mobile Analytics Overview
    2. Amazon Mobile Analytics Benefits
    3. Amazon Mobile Analytics Supported Platforms
    4. Amazon Mobile Analytics: More Details
    5. Amazon Mobile Analytics Integration with Other AWS Services
    6. Amazon Mobile Analytics Pre-Requisites
    7. Using Amazon Mobile Analytics to Understand User Engagement
  • AMAZON PINPOINT
    1. Amazon Pinpoint Overview
    2. Amazon Pinpoint Benefits
    3. Amazon Pinpoint Supported Platforms
    4. Amazon Pinpoint Interfaces
    5. Amazon Pinpoint Analytics
    6. Creating an Amazon Pinpoint Campaign
  • AWS DATA PIPELINE IN DETAIL
    1. AWS Data Pipeline Overview
    2. AWS Data Pipeline Supported Databases
    3. AWS Data Pipeline Data Stores
    4. AWS Data Pipeline Compute Services to Transform Data
    5. AWS Data Pipeline Data Nodes, Activities, Preconditions & Schedules
    6. AWS Data Pipeline Interfaces & Task Runner
    7. Steps to Complete Before You Create an AWS Data Pipeline
    8. Copy CSV Data Between Amazon S3 Buckets Using AWS Data Pipeline
  • AMAZON SIMPLE STORAGE SERVICE (S3) IN DETAIL
    1. Amazon S3 Overview
    2. Amazon S3 “Flavors”
    3. Amazon S3 Lifecycle
    4. Examples of Amazon S3’s Benefits in Large-Scale Analytics
  • AMAZON RDS AURORA DATABASE IN DETAIL
    1. Amazon RDS Aurora Overview vs. Traditional RDBMS
    2. Amazon Aurora Benefits
    3. Amazon Aurora’s Architecture
    4. Amazon Aurora Interfaces
    5. Amazon Aurora’s Integration with Other AWS Services
    6. Create an Amazon Aurora Cluster
    7. Amazon Aurora’s PostgreSQL Support
  • AMAZON DYNAMODB DATABASE IN DETAIL
    1. Amazon DynamoDB Overview
    2. Amazon DynamoDB Benefits in Analytical Processing
    3. Amazon DynamoDB Integration with Other AWS Services
    4. Amazon DynamoDB Interfaces
    5. Creating & Querying a DynamoDB Database

First Chapter can be found here.

#gottaluvAWS! #gottaluvAWSMarketplace!

Posted in Amazon Artificial Intelligence, Amazon Athena, Amazon Aurora, Amazon DynamoDB, Amazon EC2 On-Demand Instances, Amazon Elastic MapReduce, Amazon Elasticsearch Service, Amazon IAM, Amazon Kinesis Family, Amazon Machine Learning, Amazon Mobile Analytics, Amazon Pinpoint, Amazon QuickSight, Amazon Redshift Data Warehouse, Amazon Web Services, Amazon Web Services Analytic Services, AWS Analytic Services, AWS Analytics, AWS BI, AWS Data Collection, AWS Data Orchestration, AWS Data Pipeline, AWS Glue, AWS Marketplace, AWS S2 | 6 Comments

I Want to Create an eLearning Class on “All AWS Analytics Offerings”, So…

A photo from AWS re:Invent 2016 annotated by my great friend, Jen Underwood of ImpactAnalytix

A photo from AWS re:Invent 2016 annotated by my great friend, Jen Underwood of ImpactAnalytix

I Want to Create an eLearning Class on “All AWS Analytics Offerings”, but…

eLearning companies seem to want more targeted Business Intelligence (BI) & Analytical topics. It makes sense to appeal to certain demographics who want to know a specific type of BI/Analytic Solution – but, if they don’t know their options, how do they really “know” they “know” what type of BI/analytical solution is the right fit?

As a geek, I’d run to any place I could get a holistic view of  everything available insofar as Big Data BI & Analytics in the AWS cloud, & I don’t think I’m alone!

SHORT HISTORY FOR THOSE WHO NEED IT (everyone else skip a few paragraphs to “MY AWS BIG DATA BI & ANALYTIC AGENDA”):

Amazon Web Services has been serious about analytics for years, which is why they have the broadest platform for big data in the market today, with deep and rapidly expanding functionality across big data stores, data warehousing, distributed analytics, real-time streaming, machine learning, and business intelligence.

Gartner1 confirms AWS has the most diverse customer base and the broadest range of use cases, including enterprise mission-critical applications. For the sixth consecutive year, Gartner1 also confirms AWS is the overwhelming market share leader, with over 10 times more cloud compute capacity in use than the aggregate total of the other 14 providers in their Magic Quadrant!

Gartner

There’s a plethora of options on AWS Marketplace to run big data analytics software solutions available from popular vendors that are already pre-configured on an Amazon Machine Image (AMI) that solve a variety of very specific needs. You can visit the AWS Marketplace Big Data Analytics-specific site by clicking the bottom left icon on the AWS Marketplace site or by clicking here to view the premier AWS Marketplace solution providers for transforming and moving your data, processing and analyzing your data, and reporting and visualizing your data.

MY AWS BIG DATA BI & ANALYTIC AGENDA:

I’ve created an entire proposal and Table of Contents for every AWS BI & Analytic Solution, the data stores, etc. involved needed for all end-to-end AWS BI & Analytical Solutions.

My intention here is to blog about it, rather than make videos. At least people will be able to read their options without me spending as much time needed to create time-intensive video how-to’s on everything.

To me it’s a compromise to the delivery methodology for  the ultimate data-driven, metrics-driven solutions, providing companies with the kind of information they need to survive in today’s competitive business world, but DataLeader is here to help!

I will start this all-inclusive AWS BI/Analytics series of blog posts with a Table of Contents (TOC) that as I finish each part, each section of the TOC will become a link to that section.

Until next time! TOC coming in my next work break 😉

#gottaluvAWS!

#gottaluvAWSMarketplace!

Posted in Amazon Web Services, AWS Analytic Services, AWS BI, AWS Marketplace | Leave a comment

Tonight I’m Launching a Website Showcasing Next Generation Amazon Cloud Services Offerings

DataLeader.io  has launched tonight!

DataLeader.io Logo

DataLeader.io Logo

DataLeader.io brings together industry thought leaders and experts to best implement next generation AWS Cloud Solutions for both the Enterprise and small businesses.

DataLeader.io’s team is an eclectic group of artisans that paint the sky with zeroes and ones, dancing in their heads while solving your business problems.

DataLeader.io’s team is comprised of very well-known industry experts with an unquenchable passion for state-of-the-art technologies. What differentiates the team is they’re creative and innovative in a left-brained culture. The team is dedicated to delivering visionary solutions for your success in a data-driven technology world.

Data is the core of any business. Extracting insights and actionable information from data is a differentiating factor between successful businesses that can mitigate risk and respond to customer actions in near real-time vs. other businesses that will fall behind in today’s world of data deluge.

However, analyzing extensive data sets require noteworthy compute capacity that normally exceeds processing capabilities available on-premises. In addition, that compute capacity can fluctuate in size based on the data inputs and the type of analytics being performed. This characteristic of scaling workloads is perfectly suited to AWS and the AWS Marketplace’s pay-as-you-go cloud model, where applications can scale up and down (horizontally and vertically) based on demand.

DataLeader.io specializes in AWS, cloud, data and application development. The DataLeader.io team are recognized trainers in these technologies; thus training is another offering. DataLeader.io also offers Evangelism-as-a-Service and full end-to-end architecture, application development, automation, and branding.

DataLeader.io’s primary founder, Kim Schmidt, has over 50 LinkedIn Recommendations that describe Kim consistently as “innovative, passionate, charismatic, bringing structure to unstructured challenges, having tremendous impact in the broad technology landscape, a relentless force of nature who has the power of innovation and new ideas that provides amazing opportunities through new approaches while simultaneously helping people over their reluctance to technological changes.”

DataLeader.io is an AWS Partner, a Snowflake Data Warehouse Partner, a Matillion Partner, and an EDNA (Enterprise & Data Analytics) Partner.

#gottaluvAWS!
#gottaluvAWSMarketplace!

 

 

 

Posted in DataLeader.io | Tagged , , , , , , , , , , , , , , , , , , , , | Leave a comment

AWS re:Invent 2016 Product Announcements

AWS re:Invent 2016

AWS re:Invent 2016

The numerous product announcements made at the most amazing AWS re:Invent ever, 2016, are all over the internet. However, they’re so exciting I can’t help but do my own version.

AWS re:Invent 2016 Product Announcements ala Kim

AWS re:Invent 2016 Product Announcements ala Kim

There were ~32,000 attendees this year at re:Invent. It was crazy, but it was living proof of AWS Cloud superiority. And the keynotes delivered exciting announcements after announcements after announcements – keeping all these attendees not only happy but in awe. The picture below was from the first day; hallways thinned out a bit as the week progressed because of the incredible job AWS did with crowd control:

~32K Attendees at AWS re:Invent 2016!

~32K Attendees at AWS re:Invent 2016!

There was a big emphasis on “metrics” from every definition of the word applied to seemingly every Amazon Service.

AWS re:Invent 2016 Product Announcements, with Werner Vogels, CTO & VP of Amazon.com, on the Keynote Stage

AWS re:Invent 2016 Product Announcements, with Werner Vogels, CTO & VP of Amazon.com, on the Keynote Stage

So let me summarize for you the main AWS Product Announcements for you, alphabetically by product type.

ANALYTICS

  1.  Amazon Athena is a new interactive query service that makes it easy to analyze data in Amazon S3 using Standard SQL!!!!!!
Amazon Athena - AWESOME!

Amazon Athena – AWESOME!

Athena is serverless, thus having no infrastructure to manage. You pay only for the queries run. You point Athena at some data stored in Amazon S3, identify your fields, run your queries & get results in seconds!

You can read Jeff Barr’s blog post on Athena here.

2.  New feature for Amazon EMR: you can now use Amazon S3 as a data store for Apache HBase on Amazon EMR, separating your cluster’s storage & compute nodes = the new “data lake”!!!  

Data Lake Using Amazon S3 as Storage *Image courtesy of AWS properties

Data Lake Using Amazon S3 as Storage *Image courtesy of AWS properties

Bye, bye, HDFS! You can scale in compute nodes without impacting your underlying storage. You can read more about it here.

3.  Amazon QuickSight is now generally available for all customers.

Amazon QuickSight

Amazon QuickSight

QuickSight is Amazon’s own fast, cloud-powered business analytics service that makes it easy to build visualizations & perform ad-hoc analytics for Big Data. Read Jeff’s blog post here.

4.  Amazon Redshift announced four new features that improve data compression, connection management, and data loading:

Amazon Redshift

Amazon Redshift

  • Estimate storage space savings with compression with the ANALYZE COMPRESSION  command for each suggested column encoding so you can easily choose the best compression option
  • Automate compression when you create tables using CREATE TABLE AS command. This automatically compresses all columns except for sort key columns & those with boolean, real, & double precision types using LZO as the default compression type. This reduces storage space for your clusters and saves costs
  • You can now control/limit the number of database connections a user can open concurrently and the number of concurrent connections that can be made to a database using the CONNECTION LIMIT parameter when creating a user, a database, or altering a user or a database
  • When you use the COPY command for tables with a single sort key column and you insert data in sort key order, the command automatically adds new rows to the table’s sorted region, which eliminates or reduces the need to vacuum

5.  AWS Glue is a new fully managed ETL service that makes it easy to understand your data sources, prepare the data, and load it reliably to data stores.

AWS Glue

AWS Glue

It simplifies and automates data discovery, transformation, and job scheduling tasks. You can read more about it by clicking the link above.

APPLICATION SERVICES

  1.  New features were introduced for Amazon API Gateway:
Amazon API Gateway

Amazon API Gateway

  • Amazon API Gateway now integrates with the AWS Marketplace to help API owners monetize their API and meter usage for their API products, without writing any code. You can learn more about it here
  • You now can add documentation to your methods & resources for your APIs built using Amazon API Gateway, making it easy for your API consumers to understand your API (pretty cool, right?). You can read more about it here
  • Custom authorizers on Amazon API Gateway can now return additional fields in their authorization response. You can read more about it here

2. AWS Step Functions is a new service that makes it easy to coordinate the components of distributed applications and microservices using visual workflows.

AWS Step Functions

AWS Step Functions

You can read more about it on Jeff’s blog here.

ARTIFICIAL INTELLIGENCE

  1.  Amazon Lex is a new service for building conversational interfaces using voice & text:
Amazon Lex

Amazon Lex

With Lex, the same deep learning engine that powers Alexa is now available to any developer, enabling you to bring sophisticated, natural language chatbots to new & existing applications.

You can read Jeff’s blog post about Amazon Lex here.

2. Amazon Polly is a new service that turns text into lifelike speech. It supports 24 languages & 47 lifelike voices.

Amazon Polly

Amazon Polly

You can create apps that talk, allowing you to build entirely new categories of speech-enabled products.

AWS Polly Get Started

AWS Polly Get Started

You can read more about Amazon Polly in Jeff’s blog here

3.  Amazon Rekognition is a new service that makes it easy to add image analysis to your applications.

Amazon Rekognition

Amazon Rekognition

With Rekognition, you can detect objects, scenes, and faces in images. You can also search & compare faces.

You can read more about Rekognition in Jeff’s blog post here.

4. My favorite in this category is MXNet – Deep Learning Framework of Choice at AWS.

Deep Learning & MXNet

Deep Learning & MXNet

MXNet is an open source, fully featured, flexibly programmable and ultra-scalable deep learning framework supporting deep learning models including convolutional neural networks (CNNs) & long short-term memory networks (LSTMs). Running MXnet on AWS provides a highly scalable, flexible and fast model training experience for developers using deep learning on CPUs or GPUs, on clusters, servers, desktops, or mobile phones. It’s available on AWS Marketplace here.

For those interested in how “we” got to this level of  “deep learning”, refer to my article on my blog https://kimschmidtsbrain.com/ entitled “Updated with Google’s TensorFlow: Artificial Intelligence, Neural Networks, and Deep Learning”, https://kimschmidtsbrain.com/2015/10/29/artificial-intelligence-neural-networks-and-deep-learning/ written a little over a year ago. LOVE THIS TOPIC!

kimschmidtsbrain post on AI, Neural Networks, and Deep Learning

kimschmidtsbrain post on AI, Neural Networks, and Deep Learning

COMPUTE

  1.  Amazon EC2 instances in a VPC now offer native support for the IPv6 protocol. This protocol can be enabled for existing & new VPCs through the AWS Management console, API/SDK, and CLI
  2.  They introduced the new instance type for EC2, C5 instances. They’re the most powerful Compute Optimized instances, having the highest performing processors with the lowest price/compute performance in EC2. You can read Jeff’s blog post about it here
  3.  You can now add more GPU to your existing EC2 instances with Amazon EC2 Elastic GPUs. They are the most cost effective way to add graphics acceleration to Amazon EC2 instances.

 

Four Sizes for Amazon EC2 Elastic GPUs

    Four Sizes for Amazon EC2 Elastic GPUs

They’ll help applications that use machine learning, high performance databases, and genomics, for example. They’re attached similar to how EBS volumes are attached to EC2 instances.

You can read Jeff’s blog post here. This is one of my favorite new features!

4.  A new instance type, Amazon EC2 F1 is a new compute instance with programmable hardware for application acceleration.

Amazon EC2 F1 Instances

Amazon EC2 F1 Instances

With F1, you can directly access custom field programmable gate arrays (FPGA) hardware on the instance with a few clicks. These instance types can be used to speed up big data search, financial risk analysis, and genomics applications.

You can read Jeff’s blog here

5.  Amazon also announced another new instance type (no wonder they have more compute power in use than the other 16 cloud providers in Gartner’s Magic Quadrant combined!) Amazon Ec2 I3. I3 instances are the latest generation of Storage Optimized High I/O instances. They have low latency Non-Volatile Memory Express(NVMe)-based SSDs for the most demanding I/O intensive relational, NoSQL, transactional, and analytics workloads.

You can read Jeff’s blog post here

6. Amazon EC2 new instance type R4 are the latest generation of Memory Optimized instances that are 20% more price performant than R3 instances. They’re designed for today’s memory-intensive Business Intelligence, in-memory caching, and database applications, offering up to 488 GB of memory!

You can read Jeff’s blog post here

7. Amazon EC2 announced new instance types t2.xlarge & t2.2xlarge. They’re the newest burstable-performance instances. The t2.xlarge has 16 GB of memory & 4 vCPUs, and t2.2xlarge has 32 GB of memory & 8 vCPUs. These instances provide a baseline level of CPU performance with the ability to burst above the baseline.

Amazon EC2 T2 Instances for Low-Cost, Burstable Performance

Amazon EC2 T2 Instances for Low-Cost, Burstable Performance

The baseline performance and the ability to burst are governed by “CPU Credits”, which to me is an interesting read on how they work this out. You can read more about this here.

These New Amazon EC2 T2 Instances Have Burstable Performance

These New Amazon EC2 T2 Instances Have Burstable Performance

You can read Jeff’s blog post about this here.

8. Amazon Lightsail is a new service that offers the easiest way to launch & manage a virtual private server with AWS. This helps you jumpstart your project with compute, storage, & networking starting at $5/mo.

Amazon Lightsail

Amazon Lightsail

You can read Jeff’s blog post about Amazon Lightsail here.

9. The new service AWS Batch enables developers, scientists, and engineers to easily & efficiently run hundreds of thousands of batch computing jobs on AWS.

AWS Batch

AWS Batch

Use cases for AWS Batch include Financial Services (such as Post-Trade Analytics), Digital Media (such as Visual Effects Rendering), and Life Sciences (such as Drug Screening for Biopharma). Just like AWS: always on the cutting-edge of technology!

AWS Batch Use Case: Drug Screening for Biopharma Architectural Diagram *Image courtesy of AWS properties

AWS Batch Use Case: Drug Screening for Biopharma Architectural Diagram *Image courtesy of AWS properties

You can read Jeff’s blog post here.

10. There are new features for AWS Lambda:

11. This is a very promising new feature: Lambda@Edge! Lambda@Edge lets you run code at CloudFront edge locations without provisioning or managing servers! This allows developers to deliver a low latency user experience for customized web apps.

Choosing the New Edge Runtime

Choosing the New Edge Runtime

You can read more at Jeff’s blog here.

CONTAINERS

  1. They announced Blox, a collection of open source software that enables customers to build custom schedulers & integrate 3rd-party schedulers on top of ECS
Open Source Collection of Projects for Amazon ECS - Blox

Open Source Collection of Projects for Amazon ECS – Blox

You can read more about Blox on Jeff’s blog here.

DATABASES

  1.  There were two new features announced for Amazon Aurora:
Amazon Aurora

Amazon Aurora

  • This one is phenomenal! Amazon Aurora is now PostgreSQL compatible!!! You can get up to twice the performance of the typical PostgreSQL database along with all the awesome features of Amazon Aurora (if you haven’t seen my “in-progress” video series on Amazon Aurora, the first 25 videos can be found here). This adds more performance benefits and the geospatial objects associated with PostgreSQL, among other things.
Amazon Aurora's PostgreSQL Compatibility *Image courtesy of AWS properties

Amazon Aurora’s PostgreSQL Compatibility *Image courtesy of AWS properties

You can read more about this exciting new feature on Jeff’s blog here.

2. AWS Schema Conversion Tool added two more data warehouses as sources for schema conversion: Netzetta and Greenplum. It’s getting easier every day to consume data on AWS!

DESKTOP & APP STREAMING

  1.  AWS announced a new service, Amazon AppStream 2.0. This service allows you to stream desktop apps securely from the AWS cloud directly to users on the device of their choice, eliminating the need to rewrite desktop apps for the cloud. Cool.
Amazon AppStream 2.0

Amazon AppStream 2.0

2. There were three new features announced for Amazon Workspaces:

Amazon Workspaces

Amazon Workspaces

DEVELOPER TOOLS

1. AWS announced a new service called AWS CodeBuild that builds and tests code in the cloud. CodeBuild scales continuously.

AWS CodeBuild Description Page

AWS CodeBuild Description Page

You’re charged by the minute for the compute resources you use.

AWS CodeBuild

AWS CodeBuild

You can use CodeBuild with other AWS services such as AWS Code Pipeline, which automates building and testing code in CodeBuild each time you commit a change to your repository. These announcements just keep blowing my mind!

You can read Jeff’s blog post on this here.

2. Another new service announced is AWS X-RayI JUST LOVE THIS ONE!

AWS X-Ray

AWS X-Ray

AWS X-Ray helps developers analyze and debug production, distributed applications such as those built using a microservices architecture.

Architectural Diagram of How AWS X-Ray Works

Architectural Diagram of How AWS X-Ray Works

With X-Ray, you can understand how your application and its underlying services  are performing to identify and troubleshoot the root cause of performance issues and errors – even services that run for milliseconds!!!

You can read more about this awesome new service in Jeff’s blog post here.

HYBRID

1. AWS announced a new service called VMware Cloud on AWS.

VMware Cloud on AWS

VMware Cloud on AWS

This makes it easy for customers to run VMware workloads on the AWS cloud, which I think is an ingenious move for Enterprise adoption!

INTERNET OF THINGS

1. AWS Greengrass is a new service that’s software that allows you to run local compute, messaging & data caching for connected devices in a secure way.

AWS Greengrass: Secure IoT Device Communication

AWS Greengrass: Secure IoT Device Communication

You can run IoT applications seamlessly across the AWS Cloud + local devices using AWS Lambda to keep device data in sync & communicate between devices securely, even when not connected. Data is never exchanged between devices when they communicate with each other without proven identity. AWS IoT devices can be configured to communicate with one another in a Greengrass Group.

Greengrass Group Architectural Diagram * Image courtesy of AWS properties

Greengrass Group Architectural Diagram * Image courtesy of AWS properties

You can read Jeff’s blog post on AWS Greengrass here.

2. Three new features were released about AWS IoT:

  • I’ve heard nothing but accolades about the new, redesigned AWS IoT console. It’s very slick and is easier to use. This includes connecting devices, diving into details, finding key resources, and testing with the MQTT client. I don’t have any IoT devices configured, but here’s a peek at the bare console 😉
The Redesigned AWS IoT Console

The Redesigned AWS IoT Console

The AWS IoT Button Enterprise Program Home Page

The AWS IoT Button Enterprise Program Home Page

This program helps businesses expand existing applications or build new applications that connect to a simple IoT button device:

An AWS IoT Button

An AWS IoT Button

Using the Button, businesses can offer services or improve existing products using the click of the physical button seen above, & custom logic in the cloud triggers services in the cloud. Beyond cool!

  • AWS has improved the developer experience by offering as pre-order, the 2nd generation AWS IoT Button with 2x the battery life, & a new mobile app on iOS and Android for easy configuration, called the Amazon Dash Button.
Amazon Dash Button Hardware Device Home Page

Amazon Dash Button Hardware Device Home Page

As examples, you can program the button’s logic to unlock or start a car, call Uber, track medications, household appliances and more. The button connects to a WiFi network then links straight to AWS cloud.

MANAGEMENT TOOLS

1. Amazon CloudWatch Metrics has a price reduction from $0.50/metric/month to $0.30/metric/month, and a 40% reduction in New Volume-Based Pricing Tiers

2. Amazon EC2 Systems Manager is a new AMAZING service that’s a management service to help you automatically collect software inventory, apply Windows OS patches, create system images, and configure Windows & Linux OS’s.

Amazon EC2 Systems Manager Home Page

Amazon EC2 Systems Manager Home Page

This is what the “Systems Manager Services” area looks like in the Amazon EC2 Dashboard:

Amazon EC2 Systems Manager in the Amazon EC2 Dashboard

Amazon EC2 Systems Manager in the Amazon EC2 Dashboard

3. There’s a new feature for AWS Cloudformation that lets you provision & manage resources for AWS Lambda-based applications using AWS CloudFormation & the AWS Serverless Application Model (AWS SAM).

4. AWS CloudTrail now supports Amazon S3 Data Events. You can now record all API actions on S3 Objects and receive detailed information such as the AWS account of the caller, IAM user role of the caller, time of the API call, Ip address of the API & more! Considering Amazon Athena, this is a natural new feature that’ll greatly benefit customers.

5. A new service called AWS OpsWorks for Chef Automate provides a fully managed Chef server & suite of automation tools for continuous deployment, automated testing for compliance, & a user interface for visibility into your nodes.

AWS OpsWorks for Chef Automate

AWS OpsWorks for Chef Automate

0w2

This automation platform treats server configurations as code. ULTRA COOL.

6.  AWS announced a new service called AWS Personal Health Dashboard, that gives you a personalized view of AWS service health.

Viewing Your AWS Personal Health Dashboard

Viewing Your AWS Personal Health Dashboard

You’ll get alerts when services you’re using are impacted with guidance to help keep your AWS resources healthy. You can read more in Jeff’s blog post here.

AWS MARKETPLACE

1. AWS Marketplace announced formally that customers can now subscribe to SaaS and API products directly through the AWS Marketplace with SaaS Subscriptions. YIPPEE!

MESSAGING

1. You can now use Amazon SQS for applications that require strict sequence using “First-in, First-out” (FIFO) queues. In addition, the price for Amazon SQS standard queues has been reduced by up to 20%.

MIGRATION

1. Are you ready for this? AWS announced a new service called AWS Snowmobile. AWS Snowmobile is an exabyte-scale data transfer service that builds upon AWS Snowball (haha) to move extremely large amounts of data securely. As Andy Jassy stated on the keynote stage, compared to AWS Snowball, “We’re going to need a bigger box!” So they made one!

AWS Snowmobile Shown During AWS re:Invent Keynote with Andy Jassy

AWS Snowmobile Shown During AWS re:Invent Keynote with Andy Jassy

You can read Jeff’s blog post on AWS Snowmobile here.

MOBILE SERVICES

1. AWS announced a new service called Amazon Pinpoint. Amazon Pinpoint makes it easy to run targeted push notification campaigns to improve user engagement in mobile apps.

Amazon Pinpoint

Amazon Pinpoint

You can use Pinpoint to define your target segments, run your campaign, and measure results.

2. There were 2 announcements in regard to AWS Mobile Hub:

  • AWS announced SaaS Connectors feature for AWS Mobile Hub. These connectors make it easy to securely access data in 3rd-party enterprise SaaS applications from your mobile app
  • AWS Mobile Hub now includes an integration with Amazon Lex so you can build mobile apps that use speech and text, in addition to touch to your mobile app

NETWORKING & CONTENT DELIVERY

1. Amazon CloudFront has a new feature called CloudFront Regional Edge Caches that improves performance for your viewers while lowering the load on your origin resources. BONUS: There’s no additional charge for using this feature!

SECURITY, IDENTITY & COMPLIANCE

1. AWS announced a new service called AWS Artifact that provides on-demand access to AWS compliance reports to verify & demonstrate security and compliance of the cloud.

AWS Artifact

AWS Artifact

There are many audit artifacts to choose from. I’ve shrunk the page down that I took a screenshot of below to fit in as many as I could, but there’s many more:

AWS Audit Artifacts

AWS Audit Artifacts

2. A new service called AWS Organizations allows you to create groups of AWS accounts that you can use to more easily manage security and automation settings.

AWS Organizations

AWS Organizations

3. An awesome new service called AWS Shield is a managed DDoS protection service that safeguards your web apps using ELB, Amazon CloudFront, & Amazon Route 53.

AWS Shield

AWS Shield

There are two tiers of AWS Shield – Standard & Advanced – as you can see in the screenshot above.

You can read Jeff’s blog post on AWS Shield here.

STORAGE

1. Amazon EFS has a new feature that now allows access from on-prem servers.

Amazon Elastic File System

Amazon Elastic File System

This simplifies migration of file data from on-prem into Amazon EFS file systems and enables support for cloud bursting workloads.

2. Amazon Glacier announced a new feature that allows customers to use Expedited retrievals to access data in 1-5 minutes for a flat fee of $0.03/GB retrieved! Awesome!

3. Amazon announced new features to the growing portfolio of Amazon S3 Storage Management capabilities to help you better analyze & manage your storage ! YEAH! There’s 4 new features that can be further explained by clicking the link in this paragraph:

Amazon S3 New Object Tagging & New Storage Management Features Opt In

Amazon S3 New Object Tagging & New Storage Management Features Opt In

  • S3 Object Targeting
  • S3 Analytics, Storage Class Analytics
  • S3 Inventory
  • S3 CloudWatch Metrics

4. AWS collapsed the number of Amazon S3 price tiers from six to three making it easier to understand your bill and project future costs. AWS also reduced Amazon S3 & Amazon Glacier storage prices by more than 20% in most regions

5. AWS Snowball has a new feature whereby it expanded its HIPAA compliance program to include AWS Snowball so you can now transfer large amounts of Protected Health Information (PIH) into & out of AWS securely & cost effectively.

6. AWS Snowball Edge is a new service that’s a petabyte-scale data transfer service with on-board storage and compute!!! WOW!

AWS Snowball Edge

AWS Snowball Edge

Below is an image of AWS Snowball Edge:

AWS Snowball Edge Image *Image courtesy of AWS properties

AWS Snowball Edge Image *Image courtesy of AWS properties

You can read more about AWS Snowball Edge in Jeff’s blog post here.

7. AWS announced a new feature for AWS Storage Gateway, providing a virtual on-prem file server, enabling you to store & retrieve Amazon S3 objects through standard file-like storage protocols

WOW! I’m sure as impressed as I am! AWS just keeps getting better, & better, & better!

#gottaluvAWS!

Posted in Amazon Mobile Hub, Amazon Mobile Services, Amazon RDS Databases, Amazon Web Services, Andy Jassy, Architecture, AWS re:Invent 2016 New Product Announcements, Jeff Barr, SaaS AWS Marketplace, Werner Vogels | Leave a comment

AWS Marketplace Has New Announcements + What is AWS Marketplace Anyway?

AWS Marketplace Gets a Fresh New Look!

AWS Marketplace Gets a Fresh New Look!

What a nice surprise to wake up and see so many new things happening insofar as AWS Marketplace is concerned!

First of all, the website has a completely fresh look! Modern, easy to navigate, includes “Customer Success Stories” (really nice for people who don’t yet understand the immense value AWS Marketplace brings to AWS customers – it used to be on a separate page – now it’s right here, front & center!) And when you click on the small “Customer Success Story page” link at the bottom of the image below, you’re taken to an awesome page of many videos of customer success stories.

Screenshot of AWS Marketplace Customer Success Stories Section

Screenshot of AWS Marketplace Customer Success Stories Section

For those of you who aren’t familiar with AWS Marketplace, its benefits for you and what-the-heck-it-is, click here to jump to that section now, or keep reading and you’ll eventually get there! In a nutshell, it allows you to purchase pre-configured popular software from 1000+ independent software vendors with 1-Click to deploy, and you only pay for what you use. It’s really awesome!

Another BIG announcement today is that you can now find, buy, & use a variety of Software as a Service (SaaS) solutions from AWS Marketplace!

Screenshot Announcing SaaS Subscriptions from AWS Marketplace

Screenshot Announcing SaaS Subscriptions from AWS Marketplace

SaaS solutions run on AWS infrastructure. You only pay for what you consume with no monthly fees or subscription costs (true for almost every Independent Software Vendor (ISV) solution in AWS Marketplace.)

Below are “Featured SaaS Subscription Products” (it’s on the homepage if you scroll down so you can click on each one, or you can click the orange “View SaaS Products” in the above image to see them all:

Screenshot of "Featured" SaaS Subscription Products

Screenshot of “Featured” SaaS Subscription Products

Below is a list of vendors and products “to-date” – it’s growing hourly so this is just a “teaser” 🙂

Application Development & Monitoring

  • Cloudyn: Manages & optimizes AWS, multi-platform, & hybrid platform cloud deployments to give granular visibility into consumption, cost, & performance. Proactive alerts for cloud accountability across the enterprise. Actionable recommendations, right-sizing & capacity planning to optimize & efficiently grow cloud deployment
  • Cloudinary: End-to-end image and video management solution enables web and mobile app developers to deliver a superior end-user experience, speed development, reduce costs and eliminate the hassle of building and supporting a media management solution in-house, or patching together off-the-shelf solutions with limited functionality and scale. Dynamically adapt image & video properties to deliver optimal versions. Many image & video manipulations. Integration libraries and SDKs
  • Datapath.io: NP(2) – Network Performance Platform, DevOps traffic optimization, scans over 600K internet prefixes for best latency, company claims a decrease of 60%. Overrides internet’s default connection route to provide an optimized route based upon latency, bandwidth, & cost
  • New Relic Infrastructure (Pro & Essential): Monitoring capabilities for visibility across Amazon EC2 instances, on-prem & cloud. Correlates real-time health metrics with events and configuration changes so you can resolve problems quickly. Pro has more features
  • Ruxit Dynatrace Cloud-Native Monitoring (powered by AI): Save time analyzing problems, performs detailed root cause analysis & gives you the ability to replay problems visually. Explore interactive visualization of your entire app environment. No manual configuration, auto-discovery powered by AI (Artificial Intelligence)
  • Solano Labs Cl: Brings high-performance & high-availability to your critical development & deployment pipelines. Parallel execution & dynamic task distribution finishes builds up to 80x faster automatically, claimed by vendor. Revision-controlled setup for new projects & branches, code repository, popular language support
  • Solodev Enterprise CloudCMS that drives powerful, personalized, easy to manage customer experiences. Instant access to code, files & databases in  a scalable EC2 environment. Handles a diverse range of products, has custom app functionalities

Security & Log Management

  • Alert Logic Cloud InsightPrioritize vulnerabilities and misconfigurations by degree of impact, incorporate vulnerability management into DevOps pipeline to fully automate assessment & validation of issues, automated scanning of new & changed assets
  • Bitium Identity & Access Management for AWS: (Bitium Application Services Endpoint (BASE)): Authentication and Authorization-as-a-Service, token-based authentication (SAML), enterprise integration: LDAP, Active Directory, or any other IdP
  • Datadog Apps MonitoringTurnkey integrations & easy to install agent to monitor all your servers, databases, & apps in minutes. Rich out-of-box dashboards + drag & drop to create your own. Easy API to extend integrations & send metrics & events from your apps
  • Dome9 Serenity for AWS Enterprise Edition
  • Sumo Logic Log Analytics (Sumo Logic Machine Data Analytics for Logs & Metrics): AWS-hosted machine data analytics service, delivering real-time insights from logs, metrics, & event data across the entire app lifecycle & stack. It has ad how search, LogReduce & LogCompare analytics, predictive and outlier detection, Live Tail, historical & live streaming dashboards, real-time alerting, data forwarding & webhook integrations, many certifications including HIPAA.  Enterprise Plan has anomaly detection, APIs, security analytics and more
  • Trend Micro Deep Security (as a Service): Defends your AWS workloads against threats, malware & vulnerabilities. Speed regulatory compliance with multiple security controls in one product. Defends against network threats

Databases, BI, and Big Data

  • ***Added 1/6/2017 to this post***Snowflake Elastic Data Warehouse (On-Demand) is now available in AWS Marketplace!!! Snowflake is ranked the #1 Cloud Data Warehouse by Gigaom (you can read the report here), the industry leader in emerging technology research. Snowflake’s unique architecture is what makes this product so extremely popular. There’s no configuration: you just load data and query it, and Snowflake takes care of the rest! You can view a short video I created about the company – it’s a “must watch” to understand this ground-breaking technology:
  • HERE Forward Geocoder Global ServiceHigh performance REST API available worldwide with a string analyzer understanding differences between address formats across countries. Geocode large data sets in seconds. Output provides scoring & matching information, shapes for postal code boundaries, & shapes for any administrative level
  • Pitney Bowes GeoCode APIEnriches customer addresses with the most accurate location data available through a REST API. Returns lat/long coordinates. Multiple methods for individual and batch transactions
  • Qubole Data ServiceDelivers a more accessible way to perform big data analytics for data stored & growing in your AWS cloud. Multiple data engines optimized to run with minimal operational administration, leveraging extensive automation & a common metastore. Automatically provision & manages big data infrastructure based on workloads, starting & stopping clusters only when needed. Auto-scales cluster size. Leverages lulls in compute pricing cycles to run your analytic jobs when Spot Pricing makes sense

Media

  • Aspera Transfer Service: Accelerates data movement to & from the cloud at rates up to 100s of times faster than standard TCP-based transfers, claims the vendor. Provides same transfer speeds to all regions. Complete data security with in-transit & at-rest encryption, data integrity verification for each transmitted block, & automatic retry and resume from point of interruption on failure
  • NetApp Data Sync (Data Fabric Cloud Sync service):  Simple automated method to get your data into AWS, run an AWS Service (like Amazon EMR, Amazon Redshift, Amazon RDS) & get the results back on-premises. Complete data synchronization to rapidly move data to & from AWS while converting file-based NFS data to Amazon S3 object format. Accelerates data-driven decision making by taking large or small data sets for analysis then returning the results back to NFS. Continuous, active-active synchronization lowers the amount of data transferred at one time
  • Signiant Flight: For companies that need to move large files to the cloud. Cloud-native auto-scaling, global access, easy deployment & maintenance. Vendor claims uploads are 200x faster than standard IP. Comprehensive security model is built-in  and standards are audited by a 3rd party

Storage

  • Druva Phoenix (Enterprise & Business): Unified backup, disaster recovery & archiving for physical and virtual server workloads. Cloud native, global dedupe, ever incremental backups, auto storage tiering. Vendor claims a 50-90% saving over traditional server backup solutions. Highly secure AES-256 Digital Envelope Encryption

Other Business Applications & Services

  • Avalara AvaTax: Real-time rates & rules for 12,000+ U.S. tax jurisdictions. Built in address verification & geolocation with guaranteed accuracy. SKU-level taxability for millions of products & services; updates automatically for change in rates, tax laws, & nexus. Pre-built integrations to 500+ ERP, POS, & ecommerce systems with full reporting capability

Other SaaS software solutions I’ve discovered since the above list was announced this morning, in advance of the AWS re:Invent announcement are the following:

  • Hortonworks Data Cloud for AWS: Hadoop, Hive & Spark services pre-configured (lots more in the story linked to in this bullet point)
  • Terra Pixel’s Streamap Platform: Terra Pixel, an Open Geospatial imagery streaming service provider, and Carahsoft Technology Corp, the trusted government IT solutions provider, offers this solution for the U.S. Intelligence Community
  • uReveal Data Mining Tool: Manage and examine raw data in real time & view structured or unstructured data in a unified view for the U.S. Intelligence Community

Another (almost) new feature of AWS Marketplace is “Product Support Connection for AWS Marketplace Customers“:

Screenshot of AWS Marketplace Product Support Connection

Screenshot of AWS Marketplace Product Support Connection

What this essentially means is that as a consumer of software through AWS Marketplace, which manages billing for the ISVs, consolidated on their AWS bill, you can selectively share your contact information with the software vendors. This simplifies and streamlines product support requests. “As of today”, the vendors that have this enabled are listed here (just click on the orange button in the homepage screenshot shown above).

The main way to navigate AWS Marketplace is to either click on one of the “Popular Categories” links, which takes you to the general category you clicked on. The results are filtered with the most popular products listed first.

Screenshot of AWS Marketplace "Popular Categories" Section

Screenshot of AWS Marketplace “Popular Categories” Section

Another way to navigate is via the search bar at the top. Let’s say you’re looking for a firewall. Simply type in the word “firewall” in the search bar:

Screenshot of Searching AWS Marketplace by Typing in a Generic Term

Screenshot of Searching AWS Marketplace by Typing in a Generic Term

You can see the search bar makes suggestions, but let’s just say you want to see all the listings because you’re not sure which one in the drop down list to choose. So you’d just leave the word “firewall” in the search & click “Enter”. You will get a filtered list of the top products on top:

Screenshot of AWS Marketplace Search with Generic Term "Firewall"

Screenshot of AWS Marketplace Search with Generic Term “Firewall”

Of course you can drill down if you know the specific type of firewall you want by either choosing from one of the drop down options or entering your own extended search terms.

Notice that next to where you typed in “firewall”, if you decided to search for something different, there is a drop down that reads “Shop All Categories“, as shown below:

Screenshot from "Firewall Search" to "Shop All Categories"

Screenshot from “Firewall Search” to “Shop All Categories”

I’d like to point out the annotations in the screenshot above and explain what they mean:

  • The far left light violet square that reads “Your Account” shows all information about your AWS & AWS Marketplace settings
  • The bottom right magenta rectangle that reads “Your Software” takes you to your AWS Account Login page, where once logged in you see the software you’re using currently from AWS Marketplace
  • The red arrow points to the words “Sell in AWS Marketplace“. If you have software you’d like to sell on AWS Marketplace, click the link in this bullet point to take a tour showing you all you need to know about doing that
  • The top super-purple square that surrounds the words “Amazon Web Services Home” takes you to the AWS home page where you can log in to the console for your account:
Screenshot of AWS Home Page

Screenshot of AWS Home Page

Of course, with Amazon Web Services big conference, AWS re:Invent coming up in 12 days, the conference announcement is one of the carousel images. Last year in 2015, there were more than 18,000 people attending! It’s a big deal, lots of fun, lots of learning, & lots of networking. If you can’t go, I encourage you to sign up to watch the live streams, shown in the screenshot above.

Below you’ll see the updated “View Categories” drop down from the home page. I think it’s nicer!

Screenshot of Updated "View Categories" in AWS Marketplace

Screenshot of Updated “View Categories” in AWS Marketplace on Marketplace Home Page

A left menu on any page but the home page exists, and has been there a long time. I don’t know if that’s queued for a new look, but let’s go over ways you can filter your searches from there:

Screenshot of AWS Marketplace Left Filter Menu

Screenshot of AWS Marketplace Left Filter Menu

The first thing you’ll notice is under “Categories” it has links to 3 types of categories, with the number of listings available. Going from top to bottom, you then can filter by the following topics:

  • Operating Systems
  • Software Pricing Plans
  • Product Support
  • Software Free Trials (notice there’s currently 506 software products offering free trials!)
  • Delivery Method
  • Average Rating
  • Architecture
  • Region
  • Amazon EC2 Instance Type

Expanding the EC2 Instance Type, you can see the options are either Micro Instances (available for the generous AWS Free Tier trial period of 1 year), General Purpose, Memory Optimized, Storage Optimized, Compute Optimized, & GPU Optimized. AWS has computational power that’s second to none, each optimized with varying combinations of CPU, memory, storage & networking capacity to meet the need of any use case1.

Expanding the Instance Types, you can see for yourself how many types and counts of EC2 instances AWS has. It’s no wonder AWS has the broadest platform for big data today with the vast majority of big data use cases deployed in the cloud today run on AWS with compute power like this. For the sixth consecutive year, AWS is the overwhelming market share leader, with over 10 times more compute capacity in use than the aggregate total of the other 14 providers in their “Magic Quadrant”. 1

Screenshot of AWS EC2 Instance Types

Screenshot of AWS EC2 Instance Types

 

What is AWS Marketplace & What’s In It For ME?
For those of you who haven’t heard about AWS Marketplace or dismiss it (for any number of pre-conceived ideas) as another “That’s how they get you!” thought, let me explain the facts, the benefits, and how to navigate AWS Marketplace. Please read on: AWS Marketplace has more than 100,000 active customers who use 300M compute hours/month deployed on Amazon EC2, with more than 3,000 listings from over 1,000 popular software vendors (not including the new SaaS launch that occurred today.)

Gartner1 confirms that because AWS has a multi-year competitive advantage over all its competitors, it’s been able to attract over a thousand technology partners and independent software vendors from popular vendor that have licensed and packaged their software to run on AWS, have integrated their software with AWS capabilities, or to deliver add-on services all through the AWS Marketplace. The AWS Marketplace is the largest “app store” in the world, regardless of being strictly a B2B app store!

Since AWS resources can be instantiated in seconds, you can treat these as “disposable” resources – not hardware or software you’ve spent months deciding which to choose and spending a significant up-front expenditure without knowing if it will solve your problems. The “Services not Servers” mantra of AWS provides many ways to increase developer productivity, operational efficiency and the ability to “try on” various solutions available on AWS Marketplace to find the perfect fit for your business needs without commitment to long-term contracts.

1-Click, on-demand infrastructure through software solutions on AWS Marketplace allows iterative, experimental deployment and usage to take advantage of advanced analytics and emerging technologies within minutes, paying only for what you consume, by the hour or by the month.

The vast majority of big data use cases deployed in the cloud today run on AWS, with unique customer references for big data analytics, of which 67 are household names. AWS has over 50 Services and hundreds of features to support virtually any big data application and workload. When you combine the managed AWS services with software solutions available on AWS Marketplace, you can get the precise business intelligence and big data analytical solutions you want that augments and enhances your project beyond what the services themselves provide. There are over 290 big data solutions in AWS Marketplace. Therefore, you get to data-driven results faster by decreasing the time it takes to plan, forecast, and make software provisioning decisions. This greatly improves the way you build business analytics solutions and run your business.

Below are just a fraction of example solutions you can achieve when using AWS Marketplace’s software solutions with AWS big data services:

You can:

  • Launch pre-configured and pre-tested experimentation platforms for big data analysis
  • Query your data where it sits (in-datasource analysis) without moving or storing your data on an intermediate server while directly accessing the most powerful functions of the underlying database
  • Perform “ELT” (extract, load, and transform) vs. “ETL” (extract, transform, and load) your data into Amazon’s Redshift data warehouse so the data is in its original form, giving you the ability to perform multiple data warehouse transforms on the same data
  • Have long-term connectivity among many different databases
  • Ensure your data is clean and complete prior to analysis
  • Visualize millions of data points on a map
  • Develop route planning and geographic customer targeting
  • Embed visualizations in applications or stand-alone applications
  • Visualize billions of rows in seconds
  • Graph data and drill into areas of concern
  • Have built-in data science
  • Export information into any format
  • Deploy machine-learning algorithms for data mining and predictive analytics
  • Meet the needs of specialized data connector requirements
  • Create real-time geospatial visualization and interactive analytics
  • Have both OLAP and OLTP analytical processing
  • Map disparate data sources (cloud, social, Google Analytics, mobile, on-prem, big data or relational data) using high-performance massively parallel processing (MPP) with easy-to-use wizards
  • Fine-tune the type of analytical result (location, prescriptive, statistical, text, predictive, behavior, machine learning models and so on)
  • Customize the visualizations in countless views with different levels of interactivity
  • Integrate with existing SAP products
  • Deploy a new data warehouse or extend your existing one

Amazon EC2 provides an ideal platform for operating your own self-managed big data analytics applications on AWS infrastructure. Almost any software you can install on Linux or Windows virtualized environments can be run on Amazon EC2 with a pay-as-you-go pricing model with a solution available on AWS Marketplace. Amazon EC2 uses the implemented architecture to distribute computing power across parallel servers to execute the algorithms in the most efficient manner.

Some examples of self-managed big data analytics that run on Amazon EC2 include the following:

  • A Splunk Enterprise Platform, the leading software platform for real-time Operational Intelligence. Splunk software and cloud services enable organizations to search, monitor, analyze and visualize machine-generated big data coming from websites, applications, servers, networks, sensors and mobile devices. A Splunk Analytics for Hadoop, within AWS, solution is available on AWS Marketplace also. It’s called Hunk and it enables interactive exploration, analysis, and data visualization for data stored in Amazon EMR and Amazon S3
  • A Tableau Server Data Visualization Instance, for users to interact with pre-built data visualizations created using Tableau Desktop. Tableau server allows for ad-hoc querying and data discovery, supports high-volume data visualization and historical analysis, and enables the creation of reports and dashboards
  • A SAP HANA One Instance, a single-tenant SAP HANA database instance that has SAP HANA’s in-memory platform, to do transactional processing, operational reporting, online analytical processing, predictive and text analysis
  • A Geospatial AMI such as MapLarge, that brings high-performance, real-time geospatial visualization and interactive analytics. MapLarge’s visualization results are useful for plotting addresses on a map to determine demographics, analyzing law enforcement and intelligence data, delivering insight to public health information, and visualizing distances such as roads and pipelines
  • An Advanced Analytics Zementis ADAPA Decision Engine Instance, which is a platform and scoring engine to produce Data Science predictive models that integrate with other predictive models like R, Python, KNIME, SAS, SPSS, SAP, FICO and more. Zementis ADAPA Decision Engine can score data in real-time using web services or in batch mode from local files or data in Amazon S3 buckets. It provides predictive analytics through many predictive algorithms, sensor data processing (IoT), behavior analysis, and machine learning models
  • A Matillion Data Integration Instance, an ELT service natively built for Amazon Redshift, that uses Amazon Redshift’s processing for data transformations to utilize it’s blazing speed and scalability. Matillion gives the ability to orchestrate and/or transform data upon ingestion or simply load the data so it can be transformed multiple times as your business requires

Below is an AWS Marketplace brochure explaining the benefits of using Marketplace solutions for big data analytics (that can also be found on the “BI & Big Data Landing Page” if you scroll to the bottom of the page & click on “Download PDF Poster“.

Benefits of AWS Marketplace in Analytical Solutions

Poster on the Benefits of AWS Marketplace in Analytical Solutions

The Main Categories on AWS Marketplace

AWS Marketplace has solutions for big data analytics, but listed below are all of the main sections, with links to each topics’ respective landing pages:

Breaking Down the Main AWS Marketplace Categories to Specific Functionalities:

Security Solutions:

Network Infrastructure Solutions:

Storage Solutions:

BI and Big Data Solutions:

Database Solutions:

Application Development Solutions:

Content Delivery Solutions:

Mobile Solutions:

Microsoft Solutions (note: the list of  “Third-Party Software Products” is a small fraction of the AWS Marketplace solutions that run on Microsoft Servers):

  • Microsoft Workloads:
    • Windows Server (many editions, type “Windows Server” into AWS Marketplace search bar)
    • Exchange Server
    • Microsoft Dynamics (many editions, type “Microsoft Dynamics” into AWS Marketplace search bar)
    • Microsoft SQL Server  (many editions, type “SQL Server” into AWS Marketplace search bar)
    • SharePoint (many editions, type “SharePoint” into AWS Marketplace search bar)
  • Third-Party Software Products:

Migration Solutions:

I hope you read through the entire post, and that you now realize how much time, frustration, configuration, and money you can save by using the preconfigured software solutions available at AWS Marketplace, only paying for what you use.

 

#gottaluvAWS! #gottaluvAWSMarketplace!

Posted in #savingcostonAWS, AWS Marketplace List of New SaaS Partners, AWS Marketplace New UI, AWS Marketplace Product Support Connection, How to Find AWS Marketplace Category Langing Pages, How to Find AWS Marketplace Preferred Vendors, List of Main Vendors by Category AWS Marketplace, Making Your IT Life Simpler with AWS Marketplace, SaaS AWS Marketplace, Snowflake Data Warehouse, What is AWS Marketplace | Tagged , , | 4 Comments

Amazon Web Services Financials, Magic Quadrant Facts, & How AWS Started

Amazon Web Services Cloud

Amazon Web Services Cloud

Of course I’m a huge fan of AWS (can you tell by the name of my blog?)

Gartner's Magic Quadrant Image

Gartner’s Magic Quadrant Image

Why are the numbers you’re going to see so mind-blowing? There are several reasons, & I’ll list some below (elaborating on further on down in this post) including the following facts directly from Gartner, who’s “Magic Quadrants” I’m sure you’re familiar with:

  1. AWS has the broadest platform for big data in the market today, with deep and rapidly expanding functionality across big data stores, data warehousing, distributed analytics, real-time streaming, machine learning, and business intelligence.
  2. Gartner1 confirms AWS has the most diverse customer base and the broadest range of use cases, including enterprise mission-critical applications.
  3. For the sixth consecutive year, Gartner1 also confirms AWS is the overwhelming market share leader, with over 10 time more cloud compute capacity in use than the aggregate total of the other 14 providers in their Magic Quadrant!
  4. AWS has computational power that’s second to none1; each optimized with varying combinations of CPU, memory, storage and networking capacity to meet the need of any big data use case.
  5. Gartner1 confirms that because AWS has a multi-year competitive advantage over all its competitors, it’s been able to attract over a thousand technology partners and independent software vendors from popular vendor that have licensed and packaged their software to run on AWS, have integrated their software with AWS capabilities, or to deliver add-on services all through AWS Marketplace. AWS Marketplace is the largest “app store” in the world, regardless of being strictly a B2B app store!(Gartner1)

I’ll get to the numbers right away, but at the end is a great story about how AWS came to be; Andy Jassy’s innovation.

Just last year Amazon started releasing financial information on AWS. Here are some facts:

  • AWS sales rose almost 64% to $2,566B for Q1 of 2016. Compare that to Q1 of 2015, which was $1.566B.
  • Profits increased dramatically, almost 128% to $604M from $265M for the 2015 same quarter results.
  • Amazon is factoring in the cost of stock option compensation for the first time. Therefore, the profit figure for the quarter of April, 2016, would be $716M.
AWS Cloud Revenue (in Millions)

AWS Cloud Revenue (in Millions) (Image courtesy of Fortune)

(AWS Source used by Fortune, cited above)

However, it’s been stated by AWS CFO that the gross profits will fluctuate because AWS constantly cuts prices, adds new features weekly, and is going to invest in 5 new data centers to run AWS Services in more regions just this year. An interesting article on Amazon’s earnings can be found here.

More facts on AWS & AWS Marketplace Business Intelligence / Big Data Analytics:

  1. AWS provides an extensive set of managed services that help you build, secure, and scale big data analytics applications quickly and easily. Whether your applications require real-time streaming, a data warehouse solution, or batch data processing, AWS provides the infrastructure and tools to perform virtually any type of big data project
  2. When you combine the managed AWS services with software solutions available from popular software vendors on AWS Marketplace, you can get the precise business intelligence and big data analytical solutions you want that augment and enhances your project beyond what the services themselves provide. You get to data-driven results faster by decreasing the time it takes to plan, forecast, and make software provisioning decisions. This greatly improves the way you build business analytics solutions and run your business.
  3. Since AWS resources can be instantiated in seconds, you can treat these as “disposable” resources – not hardware or software you’ve spent months deciding which to choose and spending a significant up-front expenditure without knowing if it will solve your problems. The “Services not Servers” mantra of AWS provides many ways to increase developer productivity, operational efficiency and the ability to “try on” various solutions available on AWS Marketplace to find the perfect fit for your business needs without commitment to long-term contracts.

Let me now tell you how AWS came to be, referenced after the Gartner facts above. I hope you find it as fascinating as I do!

Andy Jassy, AWS CEO

Andy Jassy, AWS CEO (Image courtesy of Fortune)

Around the year 2000, Amazon faced a dilemma in its retail business when it decided to let 3rd-party vendors sell products on Amazon.com. I remember signing up for that when I first heard of it.

Although allowing 3rd party vendors to sell on Amazon was a big shift in strategy, Andy Jassy knew it would be better for its customers because it gave them more selection and better choices of prices. 

The dilemma came when they realized in order to expose its e-commerce technology to 3rd parties, it had to decouple many components of its platform through APIs. This led to having business teams communicate with each other exclusively through “hardened, well-documented APIs”, which saved time and vastly accelerated its own product development.

Jassy stated2 , “The experience of having to deliver to customers in decoupled APIs – which was way harder than we thought it was going to be – really changed the way we thought about software.”

A couple years later, Amazon realized that projects were still taking too long because the teams were spending too much time setting up storage, compute, and databases.

Around 2003, Jassy realized that not only was there internal demand for these types of infrastructure services, but none of the old-guard enterprise vendors were building them for the Amazon market. And the lightbulb went off in Jassy’s head that not only their 3rd party vendors could benefit from this, but many other companies as well. Sometime around August or September 2003, Amazon decided to create AWS to provide the infrastructure that was needed for 3rd party vendors to sell to Amazon to the world. AWS was born out of necessity, but with a brilliant mind that could see beyond what was happening and realize it was a brand new business. Jassy was then appointed to lead it.

2Courtesy of Computer Reller News (CRN)

#gottaluvAWS!

Posted in Amazon Web Services, AWS History and Cool Facts, AWS Marketplace | Tagged , , , , , , | Leave a comment

Update on Amazon Aurora’s Deep Dive Section 2.0 Series: Amazon Aurora’s Unique Architecture

It’s amazing how much a wedding can put you behind on work, but it was one of the best weekends of my life, spending it with family, friends, and loved ones.

Since I not only aim to help others learn about AWS in this blog, I’m going to add a “personal picture” to share with you of my son’s wedding, so you can “get to know and trust me” 🙂

Mr. & Mrs. Bailey Buell Schmidt - she waited ~10 years!!!

Mr. & Mrs. Bailey Buell Schmidt – she waited ~10 years!!!

Here is an update of where I am in the 2.0 Architecture video series:

  • 2.0 and 2.1: “Amazon Aurora’s Unique Architecture” – Complete
  • 2.2: “Amazon Aurora’s Clusters & Replicas” – Complete
  • 2.3: “Amazon Aurora’s Endpoints and Replication” – Complete
  • 2.4: “Understanding Amazon Aurora’s Logging, Storage, Caching, and Indexing” – Complete
  • 2.5: “Understanding Amazon Aurora’s I/O Architecture” – Complete
  • 2.6: “Understanding Amazon Aurora’s Scalability and Elasticity Architecture” – Complete
  • 2.7: “Understanding Amazon Aurora’s Fault Tolerance and Restoration Architecture”  – Complete
  • 2.8: “Amazon Aurora’s Performance” – Complete

  • 2.9: “Amazon Aurora’s Performance Benchmarking” – Complete

  • 2.10: “Understanding Amazon Aurora’s Security Architecture” – Partially Created, (image only below)
Understanding Amazon Aurora's Security Architecture

Understanding Amazon Aurora’s Security Architecture

  • With quizzes for each Video Sub-Chapter within each Sub-Chapter section & a “final quiz” for the entire Video Section at the end

The next Video Chapter 3.0 is relatively short and will be on “Amazon Aurora’s Configuration & Management“.

Following that in Video Chapter 4.0 we’ll finally get into the first DEMO, “Creating & Working with Amazon Aurora’s Clusters“.

Following that in Video Chapter 5.0 is on “Amazon Aurora’s Migrations“.

The last Video Chapter 6.0 will be another DEMO, “Data Visualization and Analytics with Amazon Aurora“.

I truly hope I get these all completed before AWS re:Invent Conference, 11/28/2016 – 12/2/2016. But like most of you that are going, this is a busy time of year pre-conference.

#gottaluvAWS!

 

Posted in Amazon Aurora | Tagged , , , , , , , , , , , , , , , , , , , , , , , , , , | 3 Comments

Amazon Aurora’s Deep Dive: Video Chapter 2.0: Amazon Aurora’s Architecture

Amazon Aurora Deep Dive 2.0: Architecture

Amazon Aurora Deep Dive 2.0: Architecture

Hello Fellow AWS Enthusiasts!

My oldest son married his girlfriend of 10 years yesterday, so it’s been a very busy week of preparation, beautiful family times, and lots of happy emotions.

However, because my time has been diverted, my Amazon Aurora Deep Dive Video Series has been put on hold.

I was almost done with the next Video Section (2.0 of 6.0), following 1.0: Amazon Aurora Overview which can be viewed here: https://www.youtube.com/playlist?list=PL2EgRNBrbhC4lbTLy9cZnHG2JvEd8wZCr) prior to changing priorities from my beloved AWS passion to my beloved family 😉

NOTE: Missed this deadline :/ This new Video Chapter will be published by the end of this week, by Sunday, October 23.

I thought I’d give you a peek at the Video Sub-Chapter Titles that are in Amazon Aurora’s Architecture Video Section:

  • 2.0: “Amazon Aurora’s Architecture Intro”
  • 2.1: “Understanding Amazon Aurora’s Unique Architecture”
  • 2.2: “Amazon Aurora’s Clusters & Replicas”
  • 2.3: “Amazon Aurora’s Endpoints and Replication”
  • 2.4: “Understanding Amazon Aurora’s Logging, Storage, Caching, and Indexing”
  • 2.5: “Understanding Amazon Aurora’s I/O Architecture”
  • 2.6: “Understanding Amazon Aurora’s Scalability and Elasticity Architecture”
  • 2.7: “Understanding Amazon Aurora’s Fault Tolerance and Restoration Architecture”
  • 2.8: “Amazon Aurora’s Performance”
  • 2.9: “Amazon Aurora’s Performance Benchmarking”
  • 2.10: “Understanding Amazon Aurora’s Security Architecture”
  • With quizzes for each Video Sub-Chapter within each Sub-Chapter section & a “final quiz” for the entire Video Section at the end

Stay tuned this week – I found putting all this awesome architectural genius together was so much fun! Wait until you see how ingenious it all is!

#gottaluvAWS!

 

Posted in Amazon Aurora, Amazon RDS Databases, Amazon Web Services | Tagged , , , , , , , , , , , , , , , | Leave a comment

First Time I’m in “AWS Week in Review”!

This is such an honor for me!

https://aws.amazon.com/blogs/aws/aws-week-in-review-september-19-2016/

AWS Week in Review

AWS Week in Review

#gottaluvAWS!

Posted in Amazon Aurora, Amazon Web Services | Tagged , , , | Leave a comment

Amazon Aurora Deep Dive Video Title 1.0: Amazon Aurora Overview is LIVE!

Amazon Aurora Deep Dive Video Title 1.0: Amazon Aurora Overview

Amazon Aurora Deep Dive Video Title 1.0: Amazon Aurora Overview

I just finished the first major “Video Topic”, Video Topic 1.0: “Amazon Aurora Overview”, with 10 different Topic Sections, listed below. Some topics are grouped together, totaling 5 separate videos. Each Topic Section has a 1 Quiz Question (if grouped together, each one will be after the grouped Topics), and there’s a 5-Question Final Quiz on the entire Video Topic at the end.

Video Topic 1.0: “Amazon Aurora Overview”

1. “What is Amazon Aurora?”

2. “The Business Value of Amazon Aurora”

3. “What Makes Amazon Aurora a ‘Game-Changer’?”

4. “What Exactly Does “MySQL-Compatible Mean, and Why is This Important?”

5. “Amazon Aurora’s High Performance Overview”

6. “Amazon Aurora’s Scalability Overview”

7. “Amazon Aurora’s High Availability and Durability Overview”

8. “Amazon Aurora’s Security Overview”

9. “Amazon Aurora is a Fully-Managed AWS Service”

10. “Amazon Aurora’s Low Cost”

Here’s the link to the playlist: http://www.youtube.com/playlist?list=PL2EgRNBrbhC4lbTLy9cZnHG2JvEd8wZCr

Next I’m working on the next Video Topic 2.0: “Amazon Aurora’s Architecture”! Stand by!

#gottaluvAWS!

Posted in Amazon Aurora, Amazon IAM, Amazon RDS Databases, Amazon VPC, Amazon Web Services, Architecture, AWS Billing and Cost Management Console, AWS Business Value, AWS History and Cool Facts, AWS Marketplace | Tagged , , , , , , , , , , , , , , , , , , | Leave a comment