Cloud Computing
Cloud computing is an on-demand delivery of IT resources over the internet with primarily pay-as-you-go pricing. With cloud computing, companies do not have to manage and maintain their own hardware and data centers. Though, this also means that they are beholden to the cloud service providers for their entire infrastructure, platforms, and services.
6 Advantages of Cloud Computing
Pay as you go
Instead of investing in data centers and hardware before you know how you are going to use them, you pay only when you use the computing resources, and pay only for how much you use.
Benefits from massive economies of scale
By using cloud computing, you can achieve a lower cost than you can get on your own. Because usage from hundreds of thousands of customers is aggregated in the cloud and thus the cloud service providers can achieve higher economies of scale, which translates into lower pay-as-you-go pricing.
Stop guessing capacity
Eliminate guessing on your infrastructure capacity needs. When you make a capacity decision prior to deploying an application, you often end up either sitting on expensive idle resources or dealing with limited capacity. With cloud computing, these problems go away. You can access as much or as little capacity as you need and scale up and down as required with only a few minutes notice.
Increase speed and agility
It resources are only a click away, which means that you reduce the time to make resources available to your developers from weeks to minutes. This results in a dramatic increase in agility for the organization, since the cost and time it takes to experiment and develop as significantly lower.
Realize cost savings
Companies can focus on projects that differentiate their business instead of maintaining data centers. Cloud computing lets you focus on your customers, rather than on the heavy lifting of racking, stacking, and powering physical infrastructure. This is often referred to as undifferentiated heavy lifting.
Global reach in minutes
Applications can be developed in multiple Regions around the world with a few clicks. This means that you can provide lower latency and a better experience for you customers at a minimal cost.
Maintain Resiliency
It is best practice to use region-scoped, managed services. These services come with availability and resiliency built into their architectures. If you cannot use such services, it is best to replicate your workload across multiple availability zones (AZs), with a minimum of two AZs. This redundancy guards against issues in which the service becomes unavailable in the AZ.
Authentication and Authorization
Authentication is the process of determining if a user is who they are claiming to be, while authorization is the process of determining what a user is allowed to do in some application.
Servers
The first building block you need to host an application is a server. Servers usually can handle Hypertext Transfer Protocol (HTTP) requests and send responses to clients following the client-server model, although an API-based communication also falls under this model. A client is a person or computer that sends a request. A server handling the requests is a computer, or collection of computers, connected to the internet serving a resource to a user, typically on the internet.
Servers power your application by providing CPU, memory, networking capacity, and other resources to process users’ requests and transform them into responses. For context, common HTTP servers include:
- Windows options, such as Internet Information Services (IIS)
- Linux options, such as Apache HTTP Web Server, Nginx, and Apache Tomcat.
To run an HTTP server an AWS, you must find a service that provides compute power in the AWS Management Console. You can log into the console and view the complete list of AWS compute services (one of which is EC2).
Amazon Web Services (AWS)
Delete AWS root user keys to stay safe
If you don’t have an access key for your AWS root user, don’t create one unless you absolutely need to. If you have an access key for your AWS root user and want to delete the keys, follow these steps:
- In the ‘AWS Management Console’, go to the ‘My Security Credentials’ page and sign in with the root user’s email address and password.
- Open the ‘Access keys’ section.
- Under ‘Actions’, choose Delete.
- Choose ‘Yes’
Identity and Authorization Management (IAM)
IAM is AWSs Identity and Access Management service. It is a service that helps to manage access to your AWS account and resources.
User
An IAM User represents a person or service that interacts with AWS. You define the user in your AWS account. Any activity done by that user is billed to your account. Once you create a user, that user can sign-in to gain access to the AWS resources inside your account.
You can also add more users to your account as needed. For example, for your cat photo application, you could create individual users in your AWS account that correspond to the people who are working on your application. Each person should have their own login credentials. Providing users with their own login credentials prevents sharing of credentials.
User Credentials
An IAM user consists of a name and a set of credentials. When you create a user, you can provide them with the following types of access:
- Access to the AWS Management Console
- Programmatic access to the AWS Command Line Interface (AWS CLI) and AWS application programming interface (AWS API)
To access the AWS Management Console, provide the user with a user name and password. For programmatic access, AWS generates a set of access keys that can be used with the AWS CLI and AWS API. IAM user credentials are considered permanent, which means that they stay with the user until there’s a forced rotation by admins.
When you create an IAM user, you can grant permissions directly at the user level. This can seem like a good idea if you have only one or a few users. However, as the number of users increases, keeping up with permissions can become more complicated. For example, if you have 3,000 users in your AWS account, administering access and getting a top-level view of who can perform what actions on which resources can be challenging.
If only there was a way to group IAM users and attach permissions at the group level instead. Guess what, there is!
Groups
An IAM group is a collection of users. All users in the group inherit the permissions assigned to the group. This makes it possible to give permissions to multiple users at once. It’s a more convenient and scalable way of managing permissions for users in you AWS account. This is why using IAM groups is a best practice.
If you have an application that you’re trying to build and you have multiple users in one account working on the application, you might organize the users by job function. For instance, you might organize your IAM groups by developers, security, and admins. You could then place all your IAM users into their respective groups.
This provides a way to see who has what permission in your organization. It also helps you scale when new people join, leave, and change roles in your organization.
Consider the following examples:
- A new developer joins your AWS account to help with your application. You create a new user and add them to the developer group, without thinking about which permissions they need.
- A developer changes jobs and becomes a security engineer. Instead of editing the user’s permissions directly, you remove them from the old group and add them to the new group that already has the correct level of access.
Keep in mind the following features of groups:
- Groups can have many users.
- Users can belong to many groups.
- Groups cannot belong to groups.
The root user can perform all actions on all resources in side an AWS account by default. This is in contrast to creating new IAM users, new groups, or new roles. New IAM identities can perform no cations inside your AWS account by default until you explicitly grant them permission.
The way you grant permissions in IAM is by using IAM policies.
Policies
To manage access and provide permission to AWS services and resources, you create IAM policies and attach them to IAM users, groups, and roles. Whenever a user or role makes a request, AWS evaluates the policies associated with them. For example, if you have a developer inside the developers group who makes a request to an AWS service, AWS evaluates any policies attached to the developers group and any policies attached to the developer user to determine if the request should be allowed or denied.
Examples
Most policies are stored in AWS as JSON documents with several policy elements. The following example provides admin access through an IAM identity-based policy.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "*",
"Resource": "*"
}
]
}
This policy has four major JSON elements: Version, Effect, Action, and Resource.
- The Version element defines the version of the policy language. It
specifies the language syntax rules that are needed by AWS to process a
policy. To use all the available policy features, include:
"Version": "2012-10-17"before the"Statement"element in your policies. - The Effect element specifies whether the statement will allow or deny access. In this policy, the Effect is “Allow”, which means you’re providing access to a particular resource.
- The Action element describes the type of action that should be allowed or denied. In the example policy, the action is "*". This is called a wildcard, and it is used to symbolize every action inside your AWS account.
- The Resource element specifies the object or objects that the policy statement covers. In the policy example, the resource is the wildcard "*". This represents all resources inside your AWS console.
Putting this information together, you have a policy that allows you to perform all actions on all resources in your AWS account. This is what we refer to as an administrator policy.
The next example shows a more granular IAM policy.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["iam: ChangePassword", "iam: GetUser"],
"Resource": "arn:aws:iam::123456789012:user/${aws:username}"
}
]
}
After looking at the JSON, you can see that this policy allows the IAM user to change their own IAM password (iam:ChangePassword) and get information about their own user (iam:GetUser). It only permits the user to access their own credentials because the resource restricts access with the variable substitution ${aws:username}.
Structure
When creating a policy, each of the elements shown in the table are required in the policy statement.
| Element | Description | Required | Example |
|---|---|---|---|
| Effect | Specifies whether the statement results in an allow or an explicit deny | x | “Effect”: “Deny” |
| Action | Describes the specific actions that will be allowed or denied | x | “Action”: “iam:createUser” |
| Resource | Specifies the object or objects that the statement covers | x | “Resource”: “arn:aws:iam::account-ID-without-hyphens: user/Bob” |
Role Based Access
This section summarizes some of the most important IAM best practices that you must be familiar with before building solutions in AWS.
Lock down the AWS root user
The root user is an all-powerful and all-knowing identity in your AWS account. If a malicious user were to gain control of the root user credentials, they would be able to access every resource in your account, including personal and billing information. To lock down the root user, you can do the following:
- Don’t share the credentials associated with the root user.
- Consider deleting the root user access keys.
- Enable MFA on the root user account.
Following the principle of least privilege
Least privilege is a standard security principle that advises you to grant only the necessary permission to do a particular job and nothing more. To implement least privilege for access control, start with the minimum set of permissions in an IAM policy and then grant additional permissions as necessary for a user, group, or role.
Use IAM appropriately
IAM is used to secure access to your AWS account and resources. It simply provides a way to create and manage users, groups, and roles to access resources in a single AWS account. IAM is NOT used for website authentication and authorization, such as providing users of a website with sign-in and sign-up functionality. IAM also does NOT support security controls for protecting operating systems and networks.
Use IAM roles when possible
Maintaining roles is more efficient than maintaining users. When you assume a role, IAM dynamically provides temporary credentials that expire after a defined period of time, between 15 minutes and 36 hours. Users, on the other hand, have long-term credentials in the form of user name and password combinations or a set of access keys.
User access keys only expire when you or the account admin rotates the keys. User login credentials expire if you applied a password policy to your account that forces users to rotate their passwords.
Consider using an identity provider
If you decide to make your cat photo application into a business and begin to have more than a handful of people working on it, consider managing employee identity information through an identity provider (IdP). Using IdP, whether it’s an AWS service such as AWS Single Sign-On or a third-party identity provider, provides a single source of truth for all identities in you organization.
You no longer have to create separate IAM users in AWS. You can instead use IAM roles to provide permissions to identities that are federated from your IdP. For example, you have an employee, Martha, who has access to multiple AWS accounts. Instead of creating and managing multiple IAM users named Martha in each of those AWS accounts, you could manage Martha in your company’s IdP. If Martha moves in the company or leaves the company, Martha can be updated in the IdP, rather than in every AWS account in the company.
Consider AWS Single Sign-On
If you have an organization that spans many employees and multiple AWS accounts, you might want your employees to sing in with a single credential.
Similar to IAM. AWS SSO offers a directory where you can create users, organize them in groups, set permissions across the groups, and grant access to AWS resources. However, AWS SSO has some advantages over IAM. For example, if you’re using a third-party IdP, you can sync your users and groups to AWS SSO. This removes the burden of having to re-create users that already exist elsewhere, and it enables you to manage the users from your IdP. More importantly, AWS SSO separates the duties between your IdP and AWS, ensuring that your cloud access management is not inside or dependent on your IdP.
Choose the right compute option
If you're responsible for setting us servers on AWS to run your infrastructure,
you have many compute options. You need to know which service to use for which
use case. At a fundamental level, three types of compute options are
available:
1. Virtual Machines (VMs)
2. Containerized Services
3. Serverless
If you have prior infrastructure knowledge, a virtual machine is often the
easiest compute option to understand. This is because a virtual machine
emulates a physical server and allows you to install an HTTP server to run your
applications. To run virtual machines, you install a hypervisor on a host
machine. The hypervisor provisions the resources to create and run your VMs.
In AWS, virtual machines are called Amazon Elastic Compute Cloud, or Amazon
EC2. Behind the scenes, AWS operates and manages the host machines and the
hypervisor layer. AWS also installs the virtual machine operating system,
called the guest operating system.
Some AWS compute services use Amazon EC2 or use vitalization concepts under the
hood. You should understand this service before advancing to container services
and serverless compute.
Amazon Elastic Compute Cloud (EC2)
Amazon EC2 is a web service that provides secure, resizeable compute capacity in the cloud. It allows you to provision virtual servers called EC2 instances. Although AWS uses the phrase “web service” to describe it, you are not limited to running just web servers on your EC2 instances.
You can create and manage EC2 instances through the AWS Management Console, the AWS Command Line Interface (CLI), AWS software development kits (SDKs), automation tools, and infrastructure orchestration services.
To create an EC2 instance, you must define the following:
- Hardware specifications, like CPU, memory, network, and storage.
- Logical configurations, like networking location, firewall rules, authentication, and the operating system of your choice.
When launching EC2 instances, the first setting you configure is which operating system you want by selecting an Amazon Machine Image (AMI).
Amazon Machine Image (AMI)
In the traditional infrastructure world, the process of spinning up a server consists of installing an operating system from installation disks, installation drives, or installation wizards over the network. In the AWS Cloud, the operating system installation is not your responsibility. Instead, It’s built into the AMI that you choose.
In addition, when you use an AMI, you can select storage mappings, the architecture type (such as 32-bit, 64-bit, or 64-bit arm), and additional software installed.
Relationship between AMIs and EC2 instances
EC2 instances are live instantiations of what is defined in an AMI, much like a cake is a live instantiation of a cake recipe. If you are familiar with software development, you can also see this kind of relationship between a class and an object.
A class is something you model and define, while an object is something you interact with. In this case, the AMI is how you model and define your instance, while the EC2 instance is the entity you interact with, where you can install your web server and serve your content to users.
When you launch a new instance, AWS allocates a virtual machine that runs on a hypervisor. Then, the AMI you selected is copied to the root device volume, which contains the image used to boot the volume. In the end, you get a server that you can connect to and install packages and additional software on. In the example, you install a web server along with the properly configured source code of your employee directory app.
One advantage of using AMIs is that they are reusable. You might choose a Linux-based AMI and configure the HTTP server, application packages, and additional software you need to run your application. If you want to create a second EC2 instance with the same configurations, you could go through the entire instance creation and configuration process to match your settings to the first instance. Or, you could create an AMI from your running instance and use the AMI to start a new instance. That way, your new instance would have the same configurations as your current instance, because the configurations set in the AMIs are the same.
Find AMIs
You can select an AMI from the following categories:
- Quick Start AMIs, which are created by AWS to help you get started quickly.
- AWS Marketplace AMIs, which provide popular open source and commercial software from third-party vendors.
- My AMIs, which are created from your EC2 instances.
- Community AMIs, which are provided by the AWS user community.
- Build your own custom image with EC2 Image Builder.
Each AMI in the AWS Management Console has an AMI ID, which is prefixed by “ami-”, followed by a random hash of numbers and letters. The IDs are unique to each AWS Region.
Amazon EC2 instance types
Amazon EC2 instances are a combination of virtual processors (vCPUs), memory, network, and, in some cases, instance storage and graphics processing units (GPUs). When you create an EC2 instance, you need to choose how much you need of each of these components.
AWS offers a variety of instances that differ based on performance. Some
instances provide more capacity than others. To get an overview of the capacity
details for a particular instance, you should look at the instance type.
Instance types consist of a prefix identifying the type of workloads they’re
optimized for, followed by a size. For example, the instance type c5.large can
be broken down as follows:
- ‘c5’ determines the instance family and generation number. Here, the instance belongs to the fifth generation of instances in an instance family that’s optimized for generic computation.
- ’large’ determines the amount of instance capacity.
Instance families
In the example c5.large, the first letter, c, stands for compute-optimized. Compute optimized is one of the many instance families that AWS offers. Each instance family is optimized to fit different use cases. The following table describes instance families and some typical workloads.
| Instance Family | Description | Use Cases |
|---|---|---|
| General purpose | Provides a balance of compute, memory, and networking resources, and can be used for a variety of workloads. | Scale out workloads, such as web servers, containerized microservices, caching fleets, distributed data stores, and development environments |
| Compute optimized | Ideal for compute-bound applications that benefit from high-performance processors. | High-performance web servers, scientific modeling, batch processing, distributed analytics, high-performance computing (HPC), machine/deep learning, ad serving, highly scalable multiplayer gaming |
| Memory optimized | Designed to deliver fast performance for workloads that process large datasets in memory. | Memory-intensive applications, such as high-performance databases, distributed web-scale in-memory caches, mid-size in-memory databases, real-time big-data analytics, and other enterprise applications |
| Accelerated computing | Use hardware accelerators or co-processors to perform functions such as floating-point number calculations, graphics processing, or data pattern matching more efficiently than is possible with conventional CPUs. | 3D visualizations, graphics-intensive remote workstations, 3D rendering, application streaming, video encoding, and other server-side graphics workloads |
| Storage optimized | Designed for workloads that require high, sequential read and write access to large datasets on local storage. They are optimized to deliver tens of thousands of low-latency random I/O operations per second (IOPS) to applications that replicate their data across different instances. | NoSQL databases, such as Cassandra, MongoDB, and Redis, in-memory databases, scale-out transactional databases, data warehousing, Elasticsearch, and analytics |
EC2 instance locations
By default, your EC2 instances are placed in a network called the default Amazon Virtual Private Cloud (Amazon VPC). This network was created so that you can easily get started with Amazon EC2 without learning how to create and configure a VPC.
Any resource you put inside the default VPC will be public and accessible by the internet, so you shouldn’t place any customer data or private information in it.
Once you get more comfortable with networking on AWS, you should change this default setting to choose your own custom VPCs and restrict access with additional routing and connectivity mechanisms.
Architect for high availability
In the network, your instance resides in an Availability Zone of your choice. As you learned previously, AWS services that are scoped at the Availability Zone level must be architected with high availability in mind.
While EC2 instances are typically reliable, two is better than one, and three is better than two. Specifying the instance size gives you an advantage when designing your architecture because you can use more smaller instances rather than a few larger ones.
If your front end only has a single instance and the instance fails, your application goes down. On the other hand, if your workload is distributed across 10 instances and one fails, you lose only 10 percent of your fleet, and your application availability is hardly affected.
When architecting any application for high availability, consider using at least two EC2 instances in two separate Availability Zones.
EC2 instance lifecycle
An EC2 instance transitions between different states from the moment you create it until it’s termination.
When you launch an instance, it enters the pending state. When an instance is pending, billing has not started. At this stage, the instance is preparing to enter the running state. Pending is where AWS performs all actions needed to set up an instance, such as copying the AMI content to the root device and allocating the necessary networking components. 2
When your instance is running, it’s ready to use. This is also the stage where billing begins. As soon as an instance is running, you can take other actions on the instance, such as reboot, terminate, stop, and stop-hibernate. 3
When you reboot an instance, it’s different than performing a stop action and then a start action. Rebooting an instance is equivalent to rebooting an operating system. The instance remains on the same host computer, and maintains its public and private IP address, in addition to any data on its instance store. 4
It typically takes a few minutes for the reboot to complete. When you stop and start an instance, your instance may be placed on a new underlying physical server. Therefore, you lose any data on the instance store that were on the previous host computer. When you stop an instance, the instance gets a new public IP address but maintains the same private IP address. 5
When you terminate an instance, the instance stores are erased, and you lose both the public IP address and private IP address of the machine. Termination of an instance means that you can no longer access the machine.
Difference between stop and stop-hibernate
When you stop an instance, it enters the stopping state until it reaches the stopped state. AWS does not charge usage or data transfer fees for your instance after you stop it, but storage for any Amazon EBS volumes is still charged. While your instance is in the stopped state, you can modify some attributes, like the instance type. When you stop your instance, the data stored in memory (RAM) is lost.
When you stop-hibernate an instance, AWS signals the operating system to perform hibernation (suspend-to-disk), which saves the contents from the instance memory (RAM) to the Amazon EBS root volume.
Consider a scenario where you build a standard three-tier application, where you have web servers, application servers, and database servers. Suddenly, the application you built becomes extremely popular. To relieve some stress on the database that supports your application, you want to implement a custom backend layer that caches database information in memory (RAM). You decide to run this custom backend caching solution on Amazon EC2.
In this scenario, the stop-hibernate feature would be instrumental in persisting storage. It would prevent you from having to manually create scripts to save the RAM data before shutting down the server.
Pricing
One of the ways to reduce costs with Amazon EC2 is to choose the right pricing option for the way your applications run. AWS offers three main purchasing options for EC2 instances – On-Demand, Reserved, and Spot Instances.
Pay as you go with On-Demand Instances
With On-Demand Instances, you pay for compute capacity with no long-term commitments. Billing begins whenever the instance is running, and billing stops when the instance is in a stopped or terminated state. The price per second for a running On-Demand Instance is fixed.
For applications that require servers to be running all the time, you are less likely to benefit from the On-Demand pricing model, simply because there is no situation where you will need to turn servers off. For example, you might want the web server hosting the front end of your corporate directory application to be running 24/7 so that users can access the website at any time. Even if no users are connected to your website, you don’t want to shut down the servers supporting the site in case of potential user activity.
In the case when servers cannot be stopped, consider using a Reserved Instance to save on costs.
Reserve capacity with Reserved Instances (RIs)
RIs provide you with a significant discount compared to On-Demand Instance pricing. RIs provide a discounted hourly rate and an optional capacity reservation for EC2 instances. You can choose between three payment options – All Upfront, Partial Upfront, or No Upfront. You can select either a 1-year or 3-year term for each of these options.
Depending on which option you choose, you are discounted differently.
All Upfront offers a higher discount than Partial Upfront instances. Partial Upfront instances offer a higher discount than No Upfront. No Upfront offers a higher discount than On-Demand.
On-Demand and No Upfront are similar, since both do not require any upfront payment. However, there is a major difference. When you choose an On-Demand Instance, you stop paying for the instance when you stop or terminate the instance. When you stop an RI, you still pay for it because you committed to a 1-year or 3-year term.
Reserved Instances are associated with an instance type and an Availability Zone depending on how you reserve it. The discount applied by a Reserved Instance purchase is not directly associated with a specific instance ID, but with an instance type.
Save on costs with Spot Instances
Another way to pay for EC2 instances is by using Spot Instances. Amazon EC2 Spot Instances allow you to take advantage of unused EC2 capacity in the AWS Cloud. They are available at up to a 90% discount compared to On-Demand prices.
With Spot Instances, you set a limit on how much you would like to pay for the instance hour. This is compared against the current Spot price that AWS determines. If the amount you pay is more than the current Spot price and there is capacity, then you will receive an instance. While they are very promising from the billing perspective, you must account for some architectural considerations to use them effectively.
One consideration is that your Spot Instance might be interrupted. For example, if AWS determines that capacity is no longer available for a particular Spot Instance or if the Spot price exceeds how much you are willing to pay, AWS will give you a 2-minute warning before it interrupts your instance. That means any application or workload that runs on a Spot Instance must be able to be interrupted.
Because of this unique consideration, inherently fault-tolerant workloads are typically good candidates to use with Spot Instances. These include big data, containerized workloads, continuous integration/continuous delivery (CI/CD), web servers, high-performance computing (HPC), image and media rendering, and other test and development workloads.
Containers
While containers are often referred to as a new technology, the idea started in the 1970s with certain UNIX kernels having the ability to separate their processes through isolation. At the time, this was configured manually, making operations complex.
With the evolution of the open source software community, containers evolved. Today, containers are used as a solution to problems of traditional compute, including the issue of getting software to run reliably when it moves from one compute environment to another.
A container is a standardized unit that packages your code and its dependencies. This package is designed to run reliably on any platform, because the container creates its own independent environment. With containers, workloads can be carried from one place to another, such as from development to production or from on premises to the cloud.
Docker
When you hear the word container, you might associate it with Docker. Docker is a popular container runtime that simplifies the management of the entire operating system stack needed for container isolation, including networking and storage. Docker helps customers create, package, deploy, and run containers.
Difference between containers and virtual machines (VMs)
Containers share the same operating system and kernel as the host they exist on, whereas virtual machines contain their own operating system. Each virtual machine must maintain a copy of an operating system, which results in a degree of wasted resources.
A container is more lightweight. They spin up quicker, almost instantly. This difference in startup time becomes instrumental when designing applications that need to scale quickly during input/output (I/O) bursts.
While containers can provide speed, virtual machines offer the full strength of an operating system and more resources, like package installation, dedicated kernel, and more.
Orchestrate containers
In AWS, containers run on EC2 instances. For example, you might have a large instance and run a few containers on that instance. While running one instance is easy to manage, it lacks high availability and scalability. Most companies and organizations run many containers on many EC2 instances across several Availability Zones.
If you’re trying to manage your compute at a large scale, you must know the following:
- How to place your containers on your instances.
- What happens if your container fails.
- What happens if your instance fails.
- How to monitor deployments of your containers
This coordination is handled by a container orchestration service. AWS offers two container orchestration services – Amazon Elastic Container Service (ECS) and Amazon Elastic Kubernetes Service (EKS).
Manage containers with Amazon Elastic Container Service (Amazon ECS)
Amazon ECS is an end-to-end container orchestration service that helps you spin up new containers and manage them across a cluster of EC2 instances.
To run and manage your containers, you need to install the Amazon ECS container agent on your EC2 instances. This agent is open source and responsible for communicating to the Amazon ECS service about cluster management details. You can run the agent on both Linux and Windows AMIs. An instance with the container agent installed is often called a container instance.
Once the Amazon ECS container instances are up and running, you can perform actions that include, but are not limited to, launching and stopping containers, getting cluster state, scaling in and out, scheduling the placement of containers across your cluster, assigning permissions, and meeting availability requirements.
To prepare your application to run on Amazon ECS, you create a task definition. The task definition is a text file, in JSON format, that describes one or more containers. A task definition is similar to a blueprint that describes the resources you need to run a container, such as CPU, memory, ports, images, storage, and networking information.
Here is a simple task definition that you can use for your corporate directory application. In this example, this runs on the Nginx web server.
{
"family": "webserver",
"containerDefinitions": [ {
"name": "web",
"image": "nginx",
"memory": "100",
"cpu": "99"
} ],
"requiresCompatibilities": [ "FARGATE" ],
"networkMode": "awsvpc",
"memory": "512",
"cpu": "256"
}
Use Kubernetes with Amazon Elastic Kubernetes Service (Amazon EKS)
Kubernetes is a portable, extensible, open source platform for managing containerized workloads and services. By bringing software development and operations together by design, Kubernetes created a rapidly growing ecosystem that is very popular and well established in the market.
If you already use Kubernetes, you can use Amazon EKS to orchestrate the workloads in the AWS Cloud. Amazon EKS is conceptually similar to Amazon ECS, but with the following differences:
- An EC2 instance with the ECS agent installed and configured is called a container instance. In Amazon EKS, it is called a worker node.
- An ECS container is called a task. In Amazon EKS, it is called a pod.
- While Amazon ECS runs on AWS native technology, Amazon EKS runs on top of Kubernetes.
If you have containers running on Kubernetes and want an advanced orchestration solution that can provide simplicity, high availability, and fine-grained control over your infrastructure, Amazon EKS could be the tool for you.
Remove the undifferentiated heavy lifting
If you run your code on Amazon EC2, AWS is responsible for the physical hardware, and you are responsible for the logical controls, such as guest operating system, security, patching, networking, and scaling.
If you run your code in containers on Amazon ECS and Amazon EKS, AWS is responsible for more of the container management, such as deploying containers across EC2 instances and managing the container cluster. However, when running ECS and EKS on EC2, you are still responsible for maintaining the underlying EC2 instances.
If you want to deploy your workloads and applications without having to manage any EC2 instances, you can do that on AWS without serverless compute.
Go serverless
Every definition of serverless mentions the following four aspects:
- There is no provisioning or managing of servers.
- Scales with usage.
- You never pay for idle resources.
- Availability and fault tolerance are built-in.
With serverless, you can spend time on the things that differentiate you application, rather than spend time on ensuring availability, scaling, and managing servers.
AWS has several serverless compute options, including AWS Fargate and AWS Lambda.
Explore serverless containers with AWS Fargate
Amazon ECS and Amazon EKS enable you to run your containers in the following two modes:
- Amazon ECS mode
- AWS Fargate mode
AWS Fargate is a purpose-built serverless compute engine for containers. Fargate scales and manages the infrastructure, allowing developers to work on what they do best - application development. It achieves this by allocating the right amount of compute, eliminating the need to choose and handle EC2 instances and cluster capacity, and scaling. Fargate supports both Amazon ECS and Amazon EKS architecture, and provides workload isolation and improves security by design.
Fargate abstracts the EC2 instance so that you’re not required to manage it. However, with Fargate, you can use all the same ECS primitives, APIs, and AWS integrations. It natively integrates with AWS Identity and Access Management (IAM) and Amazon Virtual Private Cloud (VPC). Having native integrations with Amazon VPC allows you to launch Fargate containers inside your network and control connectivity to your applications.
Lambda
Run code on AWS Lambda
If you want to deploy your workloads and applications without having to manage any EC2 instances or containers, you can use AWS Lambda.
With AWS Lambda, you can run code without provisioning or managing servers or containers. You can run code for virtually any type of application or backend service, including data processing, real-time stream processing, machine learning, WebSockets, IoT backends, mobile backends, and web apps, like your corporate directory app!
AWS Lambda requires zero administration from the user. You upload your source code, and Lambda takes care of everything required to run and scale your code with high availability. There are no servers to manage, bringing you continuous scaling with sub-second metering and consistent performance.
How AWS Lambda works
A Lambda function has three primary components -a trigger, source code, and a configuration.
The source code describes what the Lambda function should run. It can be authored in three ways:
- You create the code from scratch.
- You use a blueprint that AWS provides.
- You use some code from the AWS Serverless Application repository, a resource that contains sample applications, such as “hello world” code, Amazon Alexa Skill sample code, image resizing code, video encoding, and more.
When you create your Lambda function, you specify the runtime you want your code to run in. You can use built-in runtimes, such as Python, Nodejs, Ruby, Go, Java, or .NET Core, or you can implement your Lambda functions to run on a custom runtime.
The configuration of a Lambda function consists of information that describes how the function should run. In the configuration, you specify network placement, environment variables, memory invocation type, permission sets, and other configurations.
Triggers describe when a Lambda function should run. A trigger integrates your Lambda function with other AWS services, enabling you to run your Lambda function in response to certain API calls that occur in your AWS account. This increases your ability to respond to events in your console without having to perform manual actions. All you need is the what, how, and when of a Lambda function to have functional compute capacity that runs only when you need it to.
Amazon’s CTO, Werner Vogels, says, “No server is easier to manage than no server” (This is a re-purposing of the phrase “no code runs faster than no code”) This quote summarizes the convenience you can have when running serverless solutions, like AWS Fargate and AWS Lambda.
AWS Lambda function handler
The AWS Lambda function handler is the method in your function code that processes events. When your function is invoked, Lambda runs the handler method. When the handler exits or returns a response, it becomes available to handle another event.
You can use the following general syntax when creating a function handler in Python:
def handler_name(event, context):
...
return resultant
Lambda naming
The Lambda function handler name specified at the time you create a Lambda function is derived from the following:
- Name of the file in which the Lambda handler function is located.
- Name of the source codes handler function.
A function handler can be any name; however, the default on the Lambda console
is lambda_function.lambda_handler. This name reflects the function name as
lambda_handler, and the file where the handler code is stored in
lambda_function.py. If you choose a different name for your function
handler on the Lambda console, you must update the name on the Runtime settings
pane.
Billing granularity
AWS Lambda lets you run code without provisioning or managing servers, and you pay only for what you use. You are charged for the number of times your code is triggered (requests) and for the time your code executes, rounded to the nearest millisecond (1ms) duration.
AWS rounds up duration to the nearest millisecond with no minimum execution time. With this pricing, it can be cost effective to run functions whose execution time is very low, such as functions with durations under 100 ms or low latency APIs.
Source Code
You can find a tutorial on creating the AWS Lambda function [here](https://aws.amazon.com/blogs/compute/resize-images-on-the-fly-with-amazon-s3-aws-lambda-and amazon-api-gateway)
Networking basics
One way to think about networking is to think about sending a letter. When you send a letter, you provide the following three elements:
- The payload, or letter, inside the envelope.
- The address of the sender in the From section.
- The address of the recipient in the To section.
Each address must contain specific information, such as:
- Name of sender and recipient.
- Street
- City
- State or province
- Zip, area, or postal code
- Country
You need all parts of an address to ensure that you r letter gets to its destination. Without the correct address, postal works cannot properly deliver the letter. In the digital world, computers handle the delivery of messages in a similar way. This is called routing.
IP addresses
To properly route your messages to a location, you need an address. Just like each home has a mailing address, each computer has an IP address. However, instead of using the combination of street, city, state, zip code, and country, the IP address uses a combination of bits, 0s and 1s.
Here is an example of 32-bit address in binary format:
11000000 10101000 00000001 00011110
It’s called 32-bit because you have 32 digits. Feel free to count!
IPv4 notation
Typically, you don’t see an IP address in its binary format. Instead, it’s coveted into decimal format and noted as an ipv4 address.
In the following, the 32 bits are grouped into groups of 8 bits, also called octets or bytes. Each of these groups is converted into decimal format separated by a period.
192.168.1.30
In the end, this is what is called an Ipv4 address. This is important to know when trying to communicate to a single computer. But remember, you’re working with a network. This is where CIDR notation comes in.
CIDR notation
192.168.1.30 is a single IP address. If you want to express IP addresses between the range of 192.168.1.0 and 192.168.1.255, how could you do that?
One way is to use Classless Inter-Domain Routing (CIDR) notation. CIDR notation is a compressed way of specifying a range of IP addresses. Specifying a range determines how many IP addresses are available to you.
CIDR notation is shown here.
192.168.1.0/24
It begins with a starting IP address and is separated by a forward slash (the “/” character) followed by a number. The number at the end specifies how many of the bits of the IP address are fixed. In this example, the first 24 bits of the IP address ore fixed (the first three bytes/octets). The rest are flexible.
32 total bits subtracted by 24 fixed bits leaves 8 flexible bits. Each of these flexible bits can be either 0 or 1, because they are binary. That means that you have two choices for each of the 8 bits, providing 256 IP addresses in that IP range.
The higher the number after the ‘/’, the smaller the number of IP addresses in your network. For example, a range of 192.168.1.0/24 is smaller than 192.168.1.0/16.
When working with networks in the AWS Cloud, you choose your network size by using CIDR notation. In AWS, the smallest IP range you can have is ‘/28’, which provides 16 IP addresses. The largest IP range you can have is ‘/16’, which provides 65,536 IP addresses.
Amazon VPC
A **virtual private cloud (VPC) is an isolatd network that you create in the AWS Cloud, similar to a traditional network in a data center. When you create a VPC, you must choose three main factors:
- Name of VPC.
- Region where the VPC will live. Each VPC spans multiple Availability Zones within the selected Regions.
- IP range for the VPC in CIDR notation. This determines the size of your network. Each VPC can have up to four ‘/16’ IP ranges.
Using this information, AWS will prvision a network and IP address for that network.
Create a subnet
After you create your VPC, you must create subnets inside the network. Think of subnets as smaller networks inside your base network - or virtual local area networks (VLANs) in a traditional, on-premises network. In an on-premises network, the typical use case for subnets is to isolate or optimize network traffic. In AWS, subnets are used to provide high availability and connectivity options for your resources.
When you create a subnet, you must sepcify the following:
- VPC you want your subnet to live in. In this case: VPC (10.0.0.0/16)
- Availability Zone you want your subnet to live in. In this case: AZ1
- CIDR block for your subnet, which must be a subset of the vpc CIDR block. In this case: 10.0.0.0/24
When you launch an EC2 instance, you launch it inside a subnet, which will be located inside the Availability Zone you choose.
High availability with a VPC
When you cerate your subnets, keep high availability in mind. to maintain redundancy and fault tolerance, crete at least two subnets configured in two Availability Zones.
As you learned earlier, remember that “everything fails all of the time.” With the example network, if one of the AZs fails, you will still have your resources available in another AZ as backup.
Reserved IPs
For AWS to configure your VPC appropirately, AWS reserves five IP addresses in each subnet. These IP addressess are used for routing, Domain Name System (DNS), and network management.
For example, consider a VPC with the IP range 10.0.0.0/22. The VPC includes 1,024 total IP addresses. This is divided into four equal-sized subnets, each with a ‘/24’ IP range with 256 IP addresses. Out of each of those IP ranges, there are only 251 IP addresses that can be used because AWS reserves five.
The five reserved iP addresses can impact how you design your network. A common starting place for those who are new to teh cloud is to create a PPC with an IP range of ‘/16’ and create subnets with an IP range of ‘/24’. This provides a large amount of IP addresses to work with at both the VPC and subnet levels.
Gateways
Internet gateway (IGW)
To enable internet connectivity for your VPC, you must cerate an internet gateway. Think of the gateway as similar to a modem. Just as a modem connects your computer to the internet, the internet gateway connects your VPC to the internet. Unlike your modem at home, which sometimes goes down or offline, an internet gateway is highly available and scalable. After you create an internet gateway, you attache it to your VPC.
Virtual private gateway (VGW)
A virtual private gateway connects your AWS VPC to another private network. Once you create and attach a virtual private gateway to a VPC, the gateway acts as anchor on the AWS side of the connection. On the other side of the connection, you will need to connect a customer gateway to the other private network. A customer gateway device is a physical device or software application on your side of the connection. Once you have both gateways, you can then establish an encrypted VPN connection between the two sides.
Route Tables
Main route table
When you create a VPC, AWS creates a route table called the main route table. A route table contains a set of rules, called routes, that are used to determine where network traffic is directed. AWS assumes that when you create a new VPC with subnets, you want traffic to flow between them. therefore, the default configuration of the main route table is to allow treffic between al subnets in the local network. Below is an example of a main route tabel.
The destination and tirget are two main parts of this route table.
- The destination is a range of IP addresses where you want your traffic to go. In the example of sending a letter, you need a destination to route the letter to the appropriate place. The same is true for routing traffic. In this case, the destination is the VPC network’s IP range.
- The target is the connection through which to send the traffic. In this case, the traffic is routed through the local VPC network.
| Destination | Target | Status | Propagated |
|---|---|---|---|
| 10.2.0.0/16 | local | active | No |
Custom route tables
While the main route table is used implicit by subnet that do not have an explicit route table association, you might want to provide different routes on a per-subnet basis, for traffic to access resources outside the VPC. For example, your application might consist of a front end and a database. You can create separate subnets for the resources and provide different routes for each of them.
If you associate a custom route table with a subnet, the subnet will use it instead of the main route table. Each custom route table you create will have the local route already inside it, allowing communication to flow between all resources and subnets inside the VPC. The local route cannot be deleted.
Secure subnets with network access control lists (ACLs)
Think of a network access control list (network ACL) as a firewall at the subnet level. A network ACL enables you to control what kinds of traffic is allowed to enter or leave your subnet. You can configure this by setting up rules that define what you want to filter. Here’s an example:
Inbound
| Rule # | Type | Protocol | Port Range | Source | Allow/Deny |
|---|---|---|---|---|---|
| 100 | All IPv4 traffic | All | All | 0.0.0.0/0 | ALLOW |
| * | All IPv4 traffic | All | All | 0.0.0.0/0 | DENY |
Outbound
| Rule # | Type | Protocol | Port Range | Source | Allow/Deny |
|---|---|---|---|---|---|
| 100 | All IPv4 traffic | All | All | 0.0.0.0/0 | ALLOW |
| * | All IPv4 traffic | All | All | 0.0.0.0/0 | DENY |
The default network ACL, shown in the preceding table, allows all traffic in and out of the subnet. To allow data to flow freely to the subnet, this is a good starting place.
However, you might want to restrict data tat the subnet level. For example, if you have a web application, you might restrict your network to allow HTTPS traffic and remote desktop protocol (RDP) traffic to your web server.
Inbound
| Rule # | Source IP | Protocol | Port | Allow/Deny | Comments |
|---|---|---|---|---|---|
| 100 | All IPv4 traffic | TCP | All | ALLOW | Allows inbound HTTPS traffic from anywhere |
| 130 | 192.0.2.0/24 | TCP | All | ALLOW | Allows inbound RDP traffic to the web servers from your home network’s public IP address range (over the internet gateway) |
| * | All IPv4 traffic | All | All | DENY | Denies all inbound traffic not already handled by a preceding rule (not modifiable) |
Outbound
| Rule # | Destination IP | Protocol | Port | Allow/Deny | Comments |
|---|---|---|---|---|---|
| 120 | 0.0.0.0/0 | TCP | 1025-65535 | ALLOW | Allows outbound responses to clients on the internet (severing people visiting the web servers in the subnet). |
| * | 0.0.0.0/0 | All | All | DENY | Denies all outbound traffic not already handled by a preceding rule (not modifiable). |
Note that in the preceding network ACL example, you allow inbound 443 and outbound range 1025-65535. That’s because HTTP uses port 443 to initiate a connection and will respond to an ephemeral port. Network ACLs are considered stateless, so you need to include both the inbound and outbound ports used for the protocol. If you don’t include the outbound range, your server would respond but the traffic would never leave the subnet.
Since network ACLs are configured by default to allow incoming and outgoing traffic, you don’t need to change their initial settings unless you need additional security layers.
Secure EC2 instances with security groups
The next layer of security is for your EC2 Instances. Here, you can create a firewall called a security group. The default configuration of a security group blocks all inbound traffic and allows all outbound traffic.
You might be wondering, “Wouldn’t this block all EC2 instances from receiving the response of any customer requests?” Well, security groups are stateful. That means that they will remember if a connection is originally initiate by the EC2 instance or from the outside, and temporarily allow traffic to respond without modifying the inbound rules.
If you want your EC2 instance to accept traffic from the internet, you must open up inbound ports. If you have a web server, you might need to accept HTTP and HTTPS request to allow that type of traffic into your security group. You can create an inbound rule that will allow port 80 (HTTP) and port 443 (HTTPS), as shown in the following table:
Inbound rules
| Type | Protocol | Port Range | Source |
|---|---|---|---|
| HTTP (80) | TCP (6) | 80 | 0.0.0.0/0 |
| HTTP (80) | TCP (6) | 80 | ::/0 |
| HTTPS (443) | TCP (6) | 443 | 0.0.0.0/0 |
| HTTPS (443) | TCP (6) | 443 | ::/0 |
You learned in a previous section that subnets can be used to segregate traffic between computers in your network. Security groups can be used in the same way. A common design pattern is to organize resources into different groups and create security groups for each to control network communications between them.
Storage
AWS storage services are grouped into three categories - block, file, and object storage.
File storage
You might be familiar with file storage if you have interacted with file
storage systems like Windows File Explorer or Finder on MacOS. Files are
organized in a tree-like hierarchy that consists of folders and sub-folders.
For example, if you have hundreds of cat photos on your laptop, you might want
to create a folder called cat_photos, and place the images inside that
folder to organize them. Since you know these images will be used in an
application, you might want to place the cat photos folder inside another
folder called application_files.
Each file has metadata such as the file’s name, size, and the date that the
file was created. The file also has a path, for example,
computer/application_files/cat_photos/cat-03.png.. When you need to
retrieve a file, your system can use the path to find it in the file hierarchy.
File storage is ideal when you require centralized access to files that need to be easily shared and managed by multiple host computers. Typically, this storage is mounted onto multiple hosts, and requires file locking and integration with existing files system communication protocols.
Common use cases for file storage include:
- Large content repositories
- Development environments
- User home directories
Block storage
While file storage treats files as a singular unit, block storage splits files into fixed-size chunks of data called blocks that have their own addresses. Since each block is addressable, blocks can be retrieved efficiently.
When data is requested, the addresses are used by the storage system to organize the blocks in the correct order to form a complete file to present back to the requester. Outside of the address, no additional metadata is associated with each block. So, when you want to change a character in a file, you just change the block, or the piece of the file, that contains the character. This ease of access is why block storage solutions are fast and use less bandwidth.
Since block storage is optimized for low-latency operations, it is a typical storage choice for high-performance enterprise workloads, such as databases or enterprise resource planning (ERP) systems, that require low-latency storage.
Object storage
Objects, much like files, are treated as a single unit of data when stored. However, unlike file storage, these objects are stored in a flat structure instead of a hierarchy. Each object is a file with a unique identifier. This identifier, along with any additional metadata, is bundled with the data and stored.
With object storage, you can store almost any type of data, and there is no limit to the number of objects stored, which makes it readily scalable. Object storage is generally useful when storing large datasets; unstructured files, like media assets; and static assents, like photos.
Relate back to traditional storage systems
If you have worked with on-premises storage, you might already be familiar with block, filo, and object storage. Consider the following technologies and how they relate to systems you might have seen before:
- Block storage in the cloud is analogous to direct-attached storage (DAS) or a storage area network (SAN).
- File storage systems are often supported with a network attached storage (NAS) server.
Adding storage in a traditional data center is a rigid process - the storage solutions much be purchased, installed, and configured. With cloud computing, the process is more flexible. You can create, delete, and modify storage solutions within a matter of minutes.
Amazon EC2 instance store
Amazon EC2 instance store provides temporary block-level storage for an instance. This storage is located on disks that are physically attached to the host computer. This ties the lifecycle of the data to the lifecycle of the EC2 instance. If you delete the instance, the instance store is deleted, as well. Due to this, instance store is considered ephemeral storage. Read more about it in the AWS documentation
Instance store is ideal if you host applications that replicate data to other EC2 instances, such as Hadoop clusters. For these cluster-based workloads, having the speed of locally attached volumes and the resiliency of replicated data helps you achieve data distribution at high performance. It’s also ideal for temporary storage.
Amazon Elastic Block Storage (Amazon EBS)
As the name implies, Amazon EBS is a block-level storage device that you can attach to an Amazon EC2 instance. These storage devices are called Amazon EBS volumes. EBS volumes are essentially drives of a user-configured size attached to an EC2 instance, similar to how you might attach an external drive to your laptop. EBS volumes act similarly to how you might attach an external drive to you laptop. EBS volumes act similarly to external drives in more than one way:
- Most Amazon EBS volumes can only be connect with one computer at a time. Most EBS volumes have a one-to-one relationship with EC2 instances, so they cannot be shared by or attached to multiple instances at one time. (Recently, AWS announced the Amazon EBS multi-attach feature that enables volumes to be attached to multiple EC2 instances at one time. This feature is not available for all instance types, and all instances must be in the same Availability Zone. Read more about this scenario in the EBS documentation.)
- You can detach an EBS volume for one EC2 instance and attach it to another EC2 instance in the same Availability Zone, to access the data on it.
- The external drive is separate from the computer. That means, if an accident occurs and the computer goes down, you still have your data on your external drive. The same is true for EBS volumes.
- You’re limited to the size of the external drive, since it has a fixed limit to how scalable it can be. For example, you might have a 2TB external drive, which means you can only have 2TB of content on it. This relates to EBS as well, since a volume also has a max limitation of how much content you can store on it.
Scale Amazon EBS volumes
You can scale Amazon EBS volumes in two ways:
- Increase the volume size, as long as it doesn’t increase above the maximum size limit. For EBS volumes, the maximum amount of storage you can have is 16TB. If you provision a 5TB EBS volume, you can choose to increase the size of you volume until you get to 16TB.
- Attach multiple volumes to a single Amazon EC2 instance. EC2 has a one-to-many relationship with EBS volumes. You can add these additional volumes during or after EC2 instance creation to provide more storage capacity for your hosts.
Amazon EBS use cases
Amazon EBS is useful when you must retrieve data quickly and have data persist long-term. Volumes are commonly used in the following scenarios.
- Operating systems: Boot/root volumes to store an operating system. The root device for an instance launched for an Amazon Machine Image (AMI) is typically an Amazon EBS volume. These are commonly referred to as EBS_backed AMIs.
- Databases: A storage layer for databases running on Amazon EC2 that rely on transactional reads and writes.
- Enterprise applications: Amazon EBS provides reliable block storage to run business-critical applications.
- Throughput-intensive applications: Applications that perform long, continuous reads and writes.
Amazon EBS volume types
Amazon EBS volumes are organized into two main categories - solid-state drives (SSDs) and hard-disk drives (HDDs). SSDs provide strong performance for random input/output (I/O), while HDDs provide strong performance for sequential I/O. AWS offers two types of each volume type.
| Volume Types | Description | Use Case | Volume Size | Max IOPS | Max Throughput |
|---|---|---|---|---|---|
| EBS Provisioned IOPS SSD | Highest performance SSD Designed for latency-sensitive transactional workloads | I/O-intensive NoSQL and relational databases | 4GB - 16TB | 64,000 | 1,000 MB/s |
| EBS General Purpose S SSD | General purpose SSD that balances price and performance for a wide variety of transactional workloads | Boot volumes, low-latency interactive apps, development, and test | 1GB - 16TB | 16,000 | 250 MB/s |
| Throughput Optimized HDD | Low-cost HDD designed for frequently accessed, throughput-intensive workloads | Big data, data warehouse, log processing | 500GB - 16TB | 500 | 500 MB/s |
| Cold HDD | Lowest cost HDD designed for less frequently accessed workloads | Colder data requiring fewer scans per day | 500GB - 16TB | 250 | 250 MB/s |
Amazon EBS benefits
Here are the benefits of using Amazon EBS:
- High availability: When you create EBS volume, it is automatically replicated in its Availability Zone to prevent data loss from single points of failure.
- Data persistence: The storage persists even when your instance doesn’t.
- Data encryption: All EBS volumes support encryption.
- Flexibility: EBS volumes support on-the-fly changes. You can modify volume type, volume, size, and input/output operations per second (IOPS) capacity without stopping your instance.
- Backups: Amazon EBS provides the ability to create backups of any EBS volume.
Amazon EBS snapshots
Errors happen. One error is not backing up data and then inevitably losing it. To prevent this from happening to you, always back up your data - even in AWS.
Since your EBS volumes consist of the data from your Amazon EC2 instance, you should make backups of these volumes, called snapshots.
EBS snapshots are incremental backups that only save the blocks on the volume that have changed after your most recent snapshot. For example, if you have 10GB of data on a volume, and only 2GB of data have been modified since your last snapshot, only the 2GB that have been changed are written to Amazon Simple Storage Service (Amazon S3).
When you take snapshot of any of your EBS volumes, the backups are stored redundantly in multiple Availability Zones using Amazon S3. This aspect of storing the backup in Amazon S3 is handled by AWS, so you won’t need to interact with Amazon S3 to work with your EBS snapshots. You manage them in the Amazon EBS console, which is part of the Amazon EC2 console.
EBS snapshots can be used to create multiple new volumes, which they’re in the same Availability Zone or a different one. When you create a new volume from a snapshot, it’s an exact copy of the original volume at the time of the snapshot was taken.
Amazon S3
Unlike Amazon Elastic Block Store (Amazon EBS), Amazon Simple Storage Service (Amazon S3) is a standalone storage solution that isn’t tied to a compute service. It enables you to retrieve your data from anywhere on the web. If you have used an online storage service to back up the data from your local machine, then you most likely have used a service similar to Amazon S3. The big difference between those online storage services and Amazon S3 is the storage type.
Amazon S3 is an object storage service. Object storage stores data in a flat structure, using unique identifiers to look up objects when requested. An object is a file combined with metadata. You can store as many of these objects as you’d like. All of the characteristics of object storage are also characteristics of Amazon S3.
Concepts
In Amazon S3, you store your objects in containers called buckets. You can’t upload an object, not even a single photo, to Amazon S3 without creating a bucket first. When you create a bucket, you specify, at the very minimum, two details - the AWS Region you want the bucket to reside in and the bucket name.
To choose a Region, you will typically select a Region that you have used for other resources, such as your compute. When you choose a Region for your bucket, all objects you put inside the bucket will be redundantly stored access multiple devices, across multiple Availability Zones. This level of redundancy is designed to provide Amazon S3 customers the 99.999999999% durability and 99.99% availability for objects over a given year.
When you choose a bucket name, it must be unique across all AWS accounts AWS stops you from choosing a bucket name that has already been chosen by someone else in another AWS account. Once you choose a name, that name is yours and cannot be claimed by anyone else unless you delete the bucket, which then releases the name for others to use.
AWS uses the bucket name as part of the object identifier. In S3, each object is identified using a URL, as shown:
http://doc.s3.amazonaws.com/2006-03-01/AmazonS3.html
After the http://, you can see the bucket name. In this example, the bucket is named doc. Then, the identifier uses the s3 service name and the service provider, amazonaws. After that, you have an implied folder inside the bucket called 2006-03-01 and the object inside the folder that is named AmazonS3.html. The object name is often referred to as the key name.
You can have folders inside of buckets to help you organize objects. However, remember that no actual file hierarchy supports this on the backed. It is instead a flat structure where all files and folders live at the same level. Using buckets and folders implies a hierarchy, which creates an understandable organization for users.
Use cases
Amazon S3 is a widely used storage service, with far more use cases than could fit on one screen. The following list summarizes some of the most common ways you can use Amazon S3:
- Backup and storage: Amazon S3 is a natural place to back up files because it is highly redundant. As mentioned in the last unit, AWS stores your EBS snapshots in S3 to take advantage of its high availability.
- Media hosting: Because you can store unlimited objects, and each individual object can be up to 5TBs, Amazon S3 is an ideal location to host video, photo, and music uploads.
- Software delivery: you can use Amazon S3 to host your software applications that customers can download.
- Data lakes: Amazon S3 is an optional foundation for a data lake because of its virtually unlimited scalability. You can increase storage from gigabytes to petabytes of content, paying only for what you use. Static websites: You can configure your S3 bucket to host a static website of HTML, CSS, and client-side scripts. - Static content: Because of the limitless scaling, the support for large files, and the fact that you access any object over the web at any time, Amazon S3 is the perfect place to store static content.
Choosing the right connectivity option for resources
Everything in Amazon S3 is private by default. This means that all S3 resources, such as bucktes, folders, and objects can only be viewed by the user or AWS account that created that resource. Amazon S3 resources are all private and protected to begin with.
If you decide that you want everyone on the internet to see you photos, you can choose to make your buckets, folders, and objects public. A public resource means that eveyrone on the internet can see it. Most of the time, you don’t want your permissions to be all or nothing. Typically, you want ot be more granular about the way you provide access ot your resources.
To be more specific about who can do what with your Amazon S3 resources, Amazon S3 provides two main access maangement features - IAM ploicise and S3 buckets policies.
IAM policies
Previously, you learned about creating and using IAM policies. Now, you can apply that knowledge to Amazon S3. When IAM policies are attached to IAM users, groups, and roles, ther policies define which actions they can perform. IAM policies are not tied to any one AWS service and can be used to define access to nearly any AWS action.
You should use IAM policies for private buckets in the following two scenarios:
- You have any buckets with different permission requirements. Instead of defining many differnt S3 bucket policies, you can use IAM policies.
- You want alll policies to be in a centralized location. Using IAM policies allows you to manage all policy information in one location.
S3 bucket policies
Like IAM policies, Amazon S3 bucket policies are defined in a JSON format. The difference is IAM policios are attached to users, groups, and roles, whereas S2 bucket policies ar eanly attached to S3 buckets. S3 bucket policies specify what actions are allowed or denied on the bucket.
For example, if you have a bucket called employeebucket, you can attach an S3 bucket policy to it that allows another AWMS account to put objects in that bucket.
Or if you wanted to allow anonymous viewers to read the objcets in employee
bucket, then you can apply a policy to that bucket that allows anyone to read
objcets in the bucket using "Effect": "Allow" on the "Action": ["s3:GetObject"].
Here’s an example of what the S3 bucket policy might look like.
{
"Version": ":"
}
NOTES: Wed 03/01/2023
Stephon’s course should be harder than AWS exams.
- S3
- Security
- Loud Balancers
- Cloud Front
- Billing
- AWS private link