AWS Storage: EBS, EFS, S3

Here are some of the main storage options in AWS: 

• EBS (Elastic Block Store)
• EFS (Elastic File System)
• S3 (Simple Storage Service)

I created the following diagram to help visualize how these work:

Instance Store: An EC2 Instance has local store, similar to a physical or virtual machine. 

For a Linux-based EC2 instance (such as Amazon Linux 2), there will be the typical / (root) directory, and standard Linux directories such as /bin, /etc, /opt, /tmp

For a Windows-based EC2 Instance, there is the typical C:\ directory. 

The problem with Instance Store is that it is not persistent or durable.  If the EC2 Instance is terminated, the data that was stored in /opt is gone. 

I/O is fast because there is no network latency between the EC2 Instance and its Instance Store.

EBS: Persistent, but is attached to a single EC2 Instance.  I think of it like plugging in an additional hard drive.  You can't connect it to multiple EC2 Instances because File Systems such as EXT4, NTFS don't handle that.  I/O is fast because there is no network latency between the EC2 Instance and EBS. 

EFS: Persistent, can be shared by multiple EC2 Instances and On-Prem Servers.  I think of this like a mounted drive on a NAS.

S3: Object-based.  Objects are placed into/retrieved from buckets using HTTPS.  This is going to be much less performant than other storage mechanisms, but would be the most globally-accessible. 

ECS: Running Tasks and Task Definitions

Yay, Friday night, personal learning time!  

I'm starting to look into next steps of "how would I deploy new versions of my Spring Boot Application to ECS"?

Initial thoughts would be to do the following:

1. Update Spring Boot App
2. Test locally
3. Commit/push changes to GitHub
4. Create new local Docker Image
5. Push new Docker Image to ECR as a new version in the existing Repo
6. Create a new Revision of Task Definition (currently at Revision 1)
7. Stop current Running Task
8. Run Task using new version of Task Definition

First, taking a look at what I have running now in my ECS Cluster:

Here is the details of that cluster (when I click on the cluster-name link):

Finally, putting it all together from my previous posts, here are the details of the Task when I click on the ARN of the Task.  

And note: public IP Address of the Task is 54.237.231.47

Here is me testing that I can hit my Spring Boot App on this public IP Address:

http://54.237.231.47:8080/hello-world?name=philip

AWS VPC, Subnets, Route Tables, Internet Gateway, and CIDR Blocks

I'm a firm believer that every technology professional has stumbling blocks, something that no matter how hard they try, it's just difficult to "make that knowledge stick".

For some, it's RegEx, for others it could be DNS configuration.  

One of mine is around networking, subnets, and netmasks.  

Unfortunately, understanding these concepts is essential to AWS Virtual Private Cloud (VPC) configuration.  

Here's how I set up my VPC, Subnets, and, Route Tables, and Internet Gateways for my EC2 / ECS studies.

VPC: This is the starting point.  

I created a new VPC with the name web-apps and an IPv4 CIDR block of: 10.0.0.0/24

/24 is the equivalent of a Netmask of 255.255.255.0 

The way I've learned to think of CIDR blocks and Netmasks for IPv4 ... there are 4 Octets, and each Octet is 8 bits.  

/24 means that there are 24 1-bits in the Netmask, taking up 3 of the Octets, hence 255.255.255.0.

The higher the number (up to 32, or all 4 Octets, the more restrictive).

The lower the number (down to 0, or none of the Octets, the less restrictive).

So 0.0.0.0/0 on something like an Ingress Rule means "let the world in".  

Going back to my VPC, CIDR block of: 10.0.0.0/24

32 (all bits in an IPv4 Address) - 24 (from CIDR Block) = 8 (number of bits available for addresses in my VPC).

This means that I will have 2⁸ = 256 IP Addresses availabile to me, with a range of 10.0.0.0 -> 10.0.0.255

Note: have to subtract off 5 IP Addresses, based on Amazon's documentation: "The first four IP addresses and the last IP address in each subnet CIDR block are not available for your use, and they cannot be assigned to a resource, such as an EC2 instance.

Source: https://docs.aws.amazon.com/vpc/latest/userguide/configure-subnets.html

Subnets:

In a Production App, I would create different Subnets for different Availability Zones. Each Subnet must reside in a single Availability Zone.  

Since this is a learning project, I just used a single Subnet with a CIDR block that spanned the entire VPC.  

Route Tables:

I created a new Route Table (leaving the default one alone) and explicitly assoicated it with my web-apps-subnet.  This specifies how traffic should be routed, as the name would imply.

There are 2 routes.  It is saying "any traffic within the range of 10.0.0.0/24 is considered local traffic within the VPC.  All other traffic of 0.0.0.0/0 goes to the Internet Gateway igw-064640bbdb2f376ac

Security Group:

I have 2 inbound rules that allow for traffic on port 22 (so I can SSH into EC2 servers) and on port 8080 (port used by my Spring Boot App).

Network Interface (EC2):

My EC2 instance's Network Interface specifies the VPC, Subnet, and Security Group created in earlier steps.

EC2 and Java Spring Boot

These days, containers and ECS/EKS seem to be all the rage.  However, I wanted to make sure I could still install and run my application on an EC2 Instance.

I decided to create an EC2 Instance t2.micro) using the AMI for Amazon Linux 2.

Step 1: Installing Java 17 (Amazon Corretto)

I logged in using the private key from the Keypair I generated and installed Java 17:

login as: ec2-user

Authenticating with public key "java-demo-keypair"

Last login: Mon Dec 19 02:54:32 2022 from **********

       __|  __|_  )

       _|  (     /   Amazon Linux 2 AMI

      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/

[ec2-user@ip-10-0-0-72 ~]$ sudo yum install java-17-amazon-corretto

after all the dependencies were downloaded, installed, and this completed, I verified:

[ec2-user@ip-10-0-0-72 ~]$ java -version

openjdk version "17.0.5" 2022-10-18 LTS

OpenJDK Runtime Environment Corretto-17.0.5.8.1 (build 17.0.5+8-LTS)

OpenJDK 64-Bit Server VM Corretto-17.0.5.8.1 (build 17.0.5+8-LTS, mixed mode, sharing)

Finally, I added a Bash script to /etc/profile.d to set JAVA_HOME and PATH to include JAVA_HOME. 

[ec2-user@ip-10-0-0-72 ~]$ sudo vi /etc/profile.d/java.sh

export JAVA_HOME=/usr/lib/jvm/java-17-amazon-corretto.x86_64

and sourced this Bash script so it would take effect immediately:

[ec2-user@ip-10-0-0-72 ~]$ source /etc/profile.d/java.sh

Step 2: Create Service to run Spring Boot App

Create java-user (in java-group) with a home directory of /opt/java-apps 

[ec2-user@ip-10-0-0-72 ~]$ sudo groupadd -r java-group

[ec2-user@ip-10-0-0-72 ~]$ sudo useradd -r -m -g java-group -d /opt/java-apps -s /bin/false java-user

                             

I copied the Spring Boot JAR into /opt/java-apps/demo.jar and created the following Systemctl Service file:

[root@ip-10-0-0-72 system]# sudo vi /etc/systemd/system/springboot.service

[Unit]
Description=Java Spring Boot Demo App
After=syslog.target

[Service]
WorkingDirectory=/opt/java-apps
ExecStart=/bin/java -jar demo.jar
SuccessExitStatus=143

User=java-user
Group=java-group

[Install]
WantedBy=multi-user.target

AWS Run Task Definition in Cluster

Finally, putting everything together, we now have an Image in ECR, a Cluster in ECS, and a Task Definition.

From the Task Definition, choose Deploy -> Run task

This brought me to a page with a number of sections about the Task to run. The first 2 were not collapsable, the other 4 were. 

• Environment 
• Deployment Configuration
• Networking (collapsed)
• Task Overrides (collapsed)
• Container Overrides (collapsed)
• Tags - optional (collapsed)

Environment

I changed from FARGATE to FARGATE_SPOT. If I were running an actual Production Application, I would have used FARGATE. However, my understanding is that this is going to run on spare/deprioritized AWS Capacity and cost about 70% less.

Deployment Configuration

Networking

AWS ECS Task Definition

 Now that I have an ECR Image (built locally and pushed up from Docker Desktop) and an ECS Cluster Created, next step is to create a Task Definition.

I provided the Container Port of 8080 (same as the one I exposed in the Dockerfile of the Image). 

Here, we are going absolute minimum on CPU and Memory since it is a test app.
Note: In order to actually create Task Def and Containers, I had to go back as the Root User and create Inline Policy that contained the following IAM Actions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "iam:CreateRole",
                "iam:AttachRolePolicy"
            ],
            "Resource": "*"
        }
    ]
}

Otherwise you will see error messages like this, since AWS is trying to create a Role for running your task.

AWS ECS Cluster Creation

I created a cluster using AWS "New ECS Experience" today.  Here is the entry point:

It looks great.  I found it to be much easier and less confusing to navigate than the previous ECS interface.  

On the networking section, they had 6 subnets initially selected, and they recommend at least 3 for production.

Since this is my weekend cloud study/learning project, I don't need to distribute across multiple availability zones for high availability, so I just chose 1 subnet.

One of the big changes is that they are more explicit about letting you know that selecting EC2 instances is in addition to Fargate, not in-lieu of Fargate, as seen by the fact that AWS Fargate is checked and disabled (cannot be unselected). 

AWS Pushing Docker Image to Elastic Container Registry (ECR)

I tried to create my first Repository using the IAM user I created, java-demo. 

However, I ended up getting an error:

I fixed this by updating the ECR-PushImages Policy to include an additional action: ecr:CreateRepository.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecr:CreateRepository",
                "ecr:CompleteLayerUpload",
                "ecr:GetAuthorizationToken",
                "ecr:UploadLayerPart",
                "ecr:InitiateLayerUpload",
                "ecr:BatchCheckLayerAvailability",
                "ecr:PutImage"
            ],
            "Resource": "*"
        }
    ]
}

Note: one thing I found to be super-confusing was the Version block. I looked at it, saw a date from 2012, and figured "hmm, I should update this to the current date in 2022." 

However, that causes errors.

The Version is actually the Version of Policy language, and there are only 2 valid values:

<version_block> = "Version" : ("2008-10-17" | "2012-10-17")

Source: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_grammar.html

That allowed me to create the ECR Repository.  However, after logging back on as my IAM user, I still couldn't list the newly-created Repository. :-/  

Rather than fumble through and one-by-one try and figure out what was needed, I ended up logging back in as Root User, using the Visual Editor, looking up Service: Elastic Container Registry and manually giving the IAM user actions by Access Level ...

Once completing setting up the available ecr:actions with the Policy, I was able to tag and push my Docker Image to ECR. 

C:\>aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin ************.dkr.ecr.us-east-1.amazonaws.com
Login Succeeded

C:\>docker tag f94c25ad91cd1dabbb0dae012a0da3f50c23e050fdd1916d7bd81d5c9dbec2b9 ************.dkr.ecr.us-east-1.amazonaws.com/java-spring-cloud-demo:v0.0.1

C:\>docker push ************.dkr.ecr.us-east-1.amazonaws.com/java-spring-cloud-demo:v0.0.1
The push refers to repository [************.dkr.ecr.us-east-1.amazonaws.com/java-spring-cloud-demo]
e971bfdd6e68: Pushed
cb90fdeb280a: Pushed
15b10c92f3b2: Pushed
e5e13b0c77cb: Pushed
v0.0.1: digest: sha256:6bee10abc02d77bced7593744f31b4d373069042cb45ae4cf4a2648992b5265a size: 1161

AWS IAM User/Group/Policies

Next step in my weekend AWS / Container / Spring Boot experimentation is to try and actually get my Docker Image pushed up to ECR.  

It runs fine in my local Docker Desktop, but I want to get it running as a service in AWS ECS.  

In following AWS best practices:

"We strongly recommend that you do not use the root user for your everyday tasks, even the administrative ones. Instead, adhere to the best practice of using the root user only to create your first IAM user. Then securely lock away the root user credentials and use them to perform only a few account and service management tasks. To view the tasks that require you to sign in as the root user, see AWS Tasks That Require Root User."

Source: https://docs.aws.amazon.com/IAM/latest/UserGuide/id.html?icmpid=docs_iam_console

I created a new IAM user called java-demo and Policy called ECR-PushImages that should allow this IAM user to push to any ECR Repository.  

If I were working in an enterprise environment, I would restrict resources, but since this is home studies, I'm leaving it open.

I'm also creating a Group called java-group, adding the IAM java-demo to this Group, and assigned ECR-PushImages (and some other out-of-box AWS IAM Policies) to this Group.

Next, I installed the AWS CLI v2 (latest) and configured it using aws configure

It asked me for 4 items:

• AWS Access Key ID: <access key for java-demo IAM user>
• AWS Secret Access Key: <secret access key for java-demo IAM user>
• Default region name: us-east-1
• Default output format: json

Once this was done, I confirmed that I could connect via CLI to ECR:

 

C:\>aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin **********.dkr.ecr.us-east-1.amazonaws.com
Login Succeeded

Dockerfile and Amazon Corretto

Managed to crawl out of bed this weekend to play with Dockerfiles and Amazon Corretto (didn't feel well today).

I went to Docker Hub site for Amazon Corretto and was surprised to see Java 19 and thought about using this as my base Docker Image.  

However, went out and checked, and Java 19 is not a LTS version, so I decided to stick with Java 17. 

My Dockerfile:

FROM amazoncorretto:17.0.5-alpine3.16
RUN addgroup -S spring && adduser -S java -G spring
USER java:spring
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
EXPOSE 8080
ENTRYPOINT ["java","-jar","/app.jar"]

Then I ran the following command: 

docker build -t philiptenn/demo-app .

to create a Docker Image in my local Docker Desktop, and ran it with

docker run -p 8080:8080 philiptenn/demo-app

to create a running container.