Learn AWS by Coding (LABC)

This book was prepared as a lecture material for "Special Lectures on Information Physics and Computing", which was offered in the S1/S2 term of the 2021 academic year at the Department of Mathematical Engineering and Information Physics, the University of Tokyo.

The purpose of this book is to explain the basic knowledge and concepts of cloud computing for beginners. It provides hands-on tutorials to use real cloud environment provided by Amazon Web Services (AWS).

We assume that the readers would be students majoring science or engineering at college, or software engineers who are starting to develop cloud applications. We will introduce pracitcal steps to use the cloud for research and web application development. We plan to keep this course as interactive and practical as possible, and for that purpose, less emphasis is placed on the theories and knowledge, and more effort is placed on writing real programs. I hope that this book serves as a stepping stone for readers to use cloud computing in their future research and applications.

The book is divided into three parts:

In the first part, we explain the basic concepts and knowledge of cloud computing. Essential ideas necessary to safely and cleverly use cloud will be covered, including security and networking. In the hands-on session, we will practice setting up a simple virtual server on AWS using AWS API and AWS CDK.

In the second part, we introduce the cocenpts and techniques for running scientific computing (especially machine learning) in the cloud. In parallel, we will learn a modern virtual coumputing environment called Docker. In the first hands-on session, we will run Jupyter Notebook in the AWS cloud and run a simple machine learning program. In the second hands-on, we will create a bot that automatically generates answers to questions using natural language model powered by deep neural network. In the third hands-on, we will show how to launch a cluster with multiple GPU instances and perform massively parallel hyperparameter search for deep learning.

In the third part, we introduce the latest cloud architecture called serverless architecture. This architecture introduces radically different design concept to the cloud than the previous one (often referred to as Serverful), as it allows the processing capacity of the cloud system to be scaled up or down more flexibly depending on the load. In the first hands-on session, we will provide exercises on Lambda, DynamoDB, and S3, which are the main components of the serverless cloud. In addition, we will create a simple yet quite useful social network service (SNS) in the cloud using serverless technology.

These extensive hands-on sessions will provide you with the knowledge and skills to develop your own cloud system on AWS. All of the hands-on programs are designed to be practical, and can be customized for a variety of applications.

The philosophy of this book can be summed up in one word: "Let’s fly to space in a rocket and look at the earth once!"

What does that mean?

The "Earth" here refers to the whole picture of cloud computing. Needless to say, cloud computing is a very broad and complex concept, and it is the sum of many information technologies, hardware, and algorithms that have been elaborately woven together. Today, many parts of our society, from scientific research to everyday infrastructure, are supported by cloud technology.

The word "rocket" here refers to this lecture. In this lecture, readers will fly into space on a rocket and look at the entire earth (cloud) with their own eyes. In this journey, we do not ask deeply about the detailed machinery of the rocket (i.e. elaborate theories and algorithms). Rather, the purpose of this book is to let you actually touch the cutting edge technologies of cloud computing and realize what kind of views (and applications) are possible from there.

For this reason, this book covers a wide range of topics from the basics to advanced applications of cloud computing. The first part of the book starts with the basics of cloud computing, and the second part takes it to the next level by explaining how to execute machine learning algorithms in the cloud. In the third part, we will explain serverless architecture, a completely new cloud design that has been established in the last few years. Each of these topic is worth more than one book, but this book was written with the ambitious intention of combining them into a single volume and providing a integrative and comprehensive overview.

It may not be an easy ride, but we promise you that if you hang on to this rocket, you will get to see some very exciting sights.

This book provides hands-on tutorials to run and deploy applications on AWS. Readers must have their own AWS account to run the hands-on excercises. A brief description of how to create an AWS account is given in the appendix at the end of the book (Section 14.1), so please refer to it if necessary.

AWS offers free access to some features, and some hands-on excercises can be done for free. Other hands-on sessions (especially those dealing with machine learning) will cost a few dollars. The approximate cost of each hands-on is described at the begining of the excercise, so please be aware of the potential cost.

In addition, when using AWS in lectures at universities and other educational institutions, AWS Educate program is available. This program offers educators various teaching resources, including the AWS credits which students taking the course can use to run applications in the AWS cloud. By using AWS Educate, students can experience AWS without any financial cost. It is also possible for individuals to participate in AWS Educate without going through lectures. AWS Educate provides a variety of learning materials, and I encourage you to take advantage of them.

In this book, we will provide hands-on sessions to deploy a cloud application on AWS. The following computer environment is required to run the programs provided in this book. The installation procedure is described in the appendix at the end of the book (Section 14). Refer to the appendix as necessary and set up an environment in your local computer.

We provide a Docker image with the required programs installed, such as Python, Node.js, and AWS CDK. The source code of the hands-on program has also been included in the image. If you already know how to use Docker, then you can use this image to immediately start the hands-on tutorials without having to install anything else.

Start the the container with the followign command.

More details on this Docker image is given in the appendix (Section 14.8).

The only prerequisite knowledge required to read this book is an elementary level understanding of the computer science taught at the universities (OS, programming, etc.). No further prerequisite knowledge is assumed. There is no need to have any experience using cloud computing. However, the following prior knowledge will help you to understand more smoothly.

The source code of the hands-on tutorials is available at the following GitHub repository.

https://github.com/tomomano/learn-aws-by-coding

In addition, we provide warnings and tips in the boxes.

What is the cloud? The term "cloud" has a very broad meaning in itself, so it is difficult to give a strict definition. In academic context, The NIST Definition of Cloud Computing, published by National Institute of Standards and Technology (NIST), is often cited to define cloud computing. The definition and model of cloud described here is illustrated in Figure 2.

According to this, a cloud is a collection of hardware and software that meets the following requirements.

This may sound too abstract for you to understand. Let’s talk about it in more concrete terms.

If you wanted to upgrade the CPU on your personal computer, you would have to physically open the chassis, expose the CPU socket, and replace it with a new CPU. Or, if the storage is full, you will need to remove the old disk and insert a new one. When the computer is moved to a new location, it will not be able to connect to the network until the LAN cable of the new room is plugged in.

In the cloud, these operations can be performed by commands from a program. If you want 1000 CPUs, you can send a request to the cloud provider. Within a few minutes, you will be allocated 1000 CPUs. If you want to expand your storage from 1TB to 10TB, you can send such command (you may be familiar with this from services such as Google Drive or Dropbox). When you are done using the compute resources, you can tell the provider about it, and the allocation will be deleted immediately. The cloud provider accurately monitors the amount of computing resources used, and calculates the usage fee based on that amount.

Namely, the essence of the cloud is the virtualization and abstraction of physical hardware, and users can manage and operate physical hardware through commands as if it were a part of software. Of course, behind the scenes, a huge number of computers in data centers are running, consuming a lot of power. The cloud provider achieves this virtualization and abstraction by cleverly managing the computational resources in the data center and providing the user with a software interface. From the cloud provider’s point of view, they are able to maximize their profit margin by renting out computers to a large number of users and keeping the data center utilization rate close to 100% at all times.

In the author’s words, the key characteristics of the cloud can be defined as follows:

Coming back to The NIST Definition of Cloud Computing mentioned above, the following three forms of cloud services are defined (Figure 2).

This book mainly deals with cloud development in IaaS. In other words, it is cloud development in which the developer directly manipulates the cloud infrastructure, configures the desired network, server, and storage from scratch, and deploys the application on it. In this sense, cloud development can be divided into two steps: the step of building a program that defines the cloud infrastructure and the step of crafting an application that actually runs on the infrastructure. These two steps can be separated to some extent as a programmer’s skill set, but an understanding of both is essential to build the most efficient and optimized cloud system. This book primarily focuses on the former (operating the cloud infrastructure), but also covers the application layer. PaaS is a concept where the developer focuses on the application layer development and relies on the cloud provider for the cloud infrastructure. PaaS reduces development time by eliminating the need to develop the cloud infrastructure, but has the limitation of not being able to control the detailed behavior of the infrastructure. This book does not cover PaaS techniques and concepts.

SaaS can be considered a development "product" in the context of this book. In other words, the final goal of development is to provide a computational service or database on the available to the general public by deploying the programs on IaaS platform. As a practical demonstration, we will provide hands-on exercises such as creating a simple SNS (Section 13).

Recently, Function as a Service (FaaS) and serverless computing have been recognized as new cloud categories. These concepts will be discussed in detail in later chapters (Section 12). As will become clear as you read through this book, cloud technology is constantly and rapidly evolving. This book first touches on traditional cloud design concepts from a practical and educational point of view, and then covers the latest technologies such as serverless.

Finally, according to The NIST Definition of Cloud Computing, the following four types of cloud deployment model are defined (Figure 2). Private cloud is a cloud used only within a specific organization, group, or company. For example, universities and research institutes often operate large-scale computer servers for their members. In a private cloud, any member of the organization can run computations for free or at a very low cost. However, the upper limit of available computing resources is often limited, and there may be a lack of flexibility when expanding.

Pubclic cloud is a cloud that is offered as a commercial service to general customers. Examples of famous public cloud platforms include Google Cloud Platform (GCP) provided by Google, Azure provided by Microsoft, and Amazon Web Services (AWS) provided by Amazon. When you use a public cloud, you pay the usage cost set by the provider. In return, you get access to the computational resources of the company operating the huge data center, so it is not an exaggeration to say that the computational capacity is inexhaustible.

The third type of cloud operation is called community cloud. This refers to a cloud that is shared and operated by groups and organizations that share the same objectives and roles, such as government agencies. Finally, there is the hybrid cloud, which is a cloud composed of a combination of private, public, and community clouds. An example of hybrid cloud would be a case where some sensitive and privacy-related information is kept in the private cloud, while the rest of the system depends on the public cloud.

This book is basically about cloud development using public clouds. In particular, we will use Amazon Web Services (AWS) to learn specific techniques and concepts. Note, however, that techniques such as server scaling and virtual computing environments are common to all clouds, so you should be able to acquire knowledge that is generally applicable regardless of the cloud platform.

As mentioned above, the cloud is a computational environment where computational resources can be flexibly manipulated through programs. In this section, we would like to discuss why using the cloud is better than using a real local computing environment.

In particular, point 1 is important in research situations. In research, there are few cases in which one must keep running computations all the time. Rather, the computational load is likely to increase intensively and unexpectedly when a new algorithm is conceived, or when new data arrives. In such cases, the ability to flexibly increase computing power is a major advantage of using the cloud.

So far, we have discussed the advantages of using the cloud, but there are also some disadvantages.

In this book, AWS is used as the platform for implementing cloud applications. In this chapter, we will explain the essential knowledge of AWS that is required for hands-on tutorials.

AWS (Amazon Web Services) is a general cloud platform provided by Amazon. AWS was born in 2006 as a cloud service that leases vast computing resources that Amazon owns. In 2021, AWS holds the largest market share (about 32%) as a cloud provider (Ref). Many web-related services, including Netflix and Slack, have some or all of their server resources provided by AWS. Therefore, most of the readers would be benefiting from AWS without knowing it.

Because it has the largest market share, it offers a wider range of functions and services than any other cloud platforms. In addition, reflecting the large number of users, there are many official and third-party technical articles on the web, which is helpful in learning and debugging. In the early days, most of the users were companies engaged in web business, but recently, there is a growing number of users embracing AWS for scientific and engineering research.

Figure 4 shows a list of the major services provided by AWS at the time of writing.

The various elements required to compose a cloud, such as computation, storage, database, network, and security, are provided as independent components. Essentially, a cloud system is created by combining these components. There are also pre-packaged services for specific applications, such as machine learning, speech recognition, and augmented reality (AR) and virtual reality (VR). In total, there are more than 170 services provided.

AWS beginners often fall into a situation where they are overwhelmed by the large number of services and left at a loss. It’s not even clear what concepts to learn and in what order, and this is undoubtedly a major barrier to entry. However, the truth is that the essential components of AWS are limited to just a couple. If you know how to use the essential components, you are almost ready to start developing on AWS. Many of the other services are combinations of the basic elements that AWS has prepared as specialized packages for specific applications. Recognizing this point is the first step in learning AWS.

Here, we list the essential components for building a cloud system on AWS. You will experience them while writing programs in the hands-on sessions in later chapters. At this point, it is enough if you could just memorize the names in a corner of your mind.

One of the most important concepts you need to know when using AWS is Region and Availability Zone (AZ) (Figure 5). In the following, we will briefly describe these concepts. For more detailed information, also see official documentation "Regions, Availability Zones, and Local Zones".

A region roughly means the location of a data center. At the time of writing, AWS has data centers in 25 geographical locations around the world, as shown in Figure 6. In Japan, there are data centers in Tokyo and Osaka. Each region has a unique ID, for example, Tokyo is defined as ap-northeast-1, Ohio as us-east-2, and so on.

When you log in to the AWS console, you can select a region from the menu bar at the top right of the screen (Figure 7, circled in red). AWS resources such as EC2 are completely independent for each region. Therefore, when deploying new resources or viewing deployed resources, you need to make sure that the console region is set correctly. If you are developing a web business, you will need to deploy the cloud in various parts of the world. However, if you are using it for personal research, you are most likely fine just using the nearest region (e.g. Tokyo).

An Avaialibity Zone (AZ) is a data center that is geographically isolated within a region. Each region has two or more AZs, so that if a fire or power failure occurs in one AZ, the other AZs can cover the failure. In addition, the AZs are connected to each other by high-speed dedicated network lines, so data transfer between AZs is extremely fast. AZ is a concept that should be taken into account when server downtime is unacceptable, such as in web businesses. For personal use, there is no need to be concerned much about it. It is sufficient to know the meaning of the term.

Now that you have a general understanding of the AWS cloud, the next topic will be an overview of how to develop and deploy a cloud system on AWS.

There are two ways to perform AWS operations such as adding, editing, and deleting resources: using the console and using the API.

As mentioned in the previous section, AWS APIs can be used to create and manage any resources in the cloud. Therefore, in principle, you can construct cloud systems by combining API commands.

However, there is one practical point that needs to be considered here. The AWS API can be broadly divided into commands to manipulate resources and commands to execute tasks (Figure 9).

Manipulating resources refers to preparing static resources, such as launching an EC2 instance, creating an S3 bucket, or adding a new table to a database. Such commands need to be executed only once, when the cloud is deployed.

Commands to execute tasks refer to operations such as submitting a job to an EC2 instance or writing data to an S3 bucket. It describes the computation that should be performed within the premise of a static resource such as EC2 instance or S3 bucket. Compared to the former, the latter can be regarded as being in charge of dynamic operations.

From this point of view, it would be clever to manage programs describing the infrastructure and programs executing tasks separately. Therefore, the development of a cloud can be divided into two steps: one is to create programs that describe the static resources of the cloud, and the other is to create programs that perform dynamic operations.

CloudFormation is a mechanism for managing static resources in AWS. CloudFormation defines the blueprint of the cloud infrastructure using text files that follow the CloudFormation syntax. CloudFormation can be used to describe resource requirements, such as how many EC2 instances to launch, with what CPU power and networks configuration, and what access permissions to grant. Once a CloudFormation file has been crafted, a cloud system can be deployed on AWS with a single command. In addition, by exchanging CloudFormation files, it is possible for others to easily reproduce an identical cloud system. This concept of describing and managing cloud infrastructure programmatically is called Infrastructure as Code (IaC).

CloudFormation usually use a format called JSON (JavaScript Object Notation). The following code is an example excerpt of a CloudFormation file written in JSON.

Here, we have defined an EC2 instance named "WebServer". This is a rather long and complex description, but it specifies all necessary information to create an EC2 instance.

The source code for the hands-on is available on GitHub at handson/ec2-get-started.

First, we set up the environment for the exercise. This is a prerequisite for the hands-on sessions in later chapters as well, so make sure to do it now without mistakes.

Using Docker image for the hands-on exercises

We provide a Docker image with the required programs installed, such as Python, Node.js, and AWS CDK. The source code of the hands-on program has also been included in the image. If you already know how to use Docker, then you can use this image to immediately start the hands-on tutorials without having to install anything else.

See Section 14.8 for more instructions.

SSH (secure shell) is a tool to securely access Unix-like remote servers. In this hands-on, we will use SSH to access a virtual server. For readers who are not familiar with SSH, here we give a brief guidance.

All SSH communication is encrypted, so confidential information can be sent and received securely over the Internet. For this hands-on, you need to have an SSH client installed on your local machine to access the remote server. SSH clients come standard on Linux and Mac. For Windows, it is recommended to install WSL to use an SSH client (see [environments]).

The basic usage of the SSH command is shown below. <host name> is the IP address or DNS hostname of the server to be accessed. The <user name> is the user name of the server to be connected to.

SSH can be authenticated using plain text passwords, but for stronger security, it is strongly recommended that you use Public Key Cryptography authentication, and EC2 only allows access in this way. We do not explain the theory of public key cryptography here. The important point in this hands-on is that the EC2 instance holds the public key, and the client computer (the reader’s local machine) holds the private key. Only the computer with the private key can access the EC2 instance. Conversely, if the private key is leaked, a third party will be able to access the server, so manage the private key with care to ensure that it is never leaked.

The SSH command allows you to specify the private key file to use for login with the -i or --identity_file option. For example, use the following command.

Figure 10 shows an overview of the application we will be deploying in this hands-on.

In this application, we first set up a private virtual network environment using VPC (Virtual Private Cloud). The virtual servers of EC2 (Elastic Compute Cloud) are placed inside the public subnet of the VPC. For security purposes, access to the EC2 instance is restricted by the Security Group (SG). We will use SSH to access the virtual server and perform a simple calculation. We use AWS CDK to construct this application.

Let’s take a look at the source code of the CDK app (handson/ec2-get-started/app.py).

Let us explain each of these points in more detail.

Now that we understand the source code, let’s deploy the application on AWS. Again, it is assumed that you have finished the preparations described inSection 4.1.

This is the end of the first part of the book. We hope you have been able to follow the contents without much trouble.

In Section 2, the definition of cloud and important terminology were explained, and then the reasons for using cloud were discussed. Then, in Section 3, AWS was introduced as a platform to learn about cloud computing, and the minimum knowledge and terminology required to use AWS were explained. In the hands-on session in Section 4, we used AWS CLI and AWS CDK to set up our own private server on AWS.

You can now experience how easy it is to start up and remove virtual servers (with just a few commands!). We mentioned in Section 2 that the most important aspect of the cloud is the ability to dynamically expand and shrink computational resources. We hope that the meaning of this phrase has become clearer through the hands-on experience. Using this simple tutorial as a template, you can customize the code for your own appplications, such as creating a virtual server to host your web pages, prepare an EC2 instance with a large number of cores to run scientific computations, and many more.

In the next chapter, you will experience solving more realistic problems based on the cloud technology you have learned. Stay tuned!

The third AI boom started around 2010, and consequently machine learning is attracting a lot of attention not only in academic research but also in social and business contexts. In particular, algorithms based on multi-layered neural networks, known as deep learning, have revolutionized image recognition and natural language processing by achieving remarkably higher performance than previous algorithms.

The core feature of deep learning is its large number of parameters. As the layers become deeper, the number of weight parameters connecting the neurons between the layers increases. For example, the latest language model, GPT-3, contains as many as 175 billion parameters. With such a vast number of parameters, deep learning can achieve high expressive power and generalization performance.

Not only GPT-3, but also recent neural networks that achieve SOTA (State-of-the-Art) performance frequently contain parameters in the order of millions or billions. Naturally, training such a huge neural network is computationally expensive. As a result, it is not uncommon to see cases where the training takes more than a full day with a single workstation. With the rapid development of deep learning, the key to maximize research and business productivity is how to optimize the neural network with high throughput. The cloud is a very effective means to solve such problems! As we have seen in Section 4, the cloud can be used to dynamically launch a large number of instances, and execute computations in parallel. In addition, there are specially designed chips (e.g. GPUs) optimized for deep learning operations to accelerate the computation. By using the cloud, you gain access to inexhaustible supply of such specialized computing chips. In fact, it was reported that the training of GPT-3 was performed using Microsoft’s cloud, although the details have not been disclosed.

Here we will briefly talk about Graphics Processing Unit or GPU, which serves as an indispensable technology for deep learning.

As the name suggests, a GPU is originally a dedicated computing chip for producing computer graphics. In contrast to a CPU (Central Processing Unit) which is capable of general computation, a GPU is designed specifically for graphics operations. It can be found in familiar game consoles such as XBox and PS5, as well as in high-end notebook and desktop computers. In computer graphics, millions of pixels arranged on a screen need to be updated at video rates (30 fps) or higher. To handle this task, a single GPU chip contain hundreds to thousands of cores, each with relatively small computing power (Figure 17), and processes the pixels on the screen in parallel to achieve real-time rendering.

Although GPUs were originally developed for the purpose of computer graphics, since around 2010, some advanced programmers and engineers started to use GPU’s high parallel computing power for calculations other than graphics, such as scientific computations. This idea is called General-purpose computing on GPU or GPGPU. Due to its chip design, GPGPU is suitable for simple and regular operations such as matrix operations, and can achieve much higher speed than CPUs. Currently, GPGPU is employed in many fields such as molecular dynamics, weather simulation, and machine learning.

The operation that occurs most frequently in deep learning is the convolution operation, which transfers the output of neurons to the neurons in the next layer (Figure 18). Convolution is exactly the kind of operations that GPUs are good at, and by using GPUs instead of CPUs, learning can be dramatically accelerated, up to several hundred times.

Thus, GPUs are indispensable for machine learning calculations. However, they are quite expensive. For example, NVIDIA’s Tesla V100 chip, designed specifically for scientific computing and machine learning, is priced at about one million yen (ten thousand dollars). One million yen is quite a large investment just to start a machine learning project. The good news is, if you use the cloud, you can use GPUs with zero initial cost!

To use GPUs in AWS, you need to select an EC2 instance type equipped with GPUs, such as P2, P3, G3, and G4 instance family. Table 3 lists representative GPU-equipped instance types as of this writing.

As you can see from Table 3, the price of GPU instances is higher than the CPU-only instances. Also note that older generation GPUs (K80 compared to V100) are offered at a lower price. The number of GPUs per instance can be selected from one to a maximum of eight.

The cheapest GPU instance type is g4dn.xlarge, which is equipped with a low-cost and energy-efficient NVIDIA T4 chip. In the hands-on session in the later chapters, we will use this instance to perform deep learning calculations.

In the second hands-on session, we will launch an EC2 instance equipped with a GPU and practice training and inference of a deep learning model.

The source code for the hands-on is available on GitHub at handson/mnist.

To run this hands-on, it is assumed that the preparations described in the first hands-on (Section 4.1) have been completed. There are no other preparations required.

Figure 19 shows an overview of the application we will be deploying in this hands-on.

You will notice that many parts of the figure are the same as the application we created in the first hands-on session (Figure 10). With a few changes, we can easily build an environment to run deep learning! The three main changes are as follows.

Let’s have a look at the source code (handson/mnist/app.py). The code is almost the same as in the first hands-on. We will explain only the parts where changes were made.

Now that we understand the application source code, let’s deploy it.

The deployment procedure is almost the same as the first hands-on. Here, only the commands are listed (lines starting with # are comments). If you have forgotten the meaning of each command, review the first hands-on. You should not forget to set the access key (Section 14.3).

If the deployment is executed successfully, you should get an output like Figure 21. Note the IP address of your instance (the string following InstancePublicIp).

Let’s log in to the deployed instance using SSH. To connect to Jupyter Notebook, which we will be using later, we must log in with the port forwarding option (-L).

Port forwarding means that the connection to a specific address on the client machine is forwarded to a specific address on the remote machine via SSH encrypted communication. The option -L localhost:8931:localhost:8888 means to forward the access to localhost:8931 of your local machine to the address of localhost:8888 of the remote server (The number following : specifies the TCP/IP port number). On port 8888 of the remote server, Jupyter Notebook (described below) is running. Therefore, you can access Jupyter Notebook on the remote server by accessing localhost:8931 on the local machine (Figure 22). This type of SSH connection is called a tunnel connection.

After logging in via SSH, let’s check the status of the GPU. Run the following command.

You should get output like Figure 23. The output shows that one Tesla T4 GPU is installed. Other information such as the GPU driver, CUDA version, GPU load, and memory usage can be checked.

Jupyter Notebook is a tool for writing and running Python programs interactively. Jupyter is accessed via a web browser, and can display plots and table data beautifully as if you were writing a notebook (Figure 24). If you are familiar with Python, you have probably used it at least once.

In this hands-on session, we will run a deep learning program interactively using Jupyter Notebook. Jupyter is already installed on DLAMI, so you can start using it without any configuration.

Now, let’s start Jupyter Notebook server. On the EC2 instance where you logged in via SSH, run the following command.

When you run this command, you will see output like Figure 25. From this output, we can see that the Jupyter server is launched at the address localhost:8888 of the EC2 instance. The string ?token=XXXX following localhost:8888 is a temporary token used for accessing Jupyter.

Since the port forwarding option was added to the SSH connection, you can access localhost:8888, where Jupyter is running, from localhost:8931 on your local machine. Therefore, to access Jupyter from the local machine, you can access the following address from a web browser (Chrome, FireFox, etc.).

Remember to replace ?token=XXXX with the actual token that was issued when Jupyter server was started above.

If you access the above address, the Jupyter home screen should be loaded (Figure 26). Now, Jupyter is ready!

PyTorch is an open source deep learning library that is being developed by the Facebook AI Research LAB (FAIR). PyTorch is one of the most popular deep learning libraries at the time of writing, and is being used by Tesla in their self-driving project, to name a few. In this hands-on session, we will use PyTorch to practice deep learning.

Before we move on to some serious deep learning calculations, let’s use the PyTorch library to get a feel for what it is like to run computations on the GPU.

First, we’ll create a new notebook. Click "New" in the upper right corner of the Jupyter home screen, select the environment "conda_pytorch_p36", and create a new notebook (Figure 27). In the "conda_pytorch_p36" virtual environment, PyTorch is already installed.

Here, we will write and execute the following program (Figure 28).

First, we import PyTorch. In addition, we check that the GPU is available.

Output:

Next, let’s create a random 3x3 matrix x on CPU.

Output:

Next, we create another matrix y on GPU. We also move the matrix x on GPU.

Then, we perform the addition of the matrix x and y on GPU.

Output:

Lastly, we bring the matrix on GPU back on CPU.

Output:

The above examples are just the rudiments of GPU-based computation, but we hope you get the idea. The key is to explicitly exchange data between the CPU and GPU. This example demonstrated an operation on 3x3 matrix, so the benefit of using GPU is almost negligible. However, when the size of the matrix is in the thousands or tens of thousands, the GPU becomes much more powerful.

Let’s benchmark the speed of the GPU and the CPU and compare the performance. We will use Jupyter’s %time magic command to measure the execution time.

First, using the CPU, let’s measure the speed of computing the matrix product of a 10000x10000 matrix. Continuing from the notebook we were just workin with, paste the following code and run it.

The output should look something like shown below. This means that it took 5.8 seconds to compute the matrix product (note that the measured time varies with each run).

Next, let’s measure the speed of the same operation performed on the GPU.

The output should look something like shown below. This time, the computation was completed in 553 milliseconds!

From this benchmark, we were able to observe about 10 times speedup by using the GPU. The speed-up performance depends on the type of operation and the size of the matrix. The matrix product is one of the operations where the speedup is expected to be highest.

Now that we have covered the concepts and prerequisites for deep learning computations on AWS, it’s time to run a real deep learning application.

In this section, we will deal with one of the most elementary and famous machine learning tasks, handwritten digit recognition using the MNIST dataset (Figure 29). This is a simple task where we are given images of handwritten numbers from 0 to 9 and try to guess what the numbers are.

Here, we will use Convolutional Neural Network (CNN) to solve the MNIST task. The source code is available on GitHub at /handson/minist/pytorch/. The relevant files are mnist.ipynb and simple_mnist.py in this directory. This program is based on PyTorch’s official example project collection, with some modifications.

First, let’s upload simple_mnist.py, which contains custom classes and functions (Figure 30). Go to the home of the Jupyter, click on the "Upload" button in the upper right corner of the screen, and select the file to upload. Inside this Python program, we defined the CNN model and the parameter optimization method. We won’t explain the contents of the program, but readers interested in the subject can read the source code and learn for themselves.

Once you have uploaded simple_mnist.py, you can create a new notebook. Be sure to select the "conda_pytorch_p36" environment.

Once the new notebook is up and running, let’s import the necessary libraries first.

The torchvision package contains some useful functions, such as loading MNIST datasets. The above code also imports custom classes and functions (Model, train, evaluate) from simple_mnist.py that we will use later.

Next, we download the MNIST dataset. At the same time, we are normalizing the intensity of the images.

The MNIST dataset consists of 28x28 pixel monochrome square images and corresponding labels (numbers 0-9). Let’s extract some of the data and visualize them. You should get an output like Figure 31.

Next, we define the CNN model.

The Model class is defined in simple_mnist.py. We will use a network with two convolutional layers and two fully connected layers, as shown in Figure 32. The output layer is the Softmax function, and the loss function is the negative log likelihood function (NLL).

Next, we define an optimization algorithm to update the parameters of the CNN. We use the Stochastic Gradient Descent (SGD) method.

Now, we are ready to go. Let’s start the CNN training loop!

In this example, we are training for 5 epochs. Using a GPU, computation like this can be completed in about a minute.

The output should be a plot similar to Figure 33. You can see that the value of the loss function is decreasing (i.e. the accuracy is improving) as the iteration proceeds.

Let’s visualize the inference results of the learned CNN. By running the following code, you should get an output like Figure 34. If you closely look at this figure, the second one from the right in the bottom row looks almost like a "1", but it is correctly inferred as a "9". It looks like we have managed to create a pretty smart CNN!

Finally, we save the parameters of the trained neural network as a file named mnist_cnn.pt. This way, you can reproduce the learned model and use it for another experiment anytime in the future.

That’s it! We have experienced all the steps to set up a virtual server in the AWS cloud and perform the deep learning computation. Using the GPU instance in the cloud, we were able to train a neural network to solve the MNIST digit recognition task. Interested readers can use this hands-on as a template to run their own deep learning applications.

Now we are done with the GPU instance. Before the EC2 cost builds up, we should delete the instance we no longer use.

As in the first hands-on session, we can delete the instance using the AWS CloudFormation console, or using the AWS CLI (see Section 4.4.8).

We have been repeatedly calling for "large-scale computing systems," but what exactly does that mean? Let’s take machine learning as an example, and talk about a computer system for processing large data.

Suppose we want to train a deep learning model with a very large number of parameters, such as GPT-3 introduced in Section 5. If you want to perform such a computation, a single server will not have enough computing power. Therefore, the typical design of a computing system would be a model shown in Figure 35. Namely, a large amount of training data is distributed in small chunks across multiple machines, and the parameters of the neural network are optimized in parallel.

Or, let’s say you want to apply a trained model to a large amount of data for analysis. For example, you have a SNS platform and you are given a large number of images, and you want to label what is in each photo. In such a case, an architecture such as the one shown in Figure 36 can be considered, in which a large amount of data is divided among multiple machines, and each machine performs inference computation.

How can such applications that run multiple computers simultaneously be implemented in the cloud?

One important point is that the multiple machines running Figure 35 and Figure 36 have basically the same OS and computing environment. Here, it is possible to perform the same installation operations on each machine as one would do on an individual computer, but this would be very time-consuming and cumbersome to maintain. In other words, in order to build a large-scale computing system, it is necessary to have a mechanism that allows to easily replicate the computing environment.

To achieve this goal, a software called Docker is used.

Docker is software for running a separate computing environment independent of the host OS in a virtual environment called a container. Docker makes it possible to package all programs, including the OS, in a compact package (a packaged computing environment is called an image). Docker makes it possible to instantly replicate a computing environment on a cloud server, and to create a system for running multiple computers simultaneously, as seen in Figure 36.

Docker was developed by Solomon Hykes and his fellows in 2013, and since then it has exploded in popularity, becoming core software not only for cloud computing but also in the context of machine learning and scientific computing. Docker is available free of charge, except for enterprise products, and its core is available as an open source project. Docker is available for Linux, Windows, and Mac operating systems. Conceptually, Docker is very similar to a virtual machine (VM). Comparing Docker with VM is a very useful way to understand what Docker is, so here we take this approach.

A virtual machine (VM) is a technology that allows to run virtualized operating systems on top of a host machine (Figure 38). A VM has a layer called a hypervisor. The hypervisor first divides the physical computing resources (CPU, RAM, network, etc.) and virtualizes them. For example, if the host machine has four physical CPU cores, the hypervisor can virtually divide them into (2,2) pairs. The OS running on the VM is allocated virtualized hardware by the hypervisor. OSes running on VM are completely independent. For example, OS-A cannot access the CPU or memory space allocated to OS-B (this is called isolation). Famous software for creating VMs includes VMware, VirtualBox, and Xen. EC2, which we have used earlier, basically uses VM technology to present the user with a virtual machine with the desired specifications.

Docker, like VM, is a technology for running a virtualized OS on a host OS. In contrast to VMs, Docker does not rely on hardware-level virtualization; all virtualization is done at the software level (Figure 38). The virtual OS running on Docker relies on the host OS for much of its functionality, and as a result is very compact. Consequently, the time required to boot a virtual OS with Docker is much faster than with a VM. It is also important to note that the size of the packaged environment (i.e., image) is much smaller than that of a full OS, which greatly speeds up communication over the network. In addition, some implementations of VMs are known to have lower performance than metal (metal means OS running directly on physical hardware) due to the overhead at the hypervisor layer. Docker is designed to be able to achieve almost the same performance as metal.

There are many other differences between Docker and VM, but we will not go into details here. The important point is that Docker is a tool for creating a very compact and high-performance virtual computing environment. Because of its ease of use and lightness, Docker has been adopted in many cloud systems since its introduction in 2013, and it has become an essential core technology in the modern cloud.

The most effective way to understand what Docker is is to actually try it out. In this section, I will give a brief tutorial on Docker.

For Docker installation, please refer to Section 14.6 and official documentation. The following assumes that you have already installed Docker.

As we have explained so far, Docker is a highly versatile and powerful tool to replicate and launch a virtual computing environment. As the last topic of this section, we will talk about how to build a computing system using Docker on AWS.

Elastic Container Service (ECS) is a tool for creating Docker-based compute clusters on AWS (Figure 41). Using ECS, you can define tasks using Docker images, create a compute cluster, and add or remove instances in the compute cluster.

Figure 42 shows an overview of ECS. The ECS accepts computation jobs managed in units called tasks. When a task is submitted to the system, ECS first downloads the Docker image specified by the task from an external registry. The external registry can be Docker Hub or AWS' own image registry, ECR (Elastic Container Registry).

The next important role of ECS is task placement. By selecting a virtual instance with low computational load in a predefined cluster, ECS places a Docker image on it, and the task is started. When we say "select a virtual instance with low computational load," the specific strategy and policy for this selection depends on the parameters specified by the user.

Scaling of clusters is another important role of ECS. Scaling refers to the operation of monitoring the computational load of the instances in a cluster, and starting and stopping the instances according to the total load on the cluster. When the computational load of the entire cluster exceeds a specified threshold (e.g., 80% utilization), a new virtual instance is launched (an operation called scale-out). When the computational load is below a certain threshold, unnecessary instances are stoped (an operation called scale-in). The scaling of a cluster is achieved by the ECS cooperating with other AWS services. Specifically, ECS is most commonly paired with Auto scaling group (ASG) or Fargate. ASG and Fargate, respectively will be covered in Section 9 and Section 8.

ECS automatically manages the above explained operations for you. Once the parameters for cluster scaling and task placement are specified, the user can submit a large number of tasks, almost without thinking about behind the scenes. ECS will launch just enough instances for the amount of tasks, and after the tasks are completed, all unnecessary instances will be stopped, eliminating idling instances completely.

The theory and knowledge stuffs are over now! From the next section, let’s start building a large-scale parallel computing system using Docker and ECS!

Before getting into the hands-on exercuse, we need to learn about Fargate(Figure 43).

Let’s look again at Figure 42, which gives an overview of ECS. This figure shows a cluster under the control of ECS, and there are two choices which carry out the computation in the cluster: either EC2 or Fargate. In the case of using EC2, the instance is launched in the same way as described in the previous sections (Section 4, Section 6). However, the technical difficulty of creating and managing a compute cluster using EC2 is rather high, so we will explain it in the next section (Section 9).

Fargate is a mechanism for running container-based computational tasks, designed specifically for use in ECS. In terms of running computation, its role is similar to that of EC2, but Fargate does not have a physical entity like an EC2 instance. It means that, for example, logging in via SSH is basically not expected in Fargate, and there is no operations like "installing software". In Fargate, all computation is executed via Docker containers. Namely, to use Fargate, the user first prepares the Docker image, and then Fargate executes the computational task by using the docker run command. When Fargate is specified as an ECS cluster, operations such as scaling can be built with a simple configuration and program.

Similar to EC2, Fargate allows you to specify the size of the CPU and memory as needed. At the time of writing, you can choose between 0.25 and 4 cores for vCPU power, and 0.5 and 30 GB for RAM (for details, see Official Documentation "Amazon ECS on AWS Fargate"]). Despite the ease of scaling clusters, Fargate does not allow for a large vCPU counts or RAM capacity, nor does it allow for the use of GPUs. as in EC2 instances.

So that was an overview of Fargate, but it may not be easy to understand it all in words. From here on, let us learn how to work with ECS and Fargate by writing a real program to deploy parallel computing system.

The source code of the hands-on is available on GitHub at handson/qa-bot.

To run this hands-on, it is assumed that the preparations described in the first hands-on (Section 4.1) have been completed. It is also assumed that Docker is already installed on your local machine.

Let’s define more concretely the automatic question answering system that we will develop in this hands-on session. Assume that we are given the following context and question.

The automatic answering system we are going to create will be able to find the correct answer to such a question, given the context. To make the problem a bit easier, the answer is selected from the string contained in the context. For example, for the above question, the system should return the following answer.

While it is trivial for humans to understand such sentences, it is easy to imagine how difficult it would be for a computer to solve them. However, recent progress in natural language processing using deep learning has made remarkable progress, and it is possible to create models that can solve this problem with an extremely high accuracy.

In this hands-on, we will use the pre-trained language model provided by huggingface/transformers. This model is supported by a natural language processing model called Transformer We packaged this model in a Docker image, and the image is available at the author’s Docker Hub repository. Before we start designing the cloud system, let’s test this Docker image on the local machine.

Use the following command to download (pull) the Docker image to your local machine.

Now, let’s submit a question to this Docker image. First, define the context and question as command line variables.

Then, use the following command to run the container.

The Docker image we prepared accepts the context as the first argument and the question as the second argument. The third and fourth arguments are for implementation purposes when deploying to the cloud, so don’t worry about them for now.

When you execute this command, you should get the following output.

"score" is a number that indicates the confidence level of the answer, in the range [0,1]. "start" and "end" indicate the starting and ending position in the context where the answer is, and "answer" is the string predicted as the answer. Notice that the correct answer, "1921", was returned.

Let us ask a more difficult question.

Output:

This time, the score is 0.52, indicating that the bot is a little unsure of the answer, but it still got the right answer.

As you can see, by using a language model supported by deep learning, we have been able to create a Q&A bot that can be useful in practical applications. In the following sections, we will design a system that can automatically respond to a large number of questions by deploying this program in the cloud.

Figure 45 shows an overview of the application we are creating in this hands-on.

The summary of the system design is as follows:

Now let us take a look at the main application code (handson/qa-bot/app.py).

Now that we understand the application source code, let’s deploy it.

The deployment procedure is almost the same as the previous hands-on. Here, only the commands are listed (lines starting with # are comments). If you have forgotten the meaning of each command, review the first hands-on. You should not forget to set the access key (Section 14.3).

If the deployment is successful, you should see an output like Figure 46.

Let’s log in to the AWS console and check the contents of the deployed stack. From the console, go to the ECS page, and you should see a screen like Figure 47. Find the cluster named EcsClusterQaBot-XXXX.

Cluster is a unit that binds multiple virtual instances together, as explained earlier. In the Figure 47, check that under the word FARGATE it says 0 Running tasks and 0 Pending tasks. At this point, no tasks were submitted, so the numbers are all zero.

Next, find the item Task Definitions in the menu bar on the left of this screen, and click on it. On the destination page, find the item EcsClusterQaBotEcsClusterQaBotTaskDefXXXX and open it. Scroll down the page, and you will find the information shown in Figure 48. You can check the amount of CPU and memory used, as well as the settings related to the execution of the Docker container.

Now, let’s submit a question to the cloud!

Submitting a task to ECS is rather complicated, so I prepared a program (run_task.py) to simplify the task submission. (handson/qa-bot/run_task.py).

With the following command, you can submit a new question to the ECS cluster.

Following "ask" parameter, we supply context and questsions, in this order, as the arguments.

When you run this command, you will see the output "Waiting for the task to finish…​", and you will have to wait for a while to get an answer. During this time, ECS accepts the task, launches a new Fargate instance, and places the Docker image on the instance. Let’s monitor this sequence of events from the AWS console.

Go back to the ECS console screen, and click on the name of the cluster (EcsClusterQaBot-XXXX). Next, open the tab named "Tasks" (Figure 49). You will see a list of running tasks.

As you can see in Figure 49, the "Last status = Pending" indicates that the task is being prepared for execution at this point. It takes about 1-2 minutes to launch the Fargate instance and deploy the Docker image.

After waiting for a while, the status will change to "RUNNING" and the computation will start. When the computation is finished, the status changes to "STOPPED" and the Fargate instance is automatically shut down by ECS.

From the Figure 49 screen, click on the task ID in the "Task" column to open the task detail screen (Figure 50). The task information such as "Last status" and "Platform version" is displayed. You can also view the execution log of the container by opening the "Logs" tab.

Now, coming back to the command line where you ran run_task.py, you should see an output like Figure 51. The correct answer, "Momotaro", has been returned!

The application we have designed here can handle many questions at the same time by using ECS and Fargate. Now, let’s submit many questions at once, and observe the behavior of ECS cluster. By adding the option ask_many to run_task.py, you can send multiple questions at once. The questions are defined in handson/qa-bot/problems.json.

Run the following command.

After executing this command, go to the ECS console and look at the list of tasks (Figure 52). You can see that multiple Fargate instances have been launched and tasks are being executed in parallel.

Make sure that the status of all tasks is "STOPPED", and then get the answer to the question. To do so, execute the following command.

As a result, you will get an output like Figure 53. You can see that the bot was able to answer complex text questions with a suprisingly high accuracy.

Congratulations! You have managed to create a system that can automatically generate answers to questions using deep learning language models! Importantly, it is a highly scalable system that can handle hundreds of questions simultaneously. We didn’t prepare a GUI (Graphical User Interface) this time, but if we add a simple GUI to this system, it could be operated as a very nice web service. We didn’t add GUI to this cloud system, but with such a tweaking, this system is already useful enough for various purposes.

This concludes the third hands-on session. Finally, we must delete the stack.

To delete the stack, login to the AWS console and click the DELETE button on the CloudFormation screen. Alternatively, you can execute the following command from the command line.

Before we get into the hands-on, you need to be familiar with the technique of EC2, called Auto scaling groups (ASG).

Please take a look back at Figure 42, which gives an overview of ECS. As explained in the previous chapter (Section 8), EC2 and Fargate can be selected as the computational resource in ECS clusters. Fargate was described in the previous chapter. Using Fargate, we were able to build a highly scalable computing environment with a simple setup. However, there were some limitations, such as not being able to use GPUs. By defining a computing environment that is based on EC2, although the programming complexity increases, we can build clusters with GPUs and other more advanced and complex configurations.

A service called ASG is deployed in an EC2 cluster. An ASG constitutes a cluster by grouping multiple EC2 instances into logical units. ASGs are responsible for scaling, such as launching new instances in the cluster or stopping instances that are no longer needed. An important concept in ASG is the parameters callled desired capacity, minimum capacity, and maximum capacity. The minimum capacity and maximum capacity are parameters that specify the minimum and maximum number of instances that can be placed in a cluster, respectively. The former keeps the instances idle even when the cluster is not under load, so it can act as a buffer when the load suddenly increases. The latter prevents an excessive number of instances from being launched when the load increases unexpectedly, and serves to set an upper limit on the economic cost.

The desired capacity specifies the number of instances required by the system at a given time. The desired capacity can be set based on a fixed schedule, such as increasing or decreasing the number of instances according to a 24-hour rhythm (e.g., more during the day and less at night). Alternatively, the desired capacity can be dynamically controlled according to the load on the entire cluster. The rules that define the criteria for scaling the cluster are called scaling policies. For example, we can assume a scaling policy which maintains the utilization (load) of the entire cluster at 80% at all times. In this case, the ASG automatically removes instances from the cluster when the load of the entire cluster falls below 80%, and adds instances when the load exceeds 80%s.

After considering the above parameters, the user creates an ASG. Once ASG is created, one needs to write a program to link ASG with the ECS, which defines EC2-based ECS cluster.

As explained earlier, it is possible to construct a desired computation cluster by combining ECS and ASG. However, ECS and ASG require complicated settings, which makes programming quite tedious for both beginners and experienced users. To solve this problem, there is a service that automates the design of clusters using ECS and ASG. That service is AWS Batch.

AWS Batch, as the name implies, is designed for batch jobs (i.e., independent operations with different input data that are executed repeatedly). Many scientific calculations and machine learning can be considered as batch calculations. For example, you can run multiple simulations with different initial parameters. The advantage of using AWS Batch is that the scaling of the cluster and the allocation of jobs are all done automatically, giving the users a system where they can submit a large number of jobs without worrying about the implementation details of the cloud. However, it is important to know that the ECS/ASG/EC2 triad is working in concert behind the scenes.

In AWS Batch, the following concept is defined to facilitate job submission and management (Figure 55). First, a job is a unit of computation executed by AWS Batch. Job definitions define the specification of a job, including the address of the Docker image to be executed, the amount of CPU and RAM to be allocated, and environment variables. Each job is executed based on the job definition. When a job is executed, it is placed in job queues. Job queue is a queue of jobs waiting to be executed, and the first job in the queue is executed first. In addition, multiple queues can be arranged, and each queue can be assigned a priority value, so that jobs in the queue with the highest priority are executed first. Compute environment is a concept that is almost synonymous with the cluster, and refers to the location where computations are executed (i.e. group of EC2 or Fargate instances). In the compute environment, one needs to specify the EC2 instance types to use, and a simple scaling policy, such as the upper and lower limit on the number of instances. Job queues monitor the availability of the compute environment and place jobs to the compute environment according to the availability.

These are the concepts that you need to understand when using AWS Batch. To make a better sense of these concepts, let us actually construct an application using AWS Batch.

The hands-on source code is available on GitHub at handson/aws-batch.

To run this hands-on, it is assumed that the preparations described in the first hands-on (Section 4.1) have been completed. It is also assumed that Docker is already installed on your local machine.

At the beginning of this hands-on, we mentioned that we would be covering hyperparameter tuning in machine learning. As the simplest example, let’s take the MNIST digit recognition problem again, which was covered in Section 6.7. In Section 6.7, we trained the model using arbitrarily chosen hyperparameters. The hyperparameters used in the program include learning rate and momentum in stochastic gradient descent (SGD) algorithm. In the code, the following lines correspond to them.

The learning rate (lr=0.01) and momentum (momentum=0.5) used here are arbitrarily chosen values, and we do not know if these are the best values. This choice may happen to be the best, or there may be other hyperparameter pairs that give higher accuracy. To answer this question, let’s perform a hyperparameter search. In this article, we will take the simplest approach: hyperparameter search by grid search.

First, let’s run the Docker image used in this hands-on session locally.

The source code of the Docker image can be found on GitHub at handson/aws-batch/docker. It is based on the program we introduced in Section 6.7, with some minor changes made for this handson. Interested readers are encouraged to read the source code as well.

As an exercise, let’s start by building this Docker image on your local machine. Go to the directory where the Dockerfile is stored, and build the image with the tag mymnist.

When the image is ready, start the container with the following command and run MNIST training.

This command will start optimizing the neural network using the specified hyperparameters (learning rate given by --lr and momentum given by --momentum). The maximum number of epochs to train is specified by --epochs parameter. You will see decrease of loss values on the command line, just as we saw in Section 6 (Figure 56).

If you use the above command, the computation will be performed using the CPU. If your local computer is equipped with a GPU and you have configured nvidia-docker, you can use the following command to run the computation using the GPU.

In this command, the parameter --gpus all has been added.

You can see that the loss of the training data monotonically decreases as the number of epochs increases, regardless of whether it is run on CPU or GPU. On the other hand, you will notice that loss and accuracy of the validation data do not improve further after decreasing to a certain level. The actual plot of this behaviour should look like Figure 57.

This is a phenomenon called overfitting, which indicates that the neural network is over-fitted to the training data and the accuracy (generalization performance) for data outside the training data is not improved. To deal with such cases, a technique called early stopping is known. In early stopping, we track the loss of the validation data, and stop learning at the epoch when it turns from decreasing to increasing. Then we adopt the weight parameters at that epoch. In this hands-on session, we will use early stopping technique to determine the end of training and evaluate the performance of the model.

Figure 58 shows an overview of the application we are creating in this hands-on.

The summary of the system design is as follows:

Let us take a look at the application source code (handson/aws-batch/app.py).

Now that we understand the application source code, let’s deploy it.

The deployment procedure is almost the same as the previous hands-on. Here, only the commands are listed (lines starting with # are comments). If you have forgotten the meaning of each command, review the first hands-on. You should not forget to set the access key (Section 14.3).

After confirming that the deployment has been done successfully, let’s log in to the AWS console and check the deployed stack. Type batch in the search bar to open the AWS Batch management console (Figure 60).

The first thing you should look at is the item named SimpleBatchcompute-env in the "compute environment overview" at the bottom of the screen. Compute environment is the environment (or cluster) in which computations will be executed, as described earlier. As specified in the program, g4dn.xlarge is shown as the instance type to be used. You can also see that Minimum vCPUs is set to 0 and Maximum vCPUs is set to 64. In addition, Desired vCPUs is set to 0 because no job is running at this time. If you want to see more detailed information about the compute environment, click on the name to open the detail screen.

Next, pay attention to the item SimpleBatch-queue in the "job queue overview". Here, you can see a list of jobs waiting for execution, jobs in progress, and jobs that have completed execution. You can see that there are columns such as PENDING, RUNNING, SUCCEEDED, FAILED and so on. As the job progresses, the state of the job transitions according to these columns. We’ll come back to this later when we actually submit the job.

Finally, let’s check the job definition. Select Job definitions from the menu on the left side of the screen, and find and open the SimpleBatchjob-definition on the next screen. From here, you can see the details of the job definition (Figure 61). Among the most important information, vCPUs, Memory, and CPU define the amount of vCPU, memory, and GPU allocated to Docker, respectively. In addition, Image specifies the Docker image to be used for the job. Here, it refers to the ECR repository. Currently, this ECR is empty. The next step is to deploy the image to this ECR.

In order for Batch to execute a job, it needs to download (pull) a Docker image from a specified location. In the previous hands-on (Section 8), we pulled the image from Docker Hub, which is set to public. In this hands-on, we will adopt the design of deploying images in ECR (Elastic Container Registry), a image registry provided by AWS. The advantage of using ECR is that you can prepare a private space for images that only you can access. Batch executes its tasks by pulling images from the ECR (Figure 58).

In the source code, the following part defines the ECR.

After the first deployment, the ECR is empty. You need to push the Docker image that you use for your application to ECR.

To do so, first open the ECR screen from the AWS console (type Elastic Container Registry in the search bar). Select the Private tab and you will find a repository named simplebatch-repositoryXXXX (Figure 62).

Next, click on the name of the repository to go to the repository details page. Then, click the View push commands button on the upper right corner of the screen. This will bring up a pop-up window like Figure 63.

You can push your Docker image to ECR by executing the four commands shown in this pop-up window in order. Before pushing, make sure your AWS credentials are set. Then, navigate to the directory named docker/ in the hands-on source code. Then, execute the commands displayed in the pop-up window in order from the top.

The fourth command may take a few minutes as it uploads several gigabytes of images to ECR, but when it completes, the image has been successfully placed in ECR. If you look at the ECR console again, you can see that the image has indeed been placed (Figure 64). This completes the final preparations for executing a job using AWS Batch.

Now, we demonstrate how to submit a job to AWS Batch.

In the notebook/ directory of the hands-on source code, there is a file named run_single.ipynb (.ipynb is the file extension of Jupyter notebook). We will open this file from Jupyter notebook.

In this hands-on, Jupyter Notebook server is already installed in the virtual environment by venv. We can launch Jupyter Notebook server from the local machine by the following command.

After Jupyter Notebook server is started, open run_single.ipynb.

The first cell [1], [2], [3] defines a function to submit a job to AWS Batch (submit_job()).

Let us briefly explain the submit_job() function. In Section 9.4, when we ran the MNIST Docker container locally, we used the following command.

Here, --lr 0.1 --momentum 0.5 --epochs 10 is the argument passed to the container.

When you run a job with AWS Batch, you can also specify the command to be passed to the container by using the argument ContainerOverrides within commands parameter. The following part of the code corresponds to this.

Next, let’s move to cell [4]. Here, we submit a job with learning rate = 0.01, momentum = 0.1, and epochs = 100 using the submit_job() function.

After executing the cell [4], let’s check whether the job is actually submitted from the AWS console. If you open the AWS Batch management console, you will see a screen like Figure 65.

Pay attention to the part circled in red in Figure 65. When a job is submitted, it goes through the state of SUBMITTED and then to the state of RUNNABLE. RUNNABLE corresponds to the state of waiting for a new instance to be launched because there is not available instances in the compute environment to run the job. When the instance is ready, the status of the job goes through STARTING to RUNNING.

Next, let’s look at the Desired vCPU of the compute environment when the status of the job is RUNNING (the part circled in purple in Figure 65). The number 4 is the number of vCPU for one instance of g4dn.xlarge. You can see that the minimum number of EC2 instances required to run the job has been launched in response to the job submission. (If you are interested, you can also take a look at the EC2 console at the same time).

After a while, the status of the job will change from RUNNING to SUCCEEDED (or FAILED if an error occurs for some reason). The training of MNIST used in this hands-on should take about 10 minutes. Let’s wait until the job status becomes SUCCEEDED.

When the job completes, the training results (a CSV file containing the loss and accuracy for each epoch) will be saved in S3. You can check this from the AWS console.

If you go to the S3 console, you should find a bucket named simplebatch-bucketXXXX (the XXXX part depends on the user). If you click on it and look at the contents, you will find a CSV file named metrics_lr0.0100_m0.1000.csv (Figure 66). This is the result of training with learning rate = 0.01 and momentum = 0.1.

Now, let’s come back to run_single.ipynb. In cells [5] through [7], we are downloading the CSV file of the training results.

In [6], we define a function to download CSV data from S3 and load it as a pandas DataFrame object. Note that when you run [7], you should replace the value of the bucket_name variable with the name of your own bucket. (This is the simplebatch-bucketXXXX that we just checked from the S3 console).

Next, in cell [9], we plot the CSV data (Figure 67). We have successfully trained the MNIST model using AWS Batch, just as we did when we ran it locally!

Now, here comes the final part. Let’s use the AWS Batch system that we have built to perform real hyperparameter search.

Open the file run_sweep.ipynb in the same directory as run_single.ipynb that we just ran.

Cells [1], [2] and [3] are identical to run_single.ipynb.

A for loop in cell [4] is used to prepare a grid of hyperparameter combinations and submit the jobs to the batch. In this case, 3x3=9 jobs are created.

After executing the cell [4], open the Batch console. As before, you will see that the status of the jobs changes from SUBMITTED > RUNNABLE > STARTING > RUNNING. Finally, make sure that all 9 jobs are in the RUNNING state (Figure 68). Also, make sure that the Desired vCPUs of the compute environment is 4x9=36 (Figure 68).

Next, let’s click Jobs from the left menu of the Batch console. Here, you can see the list of running jobs (Figure 69). It is also possible to filter jobs by their status. You can see that all 9 jobs are in the RUNNING status.

Now let’s take a look at the EC2 console. Select Instances from the menu on the left, and you will see a list of running instances as shown in Figure 70. You can see that 9 instances of g4dn.xlarge are running. Batch has launched the necessary number of instances according to the job submission!

Once you have confirmed this, wait for a while until all jobs are finished (it takes about 10-15 minutes). When all the jobs are finished, you should see the number of SUCCEEDED jobs on the dashboard is 9. Also, make sure that the Desired vCPUs in the Compute environment has dropped to 0. Finally, go to the EC2 console and check that all GPU instances are stopped.

In summary, by using AWS Batch, we were able to observe a sequence of events in which EC2 instances are automatically launched in response to job submissions, and the instances are immediately stopped upon completion of the job. Since it takes about 10 minutes to complete a single job, it would take 90 minutes if 9 hyperparameter pairs were calculated sequentially. By using AWS Batch to run these computations in parallel, we were able to complete all the computations in 10 minutes!

Let’s come back to run_sweep.ipynb. In the cells after [5], the results of grid search are visualized.

The resulting plot is Figure 71.

From this plot, we can see that the accuracy is maximum when the learning rate is 0.1, although the difference is small. It can also be seen that when the learning rate is 0.1, there is no significant performance gains between different momentum values.

In this hands-on session, we experienced the steps to optimize the hyperparameters of the MNIST classification model. By using AWS Batch, we were able to build a system that can dynamically control EC2 clusters and process jobs in parallel. If you can master EC2 to this level, you will be able to solve many problems on our own!

This concludes the hands-on session. Finally, let’s delete the stack. In order to delete the stack for this hands-on, the Docker images placed in the ECR must be deleted manually. If you don’t do this, you’ll get an error when you run cdk destroy. This is a CloudFormation specification that you have to follow.

To delete a Docker image in ECR, go to the ECR console and open the repository where the image is located. Then, click the DELETE button on the upper right corner of the screen to delete it (Figure 72).

Alternatively, to perform the same operation from the AWS CLI, use the following command (replace XXXX with the name of your ECR repository).

After the image has been deleted, use the following command to delete the stack.

[sec:batch_development_and_debug] === Development and debugging of machine learning applications using the cloud

In the hands-on session described in this chapter, we used AWS Batch to run parallel neural network trainings to accelerate the model development. As the last topic in this chapter, we will discuss how to develop and debug machine learning applications using the cloud.

If you don’t have a powerful local machine with GPUs, and you have the budget to use the cloud, then a development scheme like Figure 73 would be ideal. In the first stage, create an EC2 instance with GPUa using the method described in Section 6, and experiment with various models in an interactive environment such as Jupyter Notebook. When the application is completed to some extent with Jupyter, package the application into a Docker image. Then, run docker run on EC2 to check if the created image works without errors. Next, we will perform tuning, such as hyperparameter optimization, using a computational system such as AWS Batch, which we learned in the Section 9. Once we have a good deep learning model, we will build a system to perform inference on large-scale data, using Section 8 as a reference.

In fact, the exercises in this book have been carried out along this workflow. We first experimented with a model for solving the MNIST task using Jupyter Notebook, then packaged the code into Docker, and used AWS Batch to perform a hyperparameter search. By repeating this cycle, we can proceed with the development of machine learning applications that take full advantage of the cloud.

This concludes Part II of this book. We hope you enjoyed the journey exploring the cloud technology.

In Part II, we first explained how to launch an EC2 instance with GPUs in order to run deep learning calculations in the cloud. In the hands-on session, we trained a neural network to solve the MNIST digit recognition task using a virtual server launched in the cloud (Section 6).

We also explained the steps to create a cluster using Docker and ECS as a means to build large scale machine learning applications (Section 7). As an exercise, we deployed a bot in the cloud that automatically generates answers to text questions given in English (Section 8). You should have been able to get some experience how computational resources are created and deleted dynamically in response to the submission of tasks.

Furthermore, in Section 9, we introduced a method to train neural networks in parallel using AWS Batch. Although the methods introduced here are minimal, they cover the essence of how to scale up a computer system. We hope that these hands-on experiences have given you some idea of how to apply cloud technology to solve real-world problems.

In the third part of this book, we take it a step further and explain the latest cloud design method called serverless architecture. In the hands-on session, we will implement a simple SNS service from scratch. Let’s continue our journey to enjoy the cutting-edge frontiers of cloud computing!

When you access Twitter, Facebook, YouTube, and other web services from your computer or smartphone, what is actually happening to render the contents in the page?

Many readers may already be familiar with the communication between servers and clients via HTTP, and since it would take up too much space to thoroughly explain everything, we will only cover the essentials here. In the following, we will use Twitter as a concrete example to outline the communication between the server and the client. As a sketch, Figure 74 depicts the communication between the client and the server.

As a premise, the client-server communication is done using HTTP (Hypertext Transfer Protocol). Recently, it has become a standard to use HTTPS (Hypertext Transfer Protocol Secure), which is an encrypted HTTP. In the first step, the client obtains static content from the server through HTTP(S) communication. Static content includes the main body of a web page document written in HTML (Hypertext Markup Language), page design and layout files written in CSS (Cascading Style Sheets), and programs that define the dynamic behavior of the page written in JavaScript (JS). In the design of modern web applications, including Twitter, these static files only define the "frame" of the page, and the content (e.g., the list of tweets) must be retrieved using API (Application Programming Interface). Therefore, the client sends the API request to the server according to the program defined in the JavaScript, and obtains the tweet list. JSON (JavaScript Object Notation) is often used to exchange text data. Media content such as images and videos are also retrieved by the API in the same way. The text and images retrieved in this way are embedded in the HTML document to create the final page presented to the user. Also, when posting a new tweet, the client uses the API to write the data to the server’s database.

API (Application Programming Interface) is a term that has been frequently used in this book, but we will give a more formal definition here. An API is a general term for an interface through which an application can exchange commands and data with external software. Especially in the context of web services, it refers to the list of commands that a server exposes to the outside world. The client obtains the desired data or sends data to the server by choosing the appropriate API commands.

Especially in the context of the web, APIs based on a design philosophy called REST (Representational State Transfer) are most commonly used. An API that follows the REST design guidelines is called a REST API or RESTful API.

A REST API consists of a pair of Method and URI (Universal Resource Identifier), as shown in Figure 75.

A method can be thought of as a "verb" that abstractly expresses the kind of desired operation. Methods can use any of the nine verbs defined in the HTTP standard. Among them, the five most frequently used ones are GET, POST, PUT, PATCH, and DELETE (Table 6). The operations using these five methods are collectively called CRUD (create, read, update, and delete).

On the other hand, a URI represents the target of an operation, i.e., the "object". In the context of the web, the target of an operation is often referred to as a resource. The URI often begins with the address of the web server, starting with http or https, and the path to the desired resource is specified after the / (slash). In the example of Figure 75, it means to retrieve (GET) the resource /1.1/status/home_timeline with the address https://api.twitter.com. (Note that the number 1.1 here indicates the API version.) This API request retrieves the list of tweets in the user’s home timeline.

In order to have a more realistic feeling on the web APIs, let’s take a look at Twitter’s API. A list of APIs provided by Twitter can be found at Twitter’s Developer Documentation. Some representative API endpoints are listed in Table 7.

Based on this list of APIs, let’s simulate the client-server communication that happens when you open a Twitter app or website.

When a user opens Twitter, the first API request sent to the server is GET statuses/home_timeline, which retrieves a list of tweets in the user’s home timeline. Each tweet is in JSON format and contains attributes such as id, text, user, coordinates, and entities. The id represents the unique ID of the tweet, and the text contains the body of the tweet. The user is a JSON data containing the information of the user who posted the tweet, including the name and URL of the profile image. The coordinates contains the geographic coordinates of where the tweet was posted. entities contains the links to media files (images, etc.) related to the tweet. From GET statuses/home_timeline, a list of the most recent tweets is retrieved (or a part of the list if the list is too long). If you know the ID of the tweet, you can call GET statuses/show/:id to retrieve the specific tweet specified by the :id parameter.

The GET search API is used to search tweets. The GET search API can be used to search for tweets by passing various query conditions, such as words in the tweet, hashtags, and the date, time, and location of the tweet. The API will return the tweet data in JSON format, similar to GET statuses/home_timeline.

When a user posts a new tweet, the POST statuses/update endpoint is used. The POST statuses/update endpoint receives the text of the tweet, and in the case of a reply, the ID of the tweet to which the user is replying. If you want to attach images to the tweet, use POST media/upload as well. To delete a tweet, POST statuses/destroy/:id is used.

Other frequently used operations are POST statuses/retweet/:id and POST statuses/unretweet/:id. These APIs are used to retweet or unretweet the tweet specified by :id, respectively. In addition, POST favorites/create and POST favorites/destroy can be used to add or remove a "like" to a selected tweet.

This is the sequence of operations that takes place behind Twitter applications. If you want to create your own bot, you can do so by writing a custom program that combines these APIs.

As you can see, APIs are the most fundamental element in the construction of any web service. In the following sections, the terms introduced in this section will appear many times, so please keep them in mind before reading on.

A sketch of a traditional cloud system is shown in Figure 76. The request sent from the client is first sent to the API server. In the API server, tasks are executed according to the content of the request. Some tasks can be completed by the API server alone, but in most cases, reading and writing of the database is required. In general, an independent server machine dedicated to the database is used. Large sized data, such as images and videos, are often stored on a separate storage server. These API servers, database servers, and storage servers are all independent server machines. In AWS terms, you can think of them as virtual instances of EC2.

Many web services are designed to have multiple server machines running in the cloud to handle requests from a large number of clients. The operation of distributing requests from clients to servers with enough computing capacity is called load balancing, and the machine in charge of such operation is called load balancer.

Launching a large number of instances for the purpose of distributing the computational load is fine, but it is a waste of cost and power if the computational load is too small and the most of the cluster is kept idling. Therefore, we need a mechanism that dynamically increases or decreases the number of virtual servers in a cluster according to the computational load so that all servers always maintain the certain load. Such a mechanism is called cluster scaling, and the operation of adding a new virtual instance to the cluster in response to an increase in load is called scale-out, and the operation of shutting down an instance in response to a decrease in load is called scale-in. Scaling of clusters is necessary not only for API servers, but also for database servers and storage servers. In the storage server, for example, frequently accessed data is stored in the cache area, and multiple copies of the data are made across instances. In the same way, database servers require distributed processing to prevent frequent data accesses from disrupting the system. It is necessary to adjust the load so that it is evenly distributed throughout the cloud system, and developers must spend a lot of time tuning the system. In addition, the scaling settings need to be constantly reviewed according to the number of users of the service, and continuous development is required.

What makes the matters worse, the tasks processed by the API server are non-uniform. Being non-uniform, means that, for example, task A consumes 3000 milliseconds of execution time and 512MB of memory, while another task B consumes 1000 milliseconds of execution time and 128MB of memory. Scaling a cluster becomes complex when a single server machine handles multiple tasks with different computational loads. In order to simplify this situation, it is possible to design the cluster so that only one type of task is executed by a single server, but there are many negative effects of adopting such design.

As we discussed in Section 11.1, scaling of clusters is an essential task to maximize the economic efficiency and system stability of cloud systems. Reflecting this, a lot of developer’s time and efforts have been invested in it.

Scaling a cluster is a task that all developers have done over and over again, and if some aspects could be templated and made common, it would greatly reduce the cost of development. In order to achieve this, one needs to rethink the design of cloud systems from a fundamental level. Is there a cloud system design that is simpler and more efficient by considering scaling as a first and built-in priority? Such was the motivation behind the birth of serverless architecture.

The biggest problem with conventional serverful systems is that users occupy the entire server. Namely, when an EC2 instance is launched, it is available only to the user who launched it, and the computation resources (CPU and RAM) are allocated exclusively to that user. Since a fixed allocation of computing resources has been made, the same cost will be incurred in proportion to the launch time, regardless of whether the instance’s computing load is 0% or 100%.

The starting point of serverless architecture is the complete elimination of such exclusively allocated computational resources. In a serverless architecture, all computation resources are managed by the cloud provider. Rather than renting an entire virtual instance, clients submit a program or commands to the cloud every time they need to perform a computational task. The cloud provider tries to find free space from its own huge computational resources, executes the submitted program, and returns the execution result back to the client. In other words, the cloud provider takes care of the scaling and allocation of computational resources, and the user focuses on submitting jobs. This can be illustrated as Figure 77.

In a serverless cloud, scalability is guaranteed because all scaling is taken care of by the cloud provider. Even if a client sends a large number of tasks at the same time, the cloud provider’s sophisticated system ensures that all tasks are executed without delay. Also, by using a serverless cloud, the cost of the cloud is determined by the total amount of computation. This is a big difference compared to conventional systems where the cost is determined by the launch time of the instance regardless of the total amount of computation performed.

Since serverless cloud is a fundamentally different approach from traditional cloud, the way to design the system and write code is very different. To develop and operate a serverless cloud, it is necessary to be familiar with concepts and terminology specific to serverless technology.

Now that we have an overview of serverless architecture, let us introduce you to the components that make up a serverless cloud in AWS. In particular, we will focus on Lambda, S3, and DynamoDB (Figure 78). In a serverless cloud, a system is created by integrating these components. In what follows, we will go through all the knowledge that must be kept in mind when using Lambda, S3, and DynamoDB, so it may be difficult to get a concrete image. However, in the next section (Section 12), we will provide hands-on exercises for each of them, so you can deepen your understanding.