Test

Overview

Security is an essential element of any application especially when it comes to the Restful API layer. Thousands of calls are made daily to share information via Rest APIs, making security a top concern for all organizations in all stages: designing, testing and deploying the APIs. We are living in an era where our private information is more vulnerable than ever before, so it’s very important to protect your APIs from threats and vulnerabilities that keep on increasing daily.

In addition to all the guidelines available for building a secure API, an important step is to make your API private. Attackers will not be able to launch any attack on your API if they can’t find it. Exposing your APIs to the public will add a range of security and management challenges that you can avoid.

While it’s easy to spin up simple these cloud architectures, mistakes can easily be made provisioning complex ones. Human error will always be present, especially when you can launch cloud infrastructure by clicking buttons on a web app.

The only way to avoid these kinds of errors is through automation, and Infrastructure as Code is helping engineers automatically launch cloud environments quickly and without mistakes.

Why Amazon Web Services ?

AWS Lambda and AWS API Gateway have made creating serverless APIs extremely easy. You can simply upload your decision service to AWS Lambda, configure an API Gateway, and start responding to RESTful endpoint calls.

From the security standpoint, Amazon has introduced AWS PrivateLink so you can choose to restrict all your API traffic to stay within your Amazon Virtual Private Cloud (VPC) which can be isolated from the public internet. Now you can create a private API in your Amazon API Gateway that can only be accessed from within your VPC. It eliminates the exposure of data to the public internet by providing private connectivity between VPCs, AWS services, and on-premise applications securely on the Amazon Network.

Why Terraform ?

Terraform is a tool for developing, changing and versioning infrastructure safely and efficiently. It can manage existing and popular service providers as well as custom in-house solutions. Terraform is the first multi-cloud immutable infrastructure tool that was introduced to the world by HashiCorp, released three years ago, and written in Go.

Terraform’s speed and operations are exceptional. One cool thing about it is, it’s plan command lets you see what changes you’re about to apply before you apply them. Code reuse feature and Terraform tends to make most changes faster than similar tools like CloudFormation.

Proposed Architecture

With the ability to have private API endpoints inside your own VPC, you can still use API Gateway features, while securely exposing REST APIs only to the other services and resources inside your VPC.

API Gateway private endpoints are made possible via AWS PrivateLink interface VPC endpoints. Interface endpoints work by creating elastic network interfaces in subnets that you define inside your VPC. Those network interfaces then provide access to the API Gateway running in its VPC.

API Gateway as a fully managed service runs its infrastructure in its own VPCs. When you interface with API Gateway publicly accessible endpoints, it is done through public networks. When they’re configured as private, which is the case in the proposed architecture, the public networks are not made available to route your API. Instead, your API can only be accessed using the interface endpoints that you have configured.

  • Because you configure the subnets in which your endpoints are made available, you control the availability of the access to your API Gateway hosted APIs. Make sure that you provide multiple interfaces in your VPC. In the above diagram, there is one endpoint in each subnet in each Availability Zone for which the VPC is configured.
  • Each endpoint is an elastic network interface configured in your VPC that has security groups configured. Network ACLs apply to the network interface as well.

Proposed Solution

What Should Be Pre-Installed

In order to follow this case study you will need an AWS account and to have Terraform installed. Configure your credentials so that Terraform is able to act on your behalf.

For simplicity here we will assume you are already using a set of IAM credentials with suitable access to create Lambda functions and work with API Gateway. If you aren’t sure and are working in an AWS account used only for development, the simplest approach to get started is to use credentials with full administrative access to the target AWS account.

◊ Following this case study will create objects in your AWS account that will cost you money against your AWS bill.

What You Will Do

This case study will show you how to deploy IBM ODM on AWS Lambda and generate an API Gateway to invoke it. All of that, without the need to go over the internet so mainly we’ll be using VPCs and endpoints. The guide assumes some basic familiarity with Lambda and API Gateway but does not assume any pre-existing deployment. It also assumes that you are familiar with the usual Terraform plan/apply workflow; if you’re new to Terraform itself, refer first to the Getting Started guide.

Following step-by-step instructions below, you do the following:

  1. Create the VPC

  2. Create the VPC endpoint for API Gateway

  3. Create the API

  4. Test the API

Project Directory

The project directory should contain the following files and subdirectories, we will go through each file in the rest of the article

. ├── main.tf ├── modules │ ├── lambda_tester │ │ ├── lambda.tf │ │ └── vars.tf │ ├── odm_api │ │ ├── api_gateway.tf │ │ └── lambda.tf | | └── vars.tf │ ├── vpc │ │ ├── demo_vpc.tf │ │ └── endpoint.tf | | └── network.tf | | └── vars.tf ├── providers.tf ├── README.md

1. Create the VPC

This VPC will have two private and two public subnets, one of each in an AZ, as seen in the architecture below.

Under the modules folder, create a folder named vpc in which you’ll add the following terraform files

  • demo_vpc.tf
data "aws_availability_zones" "available" {
state = "available"
}
resource "aws_vpc" "Demo-VPC" {
cidr_block = "${var.vpc_cidr_block}"
enable_dns_support = "true" #gives you an internal domain name
enable_dns_hostnames = "true" #gives you an internal host name
enable_classiclink = "false"
instance_tenancy = "default"
tags = {
Name = "Demo-VPC"
}
}
resource "aws_subnet" "publicSubnet01" {
vpc_id = "${aws_vpc.Demo-VPC.id}"
cidr_block = "${var.publicSubnet01_cidr_block}"
map_public_ip_on_launch = "true" //it makes this a public subnet
availability_zone = "${data.aws_availability_zones.available.names[0]}"
tags = {
Name = "publicSubnet01"
}
}
resource "aws_subnet" "publicSubnet02" {
vpc_id = "${aws_vpc.Demo-VPC.id}"
cidr_block = "${var.publicSubnet02_cidr_block}"
map_public_ip_on_launch = "true" //it makes this a public subnet
availability_zone = "${data.aws_availability_zones.available.names[1]}"
tags = {
Name = "publicSubnet02"
}
}

resource "aws_subnet" "privateSubnet01" {
vpc_id = "${aws_vpc.Demo-VPC.id}"
cidr_block = "${var.privateSubnet01_cidr_block}"
map_public_ip_on_launch = "false" //it makes this a private subnet
availability_zone = "${data.aws_availability_zones.available.names[1]}"
tags = {
Name = "privateSubnet01"
}
}

resource "aws_subnet" "privateSubnet02" {
vpc_id = "${aws_vpc.Demo-VPC.id}"
cidr_block = "${var.privateSubnet02_cidr_block}"
map_public_ip_on_launch = "false" //it makes this a private subnet
availability_zone = "${data.aws_availability_zones.available.names[0]}"
tags = {
Name = "privateSubnet02"
}
}

output "vpc_id" {
value = "${aws_vpc.Demo-VPC.id}"
}

output "privateSubnet1_id" {
value = "${aws_subnet.privateSubnet01.id}"
}
output "privateSubnet2_id" {
value = "${aws_subnet.privateSubnet02.id}"
}
resource "aws_internet_gateway" "DemoVPCIGW" {
vpc_id = "${aws_vpc.Demo-VPC.id}"
tags = {
Name = "DemoVPCIGW"
}
}

resource "aws_route_table" "PublicRouteTable" {
vpc_id = "${aws_vpc.Demo-VPC.id}"

route {
//associated subnet can reach everywhere
cidr_block = "0.0.0.0/0"
//CRT uses this IGW to reach internet
gateway_id = "${aws_internet_gateway.DemoVPCIGW.id}"
}

tags = {
Name = "PublicRouteTable"
}
}

resource "aws_route_table_association" "PublicSubnetRTAssociation01"{
subnet_id = "${aws_subnet.publicSubnet01.id}"
route_table_id = "${aws_route_table.PublicRouteTable.id}"
}

resource "aws_route_table_association" "PublicSubnetRTAssociation02"{
subnet_id = "${aws_subnet.publicSubnet02.id}"
route_table_id = "${aws_route_table.PublicRouteTable.id}"
}

resource "aws_route_table_association" "PrivateSubnetRTAssociation01"{
subnet_id = "${aws_subnet.privateSubnet01.id}"
route_table_id = "${aws_route_table.PublicRouteTable.id}"
}

resource "aws_route_table_association" "PrivateSubnetRTAssociation02"{
subnet_id = "${aws_subnet.privateSubnet02.id}"
route_table_id = "${aws_route_table.PublicRouteTable.id}"
}

resource "aws_security_group" "EndpointSG" {
vpc_id = "${aws_vpc.Demo-VPC.id}"

ingress {
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = ["${var.vpc_cidr_block}"]
}
egress {
from_port = 0
to_port = 65535
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}

tags = {
Name = "EndpointSG"
}
}

 
variable "vpc_cidr_block" {
default = "10.45.0.0/16"
}

variable "publicSubnet01_cidr_block" {
default = "10.45.10.0/24"
}
variable "publicSubnet02_cidr_block" {
default = "10.45.11.0/24"
}
variable "privateSubnet01_cidr_block" {
default = "10.45.225.0/24"
}

variable "privateSubnet02_cidr_block" {
default = "10.45.224.0/24"
}
2. Create the VPC endpoint for API Gateway:

Always under the vpc folder, add the following file:

data "aws_region" "current" {}

resource "aws_vpc_endpoint" "execute-api" {
vpc_id = "${aws_vpc.Demo-VPC.id}"
service_name = "com.amazonaws.${data.aws_region.current.name}.execute-api"
vpc_endpoint_type = "Interface"

security_group_ids = [
"${aws_security_group.EndpointSG.id}",
]
subnet_ids = ["${aws_subnet.privateSubnet01.id}","${aws_subnet.privateSubnet02.id}"]

private_dns_enabled = true
}

output "endpoint_id" {
value = "${aws_vpc_endpoint.execute-api.id}"
}

output "endpointDns" {
value = "${aws_vpc_endpoint.execute-api.dns_entry}"
}

Creating the endpoint takes a few moments to go through all of the interface endpoint lifecycle steps.

3. Create the API

Under the modules folder, create a folder named odm_api in which you’ll add the following terraform files:

resource "aws_lambda_function" "example" {
function_name = "hello_world"

# The bucket name as created earlier with "aws s3api create-bucket"
s3_bucket = "ibm-odm-dependencies"
s3_key = "greetings-0.0.1-SNAPSHOT-shaded.jar"
# "main" is the filename within the zip file (main.js) and "handler"
# is the name of the property under which the handler function was
# exported in that file.
handler = "com.sb.greetings.SimpleDecisionEngineRunner::handleRequest"
runtime = "java8"
timeout="10"
role = "${aws_iam_role.lambda_exec.arn}"
}

# IAM role which dictates what other AWS services the Lambda function
# may access.
resource "aws_iam_role" "lambda_exec"{
name = "hello_world_lambda"

assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}
resource "aws_api_gateway_resource" "proxy" {
rest_api_id = "${aws_api_gateway_rest_api.example.id}"
parent_id = "${aws_api_gateway_rest_api.example.root_resource_id}"
path_part = "{proxy+}"
}

resource "aws_api_gateway_method" "proxy" {
rest_api_id = "${aws_api_gateway_rest_api.example.id}"
resource_id = "${aws_api_gateway_resource.proxy.id}"
http_method = "ANY"
authorization = "NONE"
}

resource "aws_api_gateway_integration" "lambda" {
rest_api_id = "${aws_api_gateway_rest_api.example.id}"
resource_id = "${aws_api_gateway_method.proxy.resource_id}"
http_method = "${aws_api_gateway_method.proxy.http_method}"

integration_http_method = "POST"
type = "AWS"
uri = "${aws_lambda_function.example.invoke_arn}"
}

resource "aws_api_gateway_method" "proxy_root" {
rest_api_id = "${aws_api_gateway_rest_api.example.id}"
resource_id = "${aws_api_gateway_rest_api.example.root_resource_id}"
http_method = "ANY"
authorization = "NONE"
}

resource "aws_api_gateway_integration" "lambda_root" {
rest_api_id = "${aws_api_gateway_rest_api.example.id}"
resource_id = "${aws_api_gateway_method.proxy_root.resource_id}"
http_method = "${aws_api_gateway_method.proxy_root.http_method}"

integration_http_method = "POST"
type = "AWS"
uri = "${aws_lambda_function.example.invoke_arn}"
}

resource "aws_api_gateway_method_response" "root_response_200" {
rest_api_id = "${aws_api_gateway_rest_api.example.id}"
resource_id = "${aws_api_gateway_method.proxy_root.resource_id}"
http_method = "${aws_api_gateway_method.proxy_root.http_method}"
status_code = "200"
}

resource "aws_api_gateway_integration_response" "proxy_root" {
rest_api_id = "${aws_api_gateway_rest_api.example.id}"
resource_id = "${aws_api_gateway_method.proxy_root.resource_id}"
http_method = "${aws_api_gateway_method.proxy_root.http_method}"
status_code = "${aws_api_gateway_method_response.root_response_200.status_code}"
}

resource "aws_api_gateway_method_response" "proxy_response_200" {
rest_api_id = "${aws_api_gateway_rest_api.example.id}"
resource_id = "${aws_api_gateway_method.proxy.resource_id}"
http_method = "${aws_api_gateway_method.proxy.http_method}"
status_code = "200"
}

resource "aws_api_gateway_integration_response" "proxy" {
rest_api_id = "${aws_api_gateway_rest_api.example.id}"
resource_id = "${aws_api_gateway_method.proxy.resource_id}"
http_method = "${aws_api_gateway_method.proxy.http_method}"
status_code = "${aws_api_gateway_method_response.proxy_response_200.status_code}"

}

resource "aws_lambda_permission" "apigw" {
statement_id = "AllowAPIGatewayInvoke"
action = "lambda:InvokeFunction"
function_name = "${aws_lambda_function.example.function_name}"
principal = "apigateway.amazonaws.com"
# The "/*/*" portion grants access from any method on any resource
# within the API Gateway REST API.
source_arn = "${aws_api_gateway_rest_api.example.execution_arn}/*/*"
}

resource "aws_api_gateway_deployment" "example" {
depends_on = [
aws_api_gateway_integration.lambda,
aws_api_gateway_integration.lambda_root,
aws_api_gateway_integration_response.proxy,
aws_api_gateway_integration_response.proxy_root
]

rest_api_id = "${aws_api_gateway_rest_api.example.id}"
stage_name = "test"
}
  • api_gateway.tf
data "aws_caller_identity" "current" {}

resource "aws_api_gateway_rest_api" "example" {
name = "hello_world_api"
description = "Terraform Serverless Application Example"
endpoint_configuration {
types = ["PRIVATE"]
}
policy=<<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "execute-api:Invoke",
"Resource": "execute-api:/*",
"Condition": {
"StringEquals": {

"aws:sourceVpce": "${var.endpoint_id}"
}
}
}
]
}
EOF
}

output "base_url" {
value = "${aws_api_gateway_deployment.example.invoke_url}"
}

output "Api_id" {
value = "${aws_api_gateway_rest_api.example.id}"
}

 

As you can see in the “aws_api_gateway_rest_ap “ resource block and under the policy attribute, we’ve created a resource policy to allow access to the API from inside the VPC.

variable "endpoint_id" {}

Your API is now fully deployed and available from inside your VPC. Next, test to confirm that it’s working.

4. Test the API

To emphasize the privateness of this API, test it from a resource that only lives inside your VPC and has no direct network access to it, in the traditional networking sense.

Launch a Lambda function inside the VPC, with no public access. To show its ability to hit the private API endpoint, invoke it using the console. The function is launched inside the private subnets inside the VPC without access to a NAT gateway, which would be required for any internet access. This works because Lambda functions are invoked using the service API, not any direct network access to the function’s underlying resources inside your VPC.

All the code for this function is located inside of the lambda_tester folder. It creates just three resources, as shown in the architecture below:

  • A Lambda function
  • An IAM role
  • A VPC security group

 

Under the modules folder, create a folder named lambda_tester in which you’ll add the following terraform files:

 

data "aws_region" "current" {}

data "archive_file" "lambda_zip_inline" {
type = "zip"
output_path = "/tmp/lambda_zip_inline.zip"

source {
content = <<EOF
var https = require('https');
const options = {
host: '${var.endpointDns[0].dns_name}',
port: 443,
path: '/test',
method: 'GET',
headers: {
'Host':'${var.Api_id}.execute-api.${data.aws_region.current.name}.amazonaws.com'
}
};
exports.handler = (event, context, callback) => {
https.request(options, (res) => {
console.log('statusCode:', res.statusCode);
console.log('headers:', res.headers);
let data = '';
res.on('data', (d) => {
data += d;
process.stdout.write(d);
});
res.on('end', () => {
callback(null, JSON.parse(data));
});
}).on('error', (e) => {
console.error(e);
callback(null, e);
}).end();
};
EOF
filename = "main.js"
}
}
resource "aws_lambda_function" "LambdaTester" {
function_name = "LambdaTester"
filename = "${data.archive_file.lambda_zip_inline.output_path}"
source_code_hash = "${data.archive_file.lambda_zip_inline.output_base64sha256}"
# "main" is the filename within the zip file (main.js) and "handler"
# is the name of the property under which the handler function was
# exported in that file.
handler = "main.handler"
runtime = "nodejs12.x"
timeout="35"
role = "${aws_iam_role.lambda_exec.arn}"
vpc_config {

subnet_ids = ["${var.privateSubnet1_id}","${var.privateSubnet2_id}"]

security_group_ids = ["${aws_security_group.ApiTesterSG.id}"]

}
}

resource "aws_security_group" "ApiTesterSG" {
vpc_id = "${var.vpc_id}"

egress {
from_port = 0
to_port = 65535
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}

tags = {
Name = "ApiTesterSG"
}
}
resource "aws_iam_role" "lambda_exec"{
name = "APITesterFunctionRole"

assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}

# IAM role which dictates what other AWS services the Lambda function
# may access.
resource "aws_iam_policy" "lambda_exec"{
name = "APITesterFunctionRolePolicy"
policy = <<POLICY
{
"Version": "2012-10-17",
"Statement": {
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"ec2:CreateNetworkInterface",
"ec2:DescribeNetworkInterfaces",
"ec2:DeleteNetworkInterface"
],
"Resource": "*"
}
}
POLICY
}

resource "aws_iam_policy_attachment" "profile"{
name="APITesterFunctionRolePolicyAttachment"
roles=["${aws_iam_role.lambda_exec.name}"]
policy_arn="${aws_iam_policy.lambda_exec.arn}"
}
variable "vpc_id" {}
variable "privateSubnet1_id" {}
variable "privateSubnet2_id" {}
variable "endpointDns" {}
variable "endpoint_id" {}
variable "Api_id" {}

And there you go, you’re all set.

But before you run the terraform script you must add the main.tf file, to ramp up all the modules of your project into a single connected unit.

In the root folder of the project add the main.tf file with the following code.

module "vpc" {
source = "./modules/vpc"
}
module "odm_api" {
source = "./modules/odm_api"
endpoint_id="${module.vpc.endpoint_id}"
}

module "lambda_tester" {
source = "./modules/lambda_tester"
vpc_id="${module.vpc.vpc_id}"
privateSubnet1_id="${module.vpc.privateSubnet1_id}"
privateSubnet2_id="${module.vpc.privateSubnet2_id}"
endpointDns="${module.vpc.endpointDns}"
endpoint_id="${module.vpc.endpoint_id}"
Api_id="${module.odm_api.Api_id}"
}

Here you can see the use of modular approach and output variables, take some time to read the code above.

5. Running the scripts

Terraform is controlled via a very easy to use command-line interface (CLI). Terraform is only a single command-line application: Terraform.

In order to run your Terraform code, you need to follow these steps

Command: init

The terraform init command is used to initialize a working directory containing Terraform configuration files. This is the first command that should be run after writing a new Terraform configuration or cloning an existing one from version control. It is safe to run this command multiple times.

Command: plan

The terraform plan command is used to create an execution plan. Terraform performs a refresh, unless explicitly disabled, and then determines what actions are necessary to achieve the desired state specified in the configuration files.

This command is a convenient way to check whether the execution plan for a set of changes matches your expectations without making any changes to real resources or to the state. For example, terraform plan might be run before committing a change to version control, to create confidence that it will behave as expected.

Command: apply

The terraform apply command is used to apply the changes required to reach the desired state of the configuration, or the pre-determined set of actions generated by a terraform plan execution plan.

when finished you can run the terraform destroy. This command is used to destroy the Terraform-managed infrastructure.

Conclusion

API Gateway private endpoints enable use cases for building private API–based services inside your own VPCs. You can now keep both the frontend to your API (API Gateway) and the backend service (Lambda, EC2, ECS, etc.) private inside your VPC. Or you can have networks using Direct Connect networks without the need to expose them to the internet in any way. All of this without the need to manage the infrastructure that powers the API gateway itself!

A cloud in mind ?
Contact us !

How can we help ?