Prerequisites¶

projectx-prod-vpc has been created with subnets configured.
projectx-prod-websvr-public EC2 instance exists and is fully configured with all applications and tools.
My-Desktop-Key-Pair key pair exists.
AWS CLI configured with appropriate credentials.

Network Topology¶

Overview¶

This guide walks through the environment setup for AWS Lambda.

As mentioned in previous guides, AWS Lambda will be used to fetch data from open-source threat intelligence feeds. This data will then be aggregated, stored, and visualized using our web stack.

Python scripts will be used to fetch data from open-source feeds.

Before we add these Python scripts, we need to ensure our Lambda environment is prepared.

This includes adding:

Lambda Layers: Third party Python libraries packaged into a zip file. This will allow the python files to source the third party libraries.
Lambda IAM Role: Provision a dedicated Lambda Execution role with AWSLambdaBasicExecutionRole and access to S3 with GetObject and PutObject permissions via a Customer inline policy. This will allow the Lambda functions to fetch logs into our threat-intelligence-feed-bucket.
Security Groups: Create projectx-lambda-feed-SG, which will provide Outbound access to all traffic.
Create S3 Bucket: Create a new S3 bucket, threat-intelligence-feed-bucket, which will be used as a store interface between the prober and the parser Lambda functions.
Provision S3 Gateway Endpoint: An S3 Gateway Endpoint allows resources in private VPC subnets to reach S3. Without a S3 Gateway Endpoint, the parser Lambda function would not be able to access S3 bucket.

Lambda supports different runtime environment versions for Python, such as Python 3.10, 3.11, 3.12, 3.13, and 3.14.

We will use Python 3.12 for stability purposes.

Threat Intelligence Feeds¶

We will create two Lambda functions with the following purposes:

public-threat-intelligence-feed-prober¶

This function will be a public function, which will have Internet access to probe open-source threat intelligence feeds.

Each feed will be a Python script that will run asynchronously from the other, in case of fetching data failure.

Here are the feeds we will eventually have:

('security_news_rss', 'placeholder', '{"items": []}'),
('top_100_domains', 'placeholder', '{"items": []}'),
('top_ips', 'placeholder', '{"items": []}'),
('top_10_countries_by_ip', 'placeholder', '{"items": []}'),
('top_malware_hashes', 'placeholder', '{"items": []}'),
('top_iocs', 'placeholder', '{"items": []}');

👉 For CA101, we focus on templating the open-source threat intelligence feeds. In Web & Attacks 101, we will build the actual logic (through Claude Code, or through sourcing the exercise files).

private-threat-intelligence-feed-parser¶

Each Python script in the public-threat-intelligence-feed-parser Lambda function will output a .json file with its data. This .json file will be written to the S3 bucket, threat-intelligence-feed-bucket.

The private-threat-intelligence-feed-parser will take the newly written data via S3 Event Notification and send it to the projectx-websvr-public PostgreSQL database, where each .json file will be normalized and stored in the existing dashboard.panel_feed table:

We have already executed these SQL commands while setting up the database.

CREATE SCHEMA dashboard AUTHORIZATION projectx_dbadmin;

CREATE TABLE dashboard.panel_feed (
    id bigserial PRIMARY KEY,
    panel_name text NOT NULL,
    source_feed text NOT NULL,
    payload jsonb NOT NULL,
    collected_at timestamptz NOT NULL DEFAULT now()
);

Step 1: Create Lambda Layer for Dependencies¶

We'll create a shared Lambda layer containing common Python packages used by all functions. This is how the Python files know where to locate third party libraries such as requests or feedparser.

Create Layer Directory Structure¶

Note

If you do not want to reproduce these steps, use theShared Lambda Layer on GitHub Exercise Files

On your local machine, we need to ensure the correct version of python3 is installed.

As mentioned, we will be using Python 3.12 runtime environment. This means we need to compile our zip file with the third party libraries with version 3.12.

Create the following structure:

We will assume you have access to a shell-base environment. Either through native support such as Linux, macOS zsh, or Windows Subsystem for Linux.

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install -y python3.12 python3.12-venv

Verify with:

python3.12 --version

Create a new lambda layer environment.

mkdir -p lambda-layer/python/lib/python3.12/site-packages
cd lambda-layer

Create new python3.12 environment with:

python3.12 -m venv venv

Activate it:

source venv/bin/activate

Create requirements.txt¶

Create a requirements.txt file in the lambda-layer directory:

psycopg2-binary==2.9.9
requests==2.31.0
feedparser==6.0.10

Install Dependencies¶

pip install -r requirements.txt -t python/lib/python3.12/site-packages/

Create Layer ZIP¶

sudo apt install zip
zip -r lambda-layer-python312.zip python/

Upload Layer to AWS¶

Navigate to Lambda Service¶

Open the Lambda service in the AWS Console.

Create Layer¶

In the left navigation pane, select Layers (under Additional resources).
Click Create layer.

Find the .zip file.

Configure Layer¶

Configure the layer settings:

Name: threat-intel-lambda-layer
Description: Shared Lambda layer for threat intelligence feed functions
Upload a .zip file: Click Upload and select your lambda-layer.zip file
Architectures: x86
Compatible runtimes: Select Python 3.11 and Python 3.12 and Python 3.13

Create Layer¶

Click Create.

After creation, note the Layer version ARN from the layer details page - you'll need it when creating Lambda functions.

Step 2: Create `threat-intelligence-feed-bucket` S3 Bucket¶

Navigate to S3¶

Open the S3 service in the AWS Console.

Create Bucket¶

Click Create bucket.

Bucket type: General purpose.
Bucket name: threat-intelligence-feed-parser-UNIQUE_STRING.

👉UNIQUE_STRING: Use a unique string, such as user handle, account ID, lab environment name, since S3 is a global namespace.

Leave everything else default.

Navigate to VPC¶

Open the VPC service in the AWS Console.

Under PrivateLink and Lattice, choose Endpoints.

Click Create endpoint.

Name: projectx-s3-endpoint
Type: AWS services.
Services: com.amazonaws.us-east-2.s3 GATEWAY.
VPC: projectx-prod-vpc.
Route tables: Select projectx-prod-private-rt.
Policy: Full access.

Create endpoint.

Step 3: Configure IAM Role for Lambda Functions¶

Create an IAM role with the necessary permissions for Lambda functions to access S3 threat-intelligence-feed-bucket and basic Lambda Execution role.

Navigate to IAM¶

Open the IAM service in the AWS Console.

In the left navigation pane, select Roles.

Create Role¶

Click Create role.

Select Trusted Entity Type¶

Select AWS service as the trusted entity type.

Select Use Case¶

Under Use case, select Lambda.

Click Next.

Add Permissions¶

Attach AWSLambdaBasicExecutionRole permissions policy.

Name and Create Role¶

Role name: projectx-lambda-feed-exec-role
Description: Allows Lambda functions to call AWS services on your behalf.

Click Create role.

Create & Attach Inline Policy¶

Navigate to Roles projectx-lambda-feed-exec-role.

Select Add permissions Create inline policy.

Select the JSON editor.

Inside the JSON editor, paste the following:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ThreatIntelFeedBucketReadWriteObjects",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::threat-intelligence-feed-bucket-UNIQUE_STRING/*"
    }
  ]
}

👉 "arn:aws:s3:::threat-intelligence-feed-bucket-UNIQUE_STRING/*": Paste your S3 bucket's unique name.

Name the inline policy: projectx-lambda-feed-s3-read-write.

Create the Policy.

Step 4: Create Security Group for Lambda¶

We need to give the Lambda function outbound access, we can broad or precise. We only need PostgreSQL port 5432.

In the VPC console, select Security Groups in the left navigation pane.

Click Create security group.

Basic Details¶

Security group name: projectx-lambda-feed-SG
Description: Security group for Lambda functions accessing PostgreSQL
VPC: Select projectx-prod-vpc

Outbound Rules¶

In the Outbound rules section: - Type: PostgreSQL (or Custom TCP) - Port range: 5432 - Destination type: Security group - Destination: Select your EC2 security group (e.g., projectx-prod-websvr-SG) - Description: Allow Lambda to connect to PostgreSQL

Or

In the Outbound rules section: - Type: All traffic - Port range: All - Destination: 0.0.0.0/0 - Description: Allow Outbound Lambda to all traffic

Click Create security group.

Note the Security Group ID - you'll need it when configuring Lambda functions.

Ready For The Next Steps¶

We are now prepared and ready to create the next two Lambda functions, this preparation will come in handy when it is time to provision our Lambda functions.

Prerequisites¶

Network Topology¶

Overview¶

Threat Intelligence Feeds¶

public-threat-intelligence-feed-prober¶

private-threat-intelligence-feed-parser¶

Step 1: Create Lambda Layer for Dependencies¶

Create Layer Directory Structure¶

Create requirements.txt¶

Install Dependencies¶

Create Layer ZIP¶

Upload Layer to AWS¶

Navigate to Lambda Service¶

Create Layer¶

Configure Layer¶

Create Layer¶

Step 2: Create threat-intelligence-feed-bucket S3 Bucket¶

Navigate to S3¶

Create Bucket¶

Navigate to VPC¶

Step 3: Configure IAM Role for Lambda Functions¶

Navigate to IAM¶

Create Role¶

Select Trusted Entity Type¶

Select Use Case¶

Add Permissions¶

Name and Create Role¶

Create & Attach Inline Policy¶

Step 4: Create Security Group for Lambda¶

Basic Details¶

Outbound Rules¶

Ready For The Next Steps¶

Step 2: Create `threat-intelligence-feed-bucket` S3 Bucket¶