AWS Lambda Layer for KeyBERT Extraction

Project: Create Lightweight AWS Lambda Layer for KeyBERT & Keyphrase Extraction

Overview:
We need a lightweight AWS Lambda Layer that packages KeyBERT and keyphrase-vectorizers for extracting keyphrases from text. The layer should be optimized for size, deployment speed, and Lambda compatibility (Python 3.9+ runtime).

Requirements:
1. Dependencies:
• keybert
• keyphrase-vectorizers
• Any minimal additional dependencies required (scikit-learn, numpy, torch, etc.).
• Exclude unnecessary files to keep the layer lightweight (avoid heavy ML libraries not used).
2. Lambda Layer:
• Build a deployable .zip file for AWS Lambda layer.
• Ensure the size stays under AWS Lambda’s 250MB unzipped limit.
• Should support Python 3.9 and 3.10 runtimes.
• Provide instructions for publishing and attaching the layer.
3. Testing Code:
Provide a sample Lambda function that imports the layer and runs the following function:

from keybert import KeyBERT
from keyphrase_vectorizers import KeyphraseCountVectorizer

kw_model = KeyBERT()

def patternrank_keywords(text: str):
keywords = kw_model.extract_keywords(text, vectorizer=KeyphraseCountVectorizer(), top_n=15)
return [words[0].replace("-", " ") for words in keywords]

Example input: title + abstract text
Example output: list of 15 keyphrases.

4. Deliverables:
• Lambda Layer .zip file.
• Deployment instructions (Terraform/CLI preferred but not mandatory).
• Sample Lambda function for verification.

Bonus (Optional):
• Optimize dependencies to reduce cold start times.
• Provide Dockerfile (for future re-builds).

Skills Required:
• AWS Lambda & Layers
• Python packaging
• Experience with ML/NLP libraries (KeyBERT, scikit-learn)

⸻

Goal: A ready-to-use Lambda Layer that allows us to run the provided keyphrase extraction code inside Lambda without errors.