Secure LLM Router on AWS with NeMo GuardRails, Lambda, and API Gateway
This blog series explores the design and implementation of LLM (Large Language Model) agents on AWS, focusing on real-world automation and AI-driven development workflows.
The demo showcases an agent named "AWS Operator", which manages user tasks. If a request cannot be fulfilled directly, the agent automatically creates a story in Azure DevOps. The "Developer Agent" then picks up the story, implements the required changes, and submits a pull request. After a manual review and commit, the DevOps pipeline is triggered, updating the AWS Operator agent with the newly integrated tool.
Provider Support and Endpoint
The routing mechanism was implemented using LangChain's model implementations for various providers. The entire setup is serverless, leveraging AWS API Gateway and AWS Lambda. API Gateway provides a single endpoint, and the request payload determines which model is invoked.
Router Execution Flow:
The Router extracts the provider and model name from the request. The corresponding model instance is initialized dynamically. The message is processed, and the response is sent back to the caller.
Nemo Integration
Nemo Guardrails integrates with LangChain using RunnableRails. However, this implementation comes with a key limitation — it does not allow independent control over the generation process. Instead, it enforces a workflow where both the Rails logic and text generation execute together.
Why a Custom Integration Was Needed:
Separation of Concerns: The generation process needed tool integration, whereas the Rails logic did not. Flexibility in Model Selection: The default implementation forced the use of a single model for both guardrails and text generation, which was restrictive.
To address these issues, a custom package was developed: langchain-guardrails (available on PyPI).
CI/CD
The entire implementation is automated using AWS Serverless Application Model (SAM) templates and GitHub Actions.
Key CI/CD Features:
Fully automated deployment pipeline AWS Secrets Manager for securely storing API keys and credentials Lambda runtime environment retrieves secrets dynamically at execution
Next Steps
This blog series will continue with:
Enhancing Routing Logic with more advanced decision-making. Dynamic Model Selection based on task complexity and cost. Integrating Tool Execution into the agent workflow.
Router Code Repo: https://github.com/skamalj/llmprovider.git