Connecting a Lambda to Elasticache/Redis is not your typical use case. Elasticache was initially built to be used by EC2 instances. In 2016 / 2017 AWS added the ability to connect to Elasticache via VPCs and thus a way in with your Lambdas, because Lambdas can also run in VPCs. Run both the Lambda and the Elasticache instance in the same VPC with the right security groups and you’ll be able to connect to an Elasticache/Redis instance from a Lambda.
Below is a walkthrough of how to set up this connectivity with some sample code.
Walkthrough - Creating a VPC to run your Lambda and Elasticache in:
Please understand this is my freshman level setup of a VPC to allow for connecting a Lambda to an Elasticache-Redis instance. It gets the job done but should be tightened down from a security standpoint (not in this document).
From a high level you’ll need to:
1. Create the VPC
2. Create the VPCs Subnets
3. Create a Route table for the VPC
4. Create the Security group for the VPC
5. (optionally) Create Endpoints to different AWS services if needed
6. (optionally) Replace the auto created Network ACL with a properly named version
Here are the steps:
- Navigate to the VPC Dashboard in the AWS Console
- Choose the Create button to create a new VPC instance:
- Choose “Yes, Create” to create the VPC
- The above CIDR block is “private” and can be used in your environment per:
- https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Subnets.html
- http://www.faqs.org/rfcs/rfc1918.html
- You could use the 10.* or 172.16.* block spaces as well. Other than those ranges I am a noob in the VPC space.
- Create at least two subnets:
- Names are important. Probably a good idea to include the name of the Availability Zone in the name tag along with the VPC name for easy reference later.
- Choose the VPC created.
- Give it a unique CIDR block
- For example, I created AppV3_east1a and AppV3_east1b with CIDR blocks of 192.168.0.0/25 and 192.168.1.0/26 against us-east-1a and us-east-1b.
- Create a Route Table:
- After choosing Yes, Create, configure the route table as follows (add our two subnets to it):
-
- Create a VPC security group which our Lambda will use:
- Adjust Inbound Rules, then save. Use 0.0.0.0/0 to get past this:
- Note: May be a good idea to lock this down further but that is beyond the scope of this document.
- Optionally:
- The VPC will automatically create a Network ACL for your VPC with the subnets you’ve defined. I added one anyways, named it AppV4_NACL, and assigned the new one the endpoints from the auto created one and gave it the same inbound/outbound rules.
Walkthrough: Create Elasticache Instance
- Navigate to the Elasticache Dashboard in the AWS Console
- (optional) Setup your Elasticache Subnet Group ahead of time
- From the Dashboard select Subnet Groups
- Choose the Create button and enter the following:
-
- Choose “Create” to save the Subnet Group.
- Note: The “Name” field only likes hyphens for special characters. On save it will “lower case” all your text.
- Navigate to the Redis section, then choose the Create button to create a new Elasticache-Redis instance
- Enter the following info:
- Note 1: For sample and cost sake choose a small node type of cache.t2.micro
- Note 2: For sample sake use “None”
-
- Choose a Subnet group (this is the VPC that you’ll run your Lambda and Elasticache Instance from):
- If you’ve already got a VPC setup then try using that first, else if you have none then you’ll need to create one which is beyond the scope of this document. (just learned about VPCs, so go to the first Walkthrough for details on VPC setup)
- i. The trick is to run both the Lambda and Elasticache/Redis instance in this VPC.
- You can choose the group defined above in Elasticache Subnet Group instructions above or perform the Elasticache Subnet Group setup here.
-
- Setup your security so that it has enough access:
- Note 1: I chose a security group defined by organization called “Allow All”. For testing purposes I start wide open then dial it back after it is working.
- Note 2: If you’ve got a new AWS account then you’ll probably only have the default account to begin with. Try that to see if that works. Else follow the VPC setup instructions at the beginning of this document.
-
- Leave all other options as default and choose Create to create the instance.
Walkthrough: Create Lambda
This section assumes you are able to publish your Lambdas to the AWS environment from within Visual Studio.
- Create a new Lambda Project
- Add the “StackExchange.Redis” NuGet package so you can connect to the Elasticache/Redis instance from you’re Lambda. :
- Note 1: The AWSSDK.Elasticache Nuget package is meant to manage the infrastructure (Parameter Groups, Subnets, Clusters, etc) of Elasticache and does not assist with actually working with Redis SortedSets, Hashes, Strings, etc. To do that you have to use a third party NuGet package as I am not aware of any AWS sponsored packages that do this at this time. For this example, I chose “StackExchange.Redis”. For a list of others go to: https://redis.io/clients#c. AWS may have one for Memcached though but that is out of scope for this document. I want to use the SortedSet functionality, so the focus of this document is Redis.
- Note 2: Do not choose a StackExchange.Redis higher than 1.2.1 for .Net Core 1.0 deployments. Versions greater than 1.2.1 are not supported in AWS Lambdas using .Net Core 1.0. Version 1.2.3 attempts to load .netcore 1.1. AWS recently added support for .Net Framework 2.0, so Version 1.2.6 may be a good choice for that but I have not explored that option yet:
-
- For example when I choose 1.2.3 and choose Update we get the following “Review Changes” screen:
-
- I chose Cancel at this screen because I want to remain on the AWS SDK for .Net Lambdas which is 1.0.2.
- 3. Use this code to get started in your Lambda (yeah this is ugly):
using System; using System.Collections.Generic; using System.Linq; using System.Threading.Tasks; using Amazon.Lambda.Core; using StackExchange.Redis; using System.Net; // Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class. [assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.Json.JsonSerializer))] namespace TestElasticacheRedis { public class RedisTest { private ConnectionMultiplexer _elasticacheConnection = null; public void FunctionHandler(ILambdaContext context) { System.Text.StringBuilder sb = new System.Text.StringBuilder(); Console.WriteLine("Starting..."); if (_elasticacheConnection == null) { _elasticacheConnection = GetConnection(); } Console.WriteLine("After Connection"); var db = _elasticacheConnection.GetDatabase(); Console.Write("After GetDatabase, "); Console.WriteLine("isConnected:" + db.IsConnected("timeSorted") + ", dbinstance: " + db.Database); TimeSpan t = DateTime.UtcNow - new DateTime(1970, 1, 1); var score = t.TotalMilliseconds; //Add to 'timeSorted', store a value of System.Guid, and sort by t.TotalMilliseconds. // Search by t.TotalMilliseconds for fast lookup. db.SortedSetAdd("timeSorted", System.Guid.NewGuid().ToString(), score); var byScan = db.SortedSetScan("timeSorted"); Console.WriteLine("Scan: " + string.Join(",\n", byScan)); var byRank = db.SortedSetRangeByRank("timeSorted"); Console.WriteLine("Rank: " + string.Join(",\n", byRank)); var byScore = db.SortedSetRangeByScore("timeSorted", score, 1, Exclude.None, Order.Ascending); Console.WriteLine("Score: " + string.Join(",\n", byScore)); Console.WriteLine("After sortedsetadd"); } private ConnectionMultiplexer GetConnection() { //because of https://github.com/dotnet/corefx/issues/8768 var addresses = Dns.GetHostAddressesAsync(Environment.GetEnvironmentVariable("ElasticacheConnectionString")).Result; var ip4Adresses = addresses.Select(x => x.MapToIPv4()).Distinct().ToList(); var config = new ConfigurationOptions { AllowAdmin = true, AbortOnConnectFail = false, }; ip4Adresses.ForEach(ip => config.EndPoints.Add(ip, 6379)); return ConnectionMultiplexer.Connect(config); } } }- Note: Be sure to plug in the port of 6379 and the address of your Elasticache in the Environment Variables for the Lambda. You can find the address here:
-
- Setup your IAM role to give your Lambda the ability to execute in AWS and to deploy to AWS:
- Setup a role with permissions similar to the below illustration.
-
- In order to deploy from Visual Studio to AWS, you’ll need the “ec2:CreateNetworkInterface” Action which the AWSLambdaVPCAccessExecutionRole will give you.
- Feel free to cut this down and build your own policy based on what you need vs what you see above.
- 5. Deploy your Lambda to AWS:
- Choose Next
- Populate the two VPC Subnets
- Choose the appropriate Security Group
- Enter the ElasticacheConnectionString as a Variable.
- Choose an IAM role with the following permissions:
-
- In order to deploy from Visual Studio to AWS, you’ll need the “ec2:CreateNetworkInterface” Action which the AWSLambdaVPCAccessExecutionRole will give you.
- At some point, cut this down and build your own policy based on what you need vs what you see above.
- Choose Upload to push the Lambda to your environment.
Troubleshooting:
- If you get a “No connection is available to service this operation”…“UnableToConnect” error, first verify that the IP address that is trying to connect to is in the VPC IP Address range you entered. If it isn’t then your Lambda is either:
- Not pointing to the correct Elasticache-Redis instance url
- The Elasticache-Redis instance is not running on the selected VPC
- The Lambda is not running on the selected VPC.
- Remember, both your Elasticache-Redis and Lambda components need to be setup to run in the same VPC.
- If you get a “No connection is available to service this operation”…”It was not possible to connect to the redis server(s)”, try the above then the below:
- I think both are running on the same VPC but we’re missing something.
- Navigate to the Elasticache Dashboard, then modify the Redis instance, verify the VPC Security Group selected is the “Allow All” we specified earlier.:
-
- If not, select the Allow All item and choose the Modify button to save the changes.
- Retest the Lambda.
- Still no go then in the AWS Console: (The Lambda seems to maintain some state with the VPCs selected):
- remove the regions in the lambda,
- save the lambda
- Add the regions under VPC back again
- Save the lambda
- Test the lambda again
