Resolve 'Lambda.InvokeAccessDenied' for error message 'Access was denied. Ensure that the access policy allows access to the Lambda function' from firehose to destination(same or cross-account S3)

Summary

The error can come because of two reasons either Firehose IAM Role has no invoke permissions on lambda transformation or if the permissions are correct, then the ProcessingConfiguration parameter “RoleArn” is referring to the wrong IAM Role ARN.

By default, if we setup Firehose using the default Firehose IAM Role and the default transformation lambda, the necessary permissions will be correctly configured automatically by the AWS service. However, when we need to configure Firehose using Cloud Formation there can be a possibility to run into this the issue./

Pre-requisite for delivery setup from Firehose to destination(same or cross-account S3)

The pre-requisite steps require to follow links [1] and [2], for setting up necessary permissions for Firehose delivery to destination(S3). Note that, Link [2] is only used when we need to deliver to cross-account S3 bucket. For S3 delivery in same account, just follow link[1].

Resolving the issue

If links[1] and/or [2] are not followed correctly to allow Lambda transformation function to be invoked via Firehose, then records are not being processed by the transformer lambda. These records are considered as unsuccessful processed records are delivered to S3 bucket in the processing-failed/ folder.

Downloading any one of the objects will have each JSON object as below.

{"attemptsMade":4,"arrivalTimestamp":1615490876740,"errorCode":"Lambda.InvokeAccessDenied","errorMessage":"Access was denied.Ensure that the access policy allows access to the Lambda function.","attemptEndingTimestamp":1615491074350,"rawData":"AAAAAAAARDIAAAAAAAAAAAAAAAAAA","lambdaArn":"arn:aws:lambda:us-east-1:XXXXXXXXXXXX:function:lambda-transformation-test-fn"}

The corresponding error can be seen in Firehose Console > “Destination Logs” tab as shown in below image

To resolve this issue, perform below steps:

1. Check if the Firehose IAM Role under Details > Permissions have permissions to invoke lambda function as per specified in link [1]. If you have S3 in cross-account ensure that the s3:PutObjectAcl is present in the Firehose IAM Role and there is bucket policy configured as per link [2].

Also, ensure that the correct lambda function version is being called. Check for the function version under Firehose IAM Role

{
           "Effect": "Allow", 
           "Action": [
               "lambda:InvokeFunction", 
               "lambda:GetFunctionConfiguration" 
           ],
           "Resource": [
               "arn:aws:lambda:region:account-id:function:function-name:function-version"
           ]
}

If all of the above is correct, then the IAM permissions are configured correctly . So check 2. for troubleshooting.

2. Confirm that the roleARN under s3DestinationDescription and extendedS3DestinationDescription and Under processingConfiguration > processors > parameters > parameterValue for parameterName RoleArn match.

Ensure that the roleARN specified is not the transformer lambda function Execution IAM Role. The execution IAM Role is the IAM Role assigned to the lambda function Under Lambda Console > Functions > (Select lambda function used for data transformation) > Configuration Tab > Permissions > Under Execution Role.

If the parameterValue for parameterName RoleARN is set to lambda execution role, then we need to update the firehose using CLI/API. Below is the approach for using CLI as below

a. Get the firehose configuration as per below ALI call

aws firehose describe-delivery-stream --delivery-stream-name <fh-name> --region <fh-region-name>

Extract details like “VersionId”, “DestinationId” and “LambdaArn” as we are going to need them while updating the Firehose configuration.

b. Since we want to update the firehose configuration, use the below CLI call

aws firehose update-destination --delivery-stream-name <fh-name> --region  <fh-region-name> --current-delivery-stream-version-id  1  --destination-id destinationId-000000000001 --extended-s3-destination-update BucketARN=arn:aws:s3:::fh-to-s3-cross-bkt,ProcessingConfiguration={Enabled=true,Processors=[{Type=Lambda,Parameters=[{ParameterName=RoleArn,ParameterValue=arn:aws:iam::XXXXXXXXXXXX:role/service-role/KinesisFirehoseServiceRole-fh-to-s3-us-west-2-0000000000000},{ParameterName=LambdaArn,ParameterValue=arn:aws:lambda:us-west-2:XXXXXXXXXXXX:function:lambda-transformation-test-fn:$LATEST}]}]}

Cross-check the highlighted portion in the above CLI is as per the required configuration.

c. Once you update the firehose configuration, re-execute the API call in a) to confirm the details provided in b) have been update.

3. If roleArn is same as mentioned in 2) , then check if there is any explicit deny that’s causing the permissions to fail.