-
Notifications
You must be signed in to change notification settings - Fork 5.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Each DynamoDB stream event handler duplicates the IAM policy, causing "Maximum policy size exceeded" #12313
Comments
Assuming I'm looking at the right code, it's not obvious why this happens:
The above code seems to add each resource to the same statement (although without any deduping based on the resource, from what I can see). I don't see how we end up getting multiple statements. |
Could it be that CloudFormation (perhaps when doing an incremental deployment) appends rather than overwrites the policy? |
The IAM policy editor shows "Suggestions: Redundant Statement", so it agrees these statements are redundant. |
Manually deleting all sections containing |
My workaround from #12313 (comment) no longer works. I'm not sure why it ever did. Deploys are now completely blocked due to the number of This is a rather serious problem. Our serverless stack is no longer deployable and we need to consider rather undesirable workarounds, like manually merging Lambdas to reduce the size of the IAM document. |
I tried a workaround by using a custom role, however that doesn't help as the default |
I've updated to the latest version and confirmed the problem persists there:
|
…erless#12313) Each function consuming a stream event would emit its own PolicyDocument statement. This statement would contain a list of actions that doesn't change between functions. For DynamoDB streams the list is: ``` "Action": [ "dynamodb:GetRecords", "dynamodb:GetShardIterator", "dynamodb:DescribeStream", "dynamodb:ListStreams" ], ``` Duplicating these for each function causes the IAM policy to exceed the AWS limit after about 30 functions. The resource names are still duplicated, if they happen to be the same.
Upon reading the code again it's clear what's going on. We're adding the same statement (in particular, the same list of allowed actions) for each event stream processing function. This is unnecessary and only the list of resources needs to be updated per function. Fix: #12320 |
We're currently running a patched version of serverless to be able to deploy to production due to this bug. Could someone please take a look at the provided PR with the fix? |
This is a production blocking issue for larger scale users of serverless with a simple fix provided as a PR. Could someone please take a look? |
Are you certain it's a bug?
Is the issue caused by a plugin?
Are you using the latest v3 release?
Is there an existing issue for this?
Issue description
We recently started seeing a deployment error
Debugging further I noticed the same IAM Role section repeated 29 times (2 repeats shown below for demonstration purposes) in the generated
backend-xxxxxx-eu-central-1-lambdaRole
role:(The resource is the same in each section so this is truly duplicated.)
It's seem like that for each
section in the YAML config we get a new repeat of the above policy, eventually causing the size to go over the limit.
Service configuration (serverless.yml) content
(We have 29 of these in our config.)
Command name and used flags
serverless deploy --stage=xxxxxx
Command output
UPDATE_FAILED: IamRoleLambdaExecution (AWS::IAM::Role) Resource handler returned message: "Maximum policy size of 10240 bytes exceeded for role backend-xxxxxx-eu-central-1-lambdaRole (Service: Iam, Status Code: 409, Request ID: 0ab39c5b-fef4-491b-be1d-d1a730xxxxxx)" (RequestToken: 7591a9ce-5117-b2b5-c915-161cb4xxxxxx, HandlerErrorCode: ServiceLimitExceeded)
Environment information
The text was updated successfully, but these errors were encountered: