Navigating High-Volume Messaging Challenges in AWS Lambda Using ReportBatchItemFailures

Introduction

Discover the challenge of managing thousands of messages in AWS Lambda and the solution that ReportBatchItemFailures offers for better performance.

Our Composition

We want to handle high-volume messages in the below component.
But we can find some metrics are not fine which are NumberOfMessagesReceived for SQS, and Invocations for lambda.

Implement ReportBatchItemFailures in Lambda

When your Lambda function encounters an error while processing a batch, all messages in that batch become visible in the queue again by default, including messages that Lambda processed successfully.

Amazon Web Services AWS Lambda Developer Guide. Retrieved from https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting

If Lambda encounters an error, the entire batch of records is marked as failed. This means that if just one record has an issue, the whole batch is considered unsuccessful.

To avoid the above event, we can implement ReportBatchItemFailures.
In this case, what we have to do is below two things.

1. Add the function_response_types in the source mapping
2. Implement the response in Lambda

resource "aws_lambda_event_source_mapping" "example" {
  batch_size           = 5
  event_source_arn     = var.event_source_arn
  function_name        = aws_lambda_function.example.arn
  function_response_types = ["ReportBatchItemFailures"] # add
  enabled              = true
}

// SqsBatchResponse レスポンス
type SqsBatchResponse struct {
    BatchItemFailures []BatchItemFailure `json:"batchItemFailures"`
}

type BatchItemFailure struct {
    ItemIdentifier string `json:"itemIdentifier"`
}

events

Using Lambda with Amazon SQS

Decrease error in Lambda

If your function code caused the error, Lambda gradually backs off retries by reducing the amount of concurrency allocated to your Amazon SQS event source mapping

Amazon Web Services AWS Lambda Developer Guide. Retrieved from https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/with-sqs.html#example-standard-queue-message-event

This means Lambda error numbers affect concurrency. So we revise the error response in Lambda.

Conclusion

In conclusion, using ReportBatchItemFailures helps us accurately track message and invocation metrics in AWS Lambda and SQS, solving our high-volume messaging issues efficiently.

Reference

SQSから10000レコードのバッチをLambda + Goで高速で処理する方法 - voicy tech blog