New repo - CloudWatch Logs to OpenSearch Serverless Analytics

[skyjasonaws]

Description

This project demonstrates how to create an automated pipeline to stream CloudWatch Logs into OpenSearch Serverless for real-time log analytics. CloudWatch Logs collects and streams log data. Amazon Data Firehose buffers and delivers to S3. A Lambda function transforms logs into JSON for OpenSearch. S3 and SQS enable durable storage and event notifications. OpenSearch Ingestion pulls logs and loads into a Serverless domain.

language

English

runtime

Python

Level

400

Type

Examples

Use case

Observability

Primary image

https://github.com/aws-samples/cloudwatch-to-opensearch/blob/main/architecture.png

IaC framework

AWS CloudFormation

AWS Serverless services used

• Amazon API Gateway
• Amazon DynamoDB
• Amazon EventBridge
• AWS IoT
• AWS Lambda
• Amazon Rekognition
• Amazon S3
• AWS Step Functions
• Amazon SNS
• Amazon SQS
• Amazon Transcribe
• Amazon Translate

Description headline

Stream CloudWatch Logs to OpenSearch Serverless for real-time log analytics using a serverless pipeline.

Repo URL

https://github.com/aws-samples/cloudwatch-to-opensearch

Additional resources

Links to additional information:

• CloudWatch Logs
• Kinesis Data Firehose product page
• OpenSearch Service
• AWS Lambda product page
• Amazon S3 product page
• Amazon SQS product page
• OpenSearch Ingestion

Author Name

Jason Salcido

Author Image URL

https://avatars.githubusercontent.com/u/151649141?v=4

Author Bio

Jason Salcido, Senior Startup Solutions Architect at Amazon Web Services, leverages his deep expertise in cloud, analytics, integration, and data security to help clients across industries like financial services, insurance, energy, and utilities transform through technology. With nearly 30 years as a visionary technology leader, he excels at understanding needs and designing elegant solutions. Jason builds trusted advisor relationships through his technical excellence, strategic perspective, passion, and ability to inspire teams. His comprehensive enterprise architectures and innovative prototypes drive value for global organizations.

Author Twitter handle

No response

Author LinkedIn URL

https://www.linkedin.com/in/salcido/

leave

No response

[julianwood]

Added