Add a new section on how to collect any log from CloudWatch using Firehose #3887

alaudazzi · 2024-05-14T06:50:53Z

This PR:

Adds a new Firehose use case on Cloudwatch logs.
Reuses content drafted in Figure out how to collect any log from CloudWatch using Amazon Data Firehose zmoog/public-notes#86.

Closes #3882

github-actions · 2024-05-14T06:51:07Z

A documentation preview will be available soon.

🔨 Buildkite builds
📚 HTML diff
📙 Preview page

Request a new doc build by commenting

Rebuild this PR: run docs-build
Rebuild this PR and all Elastic docs: run docs-build rebuild

_{run docs-build is much faster than run docs-build rebuild. A rebuild should only be needed in rare situations.}

_{If your PR continues to fail for an unknown reason, the doc build pipeline may be broken. Elastic employees can check the pipeline status here.}

mergify · 2024-05-28T14:11:01Z

This pull request is now in conflict. Could you fix it @alaudazzi? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b cloudwatch-firehose upstream/cloudwatch-firehose
git merge upstream/main
git push upstream cloudwatch-firehose

zmoog

I expanded steps 2-5 with the new content I added to the public note we used as a raw material source for this topic.

I think it is ready for a review round! 😇

dedemorton

Added some editorial contents. Some of the comments relate to restructuring. If you need help, you can reach out in the obs-docs channel in slack. Hope this is helpful!

docs/en/observability/cloud-monitoring/aws/monitor-aws-cloudwatch-firehose.asciidoc

dedemorton · 2024-05-28T19:01:45Z

docs/en/observability/cloud-monitoring/aws/monitor-aws-cloudwatch-firehose.asciidoc

+
+In this tutorial, we will write a simple AWS Lambda-based app, collect its application logs, and forward them to Elastic. 
+
+Like many other services and platforms in AWS, Lambda functions natively log directly to CloudWatch out of the box. Lambda functions are a great tool for experimenting on AWS.


I don't think this sentence is relevant or necessary. I would remove it: "Lambda functions are a great tool for experimenting on AWS."

Yeah, I see. My original intent was to explain why we picked up lambda for this tutorial, but I agree it is irrelevant and adds too little value to stay.

docs/en/observability/cloud-monitoring/aws/monitor-aws-cloudwatch-firehose.asciidoc

Co-authored-by: DeDe Morton <dede.morton@elastic.co>

zmoog · 2024-05-28T21:03:53Z

@dedemorton, thank you for your helpful review comments and suggestions!

I added all the suggestions and addressed the comments on the structure. Let me know what you think when you have time.

These diagrams help orient users in their journey.

bmorelli25

A couple more comments for ya!

bmorelli25 · 2024-05-30T19:02:40Z

docs/en/observability/cloud-monitoring/aws/monitor-aws-cloudwatch-firehose.asciidoc

+
+1. Visit the AWS web console and open the AWS Lambda page.
+2. Click **Create function** and select the option to create a function from scratch.
+3. Select a **Function name**


For consistency with other steps

Suggested change

3. Select a **Function name**

3. Select a **Function name**.

bmorelli25 · 2024-05-30T19:04:48Z

docs/en/observability/cloud-monitoring/aws/monitor-aws-cloudwatch-firehose.asciidoc

+Set the **Parameters** in the **Destination settings** section.
+
+[discrete]
+[[firehose-cloudwatch-step-three-desination-settings-parameters]]
+=== Parameters
+
+Use the following parameters:


It feels like this line should be in the next section?

Suggested change

Set the **Parameters** in the **Destination settings** section.

[discrete]

[[firehose-cloudwatch-step-three-desination-settings-parameters]]

=== Parameters

Use the following parameters:

[discrete]

[[firehose-cloudwatch-step-three-desination-settings-parameters]]

=== Parameters

Set the **Parameters** in the **Destination settings** section.

Use the following parameters:

bmorelli25 · 2024-05-30T19:08:52Z

docs/en/observability/cloud-monitoring/aws/monitor-aws-cloudwatch-firehose.asciidoc

+
+[discrete]
+[[firehose-cloudwatch-step-four-subscription-filter-destination]]
+==== Choose destination


These headings don't display correctly but that's a CSS bug. We have a PR open in elastic/docs to fix this.

bmorelli25 · 2024-05-30T19:09:27Z

docs/en/observability/cloud-monitoring/aws/monitor-aws-cloudwatch-firehose.asciidoc

+
+On the AWS console, visit your Firehose stream and check for entries in the *Destination error logs* section.
+
+If everything is running smoothly, this list will be empty. If there's an error, you can check the details. Here is a delivery stream that fails to send records to the Elastic stack due to bad authentication settings:


This is a good example to include 👍

Start initial structure

b8193ce

alaudazzi added docs Improvements or additions to documentation enhancement New feature or request backport-8.13 Automated backport with mergify backport-8.14 Automated backport with mergify labels May 14, 2024

alaudazzi self-assigned this May 14, 2024

alaudazzi requested a review from zmoog May 14, 2024 06:51

Expand steps 2-5

f8dd023

zmoog and others added 2 commits May 28, 2024 18:15

Cleanup

632b9fb

Merge branch 'main' into cloudwatch-firehose

79b7f50

zmoog marked this pull request as ready for review May 28, 2024 16:19

zmoog requested a review from a team as a code owner May 28, 2024 16:19

zmoog requested a review from dedemorton May 28, 2024 16:19

zmoog reviewed May 28, 2024

View reviewed changes

dedemorton reviewed May 28, 2024

View reviewed changes

zmoog and others added 3 commits May 28, 2024 22:30

Apply suggestions from code review

65455e4

Co-authored-by: DeDe Morton <dede.morton@elastic.co>

Apply suggestions from code review

20d3f0b

Apply suggestions from code review

73661b0

Add progressing diagram to steps 2-5

fef51ce

These diagrams help orient users in their journey.

bmorelli25 reviewed May 30, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a new section on how to collect any log from CloudWatch using Firehose #3887

Add a new section on how to collect any log from CloudWatch using Firehose #3887

alaudazzi commented May 14, 2024 •

edited by zmoog

github-actions bot commented May 14, 2024

mergify bot commented May 28, 2024

zmoog left a comment

dedemorton left a comment

dedemorton May 28, 2024

zmoog May 28, 2024

zmoog commented May 28, 2024

bmorelli25 left a comment

bmorelli25 May 30, 2024

bmorelli25 May 30, 2024

bmorelli25 May 30, 2024

bmorelli25 May 30, 2024


		In this tutorial, we will write a simple AWS Lambda-based app, collect its application logs, and forward them to Elastic.

		Like many other services and platforms in AWS, Lambda functions natively log directly to CloudWatch out of the box. Lambda functions are a great tool for experimenting on AWS.


		On the AWS console, visit your Firehose stream and check for entries in the Destination error logs section.

		If everything is running smoothly, this list will be empty. If there's an error, you can check the details. Here is a delivery stream that fails to send records to the Elastic stack due to bad authentication settings:

Add a new section on how to collect any log from CloudWatch using Firehose #3887

Are you sure you want to change the base?

Add a new section on how to collect any log from CloudWatch using Firehose #3887

Conversation

alaudazzi commented May 14, 2024 • edited by zmoog

github-actions bot commented May 14, 2024

mergify bot commented May 28, 2024

zmoog left a comment

Choose a reason for hiding this comment

dedemorton left a comment

Choose a reason for hiding this comment

dedemorton May 28, 2024

Choose a reason for hiding this comment

zmoog May 28, 2024

Choose a reason for hiding this comment

zmoog commented May 28, 2024

bmorelli25 left a comment

Choose a reason for hiding this comment

bmorelli25 May 30, 2024

Choose a reason for hiding this comment

bmorelli25 May 30, 2024

Choose a reason for hiding this comment

bmorelli25 May 30, 2024

Choose a reason for hiding this comment

bmorelli25 May 30, 2024

Choose a reason for hiding this comment

alaudazzi commented May 14, 2024 •

edited by zmoog