Add a new section on how to collect AWS Network Firewall using Firehose #3885

alaudazzi · 2024-05-13T14:53:58Z

This PR:

is still WIP
adds a new Firehose use case on Network Firewall logs.
reuses content drafted in Figure out how to collect AWS Network Firewall using Amazon Data Firehose zmoog/public-notes#84.

Doc preview

Closes #3881

github-actions · 2024-05-13T14:54:16Z

A documentation preview will be available soon.

🔨 Buildkite builds
📚 HTML diff
📙 Preview page

Request a new doc build by commenting

Rebuild this PR: run docs-build
Rebuild this PR and all Elastic docs: run docs-build rebuild

_{run docs-build is much faster than run docs-build rebuild. A rebuild should only be needed in rare situations.}

_{If your PR continues to fail for an unknown reason, the doc build pipeline may be broken. Elastic employees can check the pipeline status here.}

zmoog · 2024-05-14T14:43:45Z

docs/en/observability/cloud-monitoring/aws/monitor-aws-firewall-firehose.asciidoc

+== Step 2: Select a resource
+
+You can either use an existing AWS Network Firewall, or create a new one.
+
+To create a new AWS Network Firewall, follow these steps:
+
+. Open the VPC service in the AWS console.
+
+The best option to create a VPC for a quick test is to use the wizard in the AWS console.
+
+. Create a VPC and other networking resources.
+
+Leave the default settings and choose a name for your VPC resources.
+
+[role="screenshot"]
+image::firehose-firewall-vpc-resources.png[Firewall VPC resources]
+
+. Create an AWS Network Firewall
+
+. Set up the firewall policy by creating a rule group.
+
+. Deploy an EC2 to generate network traffic.
+
+Launch an EC2, select the VPC you have just created, and enable *Auto-assign public IP*.


@alaudazzi, as we discussed offline, in the first version of this tutorial, we can assume the reader already has a Network Firewall producing logs they want to forward to an Elastic stack using Firehose. Creating a VPC for testing a Network Firewall is nontrivial and would take over the tutorial.

I am working on a dedicated zmoog/public-notes#87 to set up a testing Network Firewall using Terraform, so we can provide a one-step recipe to create a VPC with a Network Firewall for testing.

zmoog · 2024-05-14T15:00:14Z

docs/en/observability/cloud-monitoring/aws/monitor-aws-firewall-firehose.asciidoc

+. Click *Create Firehose stream* and choose the source and destination of your Firehose stream. Unless you are streaming data from Kinesis Data Streams, set source to `Direct PUT` and destination to `Elastic`. 
+
+. Provide a meaningful *Firehose stream name* that will allow you to identify this delivery stream later. Your Firehose name must start with the prefix `aws-waf-logs-` or it will not show up later.
+
+NOTE: For advanced use cases, source records can be transformed by invoking a custom Lambda function. When using Elastic integrations, this should not be required.
+


Creating a Firehose stream to forward Network Firewall is similar to the CloudTrail step.

Suggested change

. Click *Create Firehose stream* and choose the source and destination of your Firehose stream. Unless you are streaming data from Kinesis Data Streams, set source to `Direct PUT` and destination to `Elastic`.

. Provide a meaningful *Firehose stream name* that will allow you to identify this delivery stream later. Your Firehose name must start with the prefix `aws-waf-logs-` or it will not show up later.

NOTE: For advanced use cases, source records can be transformed by invoking a custom Lambda function. When using Elastic integrations, this should not be required.

For more information on how to set up a Amazon Data Firehose delivery stream to send data to Elastic Cloud, you can also check the <<monitor-aws-firehose,setup guide>>.

. Click *Create Firehose stream* and choose the source and destination of your Firehose stream. Set source to `Direct PUT` and destination to `Elastic`.

. Collect {es} endpoint and API key from your deployment on Elastic Cloud.

- Elastic endpoint URL: Enter the Elasticsearch endpoint URL of your Elasticsearch cluster. To find the Elasticsearch endpoint, go to the Elastic Cloud console and select *Connection details*.

- API key: Enter the encoded Elastic API key. To create an API key, go to the Elastic Cloud console, select *Connection details* and click *Create and manage API keys*. If you are using an API key with *Restrict privileges*, make sure to review the Indices privileges to provide at least "auto_configure" & "write" permissions for the indices you will be using with this delivery stream.

. Set up the delivery stream by specifying the following data:

+

- Elastic endpoint URL

- API key

- Content encoding: gzip

- Retry duration: 60 (default)

- Parameters:

- es_datastream_name: `logs-aws.firewall_logs-default`

- Backup settings: failed data only to s3 bucket

zmoog · 2024-05-14T15:32:40Z

Here are a draft for the step four:

[discrete]
[[firehose-cloudtrail-step-four]]
== Step 4: Enable logging

The AWS Network Firewall logs has logging support built in. It supports sending logs to Amazon S3, Amazon CloudWatch, and Amazon Kinesis Data Firehose.

To enable logging to Amazon Data Firehose:

- In the AWS console, navigate to the AWS Network Firewall service.
- Select the firewall you want to enable logging for.
- In the *Logging* section, click *Edit*.
- Select the *Send logs to* option and choose *Kinesis Data Firehose*.
- Select the Firehose stream you created in the previous step.
- Click *Save*.

A note on the "Select the Send logs to option and choose Kinesis Data Firehose" step. Today, the AWS console still uses the old name "Kinesis Data Firehose" instead of the updated "Amazon Data Firehose".

So this is probably one of those cases where we need to abstract the guide away from the UI details.

zmoog · 2024-05-14T15:42:39Z

docs/en/observability/cloud-monitoring/aws/monitor-aws-firewall-firehose.asciidoc

+. Set up logging.
+
+Open the *Logging* section to edit your firewall settings. If you want to quickly check your Network Firewall logs before setting up Firehose, you can enable logging on CloudWatch, and then inspect the log events:
+
+[role="screenshot"]
+image::firehose-firewall-logging.png[Firewall setup logging]
+
+. Visit CloudWatch and open your log group. If everything is working correctly, you will get the list of log events:
+
+[role="screenshot"]
+image::firehose-cloudwatch-log-events.png[CloudWatch Log events]
+
+[discrete]
+[[firehose-firewall-step-three]]


If you want to quickly check your Network Firewall logs before setting up Firehose, you can enable logging on CloudWatch, and then inspect the log events:

After rereading this part, I realized it isn't worth enabling logging on to CloudWatch. Enabling logging on CloudWatch brings value if we set up our test Network Firewall, and we want to double-check that our setup is sound and that it's actually logging data.

Since we assume the reader already has a working Network Firewall, I suggest setting up the logging to Firehose in step four.

alaudazzi · 2024-05-15T17:51:02Z

Thank you for your comments @zmoog. I'll be on PTO for the next two weeks, in case you need support you can reach out to @dedemorton (thank you DeDe!)

- Drop how to create a network firewall; it's too complex to include in guide. - Expand the guide with the missing content.

mergify · 2024-05-29T08:58:19Z

This pull request is now in conflict. Could you fix it @alaudazzi? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b network-firewall upstream/network-firewall
git merge upstream/main
git push upstream network-firewall

zmoog · 2024-05-29T09:08:16Z

docs/en/observability/cloud-monitoring/aws/monitor-aws-firewall-firehose.asciidoc

+[role="screenshot"]
+image::firehose-networkfirewall-firewall.png[AWS Network Firewall]
+
+You can either use an existing AWS Network Firewall, or create a new one for testing purposes.
+
+Creating a Network Firewall is not trivial and is beyond the scope of this guide. For more information, see the AWS documentation on the https://docs.aws.amazon.com/network-firewall/latest/developerguide/getting-started.html[Getting started with AWS Network Firewall] guide.
+


@dedemorton, explaining how to set up a Network Firewall for testing would be 2x the size of this guide.

So, I guess we have (at least) two options:

say, "it's beyond the scope of this guide."

link to an external source

I created a Terraform module at https://github.com/zmoog/integrations-cookbook/pull/1/files to test the Network Firewall for this guide. The module would allow users to set up the entire Network Firewall in one step.

However, (1) the module is in draft on (2) my personal repo. For the draft, I can finish it up quickly. But I can't link to my personal repo from the official Elastic docs.

What are our options here? Can we deliver the Terraform module through an official Elastic resource? Any suggestion is welcome!

Good question! TBH it's been quite awhile since I've provided users with links to special resources. Does it need to be a GitHub repo? I know in the past we've used download.elastic.co to provide sample data for guides, but this seems different. @bmorelli25 Do you have any ideas for the best way to deliver the Terraform module that Maurizio created for testing the network firewall?

Hi @zmoog. Yes, we have the https://github.com/elastic/observability-examples repo for exactly this purpose.

bmorelli25 · 2024-05-30T19:15:48Z

docs/en/observability/cloud-monitoring/aws/monitor-aws-firewall-firehose.asciidoc

+[[firehose-firewall-step-two]]
+== Step 2: Select a resource
+
+[role="screenshot"]


These will look better without our screenshot CSS since they're on a white background

Suggested change

[role="screenshot"]

bmorelli25 · 2024-05-30T19:16:03Z

docs/en/observability/cloud-monitoring/aws/monitor-aws-firewall-firehose.asciidoc

+[[firehose-firewall-step-three]]
+== Step 3: Create a stream in Amazon Data Firehose
+
+[role="screenshot"]


Suggested change

[role="screenshot"]

bmorelli25 · 2024-05-30T19:16:09Z

docs/en/observability/cloud-monitoring/aws/monitor-aws-firewall-firehose.asciidoc

+[[firehose-firewall-step-four]]
+== Step 4: Enable logging
+
+[role="screenshot"]


Suggested change

[role="screenshot"]

bmorelli25 · 2024-05-30T19:16:31Z

docs/en/observability/cloud-monitoring/aws/monitor-aws-firewall-firehose.asciidoc

+
+- *Visualize your logs with Discover*
+
+image::firehose-networkfirewall-discover.png[Visualize Network Firewall logs with Discover]


And we should add it here because this one is a screenshot

Suggested change

image::firehose-networkfirewall-discover.png[Visualize Network Firewall logs with Discover]

[role="screenshot"]

image::firehose-networkfirewall-discover.png[Visualize Network Firewall logs with Discover]

bmorelli25 · 2024-05-30T19:17:58Z

docs/en/observability/cloud-monitoring/aws/monitor-aws-firewall-firehose.asciidoc

+Navigate to {kib} and choose among the following monitoring options:
+
+- *Visualize your logs with Discover*


We try to avoid lists of one. How about this?

Suggested change

Navigate to {kib} and choose among the following monitoring options:

- *Visualize your logs with Discover*

Navigate to {kib} and choose *Visualize your logs with Discover*.

bmorelli25 · 2024-05-30T19:18:29Z

docs/en/observability/cloud-monitoring/aws/monitor-aws-firewall-firehose.asciidoc

+With the new logging settings in place, the Network Firewall starts sending log events to the Firehose stream.
+
+image::firehose-networkfirewall-data-stream.png[Firehose monitor Network Firewall logs]


Move image to the top like in the other sections.

Suggested change

With the new logging settings in place, the Network Firewall starts sending log events to the Firehose stream.

image::firehose-networkfirewall-data-stream.png[Firehose monitor Network Firewall logs]

image::firehose-networkfirewall-data-stream.png[Firehose monitor Network Firewall logs]

With the new logging settings in place, the Network Firewall starts sending log events to the Firehose stream.

Start initial structure

c9ea62d

alaudazzi added docs Improvements or additions to documentation enhancement New feature or request backport-8.13 Automated backport with mergify backport-8.14 Automated backport with mergify labels May 13, 2024

alaudazzi self-assigned this May 13, 2024

alaudazzi requested a review from zmoog May 13, 2024 14:54

zmoog reviewed May 14, 2024

View reviewed changes

Expand the aws network firewall guide

a1b6ad2

- Drop how to create a network firewall; it's too complex to include in guide. - Expand the guide with the missing content.

zmoog reviewed May 29, 2024

View reviewed changes

zmoog marked this pull request as ready for review May 29, 2024 09:08

zmoog requested a review from a team as a code owner May 29, 2024 09:08

Merge branch 'main' into network-firewall

afe3c2c

bmorelli25 reviewed May 30, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a new section on how to collect AWS Network Firewall using Firehose #3885

Add a new section on how to collect AWS Network Firewall using Firehose #3885

alaudazzi commented May 13, 2024 •

edited

github-actions bot commented May 13, 2024

zmoog May 14, 2024

zmoog May 14, 2024

zmoog commented May 14, 2024 •

edited

zmoog May 14, 2024

alaudazzi commented May 15, 2024

mergify bot commented May 29, 2024

zmoog May 29, 2024

dedemorton May 29, 2024

bmorelli25 May 29, 2024

bmorelli25 May 30, 2024

bmorelli25 May 30, 2024

bmorelli25 May 30, 2024

bmorelli25 May 30, 2024

bmorelli25 May 30, 2024

bmorelli25 May 30, 2024

-. Click *Create Firehose stream* and choose the source and destination of your Firehose stream. Unless you are streaming data from Kinesis Data Streams, set source to `Direct PUT` and destination to `Elastic`.
-. Provide a meaningful *Firehose stream name* that will allow you to identify this delivery stream later. Your Firehose name must start with the prefix `aws-waf-logs-` or it will not show up later.
-NOTE: For advanced use cases, source records can be transformed by invoking a custom Lambda function. When using Elastic integrations, this should not be required.
+For more information on how to set up a Amazon Data Firehose delivery stream to send data to Elastic Cloud, you can also check the <<monitor-aws-firehose,setup guide>>.
+. Click *Create Firehose stream* and choose the source and destination of your Firehose stream. Set source to `Direct PUT` and destination to `Elastic`.
+. Collect {es} endpoint and API key from your deployment on Elastic Cloud.
+- Elastic endpoint URL: Enter the Elasticsearch endpoint URL of your Elasticsearch cluster. To find the Elasticsearch endpoint, go to the Elastic Cloud console and select *Connection details*.
+- API key: Enter the encoded Elastic API key. To create an API key, go to the Elastic Cloud console, select *Connection details* and click *Create and manage API keys*. If you are using an API key with *Restrict privileges*, make sure to review the Indices privileges to provide at least "auto_configure" & "write" permissions for the indices you will be using with this delivery stream.
+. Set up the delivery stream by specifying the following data:
++
+- Elastic endpoint URL
+- API key
+- Content encoding: gzip
+- Retry duration: 60 (default)
+- Parameters:
+  - es_datastream_name: `logs-aws.firewall_logs-default`
+- Backup settings: failed data only to s3 bucket

	image::firehose-networkfirewall-discover.png[Visualize Network Firewall logs with Discover]
	[role="screenshot"]
	image::firehose-networkfirewall-discover.png[Visualize Network Firewall logs with Discover]

		Navigate to {kib} and choose among the following monitoring options:

		- Visualize your logs with Discover

		With the new logging settings in place, the Network Firewall starts sending log events to the Firehose stream.

		image::firehose-networkfirewall-data-stream.png[Firehose monitor Network Firewall logs]

Add a new section on how to collect AWS Network Firewall using Firehose #3885

Are you sure you want to change the base?

Add a new section on how to collect AWS Network Firewall using Firehose #3885

Conversation

alaudazzi commented May 13, 2024 • edited

github-actions bot commented May 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zmoog commented May 14, 2024 • edited

Choose a reason for hiding this comment

alaudazzi commented May 15, 2024

mergify bot commented May 29, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alaudazzi commented May 13, 2024 •

edited

zmoog commented May 14, 2024 •

edited