Add enhanced alerting details to host monitoring docs

[dedemorton]

Closes #3747 and #3627.

• Newbie question: It's possible I don't understand this feature fully, but it seems like users need to group by hostname when they create the alert, or the alert will not appear in the Hosts view. Is this correct? If so, I think we should mention that somewhere. Since this setting is optional, users might not realize it's required if they want to view alerts for specific hosts. WDYT? Am I missing something here?

  

TODO (after merging):

• Port to serverless docs. https://github.com/elastic/staging-serverless-observability-docs/pull/256

[github-actions]

A documentation preview will be available soon.

• 🔨 Buildkite builds
• 📚 HTML diff
• 📙 Preview page

Request a new doc build by commenting

• Rebuild this PR: run docs-build
• Rebuild this PR and all Elastic docs: run docs-build rebuild

_{run docs-build is much faster than run docs-build rebuild. A rebuild should only be needed in rare situations.}

_{If your PR continues to fail for an unknown reason, the doc build pipeline may be broken. Elastic employees can check the pipeline status here.}

[mergify]

This pull request does not have a backport label. Could you fix it @dedemorton? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-/d./d is the label to automatically backport to the /d./d branch. /d is the digit
  NOTE: backport-skip has been added to this pull request.

[dedemorton]

@roshan-elastic @crespocarlos Can you take a look when you have a chance and respond to my question here? Thanks! (Sorry didn't mean to add your names to the related issue. Just got my browser tabs messed up.)

[roshan-elastic]

Hey @dedemorton

Closes #3747 and #3627.

Newbie question: It's possible I don't understand this feature fully, but it seems like users need to group by hostname when they create the alert, or the alert will not appear in the Hosts view. Is this correct? If so, I think we should mention that somewhere. Since this setting is optional, users might not realize it's required if they want to view alerts for specific hosts. WDYT? Am I missing something here?

TODO (after merging):

• Port to serverless docs.

Yeah, that's right. There are pretty much two ways for alerts to show up against hosts:

(1) They use the 'Inventory' rule and select the 'Host'

(2) If they use a rule which is possible to 'group by' host.name then it should show as well

e.g. metric threshold rules, custom threshold rules [beta]

Sample Metrics Threshold Rule
screenshot

Sample Custom Threshold Rule
screenshot

However, let me double-check with the engineers on this.

@crespocarlos (cc @jennypavlova) - When I test the custom threshold and metric threshold rules:

1. Metric Threshold doesn't appear to fire at all (so not sure if this will show as an alert in the host view)
2. The custom threshold rule fires but doesn't show in the host view...is this because we only show 'inventory' and 'metric threshold' rules?

Example Host

Only one alert showing in the host view

Here are the alerts set up:

See alert rules

Question

• Are you able to confirm the logic on which alerts show against a host (and how we determine that)?

[crespocarlos]

Hey @roshan-elastic

• Metric Threshold doesn't appear to fire at all (so not sure if this will show as an alert in the host view)

It was using host.cpu.pct field, which has no value. Was this intentional?

After adjusting it to use system.cpu.system.norm.pct field, the alert fired and appeared in the hosts view.

• The custom threshold rule fires but doesn't show in the host view...is this because we only show 'inventory' and 'metric threshold' rules?

Yeah. We might need to adjust something in the custom threshold executor for it to show up in the alerts table. The actionable observability owns that, but I guess we can submit a PR.

[mergify]

This pull request is now in conflict. Could you fix it @dedemorton? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b issue#3747 upstream/issue#3747
git merge upstream/main
git push upstream issue#3747

[dedemorton]

We might need to adjust something in the custom threshold executor for it to show up in the alerts table.

@crespocarlos Is there something I can say in the docs to help users? Maybe something like, "To see alerts on the Hosts page, the rule triggering the alert must either select hosts in a condition or be configured to group by hostname." Is this statement correct?

[crespocarlos]

Hi @dedemorton , regarding Custom Threshold it's something we need to fix in the code first, but it will work the same way as Metrics Threshold.

The conditions for users to see alerts in the Hosts view are - Similar to what you wrote, but it might help to specify what needs to be done depending on the rule type?

Metrics Threshold - When creating the alert, inform host.name on the group by field
Custom Threshold - When creating the alert, inform host.name on the group by field (When fixed)
Inventory - Select Host in Condition

[dedemorton]

@roshan-elastic @crespocarlos I've updated the documentation. Can you confirm my changes and approve the topic if you think it's ready to merge? Thanks!

[roshan-elastic]

It was using host.cpu.pct field, which has no value. Was this intentional?

@crespocarlos - 🤦 thanks! Even I can't use ECS haha

[dedemorton]

@roshan-elastic Is this PR ready to merge? I still need SME approval. Thanks!

[roshan-elastic]

Sorry @dedemorton - completely missed this.

Might be too late for this version but approving for whenever it can go in