Event Management guidelines

Improve overall system performance and learn about efficient configuration practices by considering the following Event Management tips.

Table 1. Improve Event Management system performance
Topic Details
Business rules
  • Avoid writing business rules for event [em_event] tables as they can result in performance degradation.
  • Business rules that are written for alert [em_alert] tables must be highly efficient or they may result in performance degradation. Instead of writing a business rule, consider whether it is more appropriate to write a job.
  • Async business rules that are written for alert tables are also not recommended.
Configure for large-scale environments
  • Set the Enable multi node event processing (event_processor_enable_multi_node) property to Yes.

    Enable multi-node in production environments and set values based on the size of the deployment and expected event rate.

  • Set the Number of scheduled jobs processing events (event_processor_job_count) property to 4.
  • If you are sending events from a custom source, verify that events have Message Key or Source, Node, Type, and Resource data.
Latency issues for receiving events Check the following settings:
  • Verify that the Bucket field in the event table is set to a value that is greater than zero (0).
  • Navigate to System Scheduler > Scheduled Jobs Scheduled Jobs and search for - process events*.

    Check that all - process events* jobs exist according to the Number of scheduled jobs processing events (event_processor_job_count) property configuration. Verify that the State is Running or Ready. If the state is Queued or Error, set the job state to Ready.

Archive events

Avoid changing the default retention time for events.

To log events for a longer time, create an archive table and a job that copies new events to it. Do this by scheduling a job to regularly back up events [em_event] to a custom table.

ServiceNow does not recommend extending table rotation by adding more days.

Scaling up <new> When the Event Management system is initially configured, before increasing event throughput, check the processing duration of events. The majority of events should not take more than 100 milliseconds.

Alert guidelines <new>

  • When creating a business rule on an alert table, make sure that it does not take more than a few milliseconds to run. Check this using the slow-jobs table. Consider whether the same functionality can be achieved using a job.
  • Avoid creating alerts whose severity is Info. In some monitoring systems, Info severity is used to denote that an issue has been resolved, whereas in other monitoring systems, it is used to denote events that are not of operational significance. For the former, it is recommended to use the Clear severity instead of Info using a Mapping Rule. For the latter, it is recommended to have an Ignore rule, unless the alerts are of specific values.
  • Do not delete open alerts manually. If you do need to manually delete an alert, close it first.

Alert rules

  • A scheduled job applies alert rules to new alerts every 11 seconds. If an alert rule does not start immediately, allow 10–15 seconds before you start troubleshooting.
  • Use the Order field to control which alert rule runs if two alert rules have similar conditions set.
  • <new> Use alert rules with Task Templates to populate static values in an incident.
  • <new> Use the EvtMgmtCustomIncidentPopulator.populateFieldsFromAlert script include to dynamically populate alert values to an incident. See Create an incident or security incident from an alert.

Alert lifecycle <new>

  • An alert is opened whenever an event is not ignored or its threshold is passed by an event rule, and de-duplication does not identify the event as belonging to an existing alert.
  • An alert is closed when a closing event is sent on the same message key, or the alert is closed manually.
  • An alert is reopened if an opening alert that has the same message key is sent within the timeframe defined in properties (The default is a timeframe of an hour).
  • If an alert is opened and closed at a high rate, as defined in properties, it becomes flapping. When this opening and closing rate stops, the alert goes out of flapping state.
  • Do not close an alert when creating a corresponding incident.
  • If an incident is opened from an alert, that alert will remain open as long as the incident remains open. By default, when either the incident or the alert are closed the other is closed as well. This can be configured using properties.
  • Acknowledge is used to denote that the alert is known, and can temporarily be ignored.
  • Do not use Acknowledge to mark an alert as needing attention.
  • Do not create alerts in Closed state.
  • Do not create alerts in Info state.
  • Do not create alerts in Open state.
  • The evt_mgmt.alert_auto_close_interval property can be set to 0 to disable the feature that automatically closes alerts after the specified period. It is recommended that you do not specify 0 to disable this feature.

CI binding

  • Where possible, always attempt to bind an alert to a CI.
    Note: Alerts are bound to CIs, not events.
  • To bind a host, machine or any device with an IP, populate the event Node field with a unique host name, FQDN, IP, or MAC address. If other identifiers are necessary to identify a host, then populate the ci_identifiers field with a JSON format. The JSON format must contain the CMDB field name and value to perform the match.
    Note: The event Node field must be populated with a unique host name from the source before the event is inserted.
  • The primary binding strategy is to use the Node field. If the Node field is not pre-populated in the event, it can be populated using event rules.
  • <new> Binding should be done by event rules, not by business service rules, unless no other option is available.

De-duplication

  • The message_key field is used for De-Duplication. If reliable message key values are not provided with the source event, it is important to have a well-defined plan for constructing these identifiers.
  • If the message key is not defined, then the default message key is <Source + Node + Type + Resource + Metric Name> .
  • The guideline is to have the event source populate the <Source + Node + Type + Resource + Metric Name> fields out-of-the-box and populate the message key. This action enables a better distribution of event processing among instance workers and nodes.
  • If the source event does not have values for these fields, make sure to populate them using transform rules. This action does not affect event processing, but is used for de-duplication.

Event guidelines <new>

Event tables

  • Do not add columns to Event tables.
  • Do not create business rules on Event tables.
  • Do not change the default retention time for events. To log events for a longer time, create an archive table and create a job that copies new events to the archive table.
  • Do not add additional fields to an event by adding a custom field to the event table. Include additional fields in the Additional information field of the event only. For more information about how to include additional fields in events, see Populate custom alert fields.

Event rules

  • Write event rules to apply to the broadest number of events possible. More specific rules can then be created as necessary and should use a lower-order value.
  • If a more general rule can achieve the same outcome, avoid writing event rules that apply only to a certain subset of events.
  • When event rules are applied to events, no changes are made to the original event. All processing occurs in memory, so use the Processing Notes field and/or use the Check Process of Event UI action link to troubleshoot.
  • If you change a rule/transform that has existing mapping rules, you should review and retest with events that are either actual or simulated.
  • Establish a consistent naming convention. A common convention is: <customer acronym>.<Event Source>.<Description>. For example, ACME.OEM.Normalize
  • If two event rules have similar conditions set, use the Order field to control which Event Rule runs.
  • <new> Do not associate an alert with a CI using a business rule on the alert table. Use event rules to associate an alert with a CI.
  • Ensure that the From field value exactly matches a string in the JSON in the additional_info field. This matching happens when a rule has been configured based on information in a MIB file. If the MIB file is not uploaded, the JSON for the SNMP trap shows varbinds (variable binding) with dotted names, instead of the translated name in the MIB. The event field mapping rule then fails to be applied.
  • Establish a consistent naming convention. A common convention is: <Customer acronym>.<Event Source>.<Mapping target>. For example, ACME.OEM.Severity.

Construct event rules using these guidelines

Table 2. Effective rules should be written to achieve:
Desired result Required activity
Effective de-duplication and enabling efficient parallel event processing Populate the Source, Node, Type, Resource, Metric Name fields.
CI binding
  • Bind to host - by populating the Node field and optionally CI identifiers.
  • Binding to application and / or device – by populating the CI type field and the Additional information field.
Alert Correlation, using Service Analytics Populate the Resource and Metric Name fields.
Note: If CI is also bound, Alert Correlation is improved.

Event integration

SNMP traps

  • It is preferable to use a monitoring tool to send SNMP traps rather than sending them directly from devices.
  • Upload MIBs prior to defining event rules to avoid having to rewrite the event rules.

Web service API

  • Using a web service API for integration can reduce the number of event rules needed. This action avoids having to transform events (prepared data is sent in an event to ServiceNow).
  • Use dedicated credentials for integration. Optionally, designate credentials specific to each event source.
  • Use cURL instead of Python, Perl, and so on.

CloudWatch

  • Use dedicated credentials for integrating CloudWatch with ServiceNow.

Email

  • Use email only if the source has a low volume and other options are not available, such as, running a script or forwarding an SNMP trap.

Operational Metric collector logs and files

Operational Metric collector logs and files are located under the path $(MID_SERVER_DIR)/agent. Use these logs and files for troubleshooting and monitoring purposes.

Table 3. Location of Operational Metric collector logs and files:
Log or file Path
PowerShell metric collector log file Logs/retrieve_metrics{connector instance ID}.log
PowerShell output file work/metrics/metrics_output_{connector instance ID}.txt  
PowerShell input file work/metrics/parameters_{connector instance ID}.txt

Operational Metrics performance can be checked in the MID Server log file when the mid.log.level MID Server parameter is in debug mode.

Operational Metrics performance numbers are available in the sa_performance_statistics table. To view the performance numbers, filter the Performance Statistics list for Metric Collector.

Planning

  • Organize event source configuration of filters, modules, and so on, into multiple parallel efforts, rather than in serial.
  • Validate processed event formats to ensure that data that is parsed is aligned with desired results.
  • Test production events in a non-production environment. Integrate with non-production element managers and ServiceNow instances. If non-production element managers are not available, send events from element managers to both production and non-production environments.

Remediation

  • Always set orchestration workflow properties to the Remediation Task [em_remediation_task] table.
  • Use ECC Queue and Workflow > Live Workflow > All Contexts to find more detailed information on remediation activities.

Services and dashboard

  • Use Service Groups to group business services into logical groups to reduce the number of services displayed on the Service Health dashboard.
  • Import manually built service maps. For a description of the conversion process, see Import a business service as a manual service.

Task templates

  • Create a user called Event Management (or a similar name). Then the Created by field in a task template (for example, Incident) can be set to indicate that user was the source of the task.
  • To perform any dynamic value assignment or to override OOB dynamic value assignment, use the EvtMgmtCustomIncidentPopulator script include.