IRE is an underlying key component in Identification and Reconciliation, providing a
centralized framework to perform identification and reconciliation processes across different
data sources. IRE uses identification rules, reconciliation rules, and IRE data source rules
when processing incoming data before inserting that data to the CMDB.
IRE processes help maintain data integrity in the CMDB.
- IRE prevents duplicate CIs by uniquely identifying CIs.
- IRE reconciles CI attributes by allowing only authoritative data sources to write to
CMDB.
ServiceNow® applications such as
Service Mapping,
horizontal discovery, and pattern discovery, use APIs to apply identification and
reconciliation processes. You can also apply IRE processes to data imported by import sets.
Other data sources including third-party data sources, can leverage REST or scriptable IRE
APIs to perform identification and reconciliation.
For more information about using IRE APIs, see IdentificationEngine -
Scoped.
CI identification
The CMDB identification process relies on identification rules to uniquely identify CIs.
When possible, CIs can also be uniquely identified using source_name and
source_native_key values provided in the
sys_object_source_info section of the payload, and the Discovery Source [sys_object_source] table. If identification is successful
using that method, then it is not necessary to apply matching algorithms that rely on
identification rules, which is a slower identification method.
A unique CI identifier can be provided in the optional
sys_object_source_info object in the IRE
payload.
{
"items": [
{
"className": "cmdb_ci_win_server",
"values": {
"name": "SAMLABVM52"
},
"sys_object_source_info": {
"source_native_key": "16777219",
"source_name": "SCCM",
"source_feed": "SCCM Computer Identity",
"source_recency_timestamp": "2019-08-26 13:00:00"
}
}
]
}Payload items identification
IRE generates identifier keys for all payload items in an incoming payload and then uses
those keys when trying to match partial payload and incoming payloads. An identifier key is
based either on:
- Combination of the
source_name and source_native_key
values from the sys_object_source_info object.
- Identification criterion attributes.
IRE stores the identifier keys associated with partial items in the
cmdb_ire_partial_payloads_index table, and then uses those keys to try to match with
identifier keys of incoming payloads.
Enhanced IRE features
The
CreateOrUpdateCIEnhanced() scriptable API provides the
functionality for the following enhanced IRE features, which can be enabled or disabled as needed:
- Partial payloads
-
IRE isolates items for which data sources did not provide enough information to
uniquely identify the CI and therefore processing cannot continue. Some of these
items are identified as partial items, which get stored for potential later
processing. Other items are identified as incomplete items, which get stored for
logging purposes only.
For example: SCCM has multiple feeds such as a disk feed and a computer feed. The
disk feed might have complete information about the disk but insufficient
information about the computer CI that it depends on.
API option: partial_payloads which is enabled by default. When
partial_payloads is enabled, partial_commits and
deduplicate_payloads are automatically enabled regardless of
their setting in options.
- Partial commits
-
Errors in some items do not prevent committing the rest of the items in a payload.
Therefore, when a payload contains items with errors, IRE still commits the
remaining valid items in the
payload.
API option: partial_commits which is enabled by default.
- Deduplicate payload items
-
IRE deduplicates duplicate items within the payload, merging those duplicates into
a single payload item for processing.
API option: deduplicate_payloads which is enabled by default.
- Generate summary
-
IRE generates summaries in the output payload with processing details such as the
number of updates per class.
API option: generate_summary which is disabled by default.
Partial items
A payload item is determined to be a partial item if it contains the necessary data for
unique identification and if it has one of the following errors. Unique identification
requires that the payload item has the sys_object_source_info section with
source_name and source_native_key values, or the full
set of the identification criterion attributes specified for the CI class, or both.
IRE errors for a partial item:
- MISSING_MATCHING_ATTRIBUTES –— Item does not have identification criterion attributes
to use at least one identifier entry for matching.
- REQUIRED_ATTRIBUTE_EMPTY –— Unable to create a CI because a required attribute is
missing.
- MISSING_DEPENDENCY –— Dependent CI is missing a dependency relation which is specified
in the payload.
- INSERT_NOT_ALLOWED_FOR_SOURCE –— An IRE data source rule prevents the specified data
sources from creating CIs of the specified class.
For more details about IRE error messages, see IRE error messages.
If processing fails because payload items are determined to be partial items, then the
partial items are saved as partial payloads in the CMDB IRE Partial Payloads
[cmdb_ire_partial_payloads] table in JSON format for later potential processing. IRE uses
identifier keys to attempt to match incoming payloads with stored partial payloads.
If later, a data source sends the data that was missing in the partial item, IRE matches
the incoming payload with the stored partial payloads. IRE then merges any matching partial
payloads with the incoming payload. To resolve any conflicting attributes, IRE uses either
source_recency_timestamp (when source_native_key and
source_name are identical), or reconciliation rules specified for the
class. The result is a complete and valid payload which IRE then processes to create or
update the respective CIs.
Partial paylaods older than 90 days are deleted from the CMDB IRE Partial Payloads
[cmdb_ire_partial_payloads] table.
Sample of a partial payload:
Disk feed:
{
"items": [
{
"className": "cmdb_ci_computer",
"sys_object_source_info": {
"source_native_key": "Server001",
"source_name": "SCCM",
"source_feed": "DISK_FEED",
"source_recency_timestamp": "2019-08-26 13:00:00"
}
},
{
"className": "cmdb_ci_disk",
"values": {
"name": "disk1"
}
}
],
"relations": [{
"parent": 0,
"child": 1,
"type": "Contains::Contained by"}
]
}The
computer item in the above payload does not have any attributes and therefore IRE can't
process it. However,
source_name and
source_native_key are
provided making it a partial item. Because the computer item is partial, the disk item that
depends on the computer item, is a partial item too.
Sample of a subsequent payload that completes the previous partial payload by providing the
missing
details:
Server/Computer feed:
{
"items": [
{
"className": "cmdb_ci_linux_server",
"values": { "name": 'linux001',
"ip_address": "100.126.38.19",
"mac_address": "DSWER4587" },
"sys_object_source_info": {
"source_native_key": "Server001",
"source_name": "SCCM",
"source_feed": "COMPUTER_IDENTITY",
"source_recency_timestamp": "2019-08-26 14:00:00"
}
}
]
}
The computer in the partial payload and the server in the new payload match because they
have identical
source_name and
source_native_key.
Therefore, the partial payload and the new payload are merged, the operation is committed,
and the partial payload is deleted from the Partial Payloads table.
For more information about partial payloads, see CreateOrUpdateCIEnhanced().
Incomplete items
A payload item is determined to be an incomplete item if:
- It does not contain all the data necessary for unique identification
- It has an error that is not associated with a partial item
Unique identification is not possible if neither
source_name and
source_native_key within the
sys_object_source_info
object, nor the full set of identification criterion attributes specified for the CI class,
is provided.
Incomplete items are saved as incomplete payloads in the CMDB IRE Incomplete Payloads
[cmdb_ire_incomplete_payloads] table in JSON format. Incomplete items are stored for the
purpose of logging payloads with irrecoverable errors, and are never processed again.
Adding relationships
Add relationships by using either indices, or the optional JSON internal_id element.
Use the
relations object in the payload to add or update relationships by
referring to
internal_ids of items. Relationships can be created using main
items and related items in the payload. For example:
- Relation (parent Index, child Index, Relation Type)
- Relation (parent Internal Id, child Internal Id, Relation Type)
For more information and for code samples, see CreateOrUpdateCIEnhanced().
Adding references
Add references by using the optional JSON internal_id element, which is used to uniquely
identify payload items.
Use the referenceItems block to add or update references. You can add
references between any two items, including main items, lookup items, and related items, in
a single payload.
For more information and for code samples, see CreateOrUpdateCIEnhanced().
CI reclassification
Use the updateWithoutUpgrade, updateWithoutDowngrade, and
updateWithoutSwitch flags in the settings block in a payload, to prevent
unintentional updates to CIs' class. These flags prevent upgrading, downgrading, or
switching the class of a CI that multiple data sources unintentionally might attempt while
updating the same CI. For more information and for code samples, see CreateOrUpdateCIEnhanced().
Reclassification flags have precedence over any other system settings for CI reclassification.
