Product documentation Docs
    • English
    • Deutsch
    • 日本語
    • 한국어
    • Français
  • More Sites
    • Now Community
    • Developer Site
    • Knowledge Base
    • Product Information
    • ServiceNow.com
    • Training
    • Customer Success Center
    • ServiceNow Support Videos
  • Log in

Product documentation

  • Home
How search works:
  • Punctuation and capital letters are ignored
  • Special characters like underscores (_) are removed
  • Known synonyms are applied
  • The most relevant topics (based on weighting and matching to search terms) are listed first in search results
Topics are ranked in search results by how closely they match your search terms
  • A match on the entire phrase you typed
  • A match on part of the phrase you typed
  • A match on ALL of the terms in the phrase you typed
  • A match on ANY of the terms in the phrase you typed

Note: Matches in titles are always highly ranked.

  • Release version
    Table of Contents
    • Now Platform capabilities
Table of Contents
Choose your release version
    Home Orlando Now Platform Capabilities Now Platform capabilities Configuration Management Database CMDB Identification and Reconciliation Components and process of Identification and Reconciliation Identification and Reconciliation engine (IRE)

    Identification and Reconciliation engine (IRE)

    • Save as PDF Selected topic Topic & subtopics All topics in contents
    • Unsubscribe Log in to subscribe to topics and get notified when content changes.
    • Share this page

    Identification and Reconciliation engine (IRE)

    IRE is an underlying key component in Identification and Reconciliation, providing a centralized framework to perform identification and reconciliation processes across different data sources. IRE uses identification rules, reconciliation rules, and IRE data source rules when processing incoming data before inserting that data to the CMDB.

    IRE processes help maintain data integrity in the CMDB.
    • IRE prevents duplicate CIs by uniquely identifying CIs.
    • IRE reconciles CI attributes by allowing only authoritative data sources to write to CMDB.
    ServiceNow® applications such as Service Mapping, horizontal discovery, and pattern discovery, use APIs to apply identification and reconciliation processes. You can also apply IRE processes to data imported by import sets. Other data sources including third-party data sources, can leverage REST or scriptable IRE APIs to perform identification and reconciliation.

    For more information about using IRE APIs, see IdentificationEngine - Scoped.

    CI identification

    The CMDB identification process relies on identification rules to uniquely identify CIs. When possible, CIs can also be uniquely identified using source_name and source_native_key values provided in the sys_object_source_info section of the payload, and the Discovery Source [sys_object_source] table. If identification is successful using that method, then it is not necessary to apply matching algorithms that rely on identification rules, which is a slower identification method.

    A unique CI identifier can be provided in the optional sys_object_source_info object in the IRE payload.
    {
      "items": [
        {
          "className": "cmdb_ci_win_server",
          "values": {
            "name": "SAMLABVM52"
          },
          "sys_object_source_info": {
                "source_native_key": "16777219",
                "source_name": "SCCM",
                "source_feed": "SCCM Computer Identity",
                "source_recency_timestamp": "2019-08-26 13:00:00"
          }
         }
        ]
      }

    Payload items identification

    IRE generates identifier keys for all payload items in an incoming payload and then uses those keys when trying to match partial payload and incoming payloads. An identifier key is based either on:
    • Combination of the source_name and source_native_key values from the sys_object_source_info object.
    • Identification criterion attributes.
    IRE stores the identifier keys associated with partial items in the cmdb_ire_partial_payloads_index table, and then uses those keys to try to match with identifier keys of incoming payloads.

    Enhanced IRE features

    The CreateOrUpdateCIEnhanced() scriptable API provides the functionality for the following enhanced IRE features, which can be enabled or disabled as needed:
    Partial payloads

    IRE isolates items for which data sources did not provide enough information to uniquely identify the CI and therefore processing cannot continue. Some of these items are identified as partial items, which get stored for potential later processing. Other items are identified as incomplete items, which get stored for logging purposes only.

    For example: SCCM has multiple feeds such as a disk feed and a computer feed. The disk feed might have complete information about the disk but insufficient information about the computer CI that it depends on.

    API option: partial_payloads which is enabled by default. When partial_payloads is enabled, partial_commits and deduplicate_payloads are automatically enabled regardless of their setting in options.

    Partial commits

    Errors in some items do not prevent committing the rest of the items in a payload. Therefore, when a payload contains items with errors, IRE still commits the remaining valid items in the payload.

    API option: partial_commits which is enabled by default.

    Deduplicate payload items

    IRE deduplicates duplicate items within the payload, merging those duplicates into a single payload item for processing.

    API option: deduplicate_payloads which is enabled by default.

    Generate summary

    IRE generates summaries in the output payload with processing details such as the number of updates per class.

    API option: generate_summary which is disabled by default.

    Partial items

    A payload item is determined to be a partial item if it contains the necessary data for unique identification and if it has one of the following errors. Unique identification requires that the payload item has the sys_object_source_info section with source_name and source_native_key values, or the full set of the identification criterion attributes specified for the CI class, or both.

    IRE errors for a partial item:
    • MISSING_MATCHING_ATTRIBUTES –— Item does not have identification criterion attributes to use at least one identifier entry for matching.
    • REQUIRED_ATTRIBUTE_EMPTY –— Unable to create a CI because a required attribute is missing.
    • MISSING_DEPENDENCY –— Dependent CI is missing a dependency relation which is specified in the payload.
    • INSERT_NOT_ALLOWED_FOR_SOURCE –— An IRE data source rule prevents the specified data sources from creating CIs of the specified class.

    For more details about IRE error messages, see IRE error messages.

    If processing fails because payload items are determined to be partial items, then the partial items are saved as partial payloads in the CMDB IRE Partial Payloads [cmdb_ire_partial_payloads] table in JSON format for later potential processing. IRE uses identifier keys to attempt to match incoming payloads with stored partial payloads.

    If later, a data source sends the data that was missing in the partial item, IRE matches the incoming payload with the stored partial payloads. IRE then merges any matching partial payloads with the incoming payload. To resolve any conflicting attributes, IRE uses either source_recency_timestamp (when source_native_key and source_name are identical), or reconciliation rules specified for the class. The result is a complete and valid payload which IRE then processes to create or update the respective CIs.

    Partial paylaods older than 90 days are deleted from the CMDB IRE Partial Payloads [cmdb_ire_partial_payloads] table.

    Sample of a partial payload:
    Disk feed:
    {
      "items": [
        {
          "className": "cmdb_ci_computer",
          "sys_object_source_info": {
                "source_native_key": "Server001",
                "source_name": "SCCM",
                "source_feed": "DISK_FEED",
                "source_recency_timestamp": "2019-08-26 13:00:00"
          }
        },
        {
          "className": "cmdb_ci_disk",
          "values": {
            "name": "disk1"
          }
        }
      ],
      "relations": [{
                  "parent": 0,
                  "child": 1,
                  "type": "Contains::Contained by"}
                  ]
    }
    The computer item in the above payload does not have any attributes and therefore IRE can't process it. However, source_name and source_native_key are provided making it a partial item. Because the computer item is partial, the disk item that depends on the computer item, is a partial item too.
    Sample of a subsequent payload that completes the previous partial payload by providing the missing details:
    Server/Computer feed:
     {
      "items": [
        {
          "className": "cmdb_ci_linux_server",
          "values": { "name": 'linux001',
                    "ip_address": "100.126.38.19",
                    "mac_address": "DSWER4587" },
          "sys_object_source_info": {
                "source_native_key": "Server001",
                "source_name": "SCCM",
                "source_feed": "COMPUTER_IDENTITY",
                "source_recency_timestamp": "2019-08-26 14:00:00"
          }
        }
      ]
    }
    The computer in the partial payload and the server in the new payload match because they have identical source_name and source_native_key. Therefore, the partial payload and the new payload are merged, the operation is committed, and the partial payload is deleted from the Partial Payloads table.

    For more information about partial payloads, see CreateOrUpdateCIEnhanced().

    Incomplete items

    A payload item is determined to be an incomplete item if:
    • It does not contain all the data necessary for unique identification
    • It has an error that is not associated with a partial item
    Unique identification is not possible if neither source_name and source_native_key within the sys_object_source_info object, nor the full set of identification criterion attributes specified for the CI class, is provided.

    Incomplete items are saved as incomplete payloads in the CMDB IRE Incomplete Payloads [cmdb_ire_incomplete_payloads] table in JSON format. Incomplete items are stored for the purpose of logging payloads with irrecoverable errors, and are never processed again.

    Adding relationships

    Add relationships by using either indices, or the optional JSON internal_id element.

    Use the relations object in the payload to add or update relationships by referring to internal_ids of items. Relationships can be created using main items and related items in the payload. For example:
    • Relation (parent Index, child Index, Relation Type)
    • Relation (parent Internal Id, child Internal Id, Relation Type)

    For more information and for code samples, see CreateOrUpdateCIEnhanced().

    Adding references between payload items

    Add references between two payload items by using the optional JSON internal_id element, which uniquely identifies payload items.

    Use the referenceItems block to add or update references. You can add references between any two items, including main items, lookup items, and related items, in a single payload.

    For more information and for code samples, see CreateOrUpdateCIEnhanced().

    CI reclassification

    Use the updateWithoutUpgrade, updateWithoutDowngrade, and updateWithoutSwitch flags in the settings block in a payload, to prevent unintentional updates to CIs' class. These flags prevent upgrading, downgrading, or switching the class of a CI that multiple data sources unintentionally might attempt while updating the same CI. For more information and for code samples, see CreateOrUpdateCIEnhanced().

    Reclassification flags have precedence over any other system settings for CI reclassification.

    Tags:

    Feedback
    On this page

    Previous topic

    Next topic

    • Contact Us
    • Careers
    • Terms of Use
    • Privacy Statement
    • Sitemap
    • © ServiceNow. All rights reserved.

    Release version
    Choose your release version

      Identification and Reconciliation engine (IRE)

      • Save as PDF Selected topic Topic & subtopics All topics in contents
      • Unsubscribe Log in to subscribe to topics and get notified when content changes.
      • Share this page

      Identification and Reconciliation engine (IRE)

      IRE is an underlying key component in Identification and Reconciliation, providing a centralized framework to perform identification and reconciliation processes across different data sources. IRE uses identification rules, reconciliation rules, and IRE data source rules when processing incoming data before inserting that data to the CMDB.

      IRE processes help maintain data integrity in the CMDB.
      • IRE prevents duplicate CIs by uniquely identifying CIs.
      • IRE reconciles CI attributes by allowing only authoritative data sources to write to CMDB.
      ServiceNow® applications such as Service Mapping, horizontal discovery, and pattern discovery, use APIs to apply identification and reconciliation processes. You can also apply IRE processes to data imported by import sets. Other data sources including third-party data sources, can leverage REST or scriptable IRE APIs to perform identification and reconciliation.

      For more information about using IRE APIs, see IdentificationEngine - Scoped.

      CI identification

      The CMDB identification process relies on identification rules to uniquely identify CIs. When possible, CIs can also be uniquely identified using source_name and source_native_key values provided in the sys_object_source_info section of the payload, and the Discovery Source [sys_object_source] table. If identification is successful using that method, then it is not necessary to apply matching algorithms that rely on identification rules, which is a slower identification method.

      A unique CI identifier can be provided in the optional sys_object_source_info object in the IRE payload.
      {
        "items": [
          {
            "className": "cmdb_ci_win_server",
            "values": {
              "name": "SAMLABVM52"
            },
            "sys_object_source_info": {
                  "source_native_key": "16777219",
                  "source_name": "SCCM",
                  "source_feed": "SCCM Computer Identity",
                  "source_recency_timestamp": "2019-08-26 13:00:00"
            }
           }
          ]
        }

      Payload items identification

      IRE generates identifier keys for all payload items in an incoming payload and then uses those keys when trying to match partial payload and incoming payloads. An identifier key is based either on:
      • Combination of the source_name and source_native_key values from the sys_object_source_info object.
      • Identification criterion attributes.
      IRE stores the identifier keys associated with partial items in the cmdb_ire_partial_payloads_index table, and then uses those keys to try to match with identifier keys of incoming payloads.

      Enhanced IRE features

      The CreateOrUpdateCIEnhanced() scriptable API provides the functionality for the following enhanced IRE features, which can be enabled or disabled as needed:
      Partial payloads

      IRE isolates items for which data sources did not provide enough information to uniquely identify the CI and therefore processing cannot continue. Some of these items are identified as partial items, which get stored for potential later processing. Other items are identified as incomplete items, which get stored for logging purposes only.

      For example: SCCM has multiple feeds such as a disk feed and a computer feed. The disk feed might have complete information about the disk but insufficient information about the computer CI that it depends on.

      API option: partial_payloads which is enabled by default. When partial_payloads is enabled, partial_commits and deduplicate_payloads are automatically enabled regardless of their setting in options.

      Partial commits

      Errors in some items do not prevent committing the rest of the items in a payload. Therefore, when a payload contains items with errors, IRE still commits the remaining valid items in the payload.

      API option: partial_commits which is enabled by default.

      Deduplicate payload items

      IRE deduplicates duplicate items within the payload, merging those duplicates into a single payload item for processing.

      API option: deduplicate_payloads which is enabled by default.

      Generate summary

      IRE generates summaries in the output payload with processing details such as the number of updates per class.

      API option: generate_summary which is disabled by default.

      Partial items

      A payload item is determined to be a partial item if it contains the necessary data for unique identification and if it has one of the following errors. Unique identification requires that the payload item has the sys_object_source_info section with source_name and source_native_key values, or the full set of the identification criterion attributes specified for the CI class, or both.

      IRE errors for a partial item:
      • MISSING_MATCHING_ATTRIBUTES –— Item does not have identification criterion attributes to use at least one identifier entry for matching.
      • REQUIRED_ATTRIBUTE_EMPTY –— Unable to create a CI because a required attribute is missing.
      • MISSING_DEPENDENCY –— Dependent CI is missing a dependency relation which is specified in the payload.
      • INSERT_NOT_ALLOWED_FOR_SOURCE –— An IRE data source rule prevents the specified data sources from creating CIs of the specified class.

      For more details about IRE error messages, see IRE error messages.

      If processing fails because payload items are determined to be partial items, then the partial items are saved as partial payloads in the CMDB IRE Partial Payloads [cmdb_ire_partial_payloads] table in JSON format for later potential processing. IRE uses identifier keys to attempt to match incoming payloads with stored partial payloads.

      If later, a data source sends the data that was missing in the partial item, IRE matches the incoming payload with the stored partial payloads. IRE then merges any matching partial payloads with the incoming payload. To resolve any conflicting attributes, IRE uses either source_recency_timestamp (when source_native_key and source_name are identical), or reconciliation rules specified for the class. The result is a complete and valid payload which IRE then processes to create or update the respective CIs.

      Partial paylaods older than 90 days are deleted from the CMDB IRE Partial Payloads [cmdb_ire_partial_payloads] table.

      Sample of a partial payload:
      Disk feed:
      {
        "items": [
          {
            "className": "cmdb_ci_computer",
            "sys_object_source_info": {
                  "source_native_key": "Server001",
                  "source_name": "SCCM",
                  "source_feed": "DISK_FEED",
                  "source_recency_timestamp": "2019-08-26 13:00:00"
            }
          },
          {
            "className": "cmdb_ci_disk",
            "values": {
              "name": "disk1"
            }
          }
        ],
        "relations": [{
                    "parent": 0,
                    "child": 1,
                    "type": "Contains::Contained by"}
                    ]
      }
      The computer item in the above payload does not have any attributes and therefore IRE can't process it. However, source_name and source_native_key are provided making it a partial item. Because the computer item is partial, the disk item that depends on the computer item, is a partial item too.
      Sample of a subsequent payload that completes the previous partial payload by providing the missing details:
      Server/Computer feed:
       {
        "items": [
          {
            "className": "cmdb_ci_linux_server",
            "values": { "name": 'linux001',
                      "ip_address": "100.126.38.19",
                      "mac_address": "DSWER4587" },
            "sys_object_source_info": {
                  "source_native_key": "Server001",
                  "source_name": "SCCM",
                  "source_feed": "COMPUTER_IDENTITY",
                  "source_recency_timestamp": "2019-08-26 14:00:00"
            }
          }
        ]
      }
      The computer in the partial payload and the server in the new payload match because they have identical source_name and source_native_key. Therefore, the partial payload and the new payload are merged, the operation is committed, and the partial payload is deleted from the Partial Payloads table.

      For more information about partial payloads, see CreateOrUpdateCIEnhanced().

      Incomplete items

      A payload item is determined to be an incomplete item if:
      • It does not contain all the data necessary for unique identification
      • It has an error that is not associated with a partial item
      Unique identification is not possible if neither source_name and source_native_key within the sys_object_source_info object, nor the full set of identification criterion attributes specified for the CI class, is provided.

      Incomplete items are saved as incomplete payloads in the CMDB IRE Incomplete Payloads [cmdb_ire_incomplete_payloads] table in JSON format. Incomplete items are stored for the purpose of logging payloads with irrecoverable errors, and are never processed again.

      Adding relationships

      Add relationships by using either indices, or the optional JSON internal_id element.

      Use the relations object in the payload to add or update relationships by referring to internal_ids of items. Relationships can be created using main items and related items in the payload. For example:
      • Relation (parent Index, child Index, Relation Type)
      • Relation (parent Internal Id, child Internal Id, Relation Type)

      For more information and for code samples, see CreateOrUpdateCIEnhanced().

      Adding references between payload items

      Add references between two payload items by using the optional JSON internal_id element, which uniquely identifies payload items.

      Use the referenceItems block to add or update references. You can add references between any two items, including main items, lookup items, and related items, in a single payload.

      For more information and for code samples, see CreateOrUpdateCIEnhanced().

      CI reclassification

      Use the updateWithoutUpgrade, updateWithoutDowngrade, and updateWithoutSwitch flags in the settings block in a payload, to prevent unintentional updates to CIs' class. These flags prevent upgrading, downgrading, or switching the class of a CI that multiple data sources unintentionally might attempt while updating the same CI. For more information and for code samples, see CreateOrUpdateCIEnhanced().

      Reclassification flags have precedence over any other system settings for CI reclassification.

      Tags:

      Feedback

          Share this page

          Got it! Feel free to add a comment
          To share your product suggestions, visit the Idea Portal.
          Please let us know how to improve this content

          Check any that apply

          To share your product suggestions, visit the Idea Portal.
          Confirm

          We were unable to find "Coaching" in Jakarta. Would you like to search instead?

          No Yes
          • Contact Us
          • Careers
          • Terms of Use
          • Privacy Statement
          • Sitemap
          • © ServiceNow. All rights reserved.

          Subscribe Subscribed Unsubscribe Last updated: Tags: January February March April May June July August September October November December No Results Found Versions Search preferences successfully updated My release version successfully updated My release version successfully deleted An error has occurred. Please try again later. You have been unsubscribed from all topics. You are now subscribed to and will receive notifications if any changes are made to this page. You have been unsubscribed from this content Thank you for your feedback. Form temporarily unavailable. Please try again or contact  docfeedback@servicenow.com  to submit your comments. The topic you requested does not exist in the release. You were redirected to a related topic instead. The available release versions for this topic are listed There is no specific version for this documentation. Explore products Click to go to the page. Release notes and upgrades Click to open the dropdown menu. Delete Remove No selected version Reset This field is required You are already subscribed to this topic Attach screenshot The file you uploaded exceeds the allowed file size of 20MB. Please try again with a smaller file. Please complete the reCAPTCHA step to attach a screenshot
          Log in to personalize your search results and subscribe to topics
          No, thanks Login