Product documentation Docs
    • English
    • Deutsch
    • 日本語
    • 한국어
    • Français
  • More Sites
    • Now Community
    • Developer Site
    • Knowledge Base
    • Product Information
    • ServiceNow.com
    • Training
    • Customer Success Center
    • ServiceNow Support Videos
  • Log in

Product documentation

  • Home
How search works:
  • Punctuation and capital letters are ignored
  • Special characters like underscores (_) are removed
  • Known synonyms are applied
  • The most relevant topics (based on weighting and matching to search terms) are listed first in search results
Topics are ranked in search results by how closely they match your search terms
  • A match on the entire phrase you typed
  • A match on part of the phrase you typed
  • A match on ALL of the terms in the phrase you typed
  • A match on ANY of the terms in the phrase you typed

Note: Matches in titles are always highly ranked.

  • Release version
    Table of Contents
    • Now Platform capabilities
Table of Contents
Choose your release version
    Home Paris Now Platform Capabilities Now Platform capabilities Configuration Management Database Integrating third-party data into the CMDB IntegrationHub ETL (2.0.1)

    IntegrationHub ETL (2.0.1)

    • Save as PDF Selected topic Topic & subtopics All topics in contents
    • Unsubscribe Log in to subscribe to topics and get notified when content changes.
    • Share this page

    IntegrationHub ETL (2.0.1)

    Use the IntegrationHub ETL store app to create and manage ETL transform maps, which integrate third-party data into the CMDB without compromising the integrity of the CMDB. IntegrationHub ETL provides a simplified user interface that guides you through the integration process end-to-end, including a test integration run of sample data.

    Request apps on the Store

    Visit the ServiceNow Store website to view all the available apps and for information about submitting requests to the store. For cumulative release notes information for all released apps, see the ServiceNow Store version history release notes.

    When you create an integration, you import source data, transform data if needed, and select target CMDB classes and attributes to map the data to. Eventually, you run an integration test of the sample data, using your settings in the IntegrationHub ETL. You can then preview the integration test results and adjust any settings before scheduling recurring integration runs for large data sets. If you develop and test the ETL transform map on a development instance, then you can test and adjust the configuration before implementation on a production instance.

    For example, you can integrate data from SCCM (Microsoft System Center Configuration Manager).

    For more information, watch the IntegrationHub ETL | Importing resources into the CMDB video for an introduction and walk through of the IntegrationHub ETL tool.

    Roles required

    Users with the cmdb_inst_admin role can use IntegrationHub ETL to create integrations, or customize a pre-existing integration provided by ServiceNow or a vendor at the ServiceNow Store. A vendor can create a new integration and provide it as an application for anyone to use.

    IntegrationHub ETL and Import Sets

    Using IntegrationHub ETL and ETL transform maps has the following advantages over using Import Sets and transform maps:
    • Identification and Reconciliation Engine (IRE) processes are incorporated into the IntegrationHub ETL so all data is automatically processed by IRE as part of the integration. Using Import Sets and transform maps does not provide a simple way to apply IRE processes.
    • IntegrationHub ETL uses guided setup which provides guidance and a simple user interface for the entire process of integrating third-party data.
    • IntegrationHub ETL includes an integration test for a small data set using the new ETL transform map. This test lets you review the results and adjust configuration settings before scheduling recurring integrations.

    Terms

    The following terms are associated with the IntegrationHub ETL:
    CMDB application
    Name of the third-party vendor such as SCCM 2019. A CMDB application has two associated attributes: Name and Discovery Source. When creating a new integration, ensure to configure a discovery source for the CMDB application that you plan to use, before using the IntegrationHub ETL.
    Data source
    The source feed, such as SCCM 7.0 Computer Identity, where the raw source data is imported from. If you use various REST endpoints for different types of data, then each REST endpoint is associated with its own data source and an ETL transform map.
    ETL transform map
    The output generated by IntegrationHub ETL. You can integrate third-party data into the CMDB, using an ETL transform map which is configured for the respective integration.
    Source data
    Original, raw data that has been imported into IntegrationHub ETL. Source data can be used in its original form, or you can transform the data before mapping and integration.
    Transform
    An operation, that you can apply to a specific data column to transform the data values. For example, to transform the format of the data values. Use transforms to standardize data formats and meet other system requirements.
    Transformed data
    Some of the source data might not be compliant with the requirements of its target CMDB attributes and classes. In those cases, you can apply various types of transforms to the source data, before mapping the data to the target CMDB classes and attributes. Transforms, can for example convert data format, replace values, and concatenate values from multiple data columns.
    Robust Transform Engine
    Robust Transform Engine (RTE) is used to transform raw source data that is stored in staging tables, into the data that is mapped and integrated into the CMDB. RTE uses ETL transform maps that were created for the integration during data transformation.

    Each CMDB application can have multiple connections for retrieving raw data. Each connection that is used to retrieve a certain type of data, has its own pair of data source and an ETL transform map. Therefore, one CMDB application can have multiple ETL transform maps, and each of those ETL transform maps is associated with a single Data Source.

    For example:
    CMDB Application ETL Transform Map Data Source
    SCCM SCCM Computer Identify /sccm/2019/comp
    SCCM Disk /sccm/2019/disk
    SCCM Application /sccm/2019/appl

    Nested data payloads

    To process nested data payloads, you must first ensure that the data source that is used for the integration, is set with the Data in single column option. With that setting, you can correctly represent nested data in a JSON payload which IntegrationHub ETL then processes as nested data, rather than as flat data.

    Sample of nested data:
    {
                "u_computer_fqdn": "computer2-fqdn",
                "u_computer_id": 2,
                "u_computer_ip": "computer2-ip",
                "u_computer_location": "PDX",
                "u_computer_mac": "computer2-mac",
                "u_computer_name": "nested-payload-computer2",
                "u_computer_os": "computer2-os",
                "interfaces": [
                    {
                        "u_interface_ip": "computer2-eth1-ip",
                        "u_interface_mac": "computer2-eth1-mac",
                        "u_interface_name": "computer2-eth1",
                        "ip": ""
                    },
                    {
                        "u_interface_ip": "computer2-eth2-ip",
                        "u_interface_mac": "computer2-eth2-mac",
                        "u_interface_name": "computer2-eth2",
                        "ip": {
                            "u_ip_address": "computer2-eth2-ip",
                            "u_mac_address": "computer2-eth2-mac"
                        }
                    }
                ],
                "software": [
                    {
                        "u_software_name": "computer2-software2",
                        "u_software_version": "computer2-software2-1.0",
                        "instance": {
                            "u_software_instance_name": "computer2-software1-instance"
                        }
                    },
                    {
                        "u_software_name": "computer2-software2",
                        "u_software_version": "computer2-software2-2.0",
                        "instance": {
                            "u_software_instance_name": "computer2-software2-instance"
                        }
                    }
                ]
            },
    
    You can view the layers of nested data in a separate panel in IntegrationHub ETL, apply transforms, map, and integrate that data into the CMDB.
    When creating a nested data JSON payload, the following restrictions apply:
    • Column names can't contain special characters such as *, [,], #, $, spaces, and dot.
    • Column names can't be “temp” or “object”, which are reserved for internal use.
    • Consistently throughout the payload, you must use an array or an object to represent data in a specific level, regardless of the number of items in the level. If you use an array for multiple items in one object, you must also use an array to represent a single item in other objects.

    Process

    IntegrationHub ETL uses both, RTE and IRE, that work together to process and integrate data. Data is first imported from a data source, and is then stored in temporary staging tables in Import Sets systems. Using the data in the staging tables and the ETL transform map created by IntegrationHub ETL, RTE creates IRE payloads which are then processed by IRE. IRE applies reconciliation processes to avoid potential problems such as duplicate CIs, ensuring that the CMDB remains healthy, and then integrates the resulting data into the CMDB.

    • Create an ETL transform map

      IntegrationHub ETL provides a guided setup which walks you through the completion of all necessary tasks for creating an ETL transform map for a specific integration.

    • Duplicate an ETL transform map

      Use an existing ETL transform map if you need to create a map that is mostly similar to that existing map. Select an existing map, specify a new data source for the new duplicate map, and then change or retain other settings such as data transforms.

    Related concepts
    • Integration Commons for CMDB (2.5.0)
    • Service Graph connector for ExtraHop (2.0.3)
    • Service Graph connector for Microsoft Intune (1.1.1)
    • Service Graph connector for Jamf (2.4.0)
    • Service Graph connector for Microsoft SCCM (2.1.1)
    • Service Graph connector for SolarWinds (2.2.1)
    • Service Graph connector for VMware Workspace ONE UEM (1.0.4)

    Tags:

    Feedback
    On this page

    Previous topic

    Next topic

    • Contact Us
    • Careers
    • Terms of Use
    • Privacy Statement
    • Sitemap
    • © ServiceNow. All rights reserved.

    Release version
    Choose your release version

      IntegrationHub ETL (2.0.1)

      • Save as PDF Selected topic Topic & subtopics All topics in contents
      • Unsubscribe Log in to subscribe to topics and get notified when content changes.
      • Share this page

      IntegrationHub ETL (2.0.1)

      Use the IntegrationHub ETL store app to create and manage ETL transform maps, which integrate third-party data into the CMDB without compromising the integrity of the CMDB. IntegrationHub ETL provides a simplified user interface that guides you through the integration process end-to-end, including a test integration run of sample data.

      Request apps on the Store

      Visit the ServiceNow Store website to view all the available apps and for information about submitting requests to the store. For cumulative release notes information for all released apps, see the ServiceNow Store version history release notes.

      When you create an integration, you import source data, transform data if needed, and select target CMDB classes and attributes to map the data to. Eventually, you run an integration test of the sample data, using your settings in the IntegrationHub ETL. You can then preview the integration test results and adjust any settings before scheduling recurring integration runs for large data sets. If you develop and test the ETL transform map on a development instance, then you can test and adjust the configuration before implementation on a production instance.

      For example, you can integrate data from SCCM (Microsoft System Center Configuration Manager).

      For more information, watch the IntegrationHub ETL | Importing resources into the CMDB video for an introduction and walk through of the IntegrationHub ETL tool.

      Roles required

      Users with the cmdb_inst_admin role can use IntegrationHub ETL to create integrations, or customize a pre-existing integration provided by ServiceNow or a vendor at the ServiceNow Store. A vendor can create a new integration and provide it as an application for anyone to use.

      IntegrationHub ETL and Import Sets

      Using IntegrationHub ETL and ETL transform maps has the following advantages over using Import Sets and transform maps:
      • Identification and Reconciliation Engine (IRE) processes are incorporated into the IntegrationHub ETL so all data is automatically processed by IRE as part of the integration. Using Import Sets and transform maps does not provide a simple way to apply IRE processes.
      • IntegrationHub ETL uses guided setup which provides guidance and a simple user interface for the entire process of integrating third-party data.
      • IntegrationHub ETL includes an integration test for a small data set using the new ETL transform map. This test lets you review the results and adjust configuration settings before scheduling recurring integrations.

      Terms

      The following terms are associated with the IntegrationHub ETL:
      CMDB application
      Name of the third-party vendor such as SCCM 2019. A CMDB application has two associated attributes: Name and Discovery Source. When creating a new integration, ensure to configure a discovery source for the CMDB application that you plan to use, before using the IntegrationHub ETL.
      Data source
      The source feed, such as SCCM 7.0 Computer Identity, where the raw source data is imported from. If you use various REST endpoints for different types of data, then each REST endpoint is associated with its own data source and an ETL transform map.
      ETL transform map
      The output generated by IntegrationHub ETL. You can integrate third-party data into the CMDB, using an ETL transform map which is configured for the respective integration.
      Source data
      Original, raw data that has been imported into IntegrationHub ETL. Source data can be used in its original form, or you can transform the data before mapping and integration.
      Transform
      An operation, that you can apply to a specific data column to transform the data values. For example, to transform the format of the data values. Use transforms to standardize data formats and meet other system requirements.
      Transformed data
      Some of the source data might not be compliant with the requirements of its target CMDB attributes and classes. In those cases, you can apply various types of transforms to the source data, before mapping the data to the target CMDB classes and attributes. Transforms, can for example convert data format, replace values, and concatenate values from multiple data columns.
      Robust Transform Engine
      Robust Transform Engine (RTE) is used to transform raw source data that is stored in staging tables, into the data that is mapped and integrated into the CMDB. RTE uses ETL transform maps that were created for the integration during data transformation.

      Each CMDB application can have multiple connections for retrieving raw data. Each connection that is used to retrieve a certain type of data, has its own pair of data source and an ETL transform map. Therefore, one CMDB application can have multiple ETL transform maps, and each of those ETL transform maps is associated with a single Data Source.

      For example:
      CMDB Application ETL Transform Map Data Source
      SCCM SCCM Computer Identify /sccm/2019/comp
      SCCM Disk /sccm/2019/disk
      SCCM Application /sccm/2019/appl

      Nested data payloads

      To process nested data payloads, you must first ensure that the data source that is used for the integration, is set with the Data in single column option. With that setting, you can correctly represent nested data in a JSON payload which IntegrationHub ETL then processes as nested data, rather than as flat data.

      Sample of nested data:
      {
                  "u_computer_fqdn": "computer2-fqdn",
                  "u_computer_id": 2,
                  "u_computer_ip": "computer2-ip",
                  "u_computer_location": "PDX",
                  "u_computer_mac": "computer2-mac",
                  "u_computer_name": "nested-payload-computer2",
                  "u_computer_os": "computer2-os",
                  "interfaces": [
                      {
                          "u_interface_ip": "computer2-eth1-ip",
                          "u_interface_mac": "computer2-eth1-mac",
                          "u_interface_name": "computer2-eth1",
                          "ip": ""
                      },
                      {
                          "u_interface_ip": "computer2-eth2-ip",
                          "u_interface_mac": "computer2-eth2-mac",
                          "u_interface_name": "computer2-eth2",
                          "ip": {
                              "u_ip_address": "computer2-eth2-ip",
                              "u_mac_address": "computer2-eth2-mac"
                          }
                      }
                  ],
                  "software": [
                      {
                          "u_software_name": "computer2-software2",
                          "u_software_version": "computer2-software2-1.0",
                          "instance": {
                              "u_software_instance_name": "computer2-software1-instance"
                          }
                      },
                      {
                          "u_software_name": "computer2-software2",
                          "u_software_version": "computer2-software2-2.0",
                          "instance": {
                              "u_software_instance_name": "computer2-software2-instance"
                          }
                      }
                  ]
              },
      
      You can view the layers of nested data in a separate panel in IntegrationHub ETL, apply transforms, map, and integrate that data into the CMDB.
      When creating a nested data JSON payload, the following restrictions apply:
      • Column names can't contain special characters such as *, [,], #, $, spaces, and dot.
      • Column names can't be “temp” or “object”, which are reserved for internal use.
      • Consistently throughout the payload, you must use an array or an object to represent data in a specific level, regardless of the number of items in the level. If you use an array for multiple items in one object, you must also use an array to represent a single item in other objects.

      Process

      IntegrationHub ETL uses both, RTE and IRE, that work together to process and integrate data. Data is first imported from a data source, and is then stored in temporary staging tables in Import Sets systems. Using the data in the staging tables and the ETL transform map created by IntegrationHub ETL, RTE creates IRE payloads which are then processed by IRE. IRE applies reconciliation processes to avoid potential problems such as duplicate CIs, ensuring that the CMDB remains healthy, and then integrates the resulting data into the CMDB.

      • Create an ETL transform map

        IntegrationHub ETL provides a guided setup which walks you through the completion of all necessary tasks for creating an ETL transform map for a specific integration.

      • Duplicate an ETL transform map

        Use an existing ETL transform map if you need to create a map that is mostly similar to that existing map. Select an existing map, specify a new data source for the new duplicate map, and then change or retain other settings such as data transforms.

      Related concepts
      • Integration Commons for CMDB (2.5.0)
      • Service Graph connector for ExtraHop (2.0.3)
      • Service Graph connector for Microsoft Intune (1.1.1)
      • Service Graph connector for Jamf (2.4.0)
      • Service Graph connector for Microsoft SCCM (2.1.1)
      • Service Graph connector for SolarWinds (2.2.1)
      • Service Graph connector for VMware Workspace ONE UEM (1.0.4)

      Tags:

      Feedback

          Share this page

          Got it! Feel free to add a comment
          To share your product suggestions, visit the Idea Portal.
          Please let us know how to improve this content

          Check any that apply

          To share your product suggestions, visit the Idea Portal.
          Confirm

          We were unable to find "Coaching" in Jakarta. Would you like to search instead?

          No Yes
          • Contact Us
          • Careers
          • Terms of Use
          • Privacy Statement
          • Sitemap
          • © ServiceNow. All rights reserved.

          Subscribe Subscribed Unsubscribe Last updated: Tags: January February March April May June July August September October November December No Results Found Versions Search preferences successfully updated My release version successfully updated My release version successfully deleted An error has occurred. Please try again later. You have been unsubscribed from all topics. You are now subscribed to and will receive notifications if any changes are made to this page. You have been unsubscribed from this content Thank you for your feedback. Form temporarily unavailable. Please try again or contact  docfeedback@servicenow.com  to submit your comments. The topic you requested does not exist in the release. You were redirected to a related topic instead. The available release versions for this topic are listed There is no specific version for this documentation. Explore products Click to go to the page. Release notes and upgrades Click to open the dropdown menu. Delete Remove No selected version Reset This field is required You are already subscribed to this topic Attach screenshot The file you uploaded exceeds the allowed file size of 20MB. Please try again with a smaller file. Please complete the reCAPTCHA step to attach a screenshot
          Log in to personalize your search results and subscribe to topics
          No, thanks Login