The two key components that IntegrationHub ETL uses for processing are:

• Robust Transform Engine (RTE): Used to transform raw source data that is stored in staging tables, into the data that is mapped and integrated into the CMDB. RTE uses ETL transform maps that were created for the integration during data transformation.
• Identification and Reconciliation engine (IRE): Used as a centralized framework for identification and reconciliation processes across different data sources. IRE processes help maintain data integrity in the CMDB.

IntegrationHub ETL uses RTE and IRE which work together to process and integrate data. Data is first imported from a data source, and is then stored in temporary staging tables in Import Sets systems. Using the data in the staging tables and the ETL transform map created by IntegrationHub ETL, RTE creates IRE payloads which are then processed by IRE. IRE applies reconciliation processes to avoid potential problems such as duplicate CIs, ensuring that the CMDB remains healthy, and then integrates the resulting data into the CMDB.

Using IntegrationHub ETL and ETL transform maps has the following advantages over using Import Sets and transform maps:

• Identification and Reconciliation Engine (IRE) processes are incorporated into the IntegrationHub ETL so all data is automatically processed by IRE as part of the integration. Using Import Sets and transform maps does not provide a simple way to apply IRE processes.
• IntegrationHub ETL uses guided setup which provides guidance and a simple user interface for the entire process of integrating third-party data.
• IntegrationHub ETL includes an integration test for a small data set using the new ETL transform map. This test lets you review the results and adjust configuration settings before scheduling recurring integrations.

The following terms are associated with the IntegrationHub ETL:

CMDB application

Name of the third-party vendor such as SCCM 2019. A CMDB application has two associated attributes: Name and Discovery Source. When creating a new integration, ensure to configure a discovery source for the CMDB application that you plan to use, before using the IntegrationHub ETL.

Data source

The source feed, such as SCCM 7.0 Computer Identity, where the raw source data is imported from. If you use various REST endpoints for different types of data, then each REST endpoint is associated with its own data source and an ETL transform map.

ETL transform map

The output generated by IntegrationHub ETL. You can integrate third-party data into the CMDB, using an ETL transform map which is configured for the respective integration.

Source data

Original, raw data that has been imported into IntegrationHub ETL. Source data can be used in its original form, or you can transform the data before mapping and integration.

Transform

An operation, that you can apply to a specific data column to transform the data values. For example, to transform the format of the data values. Use transforms to standardize data formats and meet other system requirements.

Transformed data

Some of the source data might not be compliant with the requirements of its target CMDB attributes and classes. In those cases, you can apply various types of transforms to the source data, before mapping the data to the target CMDB classes and attributes. Transforms, can for example convert data format, replace values, and concatenate values from multiple data columns.

For example:

CMDB Application	ETL Transform Map	Data Source
SCCM	SCCM Computer Identify	/sccm/2019/comp
	SCCM Disk	/sccm/2019/disk
	SCCM Application	/sccm/2019/appl

Sample of nested data:

{
            "u_computer_fqdn": "computer2-fqdn",
            "u_computer_id": 2,
            "u_computer_ip": "computer2-ip",
            "u_computer_location": "PDX",
            "u_computer_mac": "computer2-mac",
            "u_computer_name": "nested-payload-computer2",
            "u_computer_os": "computer2-os",
            "interfaces": [
                {
                    "u_interface_ip": "computer2-eth1-ip",
                    "u_interface_mac": "computer2-eth1-mac",
                    "u_interface_name": "computer2-eth1",
                    "ip": ""
                },
                {
                    "u_interface_ip": "computer2-eth2-ip",
                    "u_interface_mac": "computer2-eth2-mac",
                    "u_interface_name": "computer2-eth2",
                    "ip": {
                        "u_ip_address": "computer2-eth2-ip",
                        "u_mac_address": "computer2-eth2-mac"
                    }
                }
            ],
            "software": [
                {
                    "u_software_name": "computer2-software2",
                    "u_software_version": "computer2-software2-1.0",
                    "instance": {
                        "u_software_instance_name": "computer2-software1-instance"
                    }
                },
                {
                    "u_software_name": "computer2-software2",
                    "u_software_version": "computer2-software2-2.0",
                    "instance": {
                        "u_software_instance_name": "computer2-software2-instance"
                    }
                }
            ]
        },

You can view the layers of nested data in a separate panel in IntegrationHub ETL, apply transforms, map, and integrate that data into the CMDB.

When creating a nested data JSON payload, the following restrictions apply:

• Field names must start with a letter (between A-Z or a-z) or with ‘_’, and must only contain letters (between A-Z or a-z), digits (0-9), or the ‘_’ character.

  For example, a field name can't contain special characters such as *, [,], #, $, spaces, and dot.

• Field names can't be “temp” or “object”, which are reserved for internal use.
• Consistently throughout the payload, you must use an array or an object to represent data in a specific level, regardless of the number of items in the level. If you use an array for multiple items in one object, you must also use an array to represent a single item in other objects.