Test tool integration lets you view test results in DevOps for Jenkins, Azure DevOps, GitHub, GitHub Enterprise, and GitLab unit, functional, and performance tests.

For GitLab and Jenkins, only JUnit test type integration is supported.

Note: For other test types, use the DevOps - POST /devops/tool/{capability} endpoint of the DevOps API.
  • Selenium tests run and published using TestNG are reported by the Jenkins plugin for ServiceNow DevOps.
  • Test type categorization is supported.
  • Additional tests results reported by tools, such as Apache JMeter, can be processed in DevOps using custom Workflow Studio subflows (no pipeline changes required).
Category Test type
Unit

JUnit (default)

NUnit

XUnit

Unit test

Note:
  • For GitLab and Jenkins, only the JUnit test type integration is supported.
  • For ADO, GitHub, and GitHub Enterprise, the JUnit, NUnit, XUnit, and Unit test type integrations are supported.

You can change the default test type by modifying the [sn_devops.default_test_type] DevOps property.
Functional
  • Integration
  • Regression
  • Smoke
  • System
  • User Acceptance
Performance Load

Test type mapping

The test type mapping connects the test type and entity being tested with the DevOps tool (DevOps > Integrations > Test Type Mappings module.)

Test type mappings

An accurate test type mapping ensures that the test type always appears as intended in the test summary results.

Field Description
Test type
  • JUnit
  • Nunit
  • Xunit
  • Unit test
    Note:
    • For GitLab and Jenkins, only the JUnit test type integration is supported.
    • For ADO, GitHub, and GitHub Enterprise, the JUnit, NUnit, XUnit, and Unit test type integrations are supported.
  • Integration
  • Regression
  • Smoke
  • System
  • User Acceptance
  • Load
DevOps Entity Id Table name

DevOps table name that contains the entity linked to the test results (in the test report payload).

  • Step [sn_devops_step]
  • Pipeline [sn_devops_pipeline]
Note: Only DevOps Step and Pipeline tables are supported.
Document

Name of the entity specified in the selected table.

For example, the name of the step, pipeline, artifact, or package.

Test File Paths

(Jenkins tests only)

Path to the generated test result file on the Jenkins server.

This is useful so test reports with attributes that don't conform to JUnit or TestNG implementation, such as JMeter for example, can still be leveraged by DevOps.

Separate multiple files by a comma.

Note: You must use a Workflow Studio subflow to transform a raw test payload.
Tool integration

Tool that's running the test.
DevOps Table

DevOps table that corresponds to the table name in the DevOps Entity Id setting.
Figure 1. DevOps test type mapping
DevOps test type mapping
Figure 2. Example yaml file of ADO unit test
Example yaml file of ADO unit test
Figure 3. Example yaml file of GitHub unit test
Example yaml file of GitHub unit test

Transforming a raw test payload

You can transform a raw test report (a report containing attributes that do not conform to JUnit or TestNG implementation), such as JMeter for example, by configuring the DevOps Test Subflow Policy decision table to call a custom subflow (Decision Tables > Decision Tables module).
Note: You must create the custom Workflow Studio subflow that transforms the raw test payload.

If there is more than one test type in a performance stage, you can use the DevOps Test Type Policy decision table to configure the test type for each test so the test result payloads are transformed correctly.

Figure 4. DevOps decision tables
DevOps decision tables
Table 1. DevOps decision tables
Decision table Purpose Configuration
DevOps Test Subflow Policy

To automatically call a custom subflow that transforms the raw payload received by the tool.

Decision inputs:

  • Test Result Payload
  • Test Type

Create a decision that specifies the custom subflow to call when the raw payload is received.

Set the conditions to contain fields that would be included in the raw payload.

For example, to call Jenkins BZ Performance Test custom subflow:

Conditions:
  • Test Type is Load

    (Load is the test type configured for performance tests)

  • Test Result Payload contains throughput
  • Test Result Payload contains concurrency

Answer: FLow: Jenkins BZ Performance Test
DevOps Test Type Policy

To automatically set test types when more than one type of test is configured in a performance test stage.

This is necessary so the results for the second test type get transformed correctly.

For example, when both a Load performance test and a JUnit performance test are mapped in the same DevOps step, the JUnit test results won't get formatted correctly unless a decision is created.

Decision inputs:
  • Step
  • Test Result Payload
  • Tool Integration
  • Pipeline

Create a decision for each type of test in the performance test stage to set the test type.

Load test:
  • Conditions:
    • Step is Perf Tests
    • Test Result Payload contains throughput
    • Test Result Payload contains concurrency

  • Answer: TestType: Load

JUnit test:

  • Conditions:
    • Step is Perf Tests
    • Test Result Payload does not contain throughput
    • Test Result Payload does not contain concurrency

  • Answer: TestType: JUnit
Figure 5. DevOps multiple performance test types
DevOps multiple performance test types
Figure 6. DevOps multiple test type mappings
DevOps test type mappings
Figure 7. DevOps decision table decision
DevOps decision table decision

Test summary results

You can view test summary results these ways.
Figure 8. DevOps performance test summary example
DevOps performance test summary

Expected standard JSON Notification capability payload - Test tool

Functional:
{ 
"name": "CorpSite-selenium#55", 
"duration": 78.802, 
"passedTests": 4, 
"failedTests": 0, 
"skippedTests": 0, 
"blockedTests": 0, 
"totalTests": 4, 
"startTime": "2020-06-30T18:12:31Z", 
"finishTime": "2020-06-30T18:12:31Z", 
"passingPercent": 100, 
 
 
// Use Artifact OR Package OR Build + Stage + PipelineName Attributes 

Send only one Attribute combination. For example, send Attributes of either  Artifact or Package, or the combination of Build + Stage + PipelineName.

If you send more than one Attribute, priority is given in the following order and the low priory one is ignored. For example, if you send attribute for both packages and artifacts, then attribute of package is considered and the attribute of artifacts is ignored.

1.packages
2.artifcats
3.buildNumber + stageName + pipelineName

"packages": [{"name": "CorpSite-pkg1"}], 
"artifacts": [{"name": "CorpSite-artifact", "version": "1.0.0"}], 
"buildNumber": "55", 
"stageName": "test", 
"pipelineName": "CorpSite-selenium", 
} 

Notes:
- The pipelineName attribute value must be same as the value in the Orchestration pipeline field of the Pipeline [sn_devops_pipeline] table.
- The stageName attribute value must be same as the value in the Orchestration stage field of the Step [sn_devops_step] table.
Performance:
{ 
"name": "Performance Tests", 
"url": "http://abc.com", 
"startTime": "2020-06-30T18:12:31Z", 
"finishTime": "2020-06-30T18:12:31Z", 
"duration": 78.802, 
"maximumVirtualUsers": "", 
"throughput": "", 
"maximumTime": "", 
"minimumTime": "", 
"averageTime": "", 
"ninetyPercent": "", 
"standardDeviation": "", 
 
// Use Artifact OR Package OR Build + Stage + PipelineName Attributes 

Send only one Attribute combination. For example, send Attributes of either  Artifact or Package, or the combination of Build + Stage + PipelineName.

If you send more than one Attribute, priority is given in the following order and the low priory one is ignored. For example, if you send attribute for both packages and artifacts, then attribute of package is considered and the attribute of artifacts is ignored.

1.packages
2.artifcats
3.buildNumber + stageName + pipelineName

"packages": [{"name": "CorpSite-pkg1"}], 
"artifacts": [{"name": "CorpSite-artifact", "version": "1.0.0"}], 
"buildNumber": "55", 
"stageName": "test", 
"pipelineName": "CorpSite-Performance", 
} 

Notes:
- The pipelineName attribute value must be same as the value in the Orchestration pipeline field of the Pipeline [sn_devops_pipeline] table.
- The stageName attribute value must be same as the value in the Orchestration stage field of the Step [sn_devops_step] table.