Thank you for your feedback.
Form temporarily unavailable. Please try again or contact docfeedback@servicenow.com to submit your comments.

Data Stream actions and pagination

Log in to subscribe to topics and get notified when content changes.

Data Stream actions and pagination

Send REST or SOAP requests from Flow Designer to APIs that return a stream of response data larger than 10 MB, or that return paginated results. Parse stream data into a series of complex object outputs and use the data pills in other actions in a flow.

For example, create a Data Stream action to import a large quantity of employee data from a third-party HR site. The Data Stream action sends a REST request to the third-party site and processes the response to populate records in the User [sys_user] table.

Benefits

Data Stream actions offer these benefits.

  • Parse and format a stream of response data larger than 10 MB.
  • Automatically send multiple requests to APIs that paginate results.
  • Enable flow designers to process large requests without complex coding or configuration.
  • Enable flow designers to process each object within a data stream using For each flow logic. For example, you might create a Data Stream action that imports document data from a third-party site. When you add the action to a flow, Flow Designer automatically adds the action to a For each flow logic block, enabling flow designers to easily create a record in ServiceNow for each object in the data stream. See Use a Data Stream action in a flow.

Limitations

Use Data Stream actions with these limitations in mind.

  • Running a Data Stream action on a MID Server is not currently supported.
  • Nesting Data Stream actions is not supported.
  • You cannot execute a Data Stream action from script. For example, you cannot use FlowAPI.executeAction() to run a Data Stream action.
  • Adding actions with wait conditions to Data Stream For Each flow logic blocks is not supported. For example, you cannot use the Ask for Approval or Wait for Condition actions within Data Stream For Each flow logic.

Action outline

Data Stream actions follow a set structure. Follow prompts to add and remove steps from the action outline. You cannot manually add steps to a Data Stream action.

Animation showing the Datastream Outline and selected options
Note: Clearing an option in a configuration page removes the step from the Data Stream outline and deletes all data associated with the step.

Action Preprocessing

Use the Action Preprocessing category to run a preprocessing script before the action sends the initial API request. For example, validate action inputs or set default values. Preprocessing executes once per action, before the first API request.

Selecting this option adds a script step to the Data Stream action. For more information, see Script step.

This is an optional Data Stream action component.

Request

Use the Request category to configure how the action sends API requests. The Request section executes once per page of results. Request components provide these configuration options.

Pagination Setup step

Request results in batches. Once one page of data is processed, the Data Stream action runs the request section again to return the next set of results. Use the pagination setup step to set up pagination options required by the API.

Note: For licensing purposes, each request counts as one transaction, including each request for the next page of results.

The value of the reserved, read-only getNextPage variable determines whether to request another page of results. As long as the getNextPage variable is true, the action continues to send requests for the next page.

This is an optional Request component.

Script step

Run a script before every request for the next page of results. Use this script for data validation and transformation when calling a paginated API. For example, generate a JSON payload for the next page request. Selecting this option adds a script step to the Data Stream action. For more information, see Script step.

This is an optional Request component.

REST or SOAP step

Send a REST or SOAP request to a third-party API. Select a data format to add an associated step to the Data Stream action. For more information, see REST step and SOAP step.

This is a mandatory Request component.

Parsing

Use the Parsing category to configure how the action separates data stream elements into complex data objects. Use the Splitter step to identify and separate items from an XML or JSON stream, and use the Script Parser step to transform each item into a complex object. The Parsing section executes once per item in the stream.

Splitting and parsing a stream of user records.

For more information about complex data, see Complex data. Parsing components provide these configuration options.

Splitter step

Identify the parent node in the response stream to map to a complex object. For example, identify a user element in an XML payload to create a complex object for each user in the response stream.

Select a splitter type to identify and separate repeated items in an XML or JSON data stream.

  • JSON: Identifies objects in a stream of JSON data.
  • XML: Identifies objects in an XML document.

This is a mandatory Parsing component.

Script Parser step

Use JavaScript and ServiceNow APIs to map items in the response stream to a complex object output represented by the targetObject global object. For example, map incident record elements identified in the splitter step to a complex object containing incident fields. If the data stream includes siblings to the item identified in the splitter step that you do not want mapped to a complex object, include conditions to exclude those items.

This is a mandatory Parsing component.

Data Stream outputs

When designing a Data Stream action, you must create a single output of type Object. The Script Parser step maps items in the stream to this object using the targetObject global object.

At runtime, the system splits and parses the stream of response data according to the Data Stream configuration. Each item in the stream maps to the complex object structure defined by the Script Parser step and the object output, resulting in a large series of complex objects. For more information about complex data, see Complex data.

Design considerations

Create Data Stream actions with these considerations in mind.

Write pagination logic according to third-party requirements

Evaluate and understand the format required by your third-party endpoint. For example, you may need to write a script that sets the read-only getNextPage variable to true as long as there is a nextPage token in the response. If the response does not contain the token, then set the variable to false. Access variables in script using bracket or dot notation. For example, variables['getNextPage']. This variable only accepts the Boolean data type. The default value is false.

Convert pagination variable data types to perform math operations

Pagination variables only support the string data type. To perform math operations, you must convert the value to an integer, perform any required operations, then convert it back to a string.

variables['offset'] = (parseInt(variables['offset']) + parseInt(variables['limit'])).toString();
Ensure that the pagination script has an end condition
Avoid infinite loops in pagination requests by creating a condition that sets the getNextPage variable to false. Cancel any long-running flows. Always test Data Stream actions before using them in production.
Do not add wait conditions to a Data Stream For Each logic block
Actions that wait, for example the Wait For Condition action, result in an error when used in a Data Stream For Each logic block. For example, suppose that you add a Data Stream action to a flow that processes a stream of expense records. You want to request approval for expenses over a certain dollar amount. Using the Ask for Approval action within the Data Stream For Each flow logic block is not supported.
Clear configuration page options carefully
Clearing an option in a configuration page removes the step from the Data Stream outline and deletes all data associated with the step.

MID Server support

Data Stream actions are not currently supported on a MID Server and generate an error when run from a MID Server. Data Stream actions cannot contain REST or SOAP steps that run on a MID Server. To prevent a flow from switching environments multiple times between the MID Server and the instance, avoid adding Data Stream actions between actions that run on the MID Server. For more information about when an action runs on the MID Server, see Integration steps.

Execution details

View the configuration and runtime results for each item processed by a Data Stream action. Select a record number to see its configuration and runtime details. By default, the execution details include requests for the first 1000 items. To change the number of items in the execution details, update the com.snc.process_flow.reporting.datastream.item.lastn system property.

Execution details for a Data Stream action.

Data stream summary

View an overview of the execution that includes this information.

  • Page count: Number of pages returned by a paginated API.
  • Total item count: Number of items in the response stream mapped to complex object outputs.
  • Error count: Number of errors encountered.
Page details

View runtime data for each step within the Data Stream action. Select a page to view runtime details for each request to a paginated API. By default, the execution details include requests for the first five pages. To change the number of requests in the execution details, update the com.snc.process_flow.page.reporting.lastn system property. Set the value to 0 to remove pages from the execution details and -1 to include all pages.

Note: Including all pages can affect performance and is not recommended.

Runtime details

Feedback