Send REST or SOAP requests from Flow Designer to APIs that
return a stream of response data larger than 10 MB, or that return paginated
results. Parse stream data into a series of complex object outputs and use the data
pills in other actions in a
flow.
For example, create a Data Stream action
to import a large quantity of employee data from a third-party HR site. The Data Stream action
sends a REST request to the third-party site and processes the response to populate records in
the User [sys_user] table.
Benefits
Data Stream
actions offer these benefits.
- Parse and format a stream of response data larger than 10 MB.
- Automatically send multiple requests to APIs that paginate results.
- Enable flow designers to process large requests without complex coding or
configuration.
- Enable flow designers to process each object within a data stream using
For each flow logic. For example, you might create a Data Stream action that imports document data from a third-party site. When you add the
action to a flow, Flow Designer automatically
adds the action to a For each flow logic block, enabling
flow designers to easily create a record in ServiceNow for each object in
the data stream. See Use a Data Stream action in a flow.
- Enable flow designers to reuse Data Stream
actions in multiple flows, using the same source of data in multiple ways.
Running a Data Stream
action
There are two ways to run a Data Stream
action.
- From a flow
- You can process each object within a data stream using
For each flow logic. For example, you might create a Data Stream action that imports document data from a third-party site. When you add the
action to a flow, Flow Designer automatically
adds the action to a For each flow logic block, enabling
flow designers to easily create a record in ServiceNow for each object in
the data stream. See Use a Data Stream action in a flow.
- From a script
- You can start a Data Stream
using the executeDataStreamAction() method in the FlowAPI class. For more information,
see FlowAPI.
Action outline
Data Stream
actions follow a set structure. Follow prompts to add and remove steps from the action
outline. You cannot manually add steps to a Data Stream
action.
Note: Clearing an option in a configuration page removes the
step from the Data Stream outline and deletes all data associated with the step.
Action Preprocessing
Use the Action Preprocessing category to run a preprocessing script before the action sends the
initial API request. For example, validate action inputs or set default values.
Preprocessing executes once per action, before the first API request.
Selecting this option adds a script step to the Data Stream action. For more information,
see Script step.
This is an optional Data Stream
action component that runs on either the instance or a MID Server.
Request
Use the Request category to configure how the action sends API requests. The Request
section executes once per page of results. Request components provide these configuration
options.
- Pagination Setup step
-
Request results in batches. Once one page of data is processed, the
Data Stream action runs the request section again to return the next set of
results. Use the pagination setup step to set up pagination options required by the API.
Configure the Pagination Setup step manually, or select a pre-built template to
apply common configurations. For example, apply the Limit /
Offset template to specify the number of items you want returned per
page (limit), and the starting number for the first item (offset). After applying
a template, update the values to ensure that the configuration complies with the
API's requirements.
Note: For licensing purposes, each request counts as one transaction, including each
request for the next page of results.
The value of the reserved, read-only getNextPage variable
determines whether to request another page of results. As long as the getNextPage variable is
true and the previous page contains data, the action continues to send requests
for the next page.
Note: You must explicitly set the value to true in the script or it will default to
false.
This is an optional Request component that only runs on the instance.
- Script step
-
Run a script before every request for the next page of results. Use this script for
data validation and transformation when calling a paginated API. For example,
generate a JSON payload for the next page request. Selecting this option adds a script step to the Data Stream action. For more information,
see Script step.
This is an optional Request component that runs on either the instance or a MID Server.
- REST or SOAP step
-
Send a REST or SOAP request to a third-party API. Select a data format to add an
associated step to the Data Stream action. For more information, see REST step and SOAP step.
This is a mandatory Request component that runs on either the instance or a MID Server.
Parsing
Use the Parsing category to configure how the action separates data stream elements into
complex data objects. Use the Splitter step to identify and separate items from an XML or
JSON stream, and use the Script Parser step to transform each item into a complex object.
The Parsing section executes once per item in the stream.
For more information about complex data, see Complex data. Parsing components provide these
configuration options.
- Splitter step
-
Identify the parent node in the response stream to map to a
complex object. For example, identify a user element in an XML payload to create a
complex object for each user in the response stream.
Select a splitter type to identify and separate repeated items in an XML or JSON
data stream.
- JSON: Identifies objects in a stream of JSON data.
- XML: Identifies objects in an XML document.
This is a mandatory Parsing component that only runs on the instance.
- Script Parser step
-
Use JavaScript and ServiceNow APIs to map items
in the response stream to a complex object output represented by the
targetObject global object. For example, map incident record
elements identified in the splitter step to a complex object containing incident
fields. If the data stream includes siblings to the item identified in the
splitter step that you do not want mapped to a complex object, include conditions
to exclude those items.
This is a mandatory Parsing component that only runs on the instance.
Data Stream outputs
When designing a Data Stream
action, you must create a single output of type Object. The Script Parser step maps items in
the stream to this object using the targetObject global object.
At runtime, the system splits and parses the stream of response data according to the Data Stream
configuration. Each item in the stream maps to the complex object structure defined by the
Script Parser step and the object output, resulting in a large series of complex objects.
For more information about complex data, see Complex data.
Execution details
View the configuration and runtime results for each item processed by a Data Stream
action. Select a record number to see its configuration and runtime details. By default, the
execution details include requests for the last 1000 items. To change the number of items in
the execution details, update the
com.snc.process_flow.reporting.datastream.item.lastn system
property.
- Data stream summary
-
View an overview of the execution that includes this information.
- Page count: Number of pages returned by a paginated
API.
- Total item count: Number of items in the response
stream mapped to complex object outputs.
- Error count: Number of errors encountered.
- Page details
-
View runtime data for each step within the Data Stream action. Select a page to view runtime details for each request to a paginated
API. By default, the execution details include requests for the last five pages. To
change the number of requests in the execution details, update the
com.snc.process_flow.page.reporting.lastn system property.
Set the value to 0 to remove pages from the execution details
and -1 to include all pages.
Note: Including all pages can affect performance and is not recommended.
