Starting with the Tokyo release, a new, optimized data collector is available. The log details for this data collector differ from the log details of the classic data collector.

Collection overview

The data collection job involves executing an SQL query for each indicator source that uses the data collector. The query repeats for every collection time from the start date to the stop date. Then queries run for the next indicator source. Each step of executing a query is documented in the data collection job log.
Figure 1. High-level view of data collection job
Flow chart showing the relation between the different levels of processes in a data collection job

Starting and completing collection job

Follow the data collection process as revealed in log messages. The log messages in each subprocess have been separated into their own tables.

Note: This log is for the optimized HSQL data collector as described in Optimizing data collection. If com.snc.pa.dc.hsql is "false," you are using the classic data collector. Otherwise, you are using the optimized data collector. For more information about that log, see Log details for classic data collector.
Collection job stepExample of resulting log entry
Shows the domain of the collection job Run As user. User <admin> belongs to domain global
Collection starts. Job start specifies whether scores, text index, or both are being collected. Messages also specify that the optimized HSQL data collector is in use and whether debug logs are enabled. Only Customer Service and Support can enable debug logs. Starting collection of Both scores and text index. Value of properties in system -

Using HSQL data collector: true

Debug logs enabled: true

Node Name : vsr007
Shows the domains for which scores are collected. Also shows other relevant details about the domains. DomainConfig: Testing Domain Config Collection logs - 3d359877b77000106886e664de11a97c, Type: conditions, RollUpType: roll_up_children, collectAggregate? Yes, collectChildren?: Yes, AggregateDomain: Name: 3d359877b77000106886e664de11a97c - SysID: d1659c77b77000106886e664de11a91e, Domains: Default,MSP,ACME,Cisco,Initech,Oceanic Airlines
Lists indicator sources that the job collects data for, including their unique IDs. Indicator Sources to collect: Incidents.New: b54f2c23d7030100b96d45a3ce61032f, Incidents.Open: b5cf2c23d7030100b96d45a3ce6103cc, Incidents.Closed: 2ac8dd31ff2302001e68ffffffffff3d, Incidents.Resolved: 4c207c23d7030100b96d45a3ce610308
Lists each indicator associated with the job and the indicator properties that are related to data collection:
  • Are records collected in a snapshot?
  • Are any breakdowns excluded?
  • Are the collection periods in the indicator source overridden?
  • Is the breakdown matrix collected?
  • If the breakdown matrix is collected, are any breakdown combinations excluded?
 JobIndicator: Number of open incidents - de537dc3d7131100b96d45a3ce610305 Job Indicator Properties: collectIndicator?: Yes, BreakdownCollectionType: WITH_EXCLUDE Excluded Breakdowns: Assignment Group, State Indicator Properties => CollectingRecords: Yes, Overridden Collection Periods: No, Scripted?: No, Breakdown Combinations?: Yes, Excluded Breakdown Combinations: No combination is excluded.
Iterate steps for each indicator source See table Steps iterated for each indicator source.
Show statistics for the data collection job Statistics: inserts 184, updates 0, deletes 92, errors 0, warnings 0
Complete job Collection ended

Steps iterated for each indicator source

Collection job stepExample of resulting log entry
Retrieve indicator source. Processing indicator source Incidents.Open
Repeat that optimized HSQL data collector is in use.
List indicators that have additional conditions, beyond the conditions inherited from the indicator source. Indicator: Admin Logins - [sys_id of indicator], Additional Conditions: ref_isc_login_event.login_type=login_admin, Is Collect Records: true
Iterate steps for each collection period. See table Steps iterated for each collection period
Retrieve next indicator source. Processing indicator source Incidents.New

Steps iterated for each collection period

Collection job step Example of resulting log entry
Start date of collection job. Collecting for 20191103 on Indicator Source Incidents.Open
Fetch the following fields:
  • The unique fields of the breakdown sources (typically sys_id)
  • The indicator fields used in breakdown mappings
  • The table fields used in scripted breakdown mappings
  • The indicator source fields used in text analytics, if the text index is collected (typically includes short_description)
 Fetching "short_description,sys_id, opened_at, assignment_group, description, priority, category" from "incident"
Generate SQL based on the conditions that are specified in the indicator source.
Note: If the indicator source specifies Today in one of the conditions, Today is considered relative to the period for which the data collection job is executed. For example, the Incidents.New indicator source includes the condition [Opened][on][Today]. With days defined to start at 07:00:00, when data is collected for 2019-10-28, the job produces the SQL script on the right.
 SELECT task0.`sys_id` FROM task task0 WHERE task0.`sys_class_name` = 'incident' AND (task0.`opened_at` >= '2019-10-28 07:00:00' AND task0.`opened_at` <= '2019-10-29 06:59:59')
Display the number of records fetched by the SQL. Fetched 150 rows from Indicator Source Incidents.Open (facts table: incident)
Delete previous results. Deleted previous results 1 for Indicator: Impersonations
Run the map/reduce function to calculate indicator scores. See the table Steps of the map/reduce function
If text indexing is active and has been configured for the indicator source, the data collector stores the resulting text index. Storing Text Index for indicator source Incidents.Open
Bytes used by text index: 41,984 for: Incidents.Open
Objects are offloaded to the HSQL database whenever one of the following thresholds is exceeded:
  • Number of scores
  • Number of snapshots
  • Total number of Count Distinct values

Log number of times the offload limit was exceeded for each category of object. Each time the offload limit is exceeded in one category, all objects are moved to HSQL temporary database storage. At the end of the map/reduce function, any remaining records are offloaded. The number of remaining records is usually less than the threshold, and this offloading is also logged.

 Number of times offload limit crossed threshold for scores: 275

Number of times offload limit crossed threshold for snapshots: 0

Number of times offload limit crossed threshold for distinct values: 0

Offload batch of records which didn't cross threshold: 1
Begin storing newly collected results for the indicator source. Storing collected results for indicator source Incidents.Resolved
Any indicator with a null value is logged. Stored 'Value when nil': 0.0 as score for indicator: Number of incidents deflected by knowledge
Finish storing results. Stored collected results
Specify which indicators the data collector does not collect scores for. Not collecting for Indicator: Summed age of open incidents with excluded Breakdown: Assignment Group
Finish collecting data for that indicator source for that period. Collection for 20200519 on Indicator Source Incidents.Open finished
Display the average amount of time spent on a record to process additional conditions that were set on indicators. Average time taken to process additional indicator condition for each record: 138 ms

Steps of the map/reduce function

Collection job step Example of resulting log entry
The map/reduce function runs. Applying map/reduce function for indicator source Incidents.Open
Warning if text indexing is active for the job but cannot run because of a missing or invalid text index configuration. Skipping text index collection as there is missing or invalid text index configuration
Process each record on the indicator source to calculate scores. Log messages track running total of processed records.

Each time the number of resulting scores or snapshots exceeds a limit, all objects are offloaded to the HSQL database. The offloading is logged after the map/reduce function completes.

Processing for each record involves:

  • Execute or evaluate any scripts.
  • Validate indicator conditions.
  • Calculate the indicator score.
  • For each breakdown:
    1. Calculate the breakdown score or execute the breakdown script.
    2. Retrieve all breakdown unique values.
    3. Create or update the array for scores or snapshots. The array is: Indicator, Breakdown 1, Artifact 1, Breakdown 2, Artifact 2, Domain, Value, Array of [sys_id]
 Processed 15.8% (3) records
Processed 31.6% (6) records
Processed 47.4% (9) records
Processed 63.2% (12) records
Processed 78.9% (15) records
Processed 94.7% (18) records
Processed 100% (4) records, Result Object size: 20 MB
The map/reduce function completes. Applied map/reduce function