Data archiving involves managing table size growth and archiving old data. It moves data that is no longer needed every day from primary tables to a set of archive tables.

The longer an instance runs, the more likely it is to accumulate data that is no longer relevant. For example, task records from two years ago are typically less relevant than currently active tasks. Old data may eventually cause performance issues by consuming system resources and slowing down queries and reports.

If you cannot delete this data because you need it for auditing or for historical purposes:
  1. Set up an archive rule that archives the data and removes it from immediate access to free up system resources.
  2. Set up a destroy rule to delete the data after a specified period.
  3. Configure the archive and destroy rule properties that control batch processing of the archive and destroy rules.

You can archive records in core tables such as the Task [task] table and records in custom tables that you create on the Now Platform.

To archive Configuration Management Database (CMDB) CI records, use the Working with CMDB Data Manager.

Figure 1. Sample Benefits of Archiving Data
Sample Benefits of Archiving Data

Data archiving supports Domain separation for service providers. For example, incidents that belong to a domain keep their domain designation even after they are archived.

Note: If you want to archive emails, activate the Email retention plugin and use the archive and destruction rules that come with the plugin. Do not use the archive feature to create your own archiving rules on the email table.

Tables and modules created by an archive rule

The first time you activate an archive rule, it performs the following actions:

  • Creates the archive table in the database. The archive table has the same name as the primary table with an "ar_" prefix. For example, if you archive the Incident [incident] table, then the archive table is [ar_incident].
  • Stores an xml version of each archived record in the sys_archive_log table. This archive log is the same table for all archive rules, and you cannot alter this behavior. It is also the only place where the sys_id is stored together with the display value for reference fields.

Example

For example, for ar_incident <assigned_to>Fred Luddy</assigned_to>, the sys_archive_log record is as follows: 

<assigned_to display_value="Fred Luddy">5137153cc611227c000bbd1bd8cd2005</assigned_to>
  • Converts multiple joined tables into a single flat-file archive table. The archive table no longer consists of a base and extended tables.
  • Converts reference field values (values set by references to records in other tables) into string values. The archive record contains the display value of the reference field at the time of the archive.
  • Adds a module to the Archive Tables list in the System Archiving application. The module name is a combination of the word "Archive" plus the display name for the archived table. For example, the archive module for the Attachment [sys_attachments] table is Archive Attachment. Click the module name to view records in the archive table.
  • Creates a list of the archive table using the default list view.
  • Creates a form for the archive table using the default form view. The form excludes any dot-walking fields such as Caller ID.Email.
Figure 2. Conversion of Multiple Joined Tables into a Flat Archive Table
Conversion of Multiple Joined Tables into a Flat Archive Table

Reference values converted to strings

Archived data is stored as a flat file with no reference fields to other tables. The archive process converts any references to other tables to string values.

In the case of a reference field, the string uses the display value such as the caller's user name. For example, the Caller reference field in an incident would display the string ITIL User. If the reference was a document ID and the archive rule included the option to archive related document IDs, then the string is the document ID of the related record.

It is important to note that archive records do not receive any future changes to referenced values. For example, if you change the user name for "John Smith" to "John A Smith", all active incident records automatically show the caller as "John A Smith" because of the reference between the Incident and User tables. However, all archived incident records display the user name that existed at the time of the archive. Any incident for "John Smith" continues referencing this user. Likewise, if you delete a user from the system, current incidents no longer display the deleted user as a caller. However, there can be archived incidents that still display the string "John Smith" as the user because the user existed at the time of the archive.

Query archived data

Archived tables are not optimized for ad hoc queries. They only contain index entries for the display value, creation date, and the primary key of sys_id.

For this reason, do not make on-demand queries against an archived table, such as searching for all priority 1 archived incidents. Instead, only search against the indexed fields. For example, search for incident INC100001 or incidents created on a specific date.

Archive tables and ACLs

By default, archive tables use the ACLs for the unarchived table of the same name. For example, the archived Incident [ar_incident] table uses the ACLs defined for the unarchived Incident [incident] table.

You can manage access to archive tables explicitly by creating ACLs for specific archive tables and setting the glide.security.enable_archive_table_acls property to true. The system then follows one of two paths:
  1. If one or more active ACLs are defined for an archive table, those ACLs control access to the archive table.
  2. If no ACLs are defined for an archive table, the system reverts to default behavior and uses the ACLs for the unarchived version of the table.
Note: The two paths are mutually exclusive: If archive table ACLs deny access, the system doesn't attempt to revert to the default behavior.

The read operation read is the only operation evaluated, and other operations are prevented.

The Execution Plan UI is aware of this logic and presents information accordingly. For example, adding the first ACL to an archive table shows that the archive table ACL is "masking" ACLs on the unarchived (original data) table.

If you have existing ACLs on archived tables, they are ignored unless you set the glide.security.enable_archive_table_acls property to true. Those newly activated ACLs may possibly cause access issues. To prevent this occurrence, the system sets the glide.security.enable_archive_table_acls property as follows:
  • Instances without the glide.security.enable_archive_table_acls property use the default value of false.
  • Upgraded instances don't install the property. The property must be added manually and set to true to enable the archive table ACL behavior.
  • Zbooted instances install the property and set it to true.