Thank you for your feedback.
Form temporarily unavailable. Please try again or contact docfeedback@servicenow.com to submit your comments.

MID Server upgrades

Log in to subscribe to topics and get notified when content changes.

MID Server upgrades

Upgrade MID Servers manually, or automatically through their host computers. The instance automatically tests to identify issues before MID Servers upgrade. Pin MID Servers to specific versions using system properties.

Warning: The MID Server cannot auto-upgrade on a Windows host if the Windows Application Experience service is disabled. For information on the error that is displayed and instructions for re-enabling this service, see KB0597552.

Access to the MID Server download site

The MID Server host computer must have access to the ServiceNow download site at install.service-now.com to upgrade automatically. If you have a self-hosted ServiceNow environment that blocks access to the download site, you must import the MID Server installer package into your MID Server hosts manually. For instructions, see KB0760123 in the Self-Hosted knowledge base.

Upgrade methods

  • Automatic: Allow the instance to automatically upgrade the MID Server. This functionality is available by default. Automatic upgrade occurs:
    • Every hour, when the MID Server checks with the instance to see if there is a different version available for upgrade. You cannot modify this time period.
    • When the instance is upgraded and the MID Server for that version is different than the version currently on the MID Server.
    • When the MID Server pre-upgrade test passes without an error. Any errors encountered during this automatic test prevent the upgrade from occurring until the issues are resolved. The pre-upgrade test is enabled by default, but can be disabled by adding and setting a system property.
  • Manual: Manually start the upgrade by clicking a related link on the MID Server record. Use this method when you do not want to wait until the next hourly automatic update or if your upgrade failed and you want to force an upgrade. See Upgrade the MID Server manually for instructions.

The Upgrade state

The instance initiates the upgrade by sending the autoUpgrade system command to the MID Server. Starting with MID Servers upgrading from an Istanbul version MID Server, the MID Server Status is changed to Upgrading while the upgrade is running. The Upgrading state is similar to the Paused state. This is done to avoid potential miscommunication between the new version of the instance and the previous version of the MID Server during upgrade. For the upgrade to run, MID servers must be in the Up state and must be validated.

While in the Upgrading state, you cannot resume or restart the MID Server. However, you can perform the same actions that you can when the MID Server is in the Paused state. After a successful upgrade, the queued output is sent to the instance and the MID Server starts retrieving new commands to process. The status also changes to Up.

When the instance sends the autoUpgrade system command to the MID Server, if it is Down or Paused, or if it has not been validated, the command remains in the ECC Queue until the MID Server status changes to Up. Then the command is processed.
Note: If you are using an Istanbul instance but you are upgrading a pre-Istanbul MID Server to Istanbul, these upgrade states are not available. They are only available for MID Servers that are already on Istanbul.

Failed upgrades

Failed upgrades are handled differently based on the version you are upgrading to:
  • Upgrade to another major release (such as Istanbul to the next full release): the status changes to Upgrade Failed.
  • Upgrade from a minor version within a release (such as Jakarta patch 1 to patch 2): the MID Server continues using the version it is currently running. It does not perform the upgrade and the status eventually changes to Up, assuming the MID Server was already functioning properly.

Upgrading MID Servers in the Down state

If a MID Server is in the Down state, it cannot process the upgrade command. When the MID Server changes to Up, it immediately checks to see if an upgrade is necessary. If it does need to upgrade, the upgrade process starts before the MID Server processes any other commands.

Upgrade error messages

The MID Server can display the following upgrade error messages.

Table 1. Upgrade error messages
Message Description
Unable to refresh packages Generic error when the error is not handled by a defined error message.
Failed to query instance for MID Server buildstamp Instance is unavailable or there is a major version mismatch between the MID Server and the instance.
Not a valid package buildstamp InstanceInfo returned an assigned buildstamp that was not in the correct format, such as a version mismatch.
Method failed (<instance URL>) HTTP/1.1 409 Conflict with code: 409 MID Server auto-upgrade request rejected when number of concurrent upgrades exceeds 2. The configured number of available semaphores for MID Server upgrades on the instance is set at 2 to prevent performance degradation.
Note: The MID Server re-sends the request every 20 seconds until it can get the semaphore.

MID Server Upgrade History

Use the MID Server Upgrade History module for troubleshooting problems with MID Server upgrades. The module contains a record of each instance upgrade. Those records provide step-by-step status details for each MID Server's upgrade process. If an error occurs, it is noted in the step and a message is dynamically generated with further details. For further information, see MID Server Upgrade History.

MID Server pre-upgrade check

The instance automatically tests the MID Server's ability to upgrade on your system prior to the actual upgrade, to identify issues that could cause a MID Server outage or require re-installation.

Each MID Server contains an AutoUpgrade monitor that compares the MID Server version with that of the instance to determine if the MID Server needs to upgrade. If the AutoUpgrade monitor discovers that the MID Server version is out of date, the monitor runs pre-upgrade validation tests for that MID Server. If an issue is detected, a message is logged to the MID Server Issue [ecc_agent_issue] table, and the upgrade is blocked. The AutoUpgrade monitor continues to run every hour, until all the tests pass. If there are no blocking issues, the MID Server downloads the appropriate upgrade package and begins the upgrade process.

Failed tests leave the MID Server in one of these states:
  • Upgrade Failed: For upgrades to a different release family, such as from London to Madrid.
  • Up: For upgrades within the same release family, such as an upgrade to a patch.
Errors, such as insufficient disk space for the installer or lack of connectivity to the download server, are written to both the MID Server agent log and to the MID Server Issue [ecc_agent_issue] table. These errors are published before the actual MID Server upgrade occurs and must be resolved before the upgrade can continue. You can view issues from the MID Server Issue [ecc_agent_issue] table in any of these locations:
  • MID Server Issues related list in a MID Server record.
  • MID Server > Server Issues navigation module.
  • MID Server Issues gauge on the MID Server dashboard.

Pre-upgrade tests

The pre-upgrade validation tests check these requirements:
  • At least 1 GB of free disk space
  • Access to the download site at install.service-now.com
  • Permission to execute these file operations:
    • Extract a ZIP archive to a temp folder.
    • Copy files from the temp folder to the agent folder.
    • Read a text file.
    • Delete the pre-upgrade contents.
  • For Windows, ensure that the Log On As user for the Windows service is either LocalSystem or a user that is part of the local Administrator group. By default, domain administrators are added to the local Administrator group when joining a computer to a domain. If the PowerShell script that performs this test does not return the expected output, the system logs a warning to the MID Server Issue [ecc_agent_issue] table, but the test passes.

Data provided

When the instance encounters issues during the pre-upgrade check, it populates these fields in the MID Server Issue [ecc_agent_issue] table:
Table 2. MID Server issue fields
Field Description
Last detected Date and time the issue was last detected.
Short description Contents of the generated message that specifies a possible issue with available disk space, download server access, or file permissions.
MID Server Name of the MID Server affected by a pre-upgrade test failure.
Issue source The process that identified the issue. For all issues with MID Server pre-upgrade testing, the source is UpgradeCheck.
State The current state of the issue. Possible states are:
  • New: Starting state when the instance creates the issue.
  • Acknowledged: State set by the administrator when they first examine the issue.
  • Resolved: Ending state, set by the instance, indicating that the issue has been resolved. If the scheduled job does not encounter the issue when it runs again, the instance automatically sets the state to this value.
Domain Domain for the MID Server. For all issues derived from MID Server pre-upgrade testing, the domain value is inherited from the domain of the MID Server user.
Count Number of times an issue has been detected. Each time the pre-upgrade tests run and encounter the same issue, the AutoUpgrade monitor increments this field.

Errors that block the upgrade

These messages describe issues detected by the pre-upgrade test and published to the MID Server Issue [ecc_agent_issue] table. Failure of any of these tests blocks the upgrade.
  • Not enough free disk space. The system reports <n> bytes free: This message is displayed when less than 1 GB of free disk space is detected on the MID Server host. This error is also written to the MID Server agent log.
  • Unable to download updates from the install server: This message indicates that either the MID Server host does not have permission to download the installation package from install.service-now.com, or network problems prevent connection. This error is also written to the MID Server agent log.
  • Unable to extract contents of pre upgrade check zip: This message indicates that the service account on the MID Server host does not have permission to extract the pre-upgrade ZIP archive to the temporary folder specified by the system property, java.io.tmpdir. On a UNIX host, the value for this property is typically /tmp or /var/tmp. On Microsoft Windows hosts, the path is c:\\WINNT\\TEMP.
  • Unable to create folder <upgrade check file path>: This message indicates that the MID Server service account does not have permission to create the upgradeCheck folder for the pre-upgrade checking files in the agent/package path.
  • Unable to verify file permissions: <message>: This message indicates an exception has occurred when checking file permissions, such as a file that does not exist or access failure.
  • MID Server Windows Service is not running as LocalSystem or a local Administrator: This message warns that the Windows service is not running with the desired permissions.

Non-blocking warnings

These warnings are displayed in the MID Server Issue [ecc_agent_issue] table and do not prevent a Windows MID Server from upgrading:
  • WARN: Unable to parse $logOnAsUser : This message warns that the Log On As User value for the Windows service is not in either of these expected formats:
    • user@domain.company.com
    • domain\user
  • WARN: Unable to look up Log On As user's groups: When the instance attempts to look up the logged on user's group memberships, it executes the net user <username> command. The instance expects a certain output structure by the Windows service from this command and issues this warning if the expected output does not match the actual output.
These PowerShell warnings are written only to the MID Server agent log. Because PowerShell is not required to use a MID Server, these configuration issues do not prevent a Windows MID Server from upgrading. However, these warnings might indicate issues in your environment that require attention.
  • Skipping PowerShell upgrade checks since PowerShell is not usable: PowerShell 2.0 (at a minimum) is not installed or powershell.exe is not available to the MID Server service user.
  • Continuing with upgrade, but the following issue was encountered during upgradeCheck: <exception message>: This message indicates that there was an issue running the PowerShell portion of the pre-upgrade tests.

Disabling the pre-upgrade check

A MID Server configuration parameter called mid.upgrade.run_precheck is set to true by default, which allows the automatic pre-upgrade test to run. To disable these tests for a single MID Server, add this parameter to that MID Server's config.xml file and set it to false. To disable these tests for all MID Servers, add a new record to the MID Server Property [ecc_agent_property] table called mid.upgrade.run_precheck. Set the value of this property to false and leave the MID Server field blank.

Pinning a MID Server to a specific version

You can pin all the MID Servers in your environment to a specific version by setting a system property, or you can configure specific versions for individual MID Servers.

Note: Do not pin the MID Server to a specific version for a significant amount of time, especially if you upgrade the instance. Under most circumstances, let the instance determine which MID Server version to use.

Version control properties

These system properties control the version for all MID Servers:
  • mid.buildstamp: Identifies the MID Server version with an identifier based on the date of the build. This property uses the format of mm-dd-yyyy-hhmm. The MID Server checks for version information hourly. If no override version is configured, the MID Server looks at the mid.buildstamp property for the version to use. This property resets itself to the default version (the version that matches your instance version) when the instance is restarted or upgraded, so any user changes are lost at that time. The system appends the release name and patch information to the date and time format.
    Caution: This property is not visible by default and should not be configured.
  • mid.version.override: Sets an override condition for the current version for all MID Servers in your environment. This action pins the MID Servers to a single version and disables the automatic upgrade feature. This property is not visible in the base system and must be added to the System Property [sys_properties] table when it is set. For details, see Add a system property.
    When the MID Servers check the version each hour, they look at the mid.version.override property first. If this property is empty, the MID Servers get their version information from the mid.buildstamp property. If an override version is configured, the MID Servers use this value and ignore the version information in the mid.buildstamp property. This override value remains when the instance is restarted and is passed to the MID Servers.
    Attention: The value in the mid.version.override property is cleared during an upgrade, which forces the MID Server to reset itself to the version in the mid.buildstamp property.

Version control configuration parameter

To pin specific MID Servers on a desired version, set the mid.pinned.version parameter with the name of that version in the config.xml file of each MID Server. Use the format <version>-mm-dd-yyyy. This setting overrides the property setting for the pinned MID Server version. For instructions, see Add a MID Server parameter.
Note: The value set in this parameter is not affected by an upgrade.

Upgrade the MID Server manually

You can manually upgrade MID Servers at any time if you do not want to wait for the automatic upgrade.

Before you begin

Role required: mid_server or admin

For the upgrade to run, MID servers must be in the Up state and must be validated.

About this task

The MID Server is upgraded to the version specified by build stamp on the instance, or by the upgrade property that you specify.

Procedure

  1. Navigate to Discovery > MID Servers or Orchestration > MID Server Configuration > MID Servers.
  2. Open the record of the MID Server that you want to upgrade.
  3. Click Upgrade MID under Related Links.
  4. Confirm that you want to perform the upgrade.
Feedback