Friday, April 23, 2021

SharePoint Patching Best Practices

If you only have 2 minutes, read the summary below. It documents the best practices discussed in this article in short statements. If you have more time, you should also read the rest of this article which gives more background information on the different topics.

Summary

General recommendations

  • Evaluate and install all SharePoint security updates as soon as possible and install them shortly after they have been released
  • Evaluate all SharePoint updates in a test environment which resembles the production environment before installing them in production
  • Take backups before installing SharePoint updates
  • For virtual servers you can take snapshots of all machines in the farm (important: only "cold" snapshots are supported – see below for more details) before installing new updates
  • If you are using the command line tool PSCONFIG.EXE instead the SharePoint Configuration Wizard be sure to specify all required parameters
  • After installing an additional language pack install the latest monthly update (for SharePoint Server 2016 and 2019 only the language dependent fix) again to ensure that the newly installed language components are upgraded to the same patch level as the rest of the farm
  • Keep Workflow Manager patching in sync with SharePoint patching
  • Last but not least: consider migrating your on-premises SharePoint farm to SharePoint in Microsoft 365 – here the complete patching effort and various other administrative tasks are handled by Microsoft for you.

SharePoint 2007

  • This version of SharePoint is out of support and no updates (including security updates) are released for this version anymore. Continuing to run SharePoint 2007 in production workloads puts your data at risk. Microsoft recommends upgrading to a supported version immediately.

SharePoint 2010 or 2013

  • Plan to upgrade to SharePoint Server 2016 or 2019. The newer versions of SharePoint Server improve the patching experience significantly:
    • Simplified packaging model (only 2 different packages to install compared to 30+ in earlier versions)
    • Shorter patching time (due to the simplified consolidated packaging model)
    • Zero-downtime patching (no maintenance window required)
    • Patching using Side-by-side functionality
  • Reduce patching time to a fraction using Russ Maxwell's PowerShell script
  • Apply the full server packages (also known as Uber packages) rather than individual fixes
  • Be aware that SharePoint Server 2010 and 2013 are in extended support and mainly security updates are released for these SharePoint versions. To get all the latest product improvements including non-security updates, upgrade to SharePoint Server 2016 or even better SharePoint Server 2019.
  • Patch levels older than April 2018 CU are no longer supported

SharePoint Server 2016 and 2019

SharePoint Patching Best Practices

Patching SharePoint can be challenging for SharePoint administrators for several reasons including the following:

  • Historically SharePoint consisted of a large number of components which could be patched independently
  • SharePoint patching requires two independent steps (installing the binaries and upgrading the databases)
  • SharePoint patch level needs to be in sync between different servers in the same SharePoint farm
  • SharePoint patching time is hard to predict and sometimes can take several hours
  • Microsoft recommends installing security updates as soon as possible to protect your environment from security vulnerabilities that others may try to exploit
  • There may be concerns that installing a SharePoint update could impact existing functionality
  • SharePoint updates cannot be uninstalled – so if something goes wrong there is no easy way to roll it back
  • Unlike other products like Windows or Office, it can be complex to automate the deployment of SharePoint updates across a farm

This article discusses several best practices and considerations for SharePoint administrators to simplify this process.

Best Practices around Security Updates

How quickly should security updates be installed?

Security updates should be installed as soon as possible after they are released – preferably within a couple of days. The longer your SharePoint environments operate without the latest security updates, the more risk you have that they could be compromised via an unpatched security vulnerability.

Do packages marked as security update only contain security related fixes?

No. SharePoint updates are cumulative, so each security update includes all fixes that have been released for both security and non-security related issues up to that point for the patched component.

Is it ok to install only the security updates and not the non-security updates released monthly?

Although it is supported to install only those packages which include the security updates it is recommended to keep SharePoint Server fully updated by installing the latest SharePoint updates regardless of whether they're security or non-security updates. However, it is less urgent to install non-security updates quickly after they're released unless you are impacted by one of the issues fixed in that update.

With SharePoint server 2016 and 2019 only two SharePoint update packages (one package which includes fixes for all language independent components like executables and one package which includes fixes for all language dependent components like resource files) are required to patch a SharePoint server.

If a language dependent patch isn't available for a given month, update to the latest previously available language dependent patch. For example, if applying the July 2019 Public Update for SharePoint Server 2016, install the language independent update for July 2019 and the language dependent patch from April 2019. If you do not install the language dependent patch, you may encounter missing or incorrect functionality.

Be aware that at least once a year it is mandatory to install also the non-security updates based on the Updated Product Servicing Policy for SharePoint Server 2016 and 2019

General Recommendations

Best Practices to reduce the risk of regressions

Microsoft performs rigorous validation of each fix, both internally and with a select set of partners and customers before it is released to ensure it has the highest quality. This has helped to significantly reduce the number of regressions reaching our customers in the last couple of years.

Due to the large number of components and various ways to configure them, it is impossible to test all possible configuration combinations. So it's unlikely, but possible, for a code change to introduce unwanted behavior with a specific configuration for some of our customers. Third party software and other customizations can also introduce additional variables that may be difficult to validate in our internal testing.

We recommend that customers use a test environment to validate new fixes with common use cases and business critical functionality before installing updates in their production environments. Test environments should simulate their production environments as much as possible to ensure meaningful validation.

As an additional safeguard we advise to take a backup (system state and file system backups of the SharePoint server and SQL / farm backup of the databases) before installing any updates as it is not possible to uninstall SharePoint updates after they have been installed.

If the SharePoint farm is installed on virtual machines, taking a consistent snapshot of all servers (SharePoint, SQL, etc.) could be used as an alternative. However, it would require downtime as so call "hot snapshots" are unsupported with SharePoint. The snapshots must be taken when all virtual machines are shut down to guarantee a consistent state between the SharePoint servers and the SharePoint databases.

Best practices around SharePoint Products Configuration Wizard and PSCONFIG.EXE

SharePoint patching consists of two steps: applying the binaries on each server and running the SharePoint Products Configuration Wizard on each machine in the farm to finalize the installation by upgrading the SharePoint databases and other administrative tasks. Be aware that the configuration wizard performs various tasks on the SharePoint server machines itself, including the registration of new/updated services and features and copying SharePoint binary files around to the required location. It is important that this step is performed after applying the patches otherwise SharePoint might not function correctly. If this step has to be delayed (e.g. the maintenance window is not big enough) at least the following PowerShell command has to be run on each server machine to ensure that the correct binary files are being used: Install-SPApplicationContent

Best practices for patching with language packs

SharePoint consists of language independent components (e.g. executables) and language dependent components (e.g. resource files which carry UI elements for the different languages). During patching both components have to be updated to ensure that the binaries can reference the correct resource elements. Language packs include only the language dependent components of SharePoint.

If a new language pack is installed on a SharePoint farm the patch level of the language dependent components for the new language will most likely not be the same as the patch level of the language independent components and the patch level of the other language dependent components already installed on the server. To ensure that the components installed with the language pack are upgraded to the same patch level as the rest of the farm it is required to apply the most current installed monthly patch again. For SharePoint Server 2016 and 2019 only the most current language dependent update must be applied again.

Best practice for patching Workflow Manager client on the SharePoint server machines

SharePoint updates sometimes include updates for the workflow activities for SharePoint 2013 workflows. It is important to ensure that the most current Workflow Manager updates are installed as well when new SharePoint updates are installed. This ensures that WFM is compatible with workflow activities updates included in the SharePoint update.

Also be aware that Workflow Client and Server components must be in sync. WFM client updates must be installed before WFM server is updated.

The SharePoint Configuration Wizard will update the SharePoint Workflow Activity definitions in WFM. In case this fails (e.g. because the WFM server cannot be reached because it is patched simultaneously with the SharePoint machines) the workflow definitions need to be updated manually after patching is completed using the following PowerShell command: Copy-SPActivitiesToWorkflowService

More details: Update Workflow in SharePoint Server

Best practice for patching with 3rd party components

3rd party components and other customizations cannot be considered when Microsoft tests new updates. It is important that the compatibility of 3rd party components with the latest updates is evaluated before new patches are installed in a production environment. A SharePoint administrator should contact the 3rd party vendor to get information about compatibility. In addition, the steps listed above in the Best Practices to reduce the risk of regressions section earlier in this chapter should be followed.

Best Practices for legacy versions of SharePoint

SharePoint 2007

This version of SharePoint is out of support and no updates (including security updates) are released for this version anymore. Continuing to use SharePoint 2007 in production workloads puts your data at risk.

Our advice is to remove such systems from your network and upgrade to a fully supported version of SharePoint.

SharePoint 2010 or 2013

Be aware that SharePoint Server 2010 and 2013 are in extended support and mainly security updates are released for these SharePoint versions. To get all the latest product improvements including non-security updates, upgrade to SharePoint Server 2016 or even better SharePoint Server 2019.

Also be aware that patch levels older than April 2018 CU are no longer supported.

SharePoint 2010 and 2013 have a significantly different packaging model than SharePoint Server 2016 and 2019. SharePoint 2010 and 2013 have 30+ different patchable components which all can potentially be on a different patch level in a single farm. Keeping track of the patch level of all components and ensuring that the latest updates are installed can be challenging if the full server packages (also known as Uber packages) are not applied. This is especially important for security updates where it would be dangerous if a security update for a component is missing. In contrast SharePoint Server 2016 and 2019 have only two patchable components. To ensure that all patchable components are always on the latest patch level we recommend applying full server packages (also known as Uber packages) rather than individual fixes:

Unlike SharePoint Server 2016 and 2019, patching of a SharePoint 2010 or 2013 farm cannot be done without downtime. This requires accurate planning of the required maintenance window to ensure that the patch downtime has minimum impact on users. Using the correct farm topology and the right strategy it is possible to reduce the downtime during patching of a SharePoint 2010 or 2013 farm to a minimum – but as mentioned before it is not possible to eliminate it completely.

If no further precautions are taken installing the monthly fixes on a SharePoint server can take several hours as certain Windows services are started and stopped repeatedly. The installation time for updates can be reduced to a fraction of the normal installation time (e.g.) using the PowerShell script created by Russ Maxwell which ensures that the services are only restarted once.

Important: SharePoint 2010 will reach end of support on October 13, 2020. No further support (including both security updates and non-security updates) will be provided after this date.

Our advice is to plan the migration of your SharePoint 2010 environment to one of the fully supported versions of SharePoint (SharePoint Server 2016 or 2019) or – if possible – to SharePoint in Microsoft 365 before SharePoint 2010 reaches end of support.

If your production environment is currently running on SharePoint 2013 we also recommend to evaluate SharePoint Server 2016 and 2019 as these versions include several features which improve the patching experience significantly.

Best Practices for SharePoint Server 2016 and 2019

SharePoint Server 2016 and 2019 have several improvements which enhance the patching experience significantly:

These use a simplified packaging model (only 2 different packages compared to 30+ in earlier versions) which reduces the size of the installation packages for our patches and also the installation time significantly.

Using the correct farm topology, it is possible to apply SharePoint updates for these versions without any downtime which removes the need for a maintenance window.

A side effect of zero-downtime patching is the fact that different servers in the farm are on different patch levels during the patching process. Due to these different patch levels, SharePoint servers would potentially serve different versions of the same JavaScript files to the end user. To prevent this SharePoint Server 2016 and 2019 include side-by-side functionality which – if enabled – guarantees that all servers in the farm send the same JavaScript files to the end users during patching.

Closing Note

SharePoint patching can be a challenging task. As security updates are important and applying them quickly is critical, patching of SharePoint farms is a frequent task for SharePoint administrators.

Newer versions of SharePoint have been improved to reduce the complexity of patching and to guarantee availability during the patch timeframe. SharePoint administrators should still carefully plan their patching strategy to minimize the potential risks of patch deployment.

Migrating to SharePoint in Microsoft 365 can remove this burden from customers. Microsoft ensures that Microsoft 365 contains our most advanced security capabilities and fixes to protect customer data.

References


Posted by at No comments:
Subscribe to: Posts (Atom)