Multiple VSS writers are failing and my backups are failling too. Need help !!

Very often we come across issues, in which the VSS Writers keep failing after two or three backups. It may be a single VSS Writer or many of them from the list we get on running "vssadmin list writers".
I have been analyzing some tasks and have found that in cases where after updating the drivers for VSS, deleting the Subscriptions key and re-registering the DLL's, if we still find that the VSS writers fail, then it is not the writers which are at fault. It is the Providers.

First let us understand some background of VSS.

Background data on VSS
======================

Snapshot:- This is a point in time copy (shadow copy) of a volume.

There are 3 important components involved in a snapshot creation.

1. Volume Shadow Copy Service
2. VSS Writers
3. VSS Providers
4. File System

Volume Shadow Copy Service - (volsnap.sys)
==========================

This is the service which helps in co-ordination ( communication between the OS, the VSS writers, the VSS providers ). The point to note is that VSS ( Volume Shadow Copy Service ) DOES NOT TAKE SNAPSHOTS as we all assume it does. It just helps other components ( Writers, Providers & OS ) communicate with each other effectively. Any transaction has to go through this service.

VSS Writers
===========

On a Server, we have a database for everything. For example, on a clean installed system, we have a database for the Registry, System, DCOM etc... everything that comes along with the OS.

If we add SQL to the server, then a database for SQL is created.

If we further add Exchange, database for Exchange is created.

VSS Writers are the ones who prepare the database for a snapshot ( shadow copy creation ). Again, VSS Writers do not take snapshots. They just prepare the database for a snapshot creation.

For example, there could be open transactions for Exchange, SQL, Registry, DCOM etc..
If we take a snapshot without preparing them first, the snapshot might be corrupt due to open transactions. For a perfect snapshot, these open transactions have to be closed first. The data in memory has to be flushed and further transaction have to stopped before a snapshot creation can happen.

This is achieved by VSS Writers.

The OS comes with some default writers like Registry Writer, System Writer, COM+ Writer etc. Writers like Exchange, SQL, IIS are not preinstalled on the system.

The moment we install Exchange, SharePoint,IIS or SQL, their writers are also installed. It comes with the package of the individual component.

So, the registry writer will flush everything in its own database and stop further transactions ( read/write operations in registry ) during a snapshot creation process.

Same goes for Exchange Writers. This will also flush existing transactions ( read/write operations ) in its own Exchange database before a snapshot can be taken.

The writers will not interfere with each other as they are only responsible for their own database.

VSS Providers
=============

Now, these are the ones responsible for taking a snapshot. It is the most important part of a shadow copy creation.

By default, we always have the "Microsoft Software Shadow Copy Provider" also known as "swprv.dll" installed on the system.

If we add a backup program like StorageCraft, then a provider from StorageCraft named "StorageCraft Shadow Copy Provider" will be installed on top of the default provider and will be used as the default provider.

Further if we add Symantec Backup Exec, then a provider from Symantec will also be installed on top of the other two and will be used as the default provider.

Note: When we run the command "vssadmin list providers" - The one that comes up on the top of the list, is the one installed last, and is the one used by default.

Sometimes, the providers may conflict or may malfunction. If the providers fail to do their job, it also affects the writers. It may cause the writers to fail too and generate event errors like Event id 8, 12298 in the event Viewer.

File System
===========

Also known as NTFS.sys is the most important part of an OS. Without a file system, nothing is possible. Further explanation not required for this. If you want to learn about file systems, please GOOGLE.



WHAT HAPPENS WHEN WE INITIATE A BACKUP using VSS
================================================

1. We initiate a backup using a backup application.
2. The request is sent to Volume Shadow Copy Service ( VSS ) or volsnap.sys
3. The VSS informs the respective writers that a snapshot creation is going to happen, so they must flush everything and freeze their databases for a a few seconds. The maximum time VSS writers can freeze their databases is for 60 seconds.
4. Once the VSS writers complete their job ( have flushed and freezed the database ), it informs Volume Shadow Copy Service ( VSS ), that they are ready now.
5. The VSS then informs the providers to take a snapshot. A snapshot is taken which takes a maximum of 10 secs. Once a snapshot is taken, it informs VSS that its job is done.
6. Then the VSS informs the VSS writers that a snapshot has been taken and that they can release the database.
7. Everything comes back to normal and the backup operation is started using the snapshot taken.


TROUBLESHOOTING WITH VSS
========================

In case of a backup failure, we must first check the drivers associated with the VSS backup. These drivers are :

1. Volume Shadow Copy service ( volsnap.sys )

XP, Server 2003, Server 2003 R2 and SBS 2003 - http://support.microsoft.com/kb/979457

Vista, Server 2008 and SBS 2008 - http://support.microsoft.com/kb/976329, http://support.microsoft.com/kb/2280117

Seven, Server 2008 R2 and SBS 2011 - http://support.microsoft.com/kb/2505454, http://support.microsoft.com/kb/980082/EN-US

2. NT File System ( NTFS.sys )

XP, Server 2003, Server 2003 R2 and SBS 2003 - http://support.microsoft.com/kb/973870

Vista, Server 2008 and SBS 2008 - http://support.microsoft.com/kb/972135, http://support.microsoft.com/kb/2525064

Seven, Server 2008 R2 and SBS 2011 - http://support.microsoft.com/kb/981208

3. The Microsoft Software Shadow Copy Provider ( swprv.dll )

http://support.microsoft.com/kb/975928; http://support.microsoft.com/kb/971394

http://support.microsoft.com/kb/976461

If we have other providers installed, which could be StorageCraft provider, Symantec providers and Acronis Providers, we need to make sure that their files are also updated.
Some of them we commonly find are

stcvsm.sys
symsnap.sys
snapman.sys

Others may also be there and we need to search it out. Your best friend GOOGLE, please use it.

PHASE 2
--------

This step is performed only when we find the VSS writers in a failed state. If we find the writers stable and still backups fail, or we get errors, please do not perform this step.

In case, the VSS writers fail, we may delete the following registry key, after of course backing it up and reboot the server.

HKLM/Software/Microsoft/EventSystem/{26c....../Subscriptions.

This key will automatically be re created on restart and writers should come back in stable state.  

Writers in Waiting for completion state is not a problem. This is as good as Stable.

We can do this on Server XP, 2003, Vista, 2008, Seven and 2008 R2 OS.

One more thing to note. If we have only a single writer in a failed state, which would always be either SQL, EXchange, IIS or NTDS, then we need to restart their respective service from Service Console.
If they fail repeatedly, then make sure that the application and its components are updated. Consult their respective engineers once. Make sure that the latest service packs and rollups are installed.

In case of NTDS writer failure in Server 2008, we may install : http://support.microsoft.com/kb/970770

If the issue still persists, please inform your TL and engage a Setup/Perf engineer. You may need to create a task and assign to Setup/Perf bin.
In this case we will have to enable VSS trace and check out further.

Note: After deleting the Subscriptions key and rebooting, the subscriptions key is back in registry. However when we do, vssadmin list writers, nothing may be listed. In this case we register the VSS dll files.

For the list of dll files to register please check KB : http://support.microsoft.com/kb/940184

DLL registration has to be done only on Server 2003. PLEASE DO NOT REGISTER DLL FILES IN VISTA, 2008 and above. They might break other components.

PHASE 3
=======

Doing the above two, would generally resolve 80-90 % of the issues. If we find that the issue is re occurring, or we have VSS event errors, it may be the providers causing the issue. If the snapshot creation does not happen properly, it may affect the writers and other components involved.

Please check the providers using vssadmin list providers. If we find any other provider, other than Microsoft Software Shadow Copy provider, we need to remove it from the list.

To do this: Start registry editor.

HKLM/System/CurrentControlSet/Services/VSS/Providers/GUIDS.

One of the GUIDS would be for Microsoft Software Shadow Copy provider. The other GUIDs would be for other providers like StorageCraft.
We will backup the GUID and then delete it. Then reboot the machine.
Now when we do vssadmin list providers, we will only see the default Microsoft provider.

I have seen in many cases where our StorageCraft provider does not work properly and continue to fail the writers. In such cases, if we get rid of the provider, then the issue is resolved and does not re occur.

PHASE 4
=======

Please also check that we have abundant free space. If we go by the book, a snapshot requires at least 10 % of the volume to be free.
The storage controllers ( SCSI/RAID controllers ) could also be an issue and may need to be updated.
Run a chkdsk and make sure the volumes are error free.
Evaulate the event viewer and search for errors. They might help.

Again, if we find any errors, GOOGLE it.

 So many people has informed me that, they found this article informative. Hence I have improved that and posting back again.

-------------------------------------------------------

Before you stat working on this issue make sure that the volumes do not have disk/file-system error. Run CHKDSK and make sure.

1) In case of a backup/writer failure, we must first check the drivers associated with the VSS backup.

These drivers are:

VOLSNAP.SYS

NTFS.SYS

SWPRV.DLL

--- Check the version of above mentioned 3 MS files, and update using MS Hotfix.

If we have other providers installed, which could be StorageCraft provider, Symantec providers and Acronis Providers, we need to make sure that their files are also updated.

Note: You can see the list of providers using this command:  VSSADMIN LIST PROVIDERS

Some of PROVIDER DRIVERS commonly find are:

stcvsm.sys

symsnap.sys

snapman.sys

2) This step is performed only when we find the

VSS writers in a failed state. If we find the writers stable and still backups fail, or we get errors, please do not perform this step.

In case, the VSS writers fail, we may delete the following registry key, after of course backing it up and reboot the server.

HKLM/Software/Microsoft/EventSystem/{26c....../Subscriptions.

This key will automatically be re-created on restart and writers should come back in stable state.

Writers in Waiting for completion state is not a problem. This is as good as Stable.

We can do this on Server 2003, 2008 and 2008 R2 OS as well.

However in Windows Server 2003, after deleting and restarting the server "vssadmin list writers" might not show all writers, in that case we need to register the DLLs:

For the list of dll files to register please check KB : http://support.microsoft.com/kb/940184

Note: Re-registering DLLs in Windows Vista/2008 and above is NOT recommended.

3) If we have only a/specificwriter(s) in a failed state, which would always be either SQL, EXchange, IIS or NTDS, then we need to restart their respective service from Service Console.

If they fail repeatedly, then make sure that the application and its components are updated. Consult their respective engineers once. Make sure that the latest service packs and roll-ups are installed.

4) Doing the above , would generally resolve 80-90 % of the issues. If we find that the issue is re occurring, or we have VSS event errors, it may be the providers causing the issue. If the snapshot creation does not happen properly, it may affect the writers and other components involved.

Please check the providers using "vssadmin list providers". If we find any other provider, other than Microsoft Software Shadow Copy provider, we need to remove it from the list.

To do this: Start registry editor.

HKLM/System/CurrentControlSet/Services/VSS/Providers/GUIDS.

One of the GUIDS would be for Microsoft Software Shadow Copy provider. The other GUIDs would be for other providers like StorageCraft. We will backup the GUID and then delete it. Then reboot the machine.

Now when we do vssadmin list providers, we will only see the default Microsoft provider.

I have seen in many cases where our StorageCraft provider does not work properly and continue to fail the writers. In such

cases, if we get rid of the provider, then the issue is resolved and does not re occur.

-- However doing these would hamper the functionality of StoaregCraft or any other backup software similar to this.

Hence engage respective Software tech/support for further help.

5) If a Shadow Copy is enabled on the volumes, make sure that the Storage Area is set to the default location.

For Example:

If we are taking a snapshot of C: then the storage area should be C:

We change this only if we do not have sufficient free space ( At least 300 MB ) free.

We can right click the volume, goto the Shadow Copy tab and change this accordingly.

Also note that the Storage Area minimum value should be 300 MB and maximum can be 3000 MB.

We can also put any value in between, which is a multiple of 300, like 600, 900, 1200, 1500 etc... depending upon the space available.

In case we find errors in the Event viewer about Time-Outs ( Event id : 12298, 12310, 2004, 12290 ) then we also need to set the MinDiffArea in the registry.

HKLM/System/CurrentControlSet/Services/Volsnap

Right Side, New DWORD VALUE - "MinDiffAreaFileSize" - Modify and set the value in between "300-3000 MB" or any multiple

of 300 in between. Ideally we can set it to 3000 MB which works best if we have space.

Ref: http://support.microsoft.com/kb/826936

6) Please also check that we have abundant free space. If we go by the book, a snapshot requires at least 10 % of the volume

to be free.

The storage controllers ( SCSI/RAID controllers ) could also be an issue and may need to be updated.