The source detected that the destination failed to resume - Veeam Instant recovery

When we are trying to move a VM using Instant reovery and it fails with below error.

Relocating VM Error: A general system error occurred: The source detected that the destination failed to resume.
(An error occurred restoring the virtual machine state during migration. , Failed to receive migration.)

Here are findings from the Log:

[22.02.2023 14:49:33] <11> Info [CQuickMigrationImpl] Calculated QM path: [[CQuickMigrationPath]
[22.02.2023 14:49:33] <11> Info Path: 'Relocate'
[22.02.2023 14:49:33] <11> Info Errors: [[CQuickMigrationErrors]
[22.02.2023 14:49:33] <11> Info QmType: 'VMotion'
[22.02.2023 14:49:33] <11> Info Errors: [No errors.]
[22.02.2023 14:49:33] <11> Info ]

Veeam uses the VMotion type to migrate the VM. And It fails with:

[22.02.2023 17:04:26] <22> Error Relocating VM
[22.02.2023 17:04:26] <22> Error Relocate VM failed, vmRef 'vm-947169', newHostRef 'host-221', newResPool 'resgroup-82' (System.Exception)
[22.02.2023 17:04:26] <22> Error A general system error occurred: The source detected that the destination failed to resume.
[22.02.2023 17:04:26] <22> Error (An error occurred restoring the virtual machine state during migration. , Failed to receive migration. ) (Veeam.Backup.ViSoap.ViServiceFaultException)
[22.02.2023 17:04:26] <22> Error VimApi.SystemError
[22.02.2023 17:04:26] <22> Error Relocate VM failed, vmRef 'vm-947169', newHostRef 'host-221', newResPool 'resgroup-82' (System.Exception)
[22.02.2023 17:04:26] <22> Error A general system error occurred: The source detected that the destination failed to resume.
[22.02.2023 17:04:26] <22> Error (An error occurred restoring the virtual machine state during migration. , Failed to receive migration. ) (Veeam.Backup.ViSoap.ViServiceFaultException)
[22.02.2023 17:04:26] <22> Error VimApi.SystemError
[22.02.2023 17:04:26] <22> Error [CQuickMigrationImpl] Failed to migrate VM

Cause : 

This issue occurs due to incorrect datastore information or out-of-date virtual hardware.

Solution : 

We tried some solutions given by Veeam and VMware and could not fix and found the our Vcetner which is used to move as destination is 5.5 version.

So moved to another location which has update to version of ESXi/Vcenter host and fixed.

Check the SharePoint sites / Mailboxes are backing up in Veeam

 

Recently I got request from customer that, they want to know what are the SharePoint sites and Mailboxes are backing up in Veeam server. 

The small script helped to fetch this,

SP site:

$Org = Get-VBOOrganization -Name "YourOrg.onmicrosoft.com"

$Job = Get-VBOJob -Name "SPSite_BackupJob_Name"

$Sites = Get-VBOOrganizationSite -Organization $Org 

$Sites | Select-Object Name |  Sort-Object -property Name 

---

Mailboxes :

$user = Get-VBOOrganizationUser -Organization $Org |Select-Object UserName |  Sort-Object -property UserName

Credit goes to :

https://github.com/nielsengelen/veeam-powershell/blob/master/VBO365/VBO365-ExcludePersonalSharePointSites.ps1

Service Provider does not have SSL Certificate in Veeam Cloud Connect Service Provider

 

Actual Issue

    When trying to add Veeam Cloud Connect as a Service Provider, you may get the following error (Service Provider does not have SSL Certificate):

Cause for the issue

    This is due to either a Veeam B&R Host or Gateway from the user end (where Veeam B&R is installed) firewall blocking outbound connectivity to Veeam Cloud Connect Service.

Resolution for the issue

    From the Veeam Backup & Replication or the Veeam Agent installed server to outbound connectivity on TCP Port 6180 should be allowed for the Veeam Cloud Connect gateway  server(s).

    Try to Telnet the port and check whether you have proper connectivity , if not request your network team to open the connectivity.

Veeam Explorer for SharePoint shows as empty while restore

We are unable to view any site/content in Veeam Explorer for SharePoint when I initiate restore, it looks empty under content DB path. We kept the screen more than an hour but no use. 

Here is the solution:

1.Please make sure there is no backup job running or wait for some time to complete all the backup jobs.
2.Stop Veeam console and Veeam services (also from the remote proxies):

• Veeam Backup for Microsoft Office 365 service (Veeam server)
• Veeam Backup Proxy for Microsoft Office 365 services. (Veeam server & remote proxies)

3.Go to Veeam Backup for Office 365 - %programdata%\Veeam\Backup365\config.xml and proxy.xml
4.Edit the config.xml in a text editor.
5.Edit the proxy.xml in a text editor.
 
Add the bold text as per examples below:
 
The config.xml for Backup365 should be like this:
<Veeam>
  <Archiver OperationTimeout="1200">
   ….
    <WCF OperationTimeout="1200" />
   ….
  </Archiver>
</Veeam>
 
The proxy.xml from Veeam server should be like this:
<Veeam>
    <Archiver OperationTimeout="1200">
      <RemoteConnection Timeout="10:00:00" />
      <WCF OperationTimeout="1200" />
       ….
  </Archiver>
</Veeam>
 
We`ll need to increase the timeout for the SharePoint explorer in order to have time to load all the items in the GUI. To do this we`ll need to modify the config.xml.
 
In order to view to config.xml, we need to enable extended logging in the SharePoint explorer. To enable extended logging, please do the following:
 
1)Open the Sharepoint explorer
2)Go to General Options -> Advanced
3)Click on "Enable extended logging"
 
After this go to C:\ProgramData\Veeam\Backup\SharePointExplorer and you`ll have the config.xml. Before we edit it please stop both VBO365 services:

• Veeam Backup for Microsoft Office 365 Service
• Veeam Backup Proxy for Microsoft Office 365 Service

 
Add the bold text as per examples below:
 
The SharePoint Explorer config.xml should look like this:
<Veeam>
<Archiver OperationTimeout="1200">
    <RemoteConnection Timeout="10:00:00" />
    <WCF OperationTimeout="1200" />        
….
 </Archiver>
</Veeam>
 
 
The proxy.xml from the remote proxy, should be like this:
<Veeam>
  <Archiver OperationTimeout="1200>
      <RemoteConnection Timeout="10:00:00" />
      <WCF OperationTimeout="1200" />
   ….
  </Archiver>
</Veeam>
 
The issue should fix now. If the issue persists, please change the value "1200" to "2400" and check again.

Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]

We recently faced backup failure for Application aware backup in Veeam for MSSQL server. The error is listed below,

Processing <VM_Name> Error: Unfreeze error: [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data.

A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}].

Instance ID: [{41239125-29f6-4455-9bc1-32439ceebe75}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f0].]

Normally this is caused by NTDS writers issue and we can easily fix it by restating the below services,

before restarting, I listed 'vssadmin list writers' through cmd, I found that, writer was in failed state

Writer name: 'SqlServerWriter'

Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}

Writer Instance Id: {41239125-29f6-4455-9bc1-32439ceebe75}

State: [8] Failed

Last error: Inconsistent shadow copy

So I have restarted the below services and ran above cmd  again and no error found.

COM+ Event System

Volume Shadow Copy

MSSQL VSS services.

Here backup will success after the retry.

Today, I got same error and I tried same fix, however got same error and it was surprise to me.

I checked the Log for the Veeam backup in MSSQL  server and found the below error,

                Connecting to mssql, connection string: Provider='msoledbsql';Data Source='Server_Name';Integrated Security='SSPI';Persist Security Info=False, timeout: 15

                Instance '' connection problem: Code = 0x80040e4d

         Code meaning = IDispatch error #3149

           Source = Microsoft OLE DB Driver for SQL Server

             Description = Login failed for user 'user_Name'.

We found that there were some new MSSQL instance created in this server and the service account was not added for those instance to access and backup.

Finally we provided permission to same account to newly created instances and issue got fixed.

Failed to mount NFS volume: Fault "PlatformConfigFaultFault", detail "NFS mount failed: Unable to connect to NFS server."

 When I try to restore the server from Veeam backup to Vcloud environment I got the error that, PlatformConfigFaultFault while restore the VMs. 

Detailed errors are shown below,

Unable to check if vPower NFS is mounted on the host "esxi-host.domain.local"(Veeam.Backup.Common.CRegeneratedTraceException)

Failed to connect backup datastore to the ESXi host "esxi-host.domain.local". (Veeam.Backup.Common.CRegeneratedTraceException)

Failed to add NFS datastore for NFS host 'Veeam_proxy4'. Failed to mount NFS volume (192.168.0.93:/VeeamBackup_Veeam_proxy4). 192.168.0.93: Fault "PlatformConfigFaultFault", detail "NFS mount 192.168.0.93:/VeeamBackup_Veeam_proxy4 failed: Unable to connect to NFS server." 

It is proxy issue. We need to open ports 902 and 443 between veeam proxy and ESXI host/VSPhere/VCloud end.

The Hyper-V Virtual Machine Management service encountered an unexpected error: Unspecified error (0x80004005). Error code: '32768'.

 We had a request from customer to extend the RAID 10 configured disks, however there is no option to extend without deleting because of mixed RAIDs 1 and 0.

So we decided to backup the data in Veeam server and delete the existing disks RAIDs (except OS - OS not under any RAID and it has dedicated disk)  and re-configure the RAID. 

After re-configured the RAID , OS was found as it is and and Hyper-V role was remained the same (We expected the same and it will reduce couple of hours down time.) with same configurations, however other stuffs were removed. I mean (.XML, .VMCX, .VMRS, .VMGS, .VHD, .VHDX) files.

Initial issue, we found was Veeam refused to communicate with this server and got RPC failed to communicate error,

The issue, we found was routing was missing and added in the server and after that both Veeam and servers were started to talk each other.

Then, we tried to restore the VMs but got below error,

Error Restore job failed Error: Failed to create planned VM configuration (path: 'D:\VMs\<GUIDs>.vmcx'). Job failed ('Failed to import a virtual machine.
The Hyper-V Virtual Machine Management service encountered an unexpected error: Unspecified error (0x80004005).'). Error code: '32768'.

Regarding the error - it seems Hyper-V could not import the VMs and throwing error. We believe that it was due to some issue in MicroSoft file systems. We contacted the Veeam and they suggest the same or try to restore files individually and import VM manually.

We restored the configuration and disks  (.XML, .VMCX, .VMRS, .VMGS, .VHD, .VHDX) files manually and import the VMs manually and all the VMs were started to work without any issues.

Failed to process team: with Teams. Failed to find Team mailbox owner account.

When I try to backup O365 environment (Onedrive, MS Teams, Mailboxes and SharePoint) through Veeam backup server,  I received  the error that, Failed to find Team mailbox owner account and backup failed for the particular channel. 

The detailed error is below,

Failed to process team: <example_channel> with Teams. Failed to find Team mailbox owner account.

I found this group <example_channel> is orphaned  state, none is assigned as group owner. So I assigned one Global admin as owner to fix this. 

Follow the steps to assign Group owner,

1. Sign in to the Microsoft 365 portal with an account that has administrator privileges. 

2. Choose Groups from the navigation pane at the left hand side.

2. Search and find the group that you need to assign a new owner to and click on it.

3. In the new window appeared in right navigation, Select Owners tab, choose Edit.

4. In the View Owners dialog box, choose Add Owners.

5. Assign the right person(s) as owner and close the tab. 

Note : Select minimum two recommended as by MicroSoft.

Error: Failed to call RPC function 'StartAgent': Timed out requesting agent port for client sessions.

 Error : 

Failed to call RPC function 'StartAgent': Timed out requesting agent port for client sessions.

When we try to backup VMware VCloud environment, recently we occupied with above error in our Veeam server always, though these  environment (VBR and two repositories) have 200GB RAM and 2 sockets and 48 CPU cores on each servers.

This message typically means that the proxy/repository is overloaded, however all the servers are the Apollo servers, which most powerful machines with high configuration, finally I have lowered both proxy and repository tasks again and again, but the problem just reappear.

Cause:

There are transport agents that start for VMs processing's, this load can be limited by the concurrent task limit. 

But VCD job has also vApps, and for this container also transport agents start, but they are not limited by the concurrent task limit and Veeam Agent processes are not being properly closed by the moment so it is causing the port range (2500-3300) is being occupied. 

Due to this many agents are trying to start at one moment (we set 35-40 backup job start time as same) and  transport service "Veeam Data Mover Service"  started to crash.

Also we found some errors in log,

 < 34284> rpc      | ERR |RPC method call has failed (StartAgent).

 < 34284> rpc      | >>  |Timed  out requesting agent port for client sessions.

 < 34284> rpc      | >>  |--tr:Failed to initialize agent ( specified agent path: [C:\Program Files (x86)\Veeam\Backup Transport\x64\VeeamAgent.exe])

< 34284> rpc      | >>  |--tr:Failed to initialize agent instance {f392c303-9c2b-41dd-817f-6d04907ddf9e}.

< 34284> rpc      | >>  |An exception was thrown from thread [34284].

 

How to fix this,

We have moved some jobs from current backup time to another time, since then we haven't had a single failure. As talked with Veeam, this behavior was mostly unknown to them and they will take it up with QA. 

It must be much better to, do not start the agent until it is really needed, just like is the case already for VMs.

Error: bad allocation Failed to upload disk in Veeam backup

 

Issue:

When I try to backup Hyper-V environment got the below error, 

Error: bad allocation Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: Reconnectable protocol device was closed. Unable to retrieve next block transmission command. Number of already processed blocks: [825]. Failed to download disk 'FIleName.vhdx'.  

Cause:

Bad Allocation - This issue due to high usage of either CPU or memory.

Fix:

Checked the Hyper-V and found that the VM is using too many resources (both CPU and memory), try to reduce the resource usage or reboot the server to make server normal usage.

---

Once server became normal backup started work normally.