Service Provider does not have SSL Certificate in Veeam Cloud Connect Service Provider

 

Actual Issue

    When trying to add Veeam Cloud Connect as a Service Provider, you may get the following error (Service Provider does not have SSL Certificate):

Cause for the issue

    This is due to either a Veeam B&R Host or Gateway from the user end (where Veeam B&R is installed) firewall blocking outbound connectivity to Veeam Cloud Connect Service.

Resolution for the issue

    From the Veeam Backup & Replication or the Veeam Agent installed server to outbound connectivity on TCP Port 6180 should be allowed for the Veeam Cloud Connect gateway  server(s).

    Try to Telnet the port and check whether you have proper connectivity , if not request your network team to open the connectivity.

Veeam Explorer for SharePoint shows as empty while restore

We are unable to view any site/content in Veeam Explorer for SharePoint when I initiate restore, it looks empty under content DB path. We kept the screen more than an hour but no use. 

Here is the solution:

1.Please make sure there is no backup job running or wait for some time to complete all the backup jobs.
2.Stop Veeam console and Veeam services (also from the remote proxies):

• Veeam Backup for Microsoft Office 365 service (Veeam server)
• Veeam Backup Proxy for Microsoft Office 365 services. (Veeam server & remote proxies)

3.Go to Veeam Backup for Office 365 - %programdata%\Veeam\Backup365\config.xml and proxy.xml
4.Edit the config.xml in a text editor.
5.Edit the proxy.xml in a text editor.
 
Add the bold text as per examples below:
 
The config.xml for Backup365 should be like this:
<Veeam>
  <Archiver OperationTimeout="1200">
   ….
    <WCF OperationTimeout="1200" />
   ….
  </Archiver>
</Veeam>
 
The proxy.xml from Veeam server should be like this:
<Veeam>
    <Archiver OperationTimeout="1200">
      <RemoteConnection Timeout="10:00:00" />
      <WCF OperationTimeout="1200" />
       ….
  </Archiver>
</Veeam>
 
We`ll need to increase the timeout for the SharePoint explorer in order to have time to load all the items in the GUI. To do this we`ll need to modify the config.xml.
 
In order to view to config.xml, we need to enable extended logging in the SharePoint explorer. To enable extended logging, please do the following:
 
1)Open the Sharepoint explorer
2)Go to General Options -> Advanced
3)Click on "Enable extended logging"
 
After this go to C:\ProgramData\Veeam\Backup\SharePointExplorer and you`ll have the config.xml. Before we edit it please stop both VBO365 services:

• Veeam Backup for Microsoft Office 365 Service
• Veeam Backup Proxy for Microsoft Office 365 Service

 
Add the bold text as per examples below:
 
The SharePoint Explorer config.xml should look like this:
<Veeam>
<Archiver OperationTimeout="1200">
    <RemoteConnection Timeout="10:00:00" />
    <WCF OperationTimeout="1200" />        
….
 </Archiver>
</Veeam>
 
 
The proxy.xml from the remote proxy, should be like this:
<Veeam>
  <Archiver OperationTimeout="1200>
      <RemoteConnection Timeout="10:00:00" />
      <WCF OperationTimeout="1200" />
   ….
  </Archiver>
</Veeam>
 
The issue should fix now. If the issue persists, please change the value "1200" to "2400" and check again.

Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]

We recently faced backup failure for Application aware backup in Veeam for MSSQL server. The error is listed below,

Processing <VM_Name> Error: Unfreeze error: [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data.

A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}].

Instance ID: [{41239125-29f6-4455-9bc1-32439ceebe75}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f0].]

Normally this is caused by NTDS writers issue and we can easily fix it by restating the below services,

before restarting, I listed 'vssadmin list writers' through cmd, I found that, writer was in failed state

Writer name: 'SqlServerWriter'

Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}

Writer Instance Id: {41239125-29f6-4455-9bc1-32439ceebe75}

State: [8] Failed

Last error: Inconsistent shadow copy

So I have restarted the below services and ran above cmd  again and no error found.

COM+ Event System

Volume Shadow Copy

MSSQL VSS services.

Here backup will success after the retry.

Today, I got same error and I tried same fix, however got same error and it was surprise to me.

I checked the Log for the Veeam backup in MSSQL  server and found the below error,

                Connecting to mssql, connection string: Provider='msoledbsql';Data Source='Server_Name';Integrated Security='SSPI';Persist Security Info=False, timeout: 15

                Instance '' connection problem: Code = 0x80040e4d

         Code meaning = IDispatch error #3149

           Source = Microsoft OLE DB Driver for SQL Server

             Description = Login failed for user 'user_Name'.

We found that there were some new MSSQL instance created in this server and the service account was not added for those instance to access and backup.

Finally we provided permission to same account to newly created instances and issue got fixed.

Failed to mount NFS volume: Fault "PlatformConfigFaultFault", detail "NFS mount failed: Unable to connect to NFS server."

 When I try to restore the server from Veeam backup to Vcloud environment I got the error that, PlatformConfigFaultFault while restore the VMs. 

Detailed errors are shown below,

Unable to check if vPower NFS is mounted on the host "esxi-host.domain.local"(Veeam.Backup.Common.CRegeneratedTraceException)

Failed to connect backup datastore to the ESXi host "esxi-host.domain.local". (Veeam.Backup.Common.CRegeneratedTraceException)

Failed to add NFS datastore for NFS host 'Veeam_proxy4'. Failed to mount NFS volume (192.168.0.93:/VeeamBackup_Veeam_proxy4). 192.168.0.93: Fault "PlatformConfigFaultFault", detail "NFS mount 192.168.0.93:/VeeamBackup_Veeam_proxy4 failed: Unable to connect to NFS server." 

It is proxy issue. We need to open ports 902 and 443 between veeam proxy and ESXI host/VSPhere/VCloud end.

Error 0xC1900101 – 0x20017, The installation failed in SAFE_OS phase with an error during BOOT operation

We  had a request from customer that, upgrade their server from Windows 2012 R2 to 2019 (in place upgrade) and we were facing this below error when performing OS upgrade.  

The installation failed in the Safe_OS phase with an error during boot operation. Error Code: 0xC1900101 (This indicates an issue in driver package)

We found that issue is caused y windows driver packages and tried online toturial and unable to fix this issue and finally fixed the issue using below stuffs,

 --

To check which driver caused the issue, we checked in the CompatData.xml file present under the path "C:\Windows\Panther".

 In this file, in the DriverPackages section, we can see the driver which is blocking the upgrade. We need to check the BlockMigration entry whether it is either True or False.

If the values are, 

True = Indicates the problematic driver

False = No issues.

---

For example,

<DriverPAckage Inf="oem0.inf" BlockMigration="True" HasSignedBinaries="False" />

<DriverPAckage Inf="oem6.inf" BlockMigration="True" HasSignedBinaries="False" />

 

To get the driver's name, run the below command in CMD Prompt as elevated permission,

 

dism /Online /get-drivers /format:table

 and go to "C:\Windows\INF" open the respective INF file to get the name of the exact faulty drivers.

 (in our server we found the below drivers are faulty)

For example, we found below from oem6.inf

1) oem0.inf = prnms001.inf (The Microsoft XPS Document Writer)

2) oem6.inf = vm3d.inf (VMware SVGA 3D)

 

To fix the issue we can try the following.

 

1) Remove Microsoft XPS Document Writer in the printer section.

2) Disable VMware SVGA 3D driver (Do not uninstall, we may receive black screen, cause this is the display related drivers)

Once performing the above fix, restart the server and initiate the upgrade again.

Note:

After the upgrade, both the drivers SVGA driver and Microsoft XPS document writer gets enabled automatically.

Gratitude section:

This issue was identified and fixed by one of my colleague Balumahendran Chinna Muniyandi. Thanks for his help to writing this article.

The Hyper-V Virtual Machine Management service encountered an unexpected error: Unspecified error (0x80004005). Error code: '32768'.

 We had a request from customer to extend the RAID 10 configured disks, however there is no option to extend without deleting because of mixed RAIDs 1 and 0.

So we decided to backup the data in Veeam server and delete the existing disks RAIDs (except OS - OS not under any RAID and it has dedicated disk)  and re-configure the RAID. 

After re-configured the RAID , OS was found as it is and and Hyper-V role was remained the same (We expected the same and it will reduce couple of hours down time.) with same configurations, however other stuffs were removed. I mean (.XML, .VMCX, .VMRS, .VMGS, .VHD, .VHDX) files.

Initial issue, we found was Veeam refused to communicate with this server and got RPC failed to communicate error,

The issue, we found was routing was missing and added in the server and after that both Veeam and servers were started to talk each other.

Then, we tried to restore the VMs but got below error,

Error Restore job failed Error: Failed to create planned VM configuration (path: 'D:\VMs\<GUIDs>.vmcx'). Job failed ('Failed to import a virtual machine.
The Hyper-V Virtual Machine Management service encountered an unexpected error: Unspecified error (0x80004005).'). Error code: '32768'.

Regarding the error - it seems Hyper-V could not import the VMs and throwing error. We believe that it was due to some issue in MicroSoft file systems. We contacted the Veeam and they suggest the same or try to restore files individually and import VM manually.

We restored the configuration and disks  (.XML, .VMCX, .VMRS, .VMGS, .VHD, .VHDX) files manually and import the VMs manually and all the VMs were started to work without any issues.

Failed to process team: with Teams. Failed to find Team mailbox owner account.

When I try to backup O365 environment (Onedrive, MS Teams, Mailboxes and SharePoint) through Veeam backup server,  I received  the error that, Failed to find Team mailbox owner account and backup failed for the particular channel. 

The detailed error is below,

Failed to process team: <example_channel> with Teams. Failed to find Team mailbox owner account.

I found this group <example_channel> is orphaned  state, none is assigned as group owner. So I assigned one Global admin as owner to fix this. 

Follow the steps to assign Group owner,

1. Sign in to the Microsoft 365 portal with an account that has administrator privileges. 

2. Choose Groups from the navigation pane at the left hand side.

2. Search and find the group that you need to assign a new owner to and click on it.

3. In the new window appeared in right navigation, Select Owners tab, choose Edit.

4. In the View Owners dialog box, choose Add Owners.

5. Assign the right person(s) as owner and close the tab. 

Note : Select minimum two recommended as by MicroSoft.

Problem with attaching files in Teams chat or channel

 Error:

 When user is trying to upload files using MS Teams for a channel, getting files are locked error.

Cause :

The user does not have corresponding MSTeams channel site access.

Note: Every MS Teams channel has its own SharePoint sites. So when we upload files in MS Teams it will upload in SharePoint. So the affected users can verify themselves whether they have correct permission to access the MSTeams channel site access , click POSTS tab then  three dots (...) --> Open in SharePoint option. 

Solution:

Check respective MSTeams channel's SharePoint site permission for the user and assign required permission for the user.

Error: Failed to call RPC function 'StartAgent': Timed out requesting agent port for client sessions.

 Error : 

Failed to call RPC function 'StartAgent': Timed out requesting agent port for client sessions.

When we try to backup VMware VCloud environment, recently we occupied with above error in our Veeam server always, though these  environment (VBR and two repositories) have 200GB RAM and 2 sockets and 48 CPU cores on each servers.

This message typically means that the proxy/repository is overloaded, however all the servers are the Apollo servers, which most powerful machines with high configuration, finally I have lowered both proxy and repository tasks again and again, but the problem just reappear.

Cause:

There are transport agents that start for VMs processing's, this load can be limited by the concurrent task limit. 

But VCD job has also vApps, and for this container also transport agents start, but they are not limited by the concurrent task limit and Veeam Agent processes are not being properly closed by the moment so it is causing the port range (2500-3300) is being occupied. 

Due to this many agents are trying to start at one moment (we set 35-40 backup job start time as same) and  transport service "Veeam Data Mover Service"  started to crash.

Also we found some errors in log,

 < 34284> rpc      | ERR |RPC method call has failed (StartAgent).

 < 34284> rpc      | >>  |Timed  out requesting agent port for client sessions.

 < 34284> rpc      | >>  |--tr:Failed to initialize agent ( specified agent path: [C:\Program Files (x86)\Veeam\Backup Transport\x64\VeeamAgent.exe])

< 34284> rpc      | >>  |--tr:Failed to initialize agent instance {f392c303-9c2b-41dd-817f-6d04907ddf9e}.

< 34284> rpc      | >>  |An exception was thrown from thread [34284].

 

How to fix this,

We have moved some jobs from current backup time to another time, since then we haven't had a single failure. As talked with Veeam, this behavior was mostly unknown to them and they will take it up with QA. 

It must be much better to, do not start the agent until it is really needed, just like is the case already for VMs.

Office can't be activated because it doesn't meet the requirements for running on remote

 Error :

When user tries to open office applications and gets the error,

 

Office can't be activated because it doesn't meet the requirements for running on remote

 

in Terminal/RDS server the following error may occur:

This copy of Microsoft Office 2016 cannot be used on a computer running Terminal Services. To use Office 2016 on a computer running Terminal Services, you must use a Volume License of Office.

 

Troubleshoot:

Please confirm whether the office product installed as Office 365 ProPlus on a remote desktop server through ODT deployment.

 

Fix/Solution:

If it is not installed by ODT then try to install using ODT installation method.

 

If yes,

Open the registry editor (Win+R > regedit) and go to this path:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office\ClickToRun\Configuration

Add the SharedComputerLicensing (Reg_SZ) key and set its value to 1.

(or)

Press Win+R and run gpedit.msc to open the local group policy editor.

Edit the policy or create this policy if it doesn’t exist:

Computer Configuration\Policies\Administrative Templates\Microsoft Office 2016 (Machine)\Licensing Settings

Enable the 'Use shared computer activation'.

Error: bad allocation Failed to upload disk in Veeam backup

 

Issue:

When I try to backup Hyper-V environment got the below error, 

Error: bad allocation Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: Reconnectable protocol device was closed. Unable to retrieve next block transmission command. Number of already processed blocks: [825]. Failed to download disk 'FIleName.vhdx'.  

Cause:

Bad Allocation - This issue due to high usage of either CPU or memory.

Fix:

Checked the Hyper-V and found that the VM is using too many resources (both CPU and memory), try to reduce the resource usage or reboot the server to make server normal usage.

---

Once server became normal backup started work normally.