Error 0xC1900101 – 0x20017, The installation failed in SAFE_OS phase with an error during BOOT operation

We  had a request from customer that, upgrade their server from Windows 2012 R2 to 2019 (in place upgrade) and we were facing this below error when performing OS upgrade.  

The installation failed in the Safe_OS phase with an error during boot operation. Error Code: 0xC1900101 (This indicates an issue in driver package)

We found that issue is caused y windows driver packages and tried online toturial and unable to fix this issue and finally fixed the issue using below stuffs,

 --

To check which driver caused the issue, we checked in the CompatData.xml file present under the path "C:\Windows\Panther".

 In this file, in the DriverPackages section, we can see the driver which is blocking the upgrade. We need to check the BlockMigration entry whether it is either True or False.

If the values are, 

True = Indicates the problematic driver

False = No issues.

---

For example,

<DriverPAckage Inf="oem0.inf" BlockMigration="True" HasSignedBinaries="False" />

<DriverPAckage Inf="oem6.inf" BlockMigration="True" HasSignedBinaries="False" />

 

To get the driver's name, run the below command in CMD Prompt as elevated permission,

 

dism /Online /get-drivers /format:table

 and go to "C:\Windows\INF" open the respective INF file to get the name of the exact faulty drivers.

 (in our server we found the below drivers are faulty)

For example, we found below from oem6.inf

1) oem0.inf = prnms001.inf (The Microsoft XPS Document Writer)

2) oem6.inf = vm3d.inf (VMware SVGA 3D)

 

To fix the issue we can try the following.

 

1) Remove Microsoft XPS Document Writer in the printer section.

2) Disable VMware SVGA 3D driver (Do not uninstall, we may receive black screen, cause this is the display related drivers)

Once performing the above fix, restart the server and initiate the upgrade again.

Note:

After the upgrade, both the drivers SVGA driver and Microsoft XPS document writer gets enabled automatically.

Gratitude section:

This issue was identified and fixed by one of my colleague Balumahendran Chinna Muniyandi. Thanks for his help to writing this article.

The Hyper-V Virtual Machine Management service encountered an unexpected error: Unspecified error (0x80004005). Error code: '32768'.

 We had a request from customer to extend the RAID 10 configured disks, however there is no option to extend without deleting because of mixed RAIDs 1 and 0.

So we decided to backup the data in Veeam server and delete the existing disks RAIDs (except OS - OS not under any RAID and it has dedicated disk)  and re-configure the RAID. 

After re-configured the RAID , OS was found as it is and and Hyper-V role was remained the same (We expected the same and it will reduce couple of hours down time.) with same configurations, however other stuffs were removed. I mean (.XML, .VMCX, .VMRS, .VMGS, .VHD, .VHDX) files.

Initial issue, we found was Veeam refused to communicate with this server and got RPC failed to communicate error,

The issue, we found was routing was missing and added in the server and after that both Veeam and servers were started to talk each other.

Then, we tried to restore the VMs but got below error,

Error Restore job failed Error: Failed to create planned VM configuration (path: 'D:\VMs\<GUIDs>.vmcx'). Job failed ('Failed to import a virtual machine.
The Hyper-V Virtual Machine Management service encountered an unexpected error: Unspecified error (0x80004005).'). Error code: '32768'.

Regarding the error - it seems Hyper-V could not import the VMs and throwing error. We believe that it was due to some issue in MicroSoft file systems. We contacted the Veeam and they suggest the same or try to restore files individually and import VM manually.

We restored the configuration and disks  (.XML, .VMCX, .VMRS, .VMGS, .VHD, .VHDX) files manually and import the VMs manually and all the VMs were started to work without any issues.

Failed to process team: with Teams. Failed to find Team mailbox owner account.

When I try to backup O365 environment (Onedrive, MS Teams, Mailboxes and SharePoint) through Veeam backup server,  I received  the error that, Failed to find Team mailbox owner account and backup failed for the particular channel. 

The detailed error is below,

Failed to process team: <example_channel> with Teams. Failed to find Team mailbox owner account.

I found this group <example_channel> is orphaned  state, none is assigned as group owner. So I assigned one Global admin as owner to fix this. 

Follow the steps to assign Group owner,

1. Sign in to the Microsoft 365 portal with an account that has administrator privileges. 

2. Choose Groups from the navigation pane at the left hand side.

2. Search and find the group that you need to assign a new owner to and click on it.

3. In the new window appeared in right navigation, Select Owners tab, choose Edit.

4. In the View Owners dialog box, choose Add Owners.

5. Assign the right person(s) as owner and close the tab. 

Note : Select minimum two recommended as by MicroSoft.

Problem with attaching files in Teams chat or channel

 Error:

 When user is trying to upload files using MS Teams for a channel, getting files are locked error.

Cause :

The user does not have corresponding MSTeams channel site access.

Note: Every MS Teams channel has its own SharePoint sites. So when we upload files in MS Teams it will upload in SharePoint. So the affected users can verify themselves whether they have correct permission to access the MSTeams channel site access , click POSTS tab then  three dots (...) --> Open in SharePoint option. 

Solution:

Check respective MSTeams channel's SharePoint site permission for the user and assign required permission for the user.

Error: Failed to call RPC function 'StartAgent': Timed out requesting agent port for client sessions.

 Error : 

Failed to call RPC function 'StartAgent': Timed out requesting agent port for client sessions.

When we try to backup VMware VCloud environment, recently we occupied with above error in our Veeam server always, though these  environment (VBR and two repositories) have 200GB RAM and 2 sockets and 48 CPU cores on each servers.

This message typically means that the proxy/repository is overloaded, however all the servers are the Apollo servers, which most powerful machines with high configuration, finally I have lowered both proxy and repository tasks again and again, but the problem just reappear.

Cause:

There are transport agents that start for VMs processing's, this load can be limited by the concurrent task limit. 

But VCD job has also vApps, and for this container also transport agents start, but they are not limited by the concurrent task limit and Veeam Agent processes are not being properly closed by the moment so it is causing the port range (2500-3300) is being occupied. 

Due to this many agents are trying to start at one moment (we set 35-40 backup job start time as same) and  transport service "Veeam Data Mover Service"  started to crash.

Also we found some errors in log,

 < 34284> rpc      | ERR |RPC method call has failed (StartAgent).

 < 34284> rpc      | >>  |Timed  out requesting agent port for client sessions.

 < 34284> rpc      | >>  |--tr:Failed to initialize agent ( specified agent path: [C:\Program Files (x86)\Veeam\Backup Transport\x64\VeeamAgent.exe])

< 34284> rpc      | >>  |--tr:Failed to initialize agent instance {f392c303-9c2b-41dd-817f-6d04907ddf9e}.

< 34284> rpc      | >>  |An exception was thrown from thread [34284].

 

How to fix this,

We have moved some jobs from current backup time to another time, since then we haven't had a single failure. As talked with Veeam, this behavior was mostly unknown to them and they will take it up with QA. 

It must be much better to, do not start the agent until it is really needed, just like is the case already for VMs.

Office can't be activated because it doesn't meet the requirements for running on remote

 Error :

When user tries to open office applications and gets the error,

 

Office can't be activated because it doesn't meet the requirements for running on remote

 

in Terminal/RDS server the following error may occur:

This copy of Microsoft Office 2016 cannot be used on a computer running Terminal Services. To use Office 2016 on a computer running Terminal Services, you must use a Volume License of Office.

 

Troubleshoot:

Please confirm whether the office product installed as Office 365 ProPlus on a remote desktop server through ODT deployment.

 

Fix/Solution:

If it is not installed by ODT then try to install using ODT installation method.

 

If yes,

Open the registry editor (Win+R > regedit) and go to this path:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office\ClickToRun\Configuration

Add the SharedComputerLicensing (Reg_SZ) key and set its value to 1.

(or)

Press Win+R and run gpedit.msc to open the local group policy editor.

Edit the policy or create this policy if it doesn’t exist:

Computer Configuration\Policies\Administrative Templates\Microsoft Office 2016 (Machine)\Licensing Settings

Enable the 'Use shared computer activation'.

Error: bad allocation Failed to upload disk in Veeam backup

 

Issue:

When I try to backup Hyper-V environment got the below error, 

Error: bad allocation Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: Reconnectable protocol device was closed. Unable to retrieve next block transmission command. Number of already processed blocks: [825]. Failed to download disk 'FIleName.vhdx'.  

Cause:

Bad Allocation - This issue due to high usage of either CPU or memory.

Fix:

Checked the Hyper-V and found that the VM is using too many resources (both CPU and memory), try to reduce the resource usage or reboot the server to make server normal usage.

---

Once server became normal backup started work normally.

Unable to create azure subscriptions : check the user has administrative privilages on the specific Azure account

Issue: 

I'm trying to restore whole VM in from Hyper-V backup job in to MS Azure. When I add account in to Veeam server getting below error,

"Unable to create azure subscriptions : check the user has administrative privilages on the specific Azure account "

We are not sure why Veeam is trying to create a new subscription while adding credential. Could you check and help us to check this case.

We found only below log in our log files

[19.10.2021 14:40:46] Info Connecting to Azure by Power Shell
[19.10.2021 14:40:46] Info Login to Azure Account
[19.10.2021 14:40:46] Info [PS] 'Add-AzureRmAccount'. Arguments: EnvironmentName='AzureCloud'.
[19.10.2021 14:41:04] Info [PS] 'Get-AzureRmContext'. Arguments: .
[19.10.2021 14:41:04] Info [PS] 'Get-AzureRmADApplication'. Arguments: IdentifierUri='https://VeeamAzureApp';ErrorAction='SilentlyContinue'.
[19.10.2021 14:41:05] Info [PS] 'Get-Module'. Arguments: n/a.
[19.10.2021 14:41:07] Info Registering application in AD
[19.10.2021 14:41:07] Info [PS] 'New-AzureRmADApplication'.
[19.10.2021 14:41:07] Info Disposing splash form
[19.10.2021 14:41:07] Error [Veeam Backup and Replication] Unable to add Azure subscriptions: check that user has administrative priviliges on the specified Azure account

my Finding:

I have tried to add account under the version 10.0.1.4854 and got this error, however after I upgraded to veeam 11 it works fine.

Note : I believe this is Veeam version issue and raised the issue to Veeam Team but they refused it. Someone has same issues on Veeam 10 and fixed it?

Exception of type 'Veeam.Backup.AgentProvider.AgentClosedException' was thrown.

Issue/Error: When we try to backup the VM in veeam backup server, we gets  exception like below,

Exception of type 'Veeam.Backup.AgentProvider.AgentClosedException' was thrown.

Root Cause:

When backup jobs starts and proxy will assign automatically, but assigned proxy doesn't have access for the disk, so unable to access the disk and got failed.

Fig 1 : Backup proxies for HD1 and 2 don't have access permission.

Fig 1 : Backup proxy for HD 2 doesn't have access permission.

Solution:

1. Assign the proxies which has right permission.(Change the proxies to manual in backup job settings and and assign it)

2. Check the proxies that has same IP range as you Hyper-V/Vcenter node, so access using proxies will not get any issues.

Find PID for VEEAM backup job

Kill veeam backup job using Process ID

Issue: Sometimes veeam backup job won't complete (hung) in particular time and it does not backup any data, so when we try to stop the job and it won't help us and will remain in stopping state. So killing the backup job is only option using PID.

So we can find the corresponding backup job's PID using their log file.


Solution: The default log path C:\ProgramData\Veeam\Backup and you can find .LOG file with the below format.

JOB.<JOB_NAME>.backup.log  Filter those files with modified date and pick the latest date log file and Search PID: Kill the job using taskkill option. Note: When backup job starts meanwhile in background there are a lot of MSSQL transaction will  start for the backup job, If you kill the backup job using PID it may lead to corrupt backup job, try using your own risk. So mostly this is not a recommended way to stop the running backup jobs.

This item cannot be deleted because it is still referenced by other pages in SharePoint 2013

I got the error This item cannot be deleted because it is still referenced by other pages, when I delete a Test.master and Test.html files from SharePoint 2013.

I have checked more suggestions and solutions from Google, but no one suggestions and solutions helps me.

Somebody suggest me to do  Create a folder, move the master page into the folder, delete that folder through SPD. MS blogs suggests to Hide the File using this link http://support.microsoft.com/?kbid=926812 (This trick for MOSS 3.0 i.e SP 2007). Some people told me to use SCRIPT ( http://blogs.msdn.com/b/kvosters/archive/2011/10/12/this-item-cannot-be-deleted-because-it-is-still-referenced-by-other-pages.aspx ), and many more suggestions. I spent more than Two days for this issue, but no more improvement. That time my colleague give the below Solution. It works fine.

Please try this,

1) My Issues with Test.html and Test.master

2) Go to Root Site, Site Settings->Look and Feel Section: Design Manager. Here the Master Page Html file can          be modified what changes are we make in .master file.

3) But Site Settings->Look and Feel Section: Design Manager ->Edit Master Pages Test.html was missing.(By default Seattle and Oslo will be appear)

4) Open the site collection through SharePoint Designer, Right click on Seattle.html file -> Select "Edit File in Advanced Mode" from context menu.

5) Seattle.html opens in a new Tab. Select all and copy the code.

6) Then Go to Test.html (Right Click) -> "Edit File in Advanced Mode". Opens in a new tab.

7) Replace the existing code in the "Test.html" by pasting the code from Seattle.html

8) Save the file (This is done in "Check In" status)

9) In the Browser, Go To Root Site, Site Settings -> Look and Feel: Design Manager -> "Edit Master Pages" in the left side.

10) Now here the Test.html file will appear. Delete this file.

11) Check the "Delete" Folder. Here the files will be removed automatically. Delete the Delete folder.