Error :
Failed to call RPC function 'StartAgent': Timed out requesting agent port for client sessions.
When we try to backup VMware VCloud environment, recently we occupied with above error in our Veeam server always, though these environment (VBR and two repositories) have 200GB RAM and 2 sockets and 48 CPU cores on each servers.
This message typically means that the proxy/repository is overloaded, however all the servers are the Apollo servers, which most powerful machines with high configuration, finally I have lowered both proxy and repository tasks again and again, but the problem just reappear.
Cause:
There are transport agents that start for VMs processing's, this load can be limited by the concurrent task limit.
But VCD job has also vApps, and for this container also transport agents start, but they are not limited by the concurrent task limit and Veeam Agent processes are not being properly closed by the moment so it is causing the port range (2500-3300) is being occupied.
Due to this many agents are trying to start at one moment (we set 35-40 backup job start time as same) and transport service "Veeam Data Mover Service" started to crash.
Also we found some errors in log,
< 34284> rpc | ERR |RPC method call has failed (StartAgent).
< 34284> rpc | >> |Timed out requesting agent port for client sessions.
< 34284> rpc | >> |--tr:Failed to initialize agent ( specified agent path: [C:\Program Files (x86)\Veeam\Backup Transport\x64\VeeamAgent.exe])
< 34284> rpc | >> |--tr:Failed to initialize agent instance {f392c303-9c2b-41dd-817f-6d04907ddf9e}.
< 34284> rpc | >> |An exception was thrown from thread [34284].
How to fix this,
We have moved some jobs from current backup time to another time, since then we haven't had a single failure. As talked with Veeam, this behavior was mostly unknown to them and they will take it up with QA.
It must be much better to, do not start the agent until it is really needed, just like is the case already for VMs.
No comments:
Post a Comment