com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host:

[bkrajendra]

ISSUE TYPE

 * Bug Report

COMPONENT NAME

ADD HOST

CLOUDSTACK VERSION

4.17.0

CONFIGURATION

basic network

OS / ENVIRONMENT

Ubuntu 20.04

SUMMARY

I already had the host set up with an earlier version of the cloud stack.
I had some issues with that setup, hence I decided to create new setup.
I installed a fresh management server 4.17 on separate server.
When I am trying add the host, I am getting following error.

2022-09-08 11:59:48,142 DEBUG [c.c.u.s.SSHCmdHelper] (qtp555273695-21:ctx-661e3e65 ctx-dec74d3e) (logid:21946c0b) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2022-09-08 11:59:49,778 DEBUG [c.c.u.s.SSHCmdHelper] (qtp555273695-21:ctx-661e3e65 ctx-dec74d3e) (logid:21946c0b) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
SSH command output:
cat: /etc/cloudstack/agent/cloud.csr: No such file or directory
chmod: cannot access '/etc/cloudstack/agent/cloud.csr': No such file or directory

2022-09-08 11:59:49,778 DEBUG [c.c.u.s.SSHCmdHelper] (qtp555273695-21:ctx-661e3e65 ctx-dec74d3e) (logid:21946c0b) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2022-09-08 11:59:50,381 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-488a4f32) (logid:f4dc8d86) HA health check task is running...
2022-09-08 11:59:51,437 DEBUG [c.c.u.s.SSHCmdHelper] (qtp555273695-21:ctx-661e3e65 ctx-dec74d3e) (logid:21946c0b) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
SSH command output:
cat: /etc/cloudstack/agent/cloud.csr: No such file or directory
chmod: cannot access '/etc/cloudstack/agent/cloud.csr': No such file or directory

2022-09-08 11:59:51,537 DEBUG [c.c.u.s.SSHCmdHelper] (qtp555273695-21:ctx-661e3e65 ctx-dec74d3e) (logid:21946c0b) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2022-09-08 11:59:53,153 DEBUG [c.c.u.s.SSHCmdHelper] (qtp555273695-21:ctx-661e3e65 ctx-dec74d3e) (logid:21946c0b) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
SSH command output:
cat: /etc/cloudstack/agent/cloud.csr: No such file or directory
chmod: cannot access '/etc/cloudstack/agent/cloud.csr': No such file or directory

2022-09-08 11:59:53,154 WARN  [c.c.h.k.d.LibvirtServerDiscoverer] (qtp555273695-21:ctx-661e3e65 ctx-dec74d3e) (logid:21946c0b)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: IP - Failed to setup keystore on the KVM host: IP
2022-09-08 11:59:53,156 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp555273695-21:ctx-661e3e65 ctx-dec74d3e) (logid:21946c0b)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: IP - Failed to setup keystore on the KVM host: IP
com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: IP
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:178)
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:320)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:811)
        at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:644)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy196.discoverHosts(Unknown Source)
        at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:142)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:776)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:600)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:327)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:145)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:142)
        at com.cloud.api.ApiServlet.doPost(ApiServlet.java:101)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:829)
2022-09-08 11:59:53,158 DEBUG [c.c.r.ResourceManagerImpl] (qtp555273695-21:ctx-661e3e65 ctx-dec74d3e) (logid:21946c0b) Could not add host at [http://IP] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: IP - Failed to setup
keystore on the KVM host: IP].
com.cloud.exception.DiscoveredWithErrorException:  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: IP- Failed to setup keystore on the KVM host: IP
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:375)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:811)
        at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:644)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy196.discoverHosts(Unknown Source)
        at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:142)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:776)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:600)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:327)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:145)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:142)
        at com.cloud.api.ApiServlet.doPost(ApiServlet.java:101)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: IP
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:178)
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:320)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:811)
        ... 60 more
2022-09-08 11:59:53,160 INFO  [c.c.u.e.CSExceptionErrorCode] (qtp555273695-21:ctx-661e3e65 ctx-dec74d3e) (logid:21946c0b) Could not find exception: com.cloud.exception.DiscoveryException in error code list for exceptions
2022-09-08 11:59:53,161 WARN  [o.a.c.a.c.a.h.AddHostCmd] (qtp555273695-21:ctx-661e3e65 ctx-dec74d3e) (logid:21946c0b) Exception:
com.cloud.exception.DiscoveryException: Could not add host at [http://IP] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: IP- Failed to setup keystore on the KVM host: IP].
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:818)
        at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:644)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy196.discoverHosts(Unknown Source)
        at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:142)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:776)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:600)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:327)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:145)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:142)
        at com.cloud.api.ApiServlet.doPost(ApiServlet.java:101)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.exception.DiscoveredWithErrorException:  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: IP - Failed to setup keystore on the KVM host: IP
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:375)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:811)
        at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:644)
        ... 59 more
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: IP
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:178)
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:320)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:811)
        ... 60 more
2022-09-08 11:59:53,164 INFO  [c.c.a.ApiServer] (qtp555273695-21:ctx-661e3e65 ctx-dec74d3e) (logid:21946c0b) Could not add host at [http://IP] with zone [1], pod [1] and
cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: IP - Failed to setup keystore on the KVM host: IP]. 

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[yadvr]

@bkrajendra have you allowed ssh server on the KVM host to login as root user and password (password based auth)? You can also disable ca framework all together by setting the auth strictness global setting to false and try adding the host again (or restart the cloudstack-agent if it was setup already).

[bkrajendra]

@rohityadavcloud thank you for the quick reply.
Yes, I have access to the Host from the management server. I have tried access from the terminal and it works with password-based auth.

I'll try auth strictness global setting and test again and update here.

[bkrajendra]

@rohityadavcloud
I tried the following things:

1. set ca.plugin.root.auth.strictness=false restarted management server, restarted agent - still same error
2. uninstalled agent on a host, removed /etc/cloudstack/agent folder - now I got properties not found error and after that again same error related to failed to setup Keystore.
   Should I clean and install the host and try again?

[Hudratronium]

@bkrajendra
I am correct that your user for adding the host has this config in sudoers?
cloudstack ALL=NOPASSWD: /usr/bin/cloudstack-setup-agent

Just out of curiosity:
What's happening when you connect via ssh to the host while using the 'cloudstack' user and run
sudo /usr/share/cloudstack-common/scripts/util/keystore-setup

[bkrajendra]

@Hudratronium
After trying a lot and everything I gave up, and decided to go with an officially supported distro - Centos 7x minimal
Host adding worked out of box without any issue, I am stuck with systemVMs are in starting state from last 12 hours, not sure how much time I should wait.

About what you said: (in case of Ubuntu 20.04)

I am using the root user for all operations. I don't have cloudstack user.
root ALL=(ALL) ALL
Second, when I try keystore-setup from ssh, this is what I get:

sudo /usr/share/cloudstack-common/scripts/util/keystore-setup
cat: : No such file or directory
chmod: missing operand after ‘600’
Try 'chmod --help' for more information.
chmod: missing operand after ‘600’
Try 'chmod --help' for more information.
chmod: missing operand after ‘600’
Try 'chmod --help' for more information.

[Lueton]

@Hudratronium After trying a lot and everything I gave up, and decided to go with an officially supported distro - Centos 7x minimal Host adding worked out of box without any issue, I am stuck with systemVMs are in starting state from last 12 hours, not sure how much time I should wait.

About what you said: (in case of Ubuntu 20.04)

I am using the root user for all operations. I don't have cloudstack user. root ALL=(ALL) ALL Second, when I try keystore-setup from ssh, this is what I get:

sudo /usr/share/cloudstack-common/scripts/util/keystore-setup
cat: : No such file or directory
chmod: missing operand after ‘600’
Try 'chmod --help' for more information.
chmod: missing operand after ‘600’
Try 'chmod --help' for more information.
chmod: missing operand after ‘600’
Try 'chmod --help' for more information.

same issue for me, no matter if via cloudstack, ssh or local

[mlsorensen]

@Hudratronium After trying a lot and everything I gave up, and decided to go with an officially supported distro - Centos 7x minimal Host adding worked out of box without any issue, I am stuck with systemVMs are in starting state from last 12 hours, not sure how much time I should wait.

About what you said: (in case of Ubuntu 20.04)

I am using the root user for all operations. I don't have cloudstack user. root ALL=(ALL) ALL Second, when I try keystore-setup from ssh, this is what I get:

sudo /usr/share/cloudstack-common/scripts/util/keystore-setup
cat: : No such file or directory
chmod: missing operand after ‘600’
Try 'chmod --help' for more information.
chmod: missing operand after ‘600’
Try 'chmod --help' for more information.
chmod: missing operand after ‘600’
Try 'chmod --help' for more information.

These errors when manually running keystore-setup are likely due to not providing all of the extra flags necessary to run keystore-setup script. They point to empty variables. You can possibly try something like this to see if it works on the Ubuntu system, or perhaps exposes a missing utility or path:

/usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/cloud.jks mykeystorepassword 365 /etc/cloudstack/agent/cloud.csr

As far as systemvm issues, that's separate debugging. Could be that the system image is not provisioned or it is up but the internal agent isn't reachable. Would need to find out if the VM is running and get on console, or if it isn't running find at what step it failed.

It may not strictly be necessary to use system VMs if you opt for "direct download" VM templates and don't need VNC console, but it's worth trying to find out what is wrong.

[yadvr]

@bkrajendra @Hudratronium @Lueton was Ubuntu minimal server installed? We don't support that, we want the standard Ubuntu server installed, where some of the utilities are preinstalled. (you can refer to my install notes on Ubuntu 20.04 here which works for me https://rohityadav.cloud/blog/cloudstack-kvm/)

In case you still hit issues, let me know. I'm happy to update/edit the QIG and notes on my blog. cc @wido @weizhouapache any pointers if you can see, as you're one of the major Ubuntu users.

[Lueton]

@bkrajendra @Hudratronium @Lueton was Ubuntu minimal server installed? We don't support that, we want the standard Ubuntu server installed, where some of the utilities are preinstalled. (you can refer to my install notes on Ubuntu 20.04 here which works for me https://rohityadav.cloud/blog/cloudstack-kvm/)

In case you still hit issues, let me know. I'm happy to update/edit the QIG and notes on my blog. cc @wido @weizhouapache any pointers if you can see, as you're one of the major Ubuntu users.

I installed the general Ubuntu Server 20.04.
I'll try your installation guide as soon as possible, thanks for sharing!

[Hudratronium]

@bkrajendra @Hudratronium @Lueton was Ubuntu minimal server installed? We don't support that, we want the standard Ubuntu server installed, where some of the utilities are preinstalled. (you can refer to my install notes on Ubuntu 20.04 here which works for me https://rohityadav.cloud/blog/cloudstack-kvm/)

We were using Ubuntu Server 20.04.0 - which should be completely supported as far as i am aware.

[weizhouapache]

@Hudratronium
yes, ubuntu 20.04 is offically fully supported in cloudstack.

Questions

• was the host added in other cloudstack platform before ?
• Did you clean the host before you add it to the new platform ?

[Lueton]

I gave it another try with @rohityadavcloud guide. It still doesnt work yet but i discovered that the error message com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host only occours if i use a non-root user when adding a new host to the cluster. If i use root this message will not be in the logs.

However this might not be the main error. I can also see in the logs that theres another error (see screeenshot) which occours with both root and non-root user. I could see that the management node can establish a connection to the host (verified via ssh logs, host CPU increased) but then there is a problem (i guess with libvirt?) which causes to agent on the host to stop/restart.

Side note: the ui only responds to this when using non-root user. If i use root the ui responds only until i close the add host dialog.

[weizhouapache]

I gave it another try with @rohityadavcloud guide. It still doesnt work yet but i discovered that the error message com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host only occours if i use a non-root user when adding a new host to the cluster. If i use root this message will not be in the logs.

However this might not be the main error. I can also see in the logs that theres another error (see screeenshot) which occours with both root and non-root user. I could see that the management node can establish a connection to the host (verified via ssh logs, host CPU increased) but then there is a problem (i guess with libvirt?) which causes to agent on the host to stop/restart.

Side note: the ui only responds to this when using non-root user. If i use root the ui responds only until i close the add host dialog.

@Lueton
if you add the host using user/password, this error can be ignored.

[Lueton]

@weizhouapache thats good to know, i was very confused!

this is my /agent/setup.log after trying to add this machine as a new host, maybe you can identify the issue.

root@delta:~# cat /var/log/cloudstack/agent/setup.log
DEBUG:root:execute:uname -r
DEBUG:root:execute:uname -m
DEBUG:root:execute:hostname -f
DEBUG:root:execute:kvm-ok
DEBUG:root:execute:awk '/MemTotal/ { printf "%.3f \n", $2/1024 }' /proc/meminfo
DEBUG:root:execute:ip a | grep "^\w" | grep -iv "^lo" | wc -l
DEBUG:root:execute:service apparmor status
DEBUG:root:execute:apparmor_status |grep libvirt
DEBUG:root:Failed to execute:
DEBUG:root:execute:/bin/systemctl mask libvirtd.socket
DEBUG:root:execute:/bin/systemctl mask libvirtd-ro.socket
DEBUG:root:execute:/bin/systemctl mask libvirtd-admin.socket
DEBUG:root:execute:/bin/systemctl mask libvirtd-tls.socket
DEBUG:root:execute:/bin/systemctl mask libvirtd-tcp.socket
DEBUG:root:execute:systemctl restart libvirtd
DEBUG:root:execute:ufw allow 22
DEBUG:root:execute:ufw allow 1798
DEBUG:root:execute:ufw allow 16509
DEBUG:root:execute:ufw allow 16514
DEBUG:root:execute:ufw allow proto tcp from any to any port 5900:6100
DEBUG:root:execute:ufw allow proto tcp from any to any port 49152:49216
DEBUG:root:execute:sudo /usr/sbin/service ufw status
DEBUG:root:execute:sudo /usr/sbin/service ufw stop
DEBUG:root:execute:sudo /usr/sbin/service ufw status
DEBUG:root:Failed to execute:● ufw.service - Uncomplicated firewall
     Loaded: loaded (/lib/systemd/system/ufw.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Fri 2022-09-16 10:17:25 UTC; 23ms ago
       Docs: man:ufw(8)
    Process: 2354 ExecStop=/lib/ufw/ufw-init stop (code=exited, status=0/SUCCESS)
   Main PID: 396 (code=exited, status=0/SUCCESS)

Sep 16 10:16:19 delta systemd[1]: Finished Uncomplicated firewall.
Sep 16 10:17:25 delta systemd[1]: Stopping Uncomplicated firewall...
Sep 16 10:17:25 delta systemd[1]: ufw.service: Succeeded.
Sep 16 10:17:25 delta systemd[1]: Stopped Uncomplicated firewall.
DEBUG:root:execute:sudo /usr/sbin/service ufw start
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent status
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent stop
DEBUG:root:execute:sleep 30
DEBUG:root:execute:sudo update-rc.d -f cloudstack-agent remove
DEBUG:root:execute:sudo update-rc.d -f cloudstack-agent defaults
DEBUG:root:Failed to execute:update-rc.d: error: unable to read /etc/init.d/cloudstack-agent
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent status
DEBUG:root:Failed to execute:● cloudstack-agent.service - CloudStack Agent
     Loaded: loaded (/lib/systemd/system/cloudstack-agent.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2022-09-16 10:17:28 UTC; 30s ago
       Docs: http://www.cloudstack.org/
   Main PID: 2216 (code=exited, status=143)

Sep 16 10:17:23 delta systemd[1]: Started CloudStack Agent.
Sep 16 10:17:23 delta java[2216]: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
Sep 16 10:17:23 delta java[2216]: SLF4J: Defaulting to no-operation (NOP) logger implementation
Sep 16 10:17:23 delta java[2216]: SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Sep 16 10:17:27 delta systemd[1]: Stopping CloudStack Agent...
Sep 16 10:17:28 delta systemd[1]: cloudstack-agent.service: Main process exited, code=exited, status=143/n/a
Sep 16 10:17:28 delta systemd[1]: cloudstack-agent.service: Failed with result 'exit-code'.
Sep 16 10:17:28 delta systemd[1]: Stopped CloudStack Agent.
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent start