Issue when adding VLAN Pool for Guest network

[samdejoux1]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Physical network > Guest Network > Adding VLAN Pool

CLOUDSTACK VERSION

CloudStack 4.16.1.0

CONFIGURATION

Advanced networking configuration with VLANs used on the management, guest and public networks. Using VMWare DVS for guest and public traffic. Public traffic vlan has provisioned in vmware correctly, this issue is limited to guest.

OS / ENVIRONMENT

VMware ESXi, 6.7.0, 19898906
VCenter Version 6.7.0 Build 19832280

SUMMARY

When adding a pool of VLANs to the guest network under 'Dedicated VLAN/VNI Ranges' we receive the following error:

Error: (HTTP 530, error code 9999) Index 0 out of bounds for length 0

This is seen when creating through the GUI and also when using Cloud Monkey API Push.

Management server log shows :

2022-08-17 10:43:17,515 ERROR [c.c.a.ApiServer] (qtp1747352992-77024:ctx-db233219 ctx-fc39cbd8) (logid:bb7bd18a) unhandled exception executing api command: [Ljava.lang.String;@21acc147
java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0

If you refresh the screen, the VLAN pool has been added however when creating an instance the process fails because a VLAN cannot be allocated from the pool. It feels like the VLAN pool is not being fully provisioned on the system.

STEPS TO REPRODUCE

- Configure a physical network for mgmt using vSwitch0
- Configure a physical network for guest using dvSwitch0 (distributed virtual switch)
- Configure traffic labels
- Create isolated guest network
- Configure VLAN pool
- Create Instance

EXPECTED RESULTS

VLAN allocated from the pool for isolated guest network

ACTUAL RESULTS

-Error received when adding VLAN range = 

Error: (HTTP 530, error code 9999) Index 0 out of bounds for length 0

-Error received when creating VM Instance = 

2022-08-17 11:41:26,323 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-21:ctx-a2fc1cdc job-1327/job-1328 ctx-b0fa7bd2) (logid:69e5d813) STATS: Failed to alloc resource from host: 4 reservedCpu: 0, requested cpu: 1000, reservedMem: (0 bytes) 0, requested mem: (1.00 GB) 1073741824
2022-08-17 11:41:26,386 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-21:ctx-a2fc1cdc job-1327/job-1328 ctx-b0fa7bd2) (logid:69e5d813) Cleaning up because we're unable to implement the network Ntwk[227|Guest|8]
com.cloud.exception.InsufficientVirtualNetworkCapacityException: Unable to allocate vnet as a part of network Ntwk[227|Guest|8] implement Scope=interface com.cloud.dc.DataCenter; id=5
2022-08-17 11:41:26,777 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-21:ctx-a2fc1cdc job-1327/job-1328 ctx-b0fa7bd2) (logid:69e5d813) VM instance {id: "363", name: "i-2-363-VM", uuid: "d3691fe9-7067-4b1c-a40e-ca8d10707d59", type="User"} state transited from [Starting] to [Stopped] with event [OperationFailed]. VM's original host: null, new host: null, host before state transition: Host {"id": "4", "name": "xxxxxxx.xxxxxxx.xxxxxxx", "uuid": "7dfd437b-a8bd-42a5-88df-c76bbc15150d", "type"="Routing"}
2022-08-17 11:41:26,832 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-21:ctx-a2fc1cdc job-1327/job-1328 ctx-b0fa7bd2) (logid:69e5d813) Invocation exception, caused by: com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {id: "363", name: "i-2-363-VM", uuid: "d3691fe9-7067-4b1c-a40e-ca8d10707d59", type="User"}Scope=interface com.cloud.dc.DataCenter; id=5
2022-08-17 11:41:26,832 INFO  [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-21:ctx-a2fc1cdc job-1327/job-1328 ctx-b0fa7bd2) (logid:69e5d813) Rethrow exception com.cloud.exceptio.InsufficientServerCapacityException: Unable to create a deployment for VM instance {id: "363", name: "i-2-363-VM", uuid: "d3691fe9-7067-4b1c-a40e-ca8d10707d59", type="User"}Scope=interface com.cloud.dc.DataCenter; id=5
2022-08-17 11:41:27,057 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-48:ctx-e6681fe2 job-1327) (logid:69e5d813) Complete async job-1327, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Unable to start a VM due to insufficient capacity"}

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[yadvr]

@samdejoux1 what VLAN pool/range was added (for ex. can you share the API request or parameter you passed in the API or UI)?

[samdejoux1]

@samdejoux1 what VLAN pool/range was added (for ex. can you share the API request or parameter you passed in the API or UI)?

Hi @rohityadavcloud please see the command used to create the vlans below:

(mycloud2) 🐱 > dedicateguestvlanrange physicalnetworkid=203 vlanrange=1100-1150 account=admin domain=ROOT domainid=76546040-05d7-11ed-970b-005056a12013
🙈 Error: (HTTP 530, error code 9999) Index 0 out of bounds for length 0
(mycloud2) 🐱 >

Many thanks,

[samdejoux1]

After receiving the out of bounds error, the VLANs are still added to the DB, however they are not 'available' for use when building an instance.

[shwstppr]

@samdejoux1 I'm unable to reproduce Index 0 out of bounds for length 0 error on 4.17. with a similar setup,

> dedicateguestvlanrange physicalnetworkid=201 vlanrange=2131-2135 account=admin domain=ROOT domainid=ecc52a37-1a08-11ed-981f-1e00f100018f
{
  "dedicatedguestvlanrange": {
    "account": "admin",
    "domain": "ROOT",
    "domainid": "ecc52a37-1a08-11ed-981f-1e00f100018f",
    "guestvlanrange": "2131-2135",
    "id": "d01dde50-d839-45c0-8dbc-59489c829171",
    "physicalnetworkid": 201,
    "zoneid": 1
  }
}

Can you please share output of list physicalnetworks id=203

[samdejoux1]

Thanks @shwstppr, please see the output of the list physical networks command below:

> list physicalnetworks id=203
{
  "count": 1,
  "physicalnetwork": [
    {
      "broadcastdomainrange": "ZONE",
      "id": "b83056d8-faed-4bf6-9286-84baaa4613da",
      "isolationmethods": "VLAN",
      "name": "guest + public",
      "state": "Enabled",
      "zoneid": "8908f96e-d35e-403d-8035-8bf857c5361d",
      "zonename": "au-dcxq1"
    }
  ]
}
> 

[shwstppr]

Thanks for sharing the physical network details @samdejoux1
The issue you see is due to the absence of VLAN range for the physical network. Right now I'm not sure what is the correct behaviour.
@weizhouapache @NuxRo @rohityadavcloud @DaanHoogland should it allow dedicating guest vlan range when the concerned physical network doesn't have any VLAN range itself?
For each VLAN in the rage for physical n/w an entry is created in cloud.op_dc_vnet_alloc which is missing in this case and code at https://github.com/apache/cloudstack/blob/4.17/server/src/main/java/com/cloud/network/NetworkServiceImpl.java#L4148 is throwing error

[DaanHoogland]

whether we should disallow or auto-create a vlan-range seems like a user decision to me @shwstppr . good work tracking this.