VPC network tier with internal LB issues

[chunkyen]

Hi, I have been testing Cloudstack using the Apache Cloudstack POC guide from ShapeBlue. In one of test case for VPC, it shows how to setup a frontend and backend network tier with internal LB and configuring VM using user data so as to demonstrate the frontend to backend LB and ACL capabilities.
I have performed this test sucessfully. However, when I shutdown my Cloudstack and power it on after sometime, the internal LB will get stuck in a starting state when I try to start the backend VMs. Furthermore, the backend tier guest network will show a shutdown state.
I have tried restarting VPC, restart router, delete router but I cannot get this to unstuck. I am also not able to clean up everything in the VPC.
I have tested this in a nested 1+3 nodes and a single baremetal node on version 4.21 with bridge networking and vlan isolation.
Have anyone else encountered the same issues? Are there any tips to troubleshoot and/or resolved this issue? (I cannot find anything useful from the management and agent log)
Thank you

[DaanHoogland]

@chunkyen , I am not sure but have you considered the order of starting things up again? Your hosts are VMs and after starting they may not start all the nested VMs again, including the systemVMs/VRs. Also where is the DB running? Did the log since restarting look clean or have DB errors?

[chunkyen]

This part is expected, a tier without any VMs “shuts down” and becomes “allocated” after a timeout.

I am not 100% sure but shouldn't this be a tier without VM running will be in a "allocated" state and become "shutdown" after a timeout? But I have never seen an actual network be in a shutdown state (except for this internal LB network error), only allocated and implemented.

When you say, "I have tried restarting VPC”, do you mean restart with cleanup?

Yes tried with and without cleanup and the result is the same, the internal LB and VM attached to this LB are stuck in starting and failed evetually after the timeout.

[DaanHoogland]

from Network,java, public enum State ,value Shutdown( "Indicates the network configuration is being destroyed”) , meaning it is not just being stopped.

part of the statemachine for it:

            s_fsm.addTransition(State.Implementing, Event.OperationFailed, State.Shutdown);
            s_fsm.addTransition(State.Implemented, Event.DestroyNetwork, State.Shutdown);

so no, only on explicit destroy would it go to state Shutdown. I have to test this, but I think the shutdown only happens if the router is stopped or (as an intermediate state) when the network destroyed.

[weizhouapache]

the internal LB feature is implemented by the separated vm which is named Internal LB

maybe there was something wrong with it. need reproducing

[chunkyen]

Anyway to debug when I encounter this again?

[chunkyen]

Just encountered it again, vpc network teir with internal LB will be in shutdown state while the internal lb VM is stuck in starting state. Only way to recover is to force stop internal lb and delete the internal lb. Happy to help debug if you can point me to the right direction.

[chunkyen]

Thank you for the explanation. I observed that the guest network that had the internal LB was in an allocated state upon management and host boot up. But as soon as I tried to boot up a vm in the network tier assigned as an instance to the internalLB, it will go into a shutdown state.
Also for the recovery, I do not need to destroy the VM attached to the internal LB. I just need to detatch the nic to another network, delete the internal LB and I can clean up the internal LB VM that was stucked.

[chunkyen]

For the internal LB just the default offering. I am using default Medium offering for the frontend and backend VM. For the VPC and network tiers, I am just following the Shape Blue POC guide.
For shutdown, I just stop all the VM and trigger shutdown of the management server. Whether I stop the router doesn't seem to make a difference.

[chunkyen]

Steps I use to consistently repro this.

1. Create VPC with frontend and backend (with internal lb) network tiers
2. Create internal LB
3. Create backend instance and assigned to internal LB
4. Create frontend instance and assign to associated public IP and create a LB
5. Test and everything work in accordance to the poc guide
6. Stop all instances, leave system VM and router/internalLB vm running
7. Shutdown management server using cmk
8. Shutdown management and KVM host (I am running nested)
9. Start management and KVM host
10. Ensure system vm and virtual router for VPC is running (sometime it doesn't start correctly)
11. Start a backend instance VM
12. The internal lb will start but stuck on starting
13. The backend network tier will show it is in Shutdown state

[DaanHoogland]

sorry @chunkyen , I did not have a chance to reproduce this yet (and will not for the coming few weeks probably)

[chunkyen]

It is ok, I know the collaboration conference is coming up (looking forward to join you guys next year)
We can follow up once you have the time :)

[rajujith]

@chunkyen I followed your steps in 4.22 RC3, I had to reboot/Start the VR after starting the KVM, but it works fine otherwise.