kubernetes nightly test fails

[khaledyoussef24]

Description

kubernetes automated test failed.
test can not determine the number on ready nodes

TestK8s/kubernetes 2023-11-16T01:00:09Z logger.go:66: "\n[Interface]\nAddress = 100.64.1.2\nPrivateKey = 8MGbYxFxU06JHKBbkRmtEaYzCBeWO/vAUGfuQ6AFOVg=\n[Peer]\nPublicKey = 0coWX2G3QClsViRpVRxUy+0rNoL52XbNhovUvRLcHjw=\nAllowedIPs = 10.1.0.0/16, 100.64.0.0/16\nPersistentKeepalive = 25\nEndpoint = 185.206.122.31:5061\n\t"
TestK8s/kubernetes 2023-11-16T01:00:11Z retry.go:91: terraform [output -no-color -json ygg_ip]
TestK8s/kubernetes 2023-11-16T01:00:11Z logger.go:66: Running command terraform with args [output -no-color -json ygg_ip]
TestK8s/kubernetes 2023-11-16T01:00:11Z logger.go:66: "300:487a:7ea1:5b4e:a0d1:83d1:3f8d:69"
    k8s_test.go:77: 
        	Error Trace:	/home/runner/work/terraform-provider-grid/terraform-provider-grid/integrationtests/k8s_test.go:23
        	            				/home/runner/work/terraform-provider-grid/terraform-provider-grid/integrationtests/k8s_test.go:77
        	Error:      	Should be empty, but was could not excute command on remote with output W1116 01:00:12.837132     251 loader.go:222] Config not found: /etc/rancher/k3s/k3s.yaml
        	            	E1116 01:00:12.848956     251 memcache.go:238] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
        	            	E1116 01:00:12.850910     251 memcache.go:238] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
        	            	E1116 01:00:12.851863     251 memcache.go:238] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
        	            	E1116 01:00:12.852827     251 memcache.go:238] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
        	            	E1116 01:00:12.854653     251 memcache.go:238] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
        	            	The connection to the server localhost:8080 was refused - did you specify the right host or port?
        	            	: Process exited with status 1
        	Test:       	TestK8s/kubernetes
    k8s_test.go:77: 
        	Error Trace:	/home/runner/work/terraform-provider-grid/terraform-provider-grid/integrationtests/k8s_test.go:27
        	            				/home/runner/work/terraform-provider-grid/terraform-provider-grid/integrationtests/k8s_test.go:77
        	Error:      	Should be true
        	Test:       	TestK8s/kubernetes
        	Messages:   	number of ready nodes is not equal to number of nodes only %!s(int=0) nodes are ready

[khaledyoussef24]

working on this

[Eslam-Nawara]

Investigation and Solution:

The problem now is that there's a problem with Yggdrasil network, and we can't use it to check the connection with the deployed cluster, and I couldn't get it to work and others had the same problem, so this issue is blocked until Yggdrasil issue is done

TestK8s/kubernetes 2023-12-13T01:16:14Z retry.go:91: terraform [output -no-color -json wg_config]
TestK8s/kubernetes 2023-12-13T01:16:14Z logger.go:66: Running command terraform with args [output -no-color -json wg_config]
TestK8s/kubernetes 2023-12-13T01:16:14Z logger.go:66: "\n[Interface]\nAddress = 100.64.1.2\nPrivateKey = ePw4ffcRYGHKDlldAmOKugBMQVYTFqb9LAywBQXVW1o=\n[Peer]\nPublicKey = 5YvWQkRVs1qV0JQ4T9FK4dlsLFO7NYzr3xis+zirkgo=\nAllowedIPs = 10.1.0.0/16, 100.64.0.0/16\nPersistentKeepalive = 25\nEndpoint = 185.206.122.32:7234\n\t"
    k8s_test.go:71: 
        	Error Trace:	/home/runner/work/terraform-provider-grid/terraform-provider-grid/integrationtests/k8s_test.go:71
        	Error:      	Should be true
        	Test:       	TestK8s/kubernetes
    k8s_test.go:74: 
        	Error Trace:	/home/runner/work/terraform-provider-grid/terraform-provider-grid/integrationtests/k8s_test.go:74
        	Error:      	Should be true
        	Test:       	TestK8s/kubernetes
TestK8s/kubernetes 2023-12-13T01:22:15Z retry.go:91: terraform [output -no-color -json ygg_ip]
TestK8s/kubernetes 2023-12-13T01:22:15Z logger.go:66: Running command terraform with args [output -no-color -json ygg_ip]
TestK8s/kubernetes 2023-12-13T01:22:15Z logger.go:66: "300:68f9:dc9f:d10e:c0a2:ceeb:8ac1:e137"
    k8s_test.go:77: 
        	Error Trace:	/home/runner/work/terraform-provider-grid/terraform-provider-grid/integrationtests/k8s_test.go:23
        	            				/home/runner/work/terraform-provider-grid/terraform-provider-grid/integrationtests/k8s_test.go:77
        	Error:      	Should be empty, but was could not start ssh connection: dial tcp [300:68f9:dc9f:d10e:c0a2:ceeb:8ac1:e137]:22: connect: connection timed out
        	Test:       	TestK8s/kubernetes
    k8s_test.go:77: 
        	Error Trace:	/home/runner/work/terraform-provider-grid/terraform-provider-grid/integrationtests/k8s_test.go:27
        	            				/home/runner/work/terraform-provider-grid/terraform-provider-grid/integrationtests/k8s_test.go:77
        	Error:      	Should be true
        	Test:       	TestK8s/kubernetes
        	Messages:   	number of ready nodes is not equal to number of nodes only %!s(int=0) nodes are ready

[Eslam-Nawara]

• Blocked by scheduler issue
• Details: fix k8s nightly test #808 (comment)