Skip to content

ERROR - Nitro Exception while binding group member to servicegroup errorcode=258 message=No such resource #630

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Avneetdabas opened this issue Jan 18, 2024 · 10 comments

Comments

@Avneetdabas
Copy link

Describe the bug
CIC is not able to update the pod ip as the backend in the VPX service group members.

To Reproduce

  1. We were able to reproduce by deploying the ingress with with 3 services on the backend, 2 services are working fine only one is showing down as the backend member is missing.
    2.CIC Version/Image : quay.io/citrix/citrix-k8s-ingress-controller:1.37.5

  2. Version of VPX - 14.1.12.30

  3. Environment variables (minus secrets)

Expected behavior
After deploying the Ingress all services should show show pod ip in the members so that client can reach the api hosted on those pods.

Logs
kubectl logs

2024-01-15 16:05:16,123 - ERROR - [nitrointerface.py:_configure_services_nondesired:2577] (MainThread) Nitro Exception while binding group member to servicegroup k8s-apexportal-webservice-service_54341_sgp_g6tphz7jrhk6c72t7dyqovf7cwchlvdr errorcode=258 message=No such resource [serviceGroupName, k8s-apexportal-webservice-service_54341_sgp_g6tphz7jrhk6c72t7dyqovf7cwchlvdr]
2024-01-15 16:05:16,154 - ERROR - [nitrointerface.py:_configure_services_nondesired:2577] (MainThread) Nitro Exception while binding group member to servicegroup k8s-apexportal-webservice-service_54341_sgp_g6tphz7jrhk6c72t7dyqovf7cwchlvdr errorcode=258 message=No such resource [serviceGroupName, k8s-apexportal-webservice-service_54341_sgp_g6tphz7jrhk6c72t7dyqovf7cwchlvdr]
2024-01-15 16:05:16,199 - ERROR - [nitrointerface.py:_configure_services_nondesired:2577] (MainThread) Nitro Exception while binding group member to servicegroup k8s-apexportal-webservice-service_54341_sgp_g6tphz7jrhk6c72t7dyqovf7cwchlvdr errorcode=258 message=No such resource [serviceGroupName, k8s-apexportal-webservice-service_54341_sgp_g6tphz7jrhk6c72t7dyqovf7cwchlvdr]
2024-01-15 16:06:04,053 - ERROR - [NSProfileHandler.py:bind_cipher_with_ssl_profile:352] (MainThread) Unable to bind cipher DEFAULT to SSL profile k8s-192.168.243.49_443_ssl
2024-01-15 17:39:14,301 - ERROR - [NSProfileHandler.py:bind_cipher_with_ssl_profile:352] (MainThread) Unable to bind cipher DEFAULT to SSL profile k8s-192.168.243.49_443_ssl
2024-01-15 19:10:39,618 - ERROR - [nitrointerface.py:set_ns_config:6968] (MainThread) Nitro exception during updating csvserver: error message=Profile does not exist
2024-01-15 19:32:38,235 - ERROR - [kubernetes.py:_parse_preconfigured_certs:419] (MainThread) certkey {'name': '.Apexanalytix.com2021-2022', 'type': 'Custom_SSL_Cipher_new'} does not have correct name/type
2024-01-15 19:32:38,235 - ERROR - [kubernetes.py:_parse_preconfigured_certs:421] (MainThread) preconfigured-certkey {"certs": [ {"name": "
.Apexanalytix.com2021-2022", "type": "Custom_SSL_Cipher_new"} ] } is not in correct format,It should be in below format

Ingress Yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: citrix
ingress.citrix.com/frontend-ip: "192.168.."
ingress.citrix.com/secure-service-type: "ssl"
ingress.citrix.com/secure-port: "443"
ingress.citrix.com/frontend-sslprofile: "HSTS2022-23"
ingress.citrix.com/preconfigured-certkey: '{"certs": [ {"name": "..com2021-2022", "type": "default"} ] }'
name: services-ingress
spec:
rules:
- host: services.
**
http:
paths:
- path: /api
pathType: Prefix
backend:
service:
name: -webservice-service
port:
number: 80
- path: /F
.V
***
pathType: Prefix
backend:
service:
name: **-soapservice-service
port:
number: 80
- path: /odata
pathType: Prefix
backend:
service:
name: -odata-service
port:
number: 80
tls:
- hosts:
- *******.
.com
secretName:

@Avneetdabas Avneetdabas changed the title ERROR - [nitrointerface.py:_configure_services_nondesired:2577] (MainThread) Nitro Exception while binding group member to servicegroup errorcode=258 message=No such resource ERROR - Nitro Exception while binding group member to servicegroup errorcode=258 message=No such resource Jan 18, 2024
@apoorvak-citrix
Copy link
Contributor

@Avneetdabas Could you kindly provide the YAML definition for the "apexportal-webservice-service" Kubernetes service, mainly the ports section?

@Avneetdabas
Copy link
Author

We are making 2 services, Cluster IP is for Netscaler VPX and the Node port is for us to test. The nodeport one is working fine.

apiVersion: v1
kind: Secret
metadata:
name: XXXXXXXX-webservice
type: Opaque
data:
RABBIT_USERNAME: XXXXXXXX
RABBIT_PASSWORD: XXXXXXXX

apiVersion: apps/v1
kind: Deployment
metadata:
name: XXXXXXXX-webservice
labels:
app: XXXXXXXX-webservice
spec:
selector:
matchLabels:
app: XXXXXXXX-webservice
replicas: 1
template:
metadata:
labels:
app: XXXXXXXX-webservice
spec:
nodeSelector:
kubernetes.io/os : linux
containers:
- name: XXXXXXXX-webservice
image: XXXXXXXXXXX.XXX.XXXXXXXXXXX.com/XXXXXX_dev/XXXXXXXX.webservice:dev
imagePullPolicy: Always
ports:
- containerPort: 54341

imagePullSecrets:

- name: regcred


apiVersion: v1
kind: Service
metadata:
name: XXXXXXXX-webservice-nodeport
labels:
app: XXXXXXXX-webservice
spec:
type: NodePort
selector:
app: XXXXXXXX-webservice
ports:
- protocol: TCP
name: http
port: 32003
targetPort: 54341


apiVersion: v1
kind: Service
metadata:
name: XXXXXXXX-webservice-service
labels:
app: XXXXXXXX-webservice
spec:
type: ClusterIP
selector:
app: XXXXXXXX-webservice
ports:
- protocol: TCP
name: http
port: 80
targetPort: 54341

@Avneetdabas
Copy link
Author

Ok, i was able to make it work by deleting the cic pod. But looks like there is a bug in the latest version.

@jeanz6
Copy link

jeanz6 commented Mar 17, 2025

Weve got the same problem with the nsic ingress-controller and a netscaler ADC appliance machine. We deploy a service of type loadBalancer and also put the loadbalancerIP in place. Then servicegroups, LBVSs and CS Vservers are created, but after a while the servicegroup for our endpoint is lost. I enabled the DEBUG log so whats going here are the log files and the service file for our deployment:
nsic-ingress-controller pod logs:

2025-03-13 13:05:26,080  - DEBUG - [kubernetes.py:update_ingress_for_node_modify_event:5685] (MainThread) Node hdlsvkwo02.snp.ag modified from kubernetes cluster
2025-03-13 13:06:02,267  - DEBUG - [kubernetes.py:event_handler:1019] (MainThread) Handling Service gateway.ecampus-dev event_type: MODIFIED
2025-03-13 13:06:02,267  - DEBUG - [kubernetes.py:_fill_service:5911] (MainThread) gateway service object with UID cd36981a-548b-4b68-b45a-b398d2f3ce2d has current resourceVersion number 85576144
2025-03-13 13:06:02,267  - DEBUG - [kubernetes.py:_fill_service:5923] (MainThread) Reading IP from spec.loadBalancerIP for service: gateway
2025-03-13 13:06:02,267  - DEBUG - [kubernetes.py:update_loadbalancer_ip:3489] (MainThread) Updating services gateway.ecampus-dev status with ['172.16.73.107']
2025-03-13 13:06:02,276  - DEBUG - [clienthelper.py:patch:120] (MainThread) <Response [200]>
2025-03-13 13:06:02,276  - DEBUG - [kubernetes.py:parse_service_annotations:1826] (MainThread) service annotations service.citrix.com/backend-tcpprofile values {"ws":"ENABLED", "sack" : "enabled"}
2025-03-13 13:06:02,276  - DEBUG - [kubernetes.py:parse_service_annotations:1826] (MainThread) service annotations service.citrix.com/frontend-tcpprofile values {"ws":"ENABLED", "sack" : "enabled"}
2025-03-13 13:06:02,276  - DEBUG - [kubernetes.py:parse_service_annotations:1826] (MainThread) service annotations service.citrix.com/preconfigured-certkey values _.snpgroup.com-wildcard-2025
2025-03-13 13:06:02,276  - DEBUG - [kubernetes.py:parse_service_annotations:1849] (MainThread) annotation value not of json type, no need to get contents
2025-03-13 13:06:02,276  - DEBUG - [kubernetes.py:parse_service_annotations:1826] (MainThread) service annotations service.citrix.com/service_type values SSL
2025-03-13 13:06:02,276  - DEBUG - [kubernetes.py:parse_service_annotations:1849] (MainThread) annotation value not of json type, no need to get contents
2025-03-13 13:06:02,276  - DEBUG - [kubernetes.py:parse_service_annotations:1826] (MainThread) service annotations service.citrix.com/backend-tcpprofile values {"ws":"ENABLED", "sack" : "enabled"}
2025-03-13 13:06:02,276  - DEBUG - [kubernetes.py:parse_service_annotations:1826] (MainThread) service annotations service.citrix.com/frontend-tcpprofile values {"ws":"ENABLED", "sack" : "enabled"}
2025-03-13 13:06:02,277  - DEBUG - [kubernetes.py:parse_service_annotations:1826] (MainThread) service annotations service.citrix.com/preconfigured-certkey values _.snpgroup.com-wildcard-2025
2025-03-13 13:06:02,277  - DEBUG - [kubernetes.py:parse_service_annotations:1849] (MainThread) annotation value not of json type, no need to get contents
2025-03-13 13:06:02,277  - DEBUG - [kubernetes.py:parse_service_annotations:1826] (MainThread) service annotations service.citrix.com/service_type values SSL
2025-03-13 13:06:02,277  - DEBUG - [kubernetes.py:parse_service_annotations:1849] (MainThread) annotation value not of json type, no need to get contents
2025-03-13 13:06:02,277  - DEBUG - [referencemanager.py:lookup_crd_instance:371] (MainThread) Lookup CRD ecampus-dev.Service.gateway
2025-03-13 13:06:02,277  - DEBUG - [referencetree.py:lookup_tree_node:37] (MainThread) Searching for ecampus-dev.Service.gateway
2025-03-13 13:06:02,277  - DEBUG - [kubernetes.py:is_typelb_configured:4519] (MainThread) Processing is_typelb_configured for name:gateway namespace:ecampus-dev
2025-03-13 13:06:02,277  - DEBUG - [kubernetes.py:is_typelb_configured:4531] (MainThread) Found existing cs app k8s-gateway_8080_ecampus-dev_svc with resourceVersion 81810221
2025-03-13 13:06:02,277  - DEBUG - [referencemanager.py:lookup_crd_instance:371] (MainThread) Lookup CRD ecampus-dev.Service.gateway
2025-03-13 13:06:02,277  - DEBUG - [referencetree.py:lookup_tree_node:37] (MainThread) Searching for ecampus-dev.Service.gateway
2025-03-13 13:06:02,277  - DEBUG - [Service.py:compare_spec:146] (MainThread) Service gateway Ports spec and loadBalancer IP is not modified
2025-03-13 13:06:02,277  - DEBUG - [Service.py:compare_citrix_annotations:159] (MainThread) Service gateway annotations not modified
2025-03-13 13:06:02,280  - DEBUG - [kubernetes.py:get_node_ip_for_pod_ip_by_node:2799] (MainThread) Getting node ip of pod, node given: hdlsvkwo03.snp.ag
2025-03-13 13:06:02,280  - DEBUG - [kubernetes.py:_update_servicegroup_with_endpoints:5185] (MainThread) Updating endpoints for k8s-gateway_8080_ecampus-dev
2025-03-13 13:06:02,280  - DEBUG - [kubernetes.py:_update_servicegroup_with_endpoints:5186] (MainThread) to-add: []
2025-03-13 13:06:02,280  - DEBUG - [kubernetes.py:_update_servicegroup_with_endpoints:5187] (MainThread) to-remove: []
2025-03-13 13:06:02,280  - DEBUG - [kubernetes.py:adjust_service_group_for_single_app:5077] (MainThread) Adjusting application k8s-gateway_8080_ecampus-dev_svc because of lb service k8s-gateway_8080_ecampus-dev
2025-03-13 13:06:02,280  - DEBUG - [nitrointerface.py:adjust_service_group:6557] (MainThread) Processing k8s-gateway_8080_ecampus-dev LB APP for k8s-gateway_8080_ecampus-dev_svc CS App
2025-03-13 13:06:02,281  - DEBUG - [globalfunc.py:get_entity_name:124] (MainThread) Entity name generated with k8s-gateway_8080 + _SGP_ + k8s-gateway_8080_ecampus-dev is k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3 with length53
2025-03-13 13:06:02,281  - DEBUG - [nitrointerface.py:_create_lbvserver_name:1828] (MainThread) Generated Entity name k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3 with str k8s-gateway_8080_ecampus-dev_svc hash k8s-gateway_8080_ecampus-dev_svc postfix SGP
2025-03-13 13:06:02,281  - DEBUG - [nitrointerface.py:adjust_service_group:6559] (MainThread) Adjust svcgrp members for k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3
2025-03-13 13:06:02,281  - DEBUG - [nitrointerface.py:_configure_services:2915] (MainThread) configuring service using traditional API (servicegroup:k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3)
2025-03-13 13:06:02,281  - DEBUG - [nitrointerface.py:_configure_services_nondesired:3052] (MainThread) Started  services configuration to servicegroup: k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3
2025-03-13 13:06:02,386  - DEBUG - [nitrointerface.py:_configure_services_nondesired:3063] (MainThread) ServiceGroup k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3: getting existing service member binding failed with error No such resource [serviceGroupName, k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3]
2025-03-13 13:06:02,386  - DEBUG - [nitrointerface.py:_configure_services_nondesired:3085] (MainThread) Binding :172.16.254.240:32706 from servicegroup k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3
2025-03-13 13:06:02,421  - ERROR - [nitrointerface.py:_configure_services_nondesired:3101] (MainThread) Nitro Exception while binding group member to servicegroup k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3 errorcode=258 message=No such resource [serviceGroupName, k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3]
2025-03-13 13:06:02,421  - DEBUG - [nitrointerface.py:_configure_services_nondesired:3102] (MainThread) Finished services configuration to servicegroup: k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3
2025-03-13 13:06:02,421  - DEBUG - [nitrointerface.py:adjust_service_group:6562] (MainThread) Adjusting Service group members for k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3 successful
2025-03-13 13:06:02,421  - DEBUG - [kubernetes.py:adjust_service_group_for_single_app:5077] (MainThread) Adjusting application k8s-gateway_8080_ecampus-dev_svc because of lb service k8s-gateway_8080_ecampus-dev
2025-03-13 13:06:02,421  - DEBUG - [nitrointerface.py:adjust_service_group:6557] (MainThread) Processing k8s-gateway_8080_ecampus-dev LB APP for k8s-gateway_8080_ecampus-dev_svc CS App
2025-03-13 13:06:02,421  - DEBUG - [globalfunc.py:get_entity_name:124] (MainThread) Entity name generated with k8s-gateway_8080 + _SGP_ + k8s-gateway_8080_ecampus-dev is k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3 with length53
2025-03-13 13:06:02,421  - DEBUG - [nitrointerface.py:_create_lbvserver_name:1828] (MainThread) Generated Entity name k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3 with str k8s-gateway_8080_ecampus-dev_svc hash k8s-gateway_8080_ecampus-dev_svc postfix SGP
2025-03-13 13:06:02,421  - DEBUG - [nitrointerface.py:adjust_service_group:6559] (MainThread) Adjust svcgrp members for k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3
2025-03-13 13:06:02,421  - DEBUG - [nitrointerface.py:_configure_services:2915] (MainThread) configuring service using traditional API (servicegroup:k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3)
2025-03-13 13:06:02,421  - DEBUG - [nitrointerface.py:_configure_services_nondesired:3052] (MainThread) Started  services configuration to servicegroup: k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3
2025-03-13 13:06:02,455  - DEBUG - [nitrointerface.py:_configure_services_nondesired:3063] (MainThread) ServiceGroup k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3: getting existing service member binding failed with error No such resource [serviceGroupName, k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3]
2025-03-13 13:06:02,455  - DEBUG - [nitrointerface.py:_configure_services_nondesired:3085] (MainThread) Binding :172.16.254.240:32706 from servicegroup k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3
2025-03-13 13:06:02,491  - ERROR - [nitrointerface.py:_configure_services_nondesired:3101] (MainThread) Nitro Exception while binding group member to servicegroup k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3 errorcode=258 message=No such resource [serviceGroupName, k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3]
2025-03-13 13:06:02,491  - DEBUG - [nitrointerface.py:_configure_services_nondesired:3102] (MainThread) Finished services configuration to servicegroup: k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3
2025-03-13 13:06:02,491  - DEBUG - [nitrointerface.py:adjust_service_group:6562] (MainThread) Adjusting Service group members for k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3 successful
2025-03-13 13:06:02,491  - DEBUG - [kubernetes.py:get_node_ip_for_pod_ip_by_node:2799] (MainThread) Getting node ip of pod, node given: hdlsvkwo03.snp.ag
2025-03-13 13:06:02,491  - DEBUG - [kubernetes.py:_update_servicegroup_with_endpoints:5185] (MainThread) Updating endpoints for k8s-gateway_https_ecampus-dev
2025-03-13 13:06:02,491  - DEBUG - [kubernetes.py:_update_servicegroup_with_endpoints:5186] (MainThread) to-add: []
2025-03-13 13:06:02,491  - DEBUG - [kubernetes.py:_update_servicegroup_with_endpoints:5187] (MainThread) to-remove: []
2025-03-13 13:06:02,491  - DEBUG - [kubernetes.py:adjust_service_group_for_single_app:5077] (MainThread) Adjusting application k8s-gateway_443_ecampus-dev_svc because of lb service k8s-gateway_https_ecampus-dev
2025-03-13 13:06:02,491  - DEBUG - [nitrointerface.py:adjust_service_group:6557] (MainThread) Processing k8s-gateway_https_ecampus-dev LB APP for k8s-gateway_443_ecampus-dev_svc CS App
2025-03-13 13:06:02,491  - DEBUG - [globalfunc.py:get_entity_name:124] (MainThread) Entity name generated with k8s-gateway_443 + _SGP_ + k8s-gateway_443_ecampus-dev is k8s-gateway_443_sgp_eas37bv6bfdvttq4wgod7vtdmemz4z4w with length52
2025-03-13 13:06:02,491  - DEBUG - [nitrointerface.py:_create_lbvserver_name:1828] (MainThread) Generated Entity name k8s-gateway_443_sgp_eas37bv6bfdvttq4wgod7vtdmemz4z4w with str k8s-gateway_443_ecampus-dev_svc hash k8s-gateway_443_ecampus-dev_svc postfix SGP
2025-03-13 13:06:02,491  - DEBUG - [nitrointerface.py:adjust_service_group:6559] (MainThread) Adjust svcgrp members for k8s-gateway_443_sgp_eas37bv6bfdvttq4wgod7vtdmemz4z4w
2025-03-13 13:06:02,491  - DEBUG - [nitrointerface.py:_configure_services:2915] (MainThread) configuring service using traditional API (servicegroup:k8s-gateway_443_sgp_eas37bv6bfdvttq4wgod7vtdmemz4z4w)
2025-03-13 13:06:02,491  - DEBUG - [nitrointerface.py:_configure_services_nondesired:3052] (MainThread) Started  services configuration to servicegroup: k8s-gateway_443_sgp_eas37bv6bfdvttq4wgod7vtdmemz4z4w
2025-03-13 13:06:02,526  - DEBUG - [nitrointerface.py:_configure_services_nondesired:3063] (MainThread) ServiceGroup k8s-gateway_443_sgp_eas37bv6bfdvttq4wgod7vtdmemz4z4w: getting existing service member binding failed with error No such resource [serviceGroupName, k8s-gateway_443_sgp_eas37bv6bfdvttq4wgod7vtdmemz4z4w]
2025-03-13 13:06:02,526  - DEBUG - [nitrointerface.py:_configure_services_nondesired:3085] (MainThread) Binding :172.16.254.240:31490 from servicegroup k8s-gateway_443_sgp_eas37bv6bfdvttq4wgod7vtdmemz4z4w
2025-03-13 13:06:02,560  - ERROR - [nitrointerface.py:_configure_services_nondesired:3101] (MainThread) Nitro Exception while binding group member to servicegroup k8s-gateway_443_sgp_eas37bv6bfdvttq4wgod7vtdmemz4z4w errorcode=258 message=No such resource [serviceGroupName, k8s-gateway_443_sgp_eas37bv6bfdvttq4wgod7vtdmemz4z4w]
2025-03-13 13:06:02,561  - DEBUG - [nitrointerface.py:_configure_services_nondesired:3102] (MainThread) Finished services configuration to servicegroup: k8s-gateway_443_sgp_eas37bv6bfdvttq4wgod7vtdmemz4z4w
2025-03-13 13:06:02,561  - DEBUG - [nitrointerface.py:adjust_service_group:6562] (MainThread) Adjusting Service group members for k8s-gateway_443_sgp_eas37bv6bfdvttq4wgod7vtdmemz4z4w successful
2025-03-13 13:06:02,561  - DEBUG - [kubernetes.py:adjust_service_group_for_single_app:5077] (MainThread) Adjusting application k8s-gateway_443_ecampus-dev_svc because of lb service k8s-gateway_https_ecampus-dev
2025-03-13 13:06:02,561  - DEBUG - [nitrointerface.py:adjust_service_group:6557] (MainThread) Processing k8s-gateway_https_ecampus-dev LB APP for k8s-gateway_443_ecampus-dev_svc CS App
2025-03-13 13:06:02,561  - DEBUG - [globalfunc.py:get_entity_name:124] (MainThread) Entity name generated with k8s-gateway_443 + _SGP_ + k8s-gateway_443_ecampus-dev is k8s-gateway_443_sgp_eas37bv6bfdvttq4wgod7vtdmemz4z4w with length52
2025-03-13 13:06:02,561  - DEBUG - [nitrointerface.py:_create_lbvserver_name:1828] (MainThread) Generated Entity name k8s-gateway_443_sgp_eas37bv6bfdvttq4wgod7vtdmemz4z4w with str k8s-gateway_443_ecampus-dev_svc hash k8s-gateway_443_ecampus-dev_svc postfix SGP
2025-03-13 13:06:02,561  - DEBUG - [nitrointerface.py:adjust_service_group:6559] (MainThread) Adjust svcgrp members for k8s-gateway_443_sgp_eas37bv6bfdvttq4wgod7vtdmemz4z4w
2025-03-13 13:06:02,561  - DEBUG - [nitrointerface.py:_configure_services:2915] (MainThread) configuring service using traditional API (servicegroup:k8s-gateway_443_sgp_eas37bv6bfdvttq4wgod7vtdmemz4z4w)
2025-03-13 13:06:02,561  - DEBUG - [nitrointerface.py:_configure_services_nondesired:3052] (MainThread) Started  services configuration to servicegroup: k8s-gateway_443_sgp_eas37bv6bfdvttq4wgod7vtdmemz4z4w
2025-03-13 13:06:02,595  - DEBUG - [nitrointerface.py:_configure_services_nondesired:3063] (MainThread) ServiceGroup k8s-gateway_443_sgp_eas37bv6bfdvttq4wgod7vtdmemz4z4w: getting existing service member binding failed with error No such resource [serviceGroupName, k8s-gateway_443_sgp_eas37bv6bfdvttq4wgod7vtdmemz4z4w]
2025-03-13 13:06:02,595  - DEBUG - [nitrointerface.py:_configure_services_nondesired:3085] (MainThread) Binding :172.16.254.240:31490 from servicegroup k8s-gateway_443_sgp_eas37bv6bfdvttq4wgod7vtdmemz4z4w
2025-03-13 13:06:02,629  - ERROR - [nitrointerface.py:_configure_services_nondesired:3101] (MainThread) Nitro Exception while binding group member to servicegroup k8s-gateway_443_sgp_eas37bv6bfdvttq4wgod7vtdmemz4z4w errorcode=258 message=No such resource [serviceGroupName, k8s-gateway_443_sgp_eas37bv6bfdvttq4wgod7vtdmemz4z4w]
2025-03-13 13:06:02,629  - DEBUG - [nitrointerface.py:_configure_services_nondesired:3102] (MainThread) Finished services configuration to servicegroup: k8s-gateway_443_sgp_eas37bv6bfdvttq4wgod7vtdmemz4z4w
2025-03-13 13:06:02,629  - DEBUG - [nitrointerface.py:adjust_service_group:6562] (MainThread) Adjusting Service group members for k8s-gateway_443_sgp_eas37bv6bfdvttq4wgod7vtdmemz4z4w successful
2025-03-13 13:06:02,630  - INFO - [kubernetes.py:update_cpx_for_apps:5537] (MainThread) Handling Type LoadBalancer Service Modification gateway.ecampus-dev
2025-03-13 13:06:02,630  - DEBUG - [pendingLoadbalancerServices.py:getName:143] (MainThread) returning concatenation of namespace: ecampus-dev and service:  gateway 
2025-03-13 13:06:02,630  - DEBUG - [pendingLoadbalancerServices.py:serviceExists:124] (MainThread) cheking if  Service: ecampus-dev_gateway key is present in pending_services for type load balancer
2025-03-13 13:06:02,630  - INFO - [kubernetes.py:kubernetes_service_to_nsapps:3187] (MainThread) Handling Service creation/Modification gateway.ecampus-dev
2025-03-13 13:06:02,630  - DEBUG - [kubernetes.py:kubernetes_service_to_nsapps:3225] (MainThread) Service App k8s-gateway_8080_ecampus-dev_svc with UID cd36981a-548b-4b68-b45a-b398d2f3ce2d is at kubernetes resourceVersion:85576144
2025-03-13 13:06:02,630  - DEBUG - [kubernetes.py:set_annotations_to_nsapp:1854] (MainThread) annotations name values 8080
2025-03-13 13:06:02,630  - DEBUG - [kubernetes.py:set_annotations_to_nsapp:1854] (MainThread) annotations protocol values tcp
2025-03-13 13:06:02,630  - DEBUG - [kubernetes.py:set_annotations_to_nsapp:1854] (MainThread) annotations port values 8080
2025-03-13 13:06:02,630  - DEBUG - [kubernetes.py:set_annotations_to_nsapp:1854] (MainThread) annotations targetPort values 8080
2025-03-13 13:06:02,630  - DEBUG - [kubernetes.py:set_annotations_to_nsapp:1854] (MainThread) annotations nodePort values 32706
2025-03-13 13:06:02,630  - DEBUG - [kubernetes.py:set_annotations_to_nsapp:1854] (MainThread) annotations vip values 172.16.73.107
2025-03-13 13:06:02,630  - DEBUG - [kubernetes.py:set_annotations_to_nsapp:1854] (MainThread) annotations com/backend-tcpprofile values {'ws': 'ENABLED', 'sack': 'enabled'}
2025-03-13 13:06:02,630  - DEBUG - [kubernetes.py:set_annotations_to_nsapp:1858] (MainThread) annotations: com/backend-tcpprofile
2025-03-13 13:06:02,630  - DEBUG - [kubernetes.py:set_annotations_to_nsapp:1854] (MainThread) annotations com/frontend-tcpprofile values {'ws': 'ENABLED', 'sack': 'enabled'}
2025-03-13 13:06:02,630  - DEBUG - [kubernetes.py:set_annotations_to_nsapp:1858] (MainThread) annotations: com/frontend-tcpprofile
2025-03-13 13:06:02,630  - DEBUG - [kubernetes.py:set_annotations_to_nsapp:1854] (MainThread) annotations com/preconfigured-certkey values _.snpgroup.com-wildcard-2025
2025-03-13 13:06:02,630  - DEBUG - [kubernetes.py:set_annotations_to_nsapp:1858] (MainThread) annotations: com/preconfigured-certkey
2025-03-13 13:06:02,630  - DEBUG - [kubernetes.py:set_annotations_to_nsapp:1854] (MainThread) annotations com/service_type values SSL
2025-03-13 13:06:02,630  - DEBUG - [kubernetes.py:set_annotations_to_nsapp:1858] (MainThread) annotations: com/service_type
2025-03-13 13:06:02,630  - DEBUG - [kubernetes.py:set_annotations_to_nsapp:1854] (MainThread) annotations sslcert values {}
2025-03-13 13:06:02,630  - DEBUG - [kubernetes.py:set_annotations_to_nsapp:1854] (MainThread) annotations port_allowed values edge
2025-03-13 13:06:02,630  - DEBUG - [kubernetes.py:set_annotations_to_nsapp:1854] (MainThread) annotations range-name values None
2025-03-13 13:06:02,630  - DEBUG - [kubernetes.py:kubernetes_service_to_nsapps:3242] (MainThread) Service App k8s-gateway_8080_ecampus-dev with cs appname k8s-gateway_8080_ecampus-dev_svc with UID cd36981a-548b-4b68-b45a-b398d2f3ce2d is at kubernetes resourceVersion:85576144
2025-03-13 13:06:02,630  - DEBUG - [kubernetes.py:kubernetes_service_to_nsapps:3337] (MainThread) Processing preconfigured_certkey _.snpgroup.com-wildcard-2025 provided in annotation
2025-03-13 13:06:02,630  - DEBUG - [kubernetes.py:kubernetes_service_to_nsapps:3340] (MainThread) Snienable is False, certtype is default
2025-03-13 13:06:02,630  - DEBUG - [kubernetes.py:handle_preconfigured_secret:4317] (MainThread) handle_preconfigured_secret with certkey_name _.snpgroup.com-wildcard-2025 and cert type:default
2025-03-13 13:06:02,630  - INFO - [kubernetes.py:kubernetes_service_to_nsapps:3437] (MainThread) Configuring Type LoadBalancer Service gateway:ecampus-dev port params:{'name': '8080', 'protocol': 'tcp', 'port': 8080, 'targetPort': 8080, 'nodePort': 32706, 'vip': '172.16.73.107', 'com/backend-tcpprofile': {'ws': 'ENABLED', 'sack': 'enabled'}, 'com/frontend-tcpprofile': {'ws': 'ENABLED', 'sack': 'enabled'}, 'com/preconfigured-certkey': '_.snpgroup.com-wildcard-2025', 'com/service_type': 'SSL', 'sslcert': {}, 'port_allowed': 'edge', 'range-name': None}
2025-03-13 13:06:02,630  - DEBUG - [kubernetes.py:update_loadbalancer_ip:3489] (MainThread) Updating services gateway.ecampus-dev status with ['172.16.73.107']
2025-03-13 13:06:02,634  - DEBUG - [clienthelper.py:patch:120] (MainThread) <Response [200]>

the big problem is mainly this lines here:

2025-03-13 13:06:02,281  - DEBUG - [nitrointerface.py:_configure_services_nondesired:3052] (MainThread) Started  services configuration to servicegroup: k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3
2025-03-13 13:06:02,386  - DEBUG - [nitrointerface.py:_configure_services_nondesired:3063] (MainThread) ServiceGroup k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3: getting existing service member binding failed with error No such resource [serviceGroupName, k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3]
2025-03-13 13:06:02,386  - DEBUG - [nitrointerface.py:_configure_services_nondesired:3085] (MainThread) Binding :172.16.254.240:32706 from servicegroup k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3
2025-03-13 13:06:02,421  - ERROR - [nitrointerface.py:_configure_services_nondesired:3101] (MainThread) Nitro Exception while binding group member to servicegroup k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3 errorcode=258 message=No such resource [serviceGroupName, k8s-gateway_8080_sgp_nu62ydeaafdrtsiectylj4osxkzisou3]

service gateway manifest yaml looks like this:

apiVersion: v1
kind: Service
metadata:
  annotations:
    kompose.cmd: kompose convert
    kompose.version: 1.19.0 (f63a961c)
    service.citrix.com/service_type: 'SSL'
    service.citrix.com/preconfigured-certkey: '_.cert-wildcard-2025'
    service.citrix.com/backend-tcpprofile: '{"ws":"ENABLED", "sack" : "enabled"}'
    service.citrix.com/frontend-tcpprofile: '{"ws":"ENABLED", "sack" : "enabled"}'
  labels:
    imcservicename: gateway
  name: gateway
spec:
  type: LoadBalancer
  loadBalancerIP: 172.16.73.107
  ports:
  - name: "8080"
    port: 8080
    targetPort: 8080
  - name: "https"
    port: 443
    targetPort: 8080
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 3600
  selector:
    imcservicename: gateway
status:
  loadBalancer: {}

It happens now for about 3months an I could not found the underlaying problem. After I reapply the service yaml it works again.
Not sure if its really a bug of the nsic-ingress-controller.. I upgrade it to the newest version.
Hope someone can help, feel free to ask for more information
cheers J.

@subashd
Copy link
Collaborator

subashd commented Mar 17, 2025

hi @jeanz6
Could you please fill the questionaires?

The annotations service-type takes index as well, like service.citrix.com/service-type-0: "SSL".
There are two ports 8080 and 443, do you require SSL for both the ports or one is HTTP and other is HTTPS?
While we analyze the logs, could you please update the annotation? Also, please provide the full log for troubleshooting.

Example:

apiVersion: v1
kind: Service
metadata:
  name: hotdrink-service
  annotations:
    service.citrix.com/service-type-0: 'HTTP'
    service.citrix.com/service-type-1: 'SSL'
    service.citrix.com/secret: '{"https":"hotdrink.secret"}'
spec:
  type: LoadBalancer
  loadBalancerIP: 1.1.1.1
  selector:
    app: frontend-hotdrinks 
  ports:
  - port: 80
    targetPort: 8080
    name: http
  - port: 443
    targetPort: 8443
    name: https

Regards,
Subash Dangol

@jeanz6
Copy link

jeanz6 commented Mar 17, 2025

Hi, first thanks for your fast answer, I have filled out the questionaire. We only need TLS/SSL on port 443, ill change the annotation tomorrorw, and give you an answer also share the full logs with you! Cheers J.

@jeanz6
Copy link

jeanz6 commented Mar 18, 2025

Hi, I attached the full nsic log, the timestamps are: 2025-03-10 08:35 and 2025-03-13 13:05. at these times it happened. After we recognized that it was gone we had to reapply our gateway-service.yaml manifest. Now I implement the annotations in our dev namespace. Thank you very much for your help !!! Cheers J.

nsic.log

@jeanz6
Copy link

jeanz6 commented Mar 18, 2025

So I added the annotations, here is the yaml file:

apiVersion: v1
kind: Service
metadata:
  annotations:
    kompose.cmd: kompose convert
    kompose.version: 1.19.0 (f63a961c)
    #service.citrix.com/service_type: 'SSL'
    service.citrix.com/service-type-0: 'HTTP'
    service.citrix.com/service-type-1: 'SSL'
    service.citrix.com/preconfigured-certkey: '_.cert-wildcard-2025'
    service.citrix.com/backend-tcpprofile: '{"ws":"ENABLED", "sack" : "enabled"}'
    service.citrix.com/frontend-tcpprofile: '{"ws":"ENABLED", "sack" : "enabled"}'
  creationTimestamp: null
  labels:
    imcservicename: gateway
  name: gateway
spec:
  type: LoadBalancer
  loadBalancerIP: 172.16.73.107
  ports:
  - name: "8080"
    port: 8080
    targetPort: 8080
  - name: "https"
    port: 443
    targetPort: 8080
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 3600
  selector:
    imcservicename: gateway
status:
  loadBalancer: {}

After applying the yaml everything looks fine, but thats everytime the same, the outage happens random. Do you need more information ? Cheers J.

@subashd
Copy link
Collaborator

subashd commented Mar 18, 2025

hi @jeanz6
If you are facing the issue again and again, could you please enable the debug log level in the configmap and share the full log?
Could you please share your company details along with the logs over an email to netscaler-appmodernization@cloud.com?

Regards,
Subash Dangol

@jeanz6
Copy link

jeanz6 commented Mar 18, 2025

Hi, DEBUG Level is on, as you can see in the nsic.log what I shared here:

Hi, I attached the full nsic log, the timestamps are: 2025-03-10 08:35 and 2025-03-13 13:05. at these times it happened. After we recognized that it was gone we had to reapply our gateway-service.yaml manifest. Now I implement the annotations in our dev namespace. Thank you very much for your help !!! Cheers J.

nsic.log

Ill also write an email to netscaler-appmodernization@cloud.com,
In the meantime you can analyze the log :) Cheers J.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants