CubeStore Error: No such file or directory

[g-diwakar]

Describe the bug
ERROR [cubestore::remotefs::cleanup] pid:1 error while trying to list local files: No such file or directory (os error 2)
To Reproduce
Running the cube store worker, it is throwing the above error.
I have deployed the cube as instructed here.

https://cube.dev/docs/product/deployment/core

Expected behavior
The error should not happen

Version:

Cube Store : v0.36.0
Cube Api : v0.35.57

Additional context
I have duckdb as source database.

For some reason cube router is refusing the connection request.

ERROR [cubestore::cluster] pid:1 Failed to get warmup partitions: Internal: Can't connect to cubestore_router:9999: Connection refused (os error 111)

[igorlukanin]

Hi @g-diwakar 👋

Cube Store : v0.36.0
Cube Api : v0.35.57

I would strongly advise to only run the same versions for Cube Store and Cube.

For some reason cube router is refusing the connection request.

I guess this one should be resolved first. I bet there's either some networking misconfiguration. How/where do you deploy Cube? Are you 100% sure its nodes can reach each other?

[pebabion]

I'm experiencing this issue as well - after matching the version of cubestore and cube API the errors stopped happening.

Scratch that. The error is still happening for me.

[igorlukanin]

@pebabion Which one of the two mentioned in the original message? Also, would you mind providing additional details as well?

[pebabion]

@igorlukanin the error that is happening for me is in the cubestore-worker pod.

2024-12-13 02:21:16,666 ERROR [cubestore::remotefs::cleanup] <pid:1> error while trying to list local files: No such file or directory (os error 2)

We followed the documentation and got cubestore deployed as below.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    service: cubestore-router
  name: cubestore-router
  namespace: default
  annotations:
    cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
spec:
  serviceName: cubestore-router
  selector:
    matchLabels:
      service: cubestore-router
  template:
    metadata:
      labels:
        service: cubestore-router
    spec: # Can't scale cubestore-router without cube cloud, so we'll just use a single replica
      containers:
        - name: cubestore-router
          image: cubejs/cubestore:latest
          env:
            - name: CUBESTORE_META_PORT
              value: "9999"
            - name: CUBESTORE_DATA_DIR
              value: /cube/.cubestore/data
            - name: CUBESTORE_SERVER_NAME
              value: cubestore-router.default.svc.cluster.local:9999
            - name: CUBESTORE_WORKERS
              value: cubestore-workers-0.cubestore-workers:10000,cubestore-workers-1.cubestore-workers:10000,cubestore-workers-2.cubestore-workers:10000,cubestore-workers-3.cubestore-workers:10000,cubestore-workers-4.cubestore-workers:10000
            - name: CUBESTORE_S3_BUCKET
              value: xxxxxxxx
            - name: CUBESTORE_S3_SUB_PATH
              value: "cubestore"
            - name: CUBESTORE_S3_REGION
              value: us-east-2
            - name: CUBESTORE_QUERY_TIMEOUT
              value: "600"
            - name: CUBESTORE_AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  name: aws
                  key: cubejs-key
            - name: CUBESTORE_AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: aws
                  key: cubejs-secret
          ports:
            - containerPort: 9999
          resources:
            requests:
              cpu: 800m
              memory: 2Gi
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    service: cubestore-workers
  name: cubestore-workers
  namespace: default
  annotations:
    cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
spec:
  replicas: 5 # If you edit this, make sure to edit the CUBESTORE_WORKERS as well
  serviceName: cubestore-workers
  selector:
    matchLabels:
      service: cubestore-workers
  template:
    metadata:
      labels:
        service: cubestore-workers
    spec:
      containers:
        - name: cubestore-workers
          image: cubejs/cubestore:v0.34.60
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: CUBESTORE_META_ADDR
              value: cubestore-router.default.svc.cluster.local:9999
            - name: CUBESTORE_DATA_DIR
              value: /cube/.cubestore/data
              # The $(POD_NAME) is a way of referencing dependent env vars
              # It will resolve to "cubestore-workers-n" where n is the replica number
            - name: CUBESTORE_SERVER_NAME
              value: "$(POD_NAME).cubestore-workers.default.svc.cluster.local:10000"
              # With 3 replicas the POD_NAMEs will be cubestore-workers-0,cubestore-workers-1, and cubestore-workers-2
              # These will then be needed in the CUBESTORE_WORKERS env var as an array
              # We use "cubestore-workers-0.cubestore-workers:10000" as it points to the headless service
              # See also cubestore-router-statefulset.yaml
            - name: CUBESTORE_WORKERS
              value: cubestore-workers-0.cubestore-workers.default.svc.cluster.local:10000,cubestore-workers-1.cubestore-workers.default.svc.cluster.local:10000,cubestore-workers-2.cubestore-workers.default.svc.cluster.local:10000,cubestore-workers-3.cubestore-workers.default.svc.cluster.local:10000,cubestore-workers-4.cubestore-workers.default.svc.cluster.local:10000
            - name: CUBESTORE_WORKER_PORT
              value: "10000"
            - name: CUBESTORE_S3_BUCKET
              value: xxxxxxxxxxx
            - name: CUBESTORE_S3_SUB_PATH
              value: "cubestore"
            - name: CUBESTORE_S3_REGION
              value: us-east-2
            - name: CUBESTORE_QUERY_TIMEOUT
              value: "600"
            - name: CUBESTORE_LOG_LEVEL
              value: "warn"
            - name: CUBESTORE_AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  name: aws
                  key: cubejs-key
            - name: CUBESTORE_AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: aws
                  key: cubejs-secret
          ports:
            - containerPort: 10000
          resources:
            requests:
              cpu: 250m
              memory: 500Mi
      initContainers:
        - name: init-router
          image: busybox
          command:
            [
              "sh",
              "-c",
              'until nc -vz cubestore-router:9999; do echo "Waiting for router"; sleep 2; done;',
            ]

Also, with the above deployment, we're also facing timeout when pre-aggregation is created. Like so:

[pebabion]

@igorlukanin I'm troubleshooting the above and wondering if the cubestore-workers have to be in the same region with the S3 export bucket?

I can see the .csv.gz files exported in the export bucket but the pre-aggregation build is still not successful and not sure what the reason might be (there's no log from cube worker or cube router).