Relay container failed to start

[Gnep]

Hi,

I tried to use the compose file to launch Cog. The Postgre and Cog are running fine, but Relay failed to start. The log shows:

WARN[2017-08-21T10:12:25Z] Error loading config file '/etc/relay.conf': Config file not found or is unreadable. 
INFO[2017-08-21T10:12:25Z] Using config file '/usr/local/etc/relay.conf'. 
log level: debugERRO[2017-08-21T10:12:25Z] Error configuring Relay:
  Token: non zero value required 

And /usr/local/etc/relay.conf:

# Relay config version
# Mandatory. Must equal 1.
version: 1

# Unique Relay id
# Environment variable: $RELAY_ID
# Default: none
id: aad48b34-5493-4107-9dc0-32dc710938ec

# Number of allowed concurrent command invocations
# Environment variable: $RELAY_MAX_CONCURRENT
# Default: 16
max_concurrent: 8

# Path to dynamic bundle config files
# Missing or empty value disables.
# Path will be created if it doesn't exist.
# Environment variable: $RELAY_DYNAMIC_CONFIG_ROOT
# Default: none
# Required: No
# dynamic_config_root: /tmp/bundle_configs

# Log level
# Environment variable: $RELAY_LOG_LEVEL
# Default: info
log_level: debug

# Log JSON? If false, plain text will be used.
# Environment variable: $RELAY_LOG_JSON
# Default: false
log_json: false

# Log output path
# Valid values: File path, stdout or console, stderr
# Environment variable: $RELAY_LOG_PATH
# Default: stdout
log_path: console

# Comma separated list of enabled command execution
# engines.
# Available engines: native,docker
# Environment variable: $RELAY_ENABLED_ENGINES
# Default: native,docker
# Note: At least one engine must be enabled.
enabled_engines: native,docker

# Information required to connect to Cog
cog:
  # Cog's host name or IP address
  # Environment variable: $RELAY_COG_HOST
  # Default: 127.0.0.1
  # host: 127.0.0.1

  # Cog's MQTT port
  # Environment variable: $RELAY_COG_PORT
  # Default: 1883
  # port: 1883

  # Use SSL to establish MQTT connection
  # Environment variable: $RELAY_COG_ENABLE_SSL
  # Default: false
  # enable_ssl: false

  # Cog shared secret
  # Environment variable: $RELAY_COG_TOKEN
  # Default: none
  # Required: Yes
  # token: sekrit

  # Relay will refresh its bundle and Docker images
  # on this interval. Valid time units are s (seconds),
  # m (minutes), and h (hours).
  # Environment variable: $RELAY_COG_REFRESH_INTERVAL
  # Default: 15m
  # refresh_interval: 15m

# Docker config
docker:
  # Use environment variables to set up Docker connection?
  # Uses Docker's default logic for the environment variables $DOCKER_HOST,
  # $DOCKER_TLS_VERIFY, and $DOCKER_CERT_PATH.
  # Environment variable: $RELAY_DOCKER_USE_ENV
  # Default: false
  use_env: false

  # Location of Docker's Unix socket
  # Environment variable: $RELAY_DOCKER_SOCKET_PATH
  # MUST begin with unix:// or tcp://
  # Default: unix:///var/run/docker.sock
  # socket_path: unix:///var/run/docker.sock

  # Docker registry
  # Environment variable: $RELAY_DOCKER_REGISTRY_HOST
  # Default: hub.docker.com
  # registry_host: hub.docker.com

  # Docker registry username
  # Environment variable: $RELAY_DOCKER_REGISTRY_USER
  # Default: none
  # Required: No
  # registry_user: demouser

  # Docker registry password
  # Environment variable: $RELAY_DOCKER_REGISTRY_PASSWORD
  # Default: none
  # Required: No
  # registry_password: demouser

  # Relay will clean up unused Docker resources on this
  # interval. Valid time units are s (seconds),
  # m (minutes), and h (hours).
  # Environment variable: $RELAY_DOCKER_CLEAN_INTERVAL
  # Default: 5m
  # clean_interval: 5m

[rebyn]

Do you mind pasting the content of docker-compose.override.slack.yml (or s/slack/hipchat/) with credentials retracted? There's this ENV VAR that needs being set in both the Cog and the Relay containers: RELAY_COG_TOKEN.

[Gnep]

@rebyn

Hi Tu,

11:28:07 [Gnep@Pi cog]$ cat docker-compose.override.yml 
version: '2'
services:
  cog:
    environment:
      - COG_SLACK_ENABLED=1
      - SLACK_API_TOKEN=${SLACK_API_TOKEN}
      - COG_BOOTSTRAP_USERNAME=admin
      - COG_BOOTSTRAP_PASSWORD=changeme
      - COG_BOOTSTRAP_EMAIL_ADDRESS=cog@localhost
      - COG_BOOTSTRAP_FIRST_NAME=Cog
      - COG_BOOTSTRAP_LAST_NAME=Administrator
      - RELAY_ID=00000000-0000-0000-0000-000000000000
      - RELAY_COG_TOKEN=supersecret

  relay:
    environment:
      - RELAY_ID=00000000-0000-0000-0000-000000000000
      - RELAY_COG_TOKEN=supersecret

Pls note that I comment out two lines below in docker-compose.yml:

    privileged: true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

• In Hyper, containers are fully privileged
• Hyper doesn't support docker.sock atm.

[rebyn]

@Gnep to my limited understanding, Cog needs access to docker.sock to schedule containers (a Cog chat command executes by using container). So whilst the error might have nothing to do with it, this might be the reason why you cannot pull the stack up successfully.

[christophermaier]

@Gnep Yup, @rebyn is correct; if you're using Docker-based command bundles, Relay needs access to the Docker socket.

[Gnep]

Thanks @rebyn @christophermaier

Could you elaborate a bit what is the use of docker.sock by Relay?

[christophermaier]

@Gnep Sure thing!

First, a little background. Cog can currently deal with two types of command bundles: "native" and Docker.

With "native" bundles, the commands are executed directly on the Relay's host. Furthermore, you (as the operator of the Cog infrastructure) are responsible for installing the necessary binaries on the Relay.

In contrast, Docker command bundles are packaged up as Docker images. As they are invoked, Relay will execute the commands in a container, and not directly on the host. Additionally, the Relay takes care of "installing" the bundle for you; it just pulls it down from whatever Docker repository it is pointed at. This makes dealing with Docker-based command bundles very easy. For both types of bundles, you need to upload a metadata file to the Cog server to make Cog aware of the bundle, but with Docker bundles, that's all you need to do. Native bundles require additional installation steps. That means additional configuration management you'll need to deal with; it also complicates things like upgrading bundles in Cog, which is otherwise very simple.

As a result, the overwhelming majority (all?) the bundles you're likely to find in the wild, including the ones on https://bundles.operable.io are going to be Docker bundles, because they're so much easier to deal with.

And so, to get back to the question you asked, Relay interacts with the Docker engine directly by using docker.sock. I think we tried some other ways, but this is the most efficient. It would be possible run Relay without Docker support by setting the RELAY_ENABLED_ENGINES setting to native (instead of the default of native,docker), but they you can only use the native bundles, which limits your options quite a bit, IMHO.

[Gnep]

@christophermaier Thanks Chris.

Maybe I should change the question: as Cog runs commands using Docker format, does it make sense to support Hyper.sh as the backend engine?

AFAIK, Cog requires a server or a swarm cluster to run jobs. To keep a cluster running and manage it certainly introduces more cost and ops overhead. Hyper.sh is a container-native cloud, where you can launch containers on-demand, without the underlying cluster whatnot (not that we managed for you, there is just NO servers at all). And given the fast launch speed of containers, we provide a per-second billing model, e.g. your command runs for 10s, you pay for only that 10s.

What I imagine is that people run Cog server somewhere, and commands are executed in Hyper.sh in ad-hoc manner. In another word, serverless chatops.

What do you think?

[christophermaier]

@Gnep It doesn't sound like Hyper.sh is a good fit for Relay at the moment. In addition to the need for docker.sock access, Relay also does some intelligent caching of containers to cut down startup time for frequently-executed commands, which really cuts down execution time in complex command pipelines.

I don't see a quick fix or workaround for this, and it seems like it would take a non-trivial amount of work to support this deployment model. I'm not saying that it's impossible, necessarily, just that it would take some doing.

(Additionally, trying to use "native" bundles instead by packaging the command executables up with a Relay executable (so everything is in one container and nothing additional needs to be scheduled) won't work either, because Cog isn't currently designed to have Relays come and go that way.)