Introduce better logging

[rverma-dev]

Following the guide here

Terraform workspaces created but no run configuration.

Exception:

2021-05-29T00:38:51.072Z	INFO	terraform-k8s	Checking outputs	{"Organization": "nslhub", "WorkspaceID": "ws-fkYap2hAd8ogTkxp", "RunID": ""}
2021-05-29T00:38:51.094Z	INFO	terraform-k8s	Updated outputs	{"Organization": "nslhub", "WorkspaceID": "ws-fkYap2hAd8ogTkxp"}
2021-05-29T00:38:51.094Z	INFO	terraform-k8s	Updating secrets	{"name": "greetings-outputs"}
2021-05-29T00:38:51.094Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"Workspace","namespace":"ci","name":"greetings","uid":"220cacfe-7f6c-4717-9258-ad02aec80e87","apiVersion":"app.terraform.io/v1alpha1","resourceVersion":"16909471"}, "reason": "WorkspaceEvent", "message": "Updated outputs for run "}
2021-05-29T00:38:51.564Z	INFO	terraform-k8s	Starting module backed run	{"Organization": "nslhub", "Name": "greetings", "Namespace": "ci"}
2021-05-29T00:38:51.775Z	ERROR	controller	Reconciler error	{"controller": "workspace-controller", "name": "greetings", "namespace": "ci", "error": "resource not found"}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:246
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
	/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
	/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
	/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90

This makes it very tough to understand the issue. Also observed

Status:
  Config Version ID:  
  Run ID:             
  Run Status:         
  Workspace ID:       ws-fkYap2hAd8ogTkxp

[rverma-dev]

The issue starts resurfacing again,

2021-06-12T07:59:02.740Z	ERROR	terraform-k8s	Failed to update output status	{"error": "workspaces.app.terraform.io \"s3-logs-ap-south-1\" not found"}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
github.com/hashicorp/terraform-k8s/workspacehelper.(*WorkspaceHelper).reconcileWorkspace
	/workspace/workspacehelper/workspace_helper.go:142
github.com/hashicorp/terraform-k8s/workspacehelper.(*WorkspaceHelper).Reconcile
	/workspace/workspacehelper/workspace_helper.go:442
github.com/hashicorp/terraform-k8s/controllers.(*WorkspaceReconciler).Reconcile
	/workspace/controllers/workspace_controller.go:63
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:244
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
	/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
	/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
	/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90

On further debugging its failing here

terraform-k8s/workspacehelper/workspace_helper.go

Line 141 in 973ac76

if err := r.client.Status().Update(context.TODO(), instance); err != nil {

There is no way to identify the cause of this. It seems it is failing when updating the workspace id in workspace resource.

[mottetm]

@rverma-nsl Did you ever figure out what was the root cause of your first issue? Facing the same but can't figure out what the issue is...