Tentacle reports that `ScriptState` could not be loaded from `scriptstate.json`

[sburmanoctopus]

Team

• I've assigned a team label to this issue

What happened?

Tentacle sometimes reports that ScriptState could not be loaded from scriptstate.json

Reproduction

This was noticed while monitoring for RPC retry errors. It would only happen rarely, but it was noticed.

Error and Stacktrace

Server exception: 
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Exception: ScriptState could not be loaded from C:\\Octopus\\Work\\ServerTasks-128528-e2178932-e67f-4788-ac61-00ee29973fe8\\scriptstate.json
   at Octopus.Tentacle.Scripts.ScriptStateStore.Load() in C:\\BuildAgent\\work\\639265b01610d682\\source\\Octopus.Tentacle\\Scripts\\ScriptStateStore.cs:line 58
   at Octopus.Tentacle.Services.Scripts.ScriptServiceV2.StartScript(StartScriptCommandV2 command) in C:\\BuildAgent\\work\\639265b01610d682\\source\\Octopus.Tentacle\\Services\\Scripts\\ScriptServiceV2.cs:line 36
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Halibut.ServiceModel.ServiceInvoker.Invoke(RequestMessage requestMessage)
   at Halibut.Transport.Protocol.MessageExchangeProtocol.InvokeAndWrapAnyExceptions(RequestMessage request, Func`2 incomingRequestProcessor)
   at Halibut.ServiceModel.HalibutProxyWithAsync.ThrowExceptionFromReceivedError(ServerError error, ILog logger)
   at Halibut.ServiceModel.HalibutProxyWithAsync.MakeRpcCall(MethodInfo asyncMethod, Object[] args)
   at Halibut.ServiceModel.HalibutProxyWithAsync.InvokeAsyncT[T](MethodInfo asyncMethod, Object[] args)
   at System.Reflection.AsyncDispatchProxyGenerator.InvokeAsync[T](Object[] args)
   at Octopus.Tentacle.Client.Decorators.HalibutExceptionAsyncScriptServiceV2Decorator.StartScriptAsync(StartScriptCommandV2 command, HalibutProxyRequestOptions halibutProxyRequestOptions)
   at Octopus.Tentacle.Client.Scripts.ScriptExecutionOrchestrator.<>c__DisplayClass14_1.<<ExecuteScript>b__2>d.MoveNext()
--- End of stack trace from previous location ---
   at Octopus.Tentacle.Client.Utils.AsyncHalibutFeatureExtensionMethods.WhenEnabled[T](AsyncHalibutFeatureWithResult`1 asyncHalibutFeatureWithResult, Func`1 action)
   at Octopus.Tentacle.Client.Scripts.ScriptExecutionOrchestrator.<>c__DisplayClass14_0.<<ExecuteScript>g__StartScriptAction|0>d.MoveNext()
--- End of stack trace from previous location ---
   at Octopus.Tentacle.Client.Execution.RpcCallExecutor.<>c__DisplayClass7_1`1.<<ExecuteWithRetries>b__3>d.MoveNext()
--- End of stack trace from previous location ---
   at Octopus.Tentacle.Client.Execution.RpcCallExecutor.<>c__DisplayClass7_0`1.<<ExecuteWithRetries>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Octopus.Tentacle.Client.Retries.RpcCallRetryHandler.<>c__DisplayClass11_0`1.<<ExecuteWithRetries>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Polly.Timeout.AsyncTimeoutEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Func`2 timeoutProvider, TimeoutStrategy timeoutStrategy, Func`5 onTimeoutAsync, Boolean continueOnCapturedContext)
   at Polly.Timeout.AsyncTimeoutEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Func`2 timeoutProvider, TimeoutStrategy timeoutStrategy, Func`5 onTimeoutAsync, Boolean continueOnCapturedContext)
   at Polly.AsyncPolicy.ExecuteAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at Octopus.Tentacle.Client.Retries.RpcCallRetryHandler.<>c__DisplayClass10_0`1.<<ExecuteWithRetries>g__ExecuteAction|2>d.MoveNext()
--- End of stack trace from previous location ---
   at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
   at Polly.AsyncPolicy.ExecuteAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at Octopus.Tentacle.Client.Retries.RpcCallRetryHandler.ExecuteWithRetries[T](Func`2 action, OnRetyAction onRetryAction, OnTimeoutAction onTimeoutAction, CancellationToken cancellationToken)
   at Octopus.Tentacle.Client.Retries.RpcCallRetryHandler.ExecuteWithRetries[T](Func`2 action, OnRetyAction onRetryAction, OnTimeoutAction onTimeoutAction, Boolean abandonActionOnCancellation, TimeSpan abandonAfter, CancellationToken cancellationToken)
   at Octopus.Tentacle.Client.Execution.RpcCallExecutor.ExecuteWithRetries[T](RpcCall rpcCall, Func`2 action, ILog logger, Boolean abandonActionOnCancellation, ClientOperationMetricsBuilder clientOperationMetricsBuilder, CancellationToken cancellationToken)
   at Octopus.Tentacle.Client.Scripts.ScriptExecutionOrchestrator.ExecuteScript(CancellationToken scriptExecutionCancellationToken)
   at Octopus.Tentacle.Client.TentacleClient.ExecuteScript(StartScriptCommandV2 startScriptCommand, Action`1 onScriptStatusResponseReceived, Func`2 onScriptCompleted, ILog logger, CancellationToken scriptExecutionCancellationToken)
   at Octopus.Server.Orchestration.Targets.Tentacles.TentacleRemoteEndpointFacadeV2.ExecuteCommand(StartScriptCommand startScriptCommand, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/Targets/Tentacles/TentacleRemoteEndpointFacadeV2.cs:line 71
   at Octopus.Server.Orchestration.Targets.Common.RemoteEndpointFacadeCancellationTokenDecorator.ExecuteCommand(StartScriptCommand command, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/Targets/Common/RemoteEndpointFacadeCancellationTokenDecorator.cs:line 36
   at Octopus.Server.Orchestration.Targets.Tentacles.Observability.ErrorLoggingRemoteEndpointFacadeDecorator.ExecuteCommand(StartScriptCommand command, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/Targets/Tentacles/Observability/ErrorLoggingRemoteEndpointFacadeDecorator.cs:line 75
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.Immediate.ExecutionTargets.TentacleExecutionTarget.Execute(ScriptCollection bootstrapperScripts, IReadOnlyList`1 bootstrapperArguments, IReadOnlyList`1 files, Nullable`1 forceIsolationLevel, Boolean raw, ITaskLog taskLog, String isolationMutexName, CancellationToken cancellationToken, Nullable`1 isolationMutexTimeout) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionExecution/Immediate/ExecutionTargets/TentacleExecutionTarget.cs:line 70
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.Immediate.ImmediateExecutor.HasDeploymentPackageBeenInstalledWhileWaitingForLock(IDeploymentPackageVersion packageVersion, IBootstrapperScriptGenerator bootstrapperScriptGenerator, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionExecution/Immediate/ImmediateExecutor.cs:line 454
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.Immediate.ImmediateExecutor.InstallDeploymentPackage(IDeploymentPackageVersion packageVersion, IBootstrapperScriptGenerator bootstrapperScriptGenerator, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionExecution/Immediate/ImmediateExecutor.cs:line 400
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.Immediate.ImmediateExecutor.ResolveRequiredToolsMessages(List`1 serviceMessages, String toolsPlatform, IBootstrapperScriptGenerator bootstrapperScriptGenerator, CalamariFlavour calamariFlavour, IReadOnlyList`1 deploymentTools, ExecutionTargetDetails executionTargetDetails, CalamariPlatformConstraint calamariPlatformConstraint, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionExecution/Immediate/ImmediateExecutor.cs:line 375
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.Immediate.ImmediateExecutor.RunOnTarget(ScriptCollection finalBootstrapperScripts, String platform, IBootstrapperScriptGenerator bootstrapperScriptGenerator, IReadOnlyList`1 bootstrapperArguments, IReadOnlyList`1 allFiles, Nullable`1 isolation, Nullable`1 isolationMutexTimeout, String isolationMutexName, CalamariFlavour flavour, IReadOnlyList`1 deploymentTools, ExecutionTargetDetails executionTargetDetails, CalamariPlatformConstraint calamariPlatformConstraint, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionExecution/Immediate/ImmediateExecutor.cs:line 304
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.Immediate.ImmediateExecutor.ExecuteCalamari(CalamariFlavour calamariFlavour, String calamariCommand, IReadOnlyList`1 calamariArguments, IReadOnlyList`1 files, IReadOnlyList`1 deploymentTools, VariableCollection extraVariables, TargetManifest targetManifest, CalamariPlatformConstraint calamariPlatformConstraint, Nullable`1 isolationMutexTimeout, String isolationMutexName, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionExecution/Immediate/ImmediateExecutor.cs:line 224
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.CommandBuilders.CalamariCommandBuilder.Execute(ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionExecution/CommandBuilders/CalamariCommandBuilder.cs:line 162
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.PackageRetention.PackageRetentionRunner.<>c__DisplayClass8_0.<<DispatchWithTransientErrorHandling>b__0>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/PackageRetention/PackageRetentionRunner.cs:line 85
--- End of stack trace from previous location ---
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.InvokeActionHandler(Machine target, ActionHandlerInvocation actionHandler, ActionAndTargetScopedVariables actionAndTargetScopedVariables, IExecutor executor, TargetManifest targetManifest, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionDispatch/AdHocActionDispatcher.cs:line 238
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.ExecuteOnDeploymentTarget(DeploymentTarget deploymentTarget, ActionHandlerInvocation actionHandler, TargetManifest targetManifest, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionDispatch/AdHocActionDispatcher.cs:line 195
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.Dispatch(Machine machine, ActionHandlerInvocation actionHandler, ITaskLog taskLog, CancellationToken cancellationToken, VariableCollection variables) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionDispatch/AdHocActionDispatcher.cs:line 80
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.PackageRetention.PackageRetentionRunner.DispatchWithTransientErrorHandling(Planned`1 planned, IAdHocActionDispatcher actionDispatcher, IMachineService machineService, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/PackageRetention/PackageRetentionRunner.cs:line 104
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.PackageRetention.PackageRetentionRunner.DispatchWithTransientErrorHandling(Planned`1 planned, IAdHocActionDispatcher actionDispatcher, IMachineService machineService, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/PackageRetention/PackageRetentionRunner.cs:line 104
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.PackageRetention.PackageRetentionRunner.<>c__DisplayClass7_1.<<Run>b__2>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/PackageRetention/PackageRetentionRunner.cs:line 53
--- End of stack trace from previous location ---
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Guidance.ExecuteWithoutGuidance(Func`2 callback, String actionName, Boolean actionIsRequiredToRun, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Guidance.cs:line 149
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Guidance.Execute(Func`2 callback, String actionName, Boolean actionIsRequiredToRun, ITaskLog taskLog, Action callbackOnExclude, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Guidance.cs:line 79
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.PackageRetention.PackageRetentionRunner.<>c__DisplayClass7_1.<<Run>b__1>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/PackageRetention/PackageRetentionRunner.cs:line 58
--- End of stack trace from previous location ---
   at Octopus.Server.Orchestration.ServerTasks.Deploy.TransientErrorDetectionExecutor.Execute(Func`2 action, ExecutionPlan plan, ITaskLog taskLog, CancellationToken cancellationToken, DeploymentTarget deploymentTarget) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/TransientErrorDetectionExecutor.cs:line 50
   at Octopus.Server.Orchestration.ServerTasks.Deploy.TransientErrorDetectionExecutor.Execute(Func`2 action, ExecutionPlan plan, ITaskLog taskLog, CancellationToken cancellationToken, DeploymentTarget deploymentTarget) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/TransientErrorDetectionExecutor.cs:line 50
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.PackageRetention.PackageRetentionRunner.<>c__DisplayClass7_0.<<Run>b__0>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/PackageRetention/PackageRetentionRunner.cs:line 62
--- End of stack trace from previous location ---
   at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.<>c__DisplayClass7_0`4.<<Execute>b__0>d.MoveNext() in ./source/Octopus.Server/Infrastructure/Orchestration/UnitsOfWork/UnitOfWorkExecutor.cs:line 124
--- End of stack trace from previous location ---
   at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.Do(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in ./source/Octopus.Core/Infrastructure/UnitsOfWork/UnitOfWorkExtensionMethods.cs:line 58
   at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.Do(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in ./source/Octopus.Core/Infrastructure/UnitsOfWork/UnitOfWorkExtensionMethods.cs:line 58
   at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.Execute[T1,T2,T3,T4](Func`6 action, CancellationToken cancellationToken, String name) in ./source/Octopus.Server/Infrastructure/Orchestration/UnitsOfWork/UnitOfWorkExecutor.cs:line 127
   at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.Execute[T1,T2,T3,T4](Func`6 action, CancellationToken cancellationToken, String name) in ./source/Octopus.Server/Infrastructure/Orchestration/UnitsOfWork/UnitOfWorkExecutor.cs:line 127
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.PackageRetention.PackageRetentionRunner.Run(ExecutionPlan plan, Planned`1 planned, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/PackageRetention/PackageRetentionRunner.cs:line 65
   at Octopus.Server.Orchestration.ServerTasks.ParallelWorkOrderAsync`1.<>c__DisplayClass7_1.<<Execute>b__3>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/ParallelWorkOrderAsync.cs:line 61
--- End of stack trace from previous location ---
   at Octopus.Server.Orchestration.ServerTasks.OctoThreadClosureAsync`1.Execute(CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/OctoThreadClosureAsync.cs:line 61

More Information

No response

Workaround

No response