Merge pull request #8490 from marciw/mw-ts-move

[Docs] Restore troubleshooting content

Author: marciw

docs/reference/images/elasticsearch-client-net-api-capture-requests-localhost.png

@@ -20,4 +20,15 @@ toc:
       - file: query.md
       - file: recommendations.md
       - file: transport.md
-      - file: migration-guide.md
+      - file: migration-guide.md
+  - file: troubleshoot/index.md
+    children:
+      - file: troubleshoot/logging.md
+        children:
+          - file: troubleshoot/logging-with-onrequestcompleted.md
+          - file: troubleshoot/logging-with-fiddler.md
+      - file: troubleshoot/debugging.md
+        children:
+          - file: troubleshoot/audit-trail.md
+          - file: troubleshoot/debug-information.md
+          - file: troubleshoot/debug-mode.md

docs/reference/images/elasticsearch-client-net-api-capture-requests-remotehost.png

@@ -0,0 +1,66 @@
+---
+mapped_pages:
+  - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/audit-trail.html
+---
+
+# Audit trail [audit-trail]
+
+Elasticsearch.Net and NEST provide an audit trail for the events within the request pipeline that occur when a request is made. This audit trail is available on the response as demonstrated in the following example.
+
+We’ll use a Sniffing connection pool here since it sniffs on startup and pings before first usage, so we can get an audit trail with a few events out
+
+```csharp
+var pool = new SniffingConnectionPool(new []{ TestConnectionSettings.CreateUri() });
+var connectionSettings = new ConnectionSettings(pool)
+    .DefaultMappingFor<Project>(i => i
+        .IndexName("project")
+    );
+
+var client = new ElasticClient(connectionSettings);
+```
+
+After issuing the following request
+
+```csharp
+var response = client.Search<Project>(s => s
+    .MatchAll()
+);
+```
+
+The audit trail is provided in the [Debug information](debug-information.md) in a human readable fashion, similar to
+
+```
+Valid NEST response built from a successful low level call on POST: /project/doc/_search
+# Audit trail of this API call:
+ - [1] SniffOnStartup: Took: 00:00:00.0360264
+ - [2] SniffSuccess: Node: http://localhost:9200/ Took: 00:00:00.0310228
+ - [3] PingSuccess: Node: http://127.0.0.1:9200/ Took: 00:00:00.0115074
+ - [4] HealthyResponse: Node: http://127.0.0.1:9200/ Took: 00:00:00.1477640
+# Request:
+<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
+# Response:
+<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
+```
+to help with troubleshootin
+
+```csharp
+var debug = response.DebugInformation;
+```
+
+But can also be accessed manually:
+
+```csharp
+response.ApiCall.AuditTrail.Count.Should().Be(4, "{0}", debug);
+response.ApiCall.AuditTrail[0].Event.Should().Be(SniffOnStartup, "{0}", debug);
+response.ApiCall.AuditTrail[1].Event.Should().Be(SniffSuccess, "{0}", debug);
+response.ApiCall.AuditTrail[2].Event.Should().Be(PingSuccess, "{0}", debug);
+response.ApiCall.AuditTrail[3].Event.Should().Be(HealthyResponse, "{0}", debug);
+```
+
+Each audit has a started and ended `DateTime` on it that will provide some understanding of how long it took
+
+```csharp
+response.ApiCall.AuditTrail
+    .Should().OnlyContain(a => a.Ended - a.Started >= TimeSpan.Zero);
+```
+

docs/reference/images/elasticsearch-client-net-api-inspect-requests.png

@@ -0,0 +1,165 @@
+---
+mapped_pages:
+  - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/debug-information.html
+---
+
+# Debug information [debug-information]
+
+Every response from Elasticsearch.Net and NEST contains a `DebugInformation` property that provides a human readable description of what happened during the request for both successful and failed requests
+
+```csharp
+var response = client.Search<Project>(s => s
+    .Query(q => q
+        .MatchAll()
+    )
+);
+
+response.DebugInformation.Should().Contain("Valid NEST response");
+```
+
+This can be useful in tracking down numerous problems and can also be useful when filing an [issue](https://github.com/elastic/elasticsearch-net/issues) on the GitHub repository.
+
+## Request and response bytes [_request_and_response_bytes]
+
+By default, the request and response bytes are not available within the debug information, but can be enabled globally on Connection Settings by setting `DisableDirectStreaming`. This disables direct streaming of
+
+1. the serialized request type to the request stream
+2. the response stream to a deserialized response type
+
+```csharp
+var connectionPool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
+
+var settings = new ConnectionSettings(connectionPool)
+    .DisableDirectStreaming(); <1>
+
+var client = new ElasticClient(settings);
+```
+
+1. disable direct streaming for **all** requests
+
+
+or on a *per request* basis
+
+```csharp
+var response = client.Search<Project>(s => s
+    .RequestConfiguration(r => r
+        .DisableDirectStreaming() <1>
+    )
+    .Query(q => q
+        .MatchAll()
+    )
+);
+```
+
+1. disable direct streaming for **this** request only
+
+
+Configuring `DisableDirectStreaming` on an individual request takes precedence over any global configuration.
+
+There is typically a performance and allocation cost associated with disabling direct streaming since both the request and response bytes must be buffered in memory, to allow them to be exposed on the response call details.
+
+
+## TCP statistics [_tcp_statistics]
+
+It can often be useful to see the statistics for active TCP connections, particularly when trying to diagnose issues with the client. The client can collect the states of active TCP connections just before making a request, and expose these on the response and in the debug information.
+
+Similarly to `DisableDirectStreaming`, TCP statistics can be collected for every request by configuring on `ConnectionSettings`
+
+```csharp
+var connectionPool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
+
+var settings = new ConnectionSettings(connectionPool)
+    .EnableTcpStats(); <1>
+
+var client = new ElasticClient(settings);
+```
+
+1. collect TCP statistics for **all** requests
+
+
+or on a *per request* basis
+
+```csharp
+var response = client.Search<Project>(s => s
+    .RequestConfiguration(r => r
+        .EnableTcpStats() <1>
+    )
+    .Query(q => q
+        .MatchAll()
+    )
+);
+
+var debugInformation = response.DebugInformation;
+```
+
+1. collect TCP statistics for **this** request only
+
+
+With `EnableTcpStats` set, the states of active TCP connections will now be included on the response and in the debug information.
+
+The client includes a `TcpStats` class to help with retrieving more detail about active TCP connections should it be required
+
+```csharp
+var tcpStatistics = TcpStats.GetActiveTcpConnections(); <1>
+var ipv4Stats = TcpStats.GetTcpStatistics(NetworkInterfaceComponent.IPv4); <2>
+var ipv6Stats = TcpStats.GetTcpStatistics(NetworkInterfaceComponent.IPv6); <3>
+
+var response = client.Search<Project>(s => s
+    .Query(q => q
+        .MatchAll()
+    )
+);
+```
+
+1. Retrieve details about active TCP connections, including local and remote addresses and ports
+2. Retrieve statistics about IPv4
+3. Retrieve statistics about IPv6
+
+
+::::{note} 
+Collecting TCP statistics may not be accessible in all environments, for example, Azure App Services. When this is the case, `TcpStats.GetActiveTcpConnections()` returns `null`.
+
+::::
+
+
+
+## ThreadPool statistics [_threadpool_statistics]
+
+It can often be useful to see the statistics for thread pool threads, particularly when trying to diagnose issues with the client. The client can collect statistics for both worker threads and asynchronous I/O threads, and expose these on the response and in debug information.
+
+Similar to collecting TCP statistics, ThreadPool statistics can be collected for all requests by configuring `EnableThreadPoolStats` on `ConnectionSettings`
+
+```csharp
+var connectionPool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
+
+var settings = new ConnectionSettings(connectionPool)
+     .EnableThreadPoolStats(); <1>
+
+var client = new ElasticClient(settings);
+```
+
+1. collect thread pool statistics for **all** requests
+
+
+or on a *per request* basis
+
+```csharp
+var response = client.Search<Project>(s => s
+     .RequestConfiguration(r => r
+             .EnableThreadPoolStats() <1>
+     )
+     .Query(q => q
+         .MatchAll()
+     )
+ );
+
+var debugInformation = response.DebugInformation; <2>
+```
+
+1. collect thread pool statistics for **this** request only
+2. contains thread pool statistics
+
+
+With `EnableThreadPoolStats` set, the statistics of thread pool threads will now be included on the response and in the debug information.
+
+

docs/reference/toc.yml

@@ -0,0 +1,50 @@
+---
+mapped_pages:
+  - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/debug-mode.html
+---
+
+# Debug mode [debug-mode]
+
+The [Debug information](debug-information.md) explains that every response from Elasticsearch.Net and NEST contains a `DebugInformation` property, and properties on `ConnectionSettings` and `RequestConfiguration` can control which additional information is included in debug information, for all requests or on a per request basis, respectively.
+
+During development, it can be useful to enable the most verbose debug information, to help identify and troubleshoot problems, or simply ensure that the client is behaving as expected. The `EnableDebugMode` setting on `ConnectionSettings` is a convenient shorthand for enabling verbose debug information, configuring a number of settings like
+
+* disabling direct streaming to capture request and response bytes
+* prettyfying JSON responses from Elasticsearch
+* collecting TCP statistics when a request is made
+* collecting thread pool statistics when a request is made
+* including the Elasticsearch stack trace in the response if there is a an error on the server side
+
+```csharp
+IConnectionPool pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
+
+var settings = new ConnectionSettings(pool)
+    .EnableDebugMode(); <1>
+
+var client = new ElasticClient(settings);
+
+var response = client.Search<Project>(s => s
+    .Query(q => q
+        .MatchAll()
+    )
+);
+
+var debugInformation = response.DebugInformation; <2>
+```
+
+1. configure debug mode
+2. verbose debug information
+
+
+In addition to exposing debug information on the response, debug mode will also cause the debug information to be written to the trace listeners in the `System.Diagnostics.Debug.Listeners` collection by default, when the request has completed. A delegate can be passed when enabling debug mode to perform a different action when a request has completed, using [`OnRequestCompleted`](logging-with-onrequestcompleted.md)
+
+```csharp
+var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
+var client = new ElasticClient(new ConnectionSettings(pool)
+    .EnableDebugMode(apiCallDetails =>
+    {
+        // do something with the call details e.g. send with logging framework
+    })
+);
+```
+

docs/reference/troubleshoot/audit-trail.md

@@ -0,0 +1,14 @@
+---
+mapped_pages:
+  - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/debugging.html
+---
+
+# Debugging [debugging]
+
+Elasticsearch.Net and NEST provide an audit trail and debug information to help you resolve issues:
+
+* [](audit-trail.md)
+* [](debug-information.md)
+* [](debug-mode.md)
+
+

docs/reference/troubleshoot/debug-information.md

@@ -0,0 +1,13 @@
+---
+navigation_title: Troubleshoot
+mapped_pages:
+  - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/troubleshooting.html
+---
+
+# Troubleshoot: {{es}} .NET client [troubleshooting]
+
+The client can provide rich details about what occurred in the request pipeline during the process of making a request, and can also provide the raw request and response JSON.
+
+* [Logging](logging.md)
+* [Debugging](debugging.md)
+

docs/reference/troubleshoot/debug-mode.md

@@ -0,0 +1,50 @@
+---
+mapped_pages:
+  - https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/logging-with-fiddler.html
+---
+
+# Logging with Fiddler [logging-with-fiddler]
+
+A web debugging proxy such as [Fiddler](http://www.telerik.com/fiddler) is a useful way to capture HTTP traffic from a machine, particularly whilst developing against a local Elasticsearch cluster.
+
+## Capturing traffic to a remote cluster [_capturing_traffic_to_a_remote_cluster]
+
+To capture traffic against a remote cluster is as simple as launching Fiddler! You may want to also filter traffic to only show requests to the remote cluster by using the filters tab
+
+:::{image} ../images/elasticsearch-client-net-api-capture-requests-remotehost.png
+:alt: Capturing requests to a remote host
+:::
+
+
+## Capturing traffic to a local cluster [_capturing_traffic_to_a_local_cluster]
+
+The .NET Framework is hardcoded not to send requests for `localhost` through any proxies and as a proxy Fiddler will not receive such traffic.
+
+This is easily circumvented by using `ipv4.fiddler` as the hostname instead of `localhost`
+
+```csharp
+var isFiddlerRunning = Process.GetProcessesByName("fiddler").Any();
+var host = isFiddlerRunning ? "ipv4.fiddler" : "localhost";
+
+var connectionSettings = new ConnectionSettings(new Uri($"http://{host}:9200"))
+    .PrettyJson(); <1>
+
+var client = new ElasticClient(connectionSettings);
+```
+
+1. prettify json requests and responses to make them easier to read in Fiddler
+
+
+With Fiddler running, the requests and responses will now be captured and can be inspected in the Inspectors tab
+
+:::{image} ../images/elasticsearch-client-net-api-inspect-requests.png
+:alt: Inspecting requests and responses
+:::
+
+As before, you may also want to filter traffic to only show requests to `ipv4.fiddler` on the port on which you are running Elasticsearch.
+
+:::{image} ../images/elasticsearch-client-net-api-capture-requests-localhost.png
+:alt: Capturing requests to localhost
+:::
+
+