Open Telemetry

[AllanOricil]

Discuss about using Open Telemetry with Nebula Logger

[jongpie]

@AllanOricil completely agree, my plan/hope is to use OpenTelemetry for bidirectional logging

• Issue Add an Apex RestResource class to provide logging capabilities to external inbound integrations #635 is an enhancement to provide the ability for external integrations to log data in a Salesforce org
• I am also considering creating a batchable or queueable job that will take Nebula Logger data, convert it to OpenTelemetry's format, and send it to an OpenTelemetry endpoint

These 2 enhancements would let you send OpenTelemetry-formatted logging data to or from a Salesforce org. I don't have a timeline of when this will be added, but it's definitely on the roadmap.

[manwe651]

I am also considering creating a batchable or queueable job that will take Nebula Logger data, convert it to OpenTelemetry's format, and send it to an OpenTelemetry endpoint

This would be absolutely incredible, and I'm very interested in the progress. We have a lot of external processing for our Salesforce instance in AWS, and I'd love to have a unified view of all of our logs.

[jongpie]

@AllanOricil certainly some of that could be built off platform, and there's nothing preventing someone from using that approach - but that is not something that I'm planning to provide as part of Nebula Logger. One of Nebula Logger's advantages is that it provides native observability directly in Salesforce, without any external dependencies. This could one day change, but I don't see a need to go off platform to provide OTel support.

You could just create and maintain a receiver, and base transformers for the most important tools out there.

The word "just" is also doing a lot of work in that sentence - you're proposing building & hosting a service, including providing tool-specific transformations. That is a huge effort, and in my opinion, building anything that's vendor-specific or tool-specific defeats the purpose of trying to leverage OpenTelemetry as a standard.

My plan is for Nebula Logger to have options for accessing logging data in OTel format, using REST/HTTP and JSON formatted according to OTel's logs data model.

• Push OTel data out of Salesforce: Having Nebula Logger push OTel data to an OTel endpoint will be one option
• Pull OTel data out of Salesforce: There will also be a custom endpoint that transforms & returns the data in Nebula Logger's custom objects in OTel's format

This will provide a consistent transformation of Nebula Logger's data model into OTel's data model, regardless of what external tool people are using. For some observability tools, there may be some additional transformations needed, but I'm considering that kind of work to be outside of the scope of Nebula Logger's role.

[AllanOricil]

he word "just" is also doing a lot of work in that sentence - you're proposing building & hosting a service, including providing tool-specific transformations.

The word "just" was refering to the amount of work if you follow my suggestion, which I believe reduces the amount of work because it wouldn't be necessary to add changes to your logger library, only create the 3 components (receiver, processors, exporters) of a collector. Let me try to explain my reasoning for this. Please answer the following questions separately.

A use case, for context.

I have 2 services that belong to the same transaction. One runs in Salesforce, and the other in AWS, which is a very simple lambda function. The transaction starts in Salesforce, goes to this service, and comes back to Salesforce. This transaction creates logs for both services. It is a pain to read and aggregate logs produced by 2 services running on different places, each using its own logger library. To solve this problem, I need to format, aggregate and send those logs to a single place, such as Signoz or DataDog, where I will be able to read, and relate them.

To extract, format, aggregate and export those logs, I decided to use Open Telemetry, and its series of open source components: receiver, processors and exporters. I'm going to setup a EC2 and start an "otel collector" agent. I'm going to configure the collector's manifest to scrape logs from both Salesforce and AWS, as they are, using their respectively "receiver". The logs generated by each service are using different formats, because they are both using different logger libraries. The logs from Salesforce are created by NebulaLogger, and the ones from AWS Winston. In this manifest, I will also configure a log exporter to send those logs to my tool, which is Signoz but it could also be DataDog.

Questions:

For logs from AWS, I don't need to deploy a job that is going to "prepare" and push logs to Signoz. The "otel collector" agent that is running in my EC2 was configured to scrape those logs using a "receiver". For Salesforce, the same will happen. Its "receiver" will scrape logs from Salesforce using the "tooling api". The "processor" will then format those, and finally the "exporter" will send them to the tool I configured, which is Signoz. Such as in the picture below

As explained above, if the whole log extraction, formatting, aggregation, export can happen in the "collector", using each of its components, and both native APIs (tooling and cloudwatch), why do I need an extra API and an extra job running in Salesforce to process those logs? If a problem can be solved in a single place, using a single tool, and native apis, I don't see a reason for having part of the job running in Salesforce. That is why I'm seeing duplicate effort. As soon as someone creates a receiver which uses the native tooling api, and a processor that can process NebulaLogger logs, there is no need for also doing this job inside Salesforce.

Do you still think a job and an extra api is required? If so, can you explain why, and how I would configure it in my otel collector manifest, please?

Additionaly, Salesforce has tons of limits, could you predict the throughput of logs comming out of Salesforce if the "extraction" uses a "push" mechanism? I don't want logs to arrive to my tool with a huge delay, or even arrive in the next day. How much bandwidth from other jobs already running in Salesforce, will this processing and push jobs consume? Maybe orgs already have too many jobs running that adding another one will be just "too much", causing process that must run once a day to be delayed.

I want it to be plug and play. I don't want to manage jobs inside Salesforce to "pre process" my logs, or push them to somewhere. My "processor", running in the otel collector agent, in the EC2, will do all the full pipeline to me. The same way it does for my Cloud Watch logs.

That is a huge effort, and in my opinion, building anything that's vendor-specific or tool-specific defeats the purpose of trying to leverage OpenTelemetry as a standard.

I'm not sure why you understood I said that. I was trying to explain that we "just" need a "processor" that can process "NebulaLogger logs", which produces a specific log format. You would create this "processor" to prepare the logs to be exported to anywhere, like in the image below.

• Push OTel data out of Salesforce: Having Nebula Logger push OTel data to an OTel endpoint will be one option

This shouldn't be an option. As soon as receiver and a processor is created, which will run outside of Salesforce, in a environment with way less processing and mem restrictions, why would a someone opt for the "push" job running inside Salesforce? What are the advantages? I can only see disavantages, and no trade-offs

I also don't see a reason for an "Open Telemetry compliant rest API" to receive logs inside Salesforce. Why would I use Salesforce to read logs? What is the use case?

[jamessimone]

I'm not sure if you meant to attach the stock image from OTel's Collector documentation page twice - I'm also not sure if the image itself is really unpacking much for the audience. Hidden, for example, in your "just ...create the 3 components (receiver, processors, exporters) of a collector" quote are little pieces of complexity like:

• authentication - I've always loved describing authentication as simple
• the actual nuances of building receivers - nothing screams "simple" like 14 subheadings on a page
• the actual nuances of exporting (I'll just use the Cloudwatch exporter as an example) aren't exactly terse even when working with something as simple as JSON from the get-go

So if we are to take the "receiver, processors, and exporters" part of that quote, 2 out of the 3 mentioned steps are actually fairly non-trivial (and that's prior to handling the authentication piece). It doesn't take a lot of investigative work to understand why the OTel registry has dozens of entries for popular languages like C#, Rust, and Java for wrapping common processes - like AWS Lambda invocations - within OTel-compliant logs: because it's much easier and far less complicated to transform data in the host language prior to pushing it to an endpoint (and the same holds true for importing data from Salesforce using an OTel-compliant endpoint, if we can do something as "simple" as assume authentication has been handled).

On the subject of complexity - it's my personal opinion that representing complex tasks as simple serves to obfuscate both the time commitment in terms of authoring functionality, as well as the maintenance time spent on existing code (which, in general, tends to far outweigh the original time costs of any piece of authored code/configuration - there are dozens if not hundreds of studies that have been done about this). In our role as software engineers, accurately and without ego representing complexity is extremely important. I don't think anybody minds when somebody takes their own time (however long that may take) to contribute something they believe is important to open source; conversely, I think many people mind when it's suggested that somebody else "should" do that work in an opinionated way.

I'd caution you against using words like "should" or "shouldn't" (citation) when talking about possibilities. I think we can all agree that because this discussion is occurring within the realm of programming (where most things are possible), there are many ways this functionality could be implemented. The introduction of "should" and "shouldn't" doesn't serve anybody. Luckily, since this is open source, you're always free to fork and/or contribute the functionality you'd like to see in the way you'd like for it to be implemented.

---

As an aside, there are plenty of counter-examples when it comes to answering questions like: "Why would I use Salesforce to read logs?" I've worked for several companies that wanted their Salesforce org to serve as the source of truth for combined logging; I've worked at places where exporting logs to a log aggregator was also common. That's really apropos of nothing, other than to say that if we consider the two of us, it's pretty telling if 50% of the sample size has had experience with those use-cases.

[arunkumar15-shell]

"The ability to pull/retrieve Nebula Logger data from your org, and return the data in OpenTelemetry's format"
Is this feature available now?

[AllanOricil]

You can write a service that runs on a certain cadence to pull logs from Salesforce, convert to OTLP format, and post to your OTLP collector using the built in OTLP receiver.