Skip to content

Commit 301e1f9

Browse files
authored
Added Anonymous Telemetry (#223)
1 parent f39a292 commit 301e1f9

File tree

21 files changed

+371
-172
lines changed

21 files changed

+371
-172
lines changed

README.md

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# 🪡 orra
22

3-
Move beyond simple Crews and Agents. Use orra to build production-ready multi-agent applications that handle complex real-world interactions.
3+
Move beyond simple Crews and Agents. Use orra's production-grade planning to reliably run multi-agent workflows that require complex real-world interactions.
44

55
![](images/orra-diagram.png)
66

@@ -9,11 +9,11 @@ orra coordinates tasks across your existing stack, agents and any tools run as s
99
* 🧠 Smart pre-evaluated execution plans
1010
* 🎯 Domain grounded
1111
* 🗿 Durable execution
12-
* 🚀 Go fast with tools as services
12+
* 🚀 Go fast and save cost with tools as services
1313
* ↩️ Revert state to handle failures
1414
* ⛑️ Automatic service health monitoring
1515
* 🔮 Real-time status tracking
16-
* 🪝 Webhooks for completion and failure notifications
16+
* 🪝 Webhooks for completion and failure monitoring
1717

1818
[Learn why we built orra →](https://tinyurl.com/orra-launch-blog-post)
1919

@@ -33,6 +33,7 @@ orra coordinates tasks across your existing stack, agents and any tools run as s
3333
- [Docs](#docs)
3434
- [Self Hosting & On-premises Deployment](#self-hosting--on-premises-deployment)
3535
- [Support](#support)
36+
- [Telemetry](#telemetry)
3637
- [License](#license)
3738

3839
## Installation
@@ -96,11 +97,11 @@ Download the latest CLI binary for your platform from our [releases page](https:
9697

9798
```shell
9899
# macOS
99-
curl -L https://github.com/orra-dev/orra/releases/download/v0.2.5/orra-darwin-arm64 -o /usr/local/bin/orra
100+
curl -L https://github.com/orra-dev/orra/releases/download/v0.2.6/orra-darwin-arm64 -o /usr/local/bin/orra
100101
chmod +x /usr/local/bin/orra
101102

102103
# Linux
103-
curl -L https://github.com/ezodude/orra/releases/download/v0.2.5/orra-linux-amd64 -o /usr/local/bin/orra
104+
curl -L https://github.com/ezodude/orra/releases/download/v0.2.6/orra-linux-amd64 -o /usr/local/bin/orra
104105
chmod +x /usr/local/bin/orra
105106

106107
# Verify installation
@@ -291,6 +292,10 @@ Need help? We're here to support you:
291292
- Report a bug or request a feature by creating an [issue](https://github.com/orra-dev/orra/issues/new?template=bug-report-feature-request.yml)
292293
- Start a [discussion](https://github.com/orra-dev/orra/discussions) about your ideas or questions
293294

295+
## Telemetry
296+
297+
See [telemetry.md](./docs/telemetry.md) for details on what is collected and how to opt out.
298+
294299
## License
295300

296301
Orra is MPL-2.0 licensed.

docs/telemetry.md

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Telemetry
2+
3+
To help us improve orra, we collect minimal, privacy-preserving usage analytics. **Telemetry is anonymous by design, and you can easily opt out.**
4+
5+
## What We Collect
6+
7+
- **[Anonymous usage events](../planengine/events.go)** (e.g., server start/stop, project/service changes, execution plan outcomes).
8+
- **Non-identifiable execution plan and project tracking:**
9+
- For events related to projects, we only track a **hashed version of the project ID**.
10+
- For execution plan events, we only track a **hashed version of the execution plan ID**.
11+
- **No personal or sensitive data** is ever collected.
12+
- **No IP addresses** or environment details are sent.
13+
14+
## How IDs Are Handled
15+
16+
- We use a **SHA-256 hash** of each project or execution plan ID.
17+
This ensures IDs cannot be reversed or linked to your actual data.
18+
19+
- Each orra instance also generates a random, anonymous identifier stored locally, used solely to distinguish unique installations.
20+
21+
## How to Opt Out
22+
23+
You can fully disable telemetry at any time:
24+
25+
- **Via environment variable:**
26+
Set the environment variable before starting orra:
27+
```shell
28+
export ANONYMIZED_TELEMETRY=false
29+
```
30+
- **Via `.env` file:**
31+
Add the following line to your project's `.env` file:
32+
```text
33+
ANONYMIZED_TELEMETRY=false
34+
```
35+
36+
## Transparency and Trust
37+
38+
- All telemetry code is open source and can be reviewed [here](../planengine/telemetry.go).
39+
- We follow industry best practices for privacy and transparency.
40+
- Telemetry helps us prioritize features, fix bugs, and ensure orra works reliably for everyone.
41+
42+
**Thank you for helping us make orra better!**

planengine/_env

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,8 +30,18 @@ EMBEDDINGS_API_BASE_URL=https://api.openai.com/v1 # Default for OpenAI, change
3030
# Server port (default: 8005)
3131
# PORT=8005
3232

33+
#========================================
34+
# PDDL CONFIGURATION
35+
#========================================
36+
3337
# PDDL validator path (default: /usr/local/bin/Validate)
3438
# PDDL_VALIDATOR_PATH=/usr/local/bin/Validate
3539

3640
# PDDL validation timeout (default: 30s)
3741
# PDDL_VALIDATION_TIMEOUT=30s
42+
43+
#========================================
44+
# Telemetry CONFIGURATION
45+
#========================================
46+
47+
ANONYMIZED_TELEMETRY=true

planengine/app.go

Lines changed: 16 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -25,13 +25,14 @@ import (
2525
)
2626

2727
type App struct {
28-
Engine *PlanEngine
29-
Router *mux.Router
30-
Db *BadgerDB
31-
Cfg Config
32-
RootCtx context.Context
33-
RootCancel context.CancelFunc
34-
Logger zerolog.Logger
28+
Engine *PlanEngine
29+
Router *mux.Router
30+
Db *BadgerDB
31+
Cfg Config
32+
TelemetrySvc *TelemetryService
33+
RootCtx context.Context
34+
RootCancel context.CancelFunc
35+
Logger zerolog.Logger
3536
}
3637

3738
func NewApp(cfg Config, args []string) (*App, error) {
@@ -125,6 +126,10 @@ func (app *App) Run() {
125126
if err := srv.ListenAndServe(); err != nil {
126127
app.Logger.Info().Msg(err.Error())
127128
}
129+
130+
app.TelemetrySvc.TrackEvent(EventServerStart, map[string]any{
131+
"version": Version,
132+
})
128133
}()
129134

130135
c := make(chan os.Signal, 1)
@@ -162,6 +167,10 @@ func (app *App) gracefulShutdown(srv *http.Server, ctx context.Context) {
162167
app.Logger.Error().Err(err).Msg("Error shutting down plan engine server")
163168
}
164169
app.Logger.Debug().Msg("http: All connections drained")
170+
171+
app.TelemetrySvc.TrackEvent(EventServerStop, map[string]any{
172+
"version": Version,
173+
})
165174
}
166175

167176
func (app *App) RegisterProject(w http.ResponseWriter, r *http.Request) {

planengine/app_test.go

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -47,13 +47,20 @@ func setupTestApp(t *testing.T) (*App, *Project, func()) {
4747
assert.NoError(t, err)
4848
}
4949

50+
telemetrySvc := &TelemetryService{
51+
client: nil,
52+
enabled: false,
53+
logger: logger,
54+
}
55+
5056
plane := NewPlanEngine()
51-
plane.Initialise(context.Background(), db, db, db, db, db, nil, nil, nil, &fakePddlValidator{}, nil, logger)
57+
plane.Initialise(context.Background(), db, db, db, db, db, nil, nil, nil, &fakePddlValidator{}, nil, logger, telemetrySvc)
5258

5359
app := &App{
54-
Router: mux.NewRouter(),
55-
Engine: plane,
56-
Logger: logger,
60+
Router: mux.NewRouter(),
61+
Engine: plane,
62+
Logger: logger,
63+
TelemetrySvc: telemetrySvc,
5764
}
5865

5966
app.configureRoutes()

planengine/compworker.go

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,11 @@ func NewCompensationWorker(projectID, orchestrationID string, logManager *LogMan
4747
}
4848

4949
func (w *CompensationWorker) Start(ctx context.Context, orchestrationID string) {
50+
w.LogManager.planEngine.TelemetrySvc.TrackEvent(EventCompensationStarted, map[string]any{
51+
"version": Version,
52+
"execution_plan_id": HashUUID(orchestrationID),
53+
})
54+
5055
for _, candidate := range w.Candidates {
5156
if err := w.processCandidate(ctx, candidate); err != nil {
5257
w.LogManager.Logger.Error().
@@ -61,6 +66,15 @@ func (w *CompensationWorker) Start(ctx context.Context, orchestrationID string)
6166
if strings.Contains(err.Error(), "expired") {
6267
status = CompensationExpired
6368
logType = CompensationExpiredLogType
69+
w.LogManager.planEngine.TelemetrySvc.TrackEvent(EventCompensationExpired, map[string]any{
70+
"version": Version,
71+
"execution_plan_id": HashUUID(orchestrationID),
72+
})
73+
} else {
74+
w.LogManager.planEngine.TelemetrySvc.TrackEvent(EventCompensationFailed, map[string]any{
75+
"version": Version,
76+
"execution_plan_id": HashUUID(orchestrationID),
77+
})
6478
}
6579

6680
// Log the compensation failure
@@ -109,6 +123,11 @@ func (w *CompensationWorker) Start(ctx context.Context, orchestrationID string)
109123
go w.sendWebhookNotification(webhook, payload)
110124
}
111125
}
126+
} else {
127+
w.LogManager.planEngine.TelemetrySvc.TrackEvent(EventCompensationCompleted, map[string]any{
128+
"version": Version,
129+
"execution_plan_id": HashUUID(orchestrationID),
130+
})
112131
}
113132
}
114133
}

planengine/config.go

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ const (
4141
PauseExecutionCode = "PAUSE_EXECUTION"
4242
StartJsonMarker = "```json"
4343
EndJsonMarker = "```"
44+
AnonymouseIDFilename = "orra.telemetry.uuid"
4445
)
4546

4647
// Supported LLM and Embeddings models
@@ -100,6 +101,7 @@ type Config struct {
100101
PddlValidatorPath string `envconfig:"default=/usr/local/bin/Validate"`
101102
PddlValidationTimeout time.Duration `envconfig:"default=30s"`
102103
StoragePath string `envconfig:"optional"`
104+
AnonymizedTelemetry bool `envconfig:"default=true"`
103105
}
104106

105107
func Load() (Config, error) {

planengine/engine.go

Lines changed: 21 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -51,20 +51,7 @@ func NewPlanEngine() *PlanEngine {
5151
return plane
5252
}
5353

54-
func (p *PlanEngine) Initialise(
55-
ctx context.Context,
56-
pStorage ProjectStorage,
57-
svcStorage ServiceStorage,
58-
orchestrationStorage OrchestrationStorage,
59-
groundingStorage GroundingStorage,
60-
failedCompStorage FailedCompensationStorage,
61-
logMgr *LogManager,
62-
wsManager *WebSocketManager,
63-
vCache *VectorCache,
64-
pddlValid PddlValidator,
65-
matcher SimilarityMatcher,
66-
Logger zerolog.Logger,
67-
) {
54+
func (p *PlanEngine) Initialise(ctx context.Context, pStorage ProjectStorage, svcStorage ServiceStorage, orchestrationStorage OrchestrationStorage, groundingStorage GroundingStorage, failedCompStorage FailedCompensationStorage, logMgr *LogManager, wsManager *WebSocketManager, vCache *VectorCache, pddlValid PddlValidator, matcher SimilarityMatcher, Logger zerolog.Logger, telemetry *TelemetryService) {
6855
p.pStorage = pStorage
6956
p.svcStorage = svcStorage
7057
p.orchestrationStorage = orchestrationStorage
@@ -76,6 +63,7 @@ func (p *PlanEngine) Initialise(
7663
p.VectorCache = vCache
7764
p.PddlValidator = pddlValid
7865
p.SimilarityMatcher = matcher
66+
p.TelemetrySvc = telemetry
7967

8068
if projects, err := pStorage.ListProjects(); err == nil {
8169
p.Logger.Trace().Interface("Projects", projects).Msg("Loaded projects from DB")
@@ -180,6 +168,10 @@ func (p *PlanEngine) RegisterOrUpdateService(service *ServiceInfo) error {
180168
Str("ProjectID", service.ProjectID).
181169
Str("ServiceName", service.Name).
182170
Msgf("Generating new service ID")
171+
p.TelemetrySvc.TrackEvent(EventProjectServiceRegistered, map[string]any{
172+
"version": Version,
173+
"project_id": HashUUID(service.ProjectID),
174+
})
183175
} else {
184176
// Load existing service
185177
existingService, err := p.svcStorage.LoadServiceByProjectID(service.ProjectID, service.ID)
@@ -194,6 +186,10 @@ func (p *PlanEngine) RegisterOrUpdateService(service *ServiceInfo) error {
194186
Str("ServiceName", service.Name).
195187
Int64("ServiceVersion", service.Version).
196188
Msgf("Updating existing service")
189+
p.TelemetrySvc.TrackEvent(EventProjectServiceUpdated, map[string]any{
190+
"version": Version,
191+
"project_id": HashUUID(service.ProjectID),
192+
})
197193
}
198194

199195
if err := p.svcStorage.StoreService(service); err != nil {
@@ -341,6 +337,11 @@ func (p *PlanEngine) ApplyGroundingSpec(ctx context.Context, spec *GroundingSpec
341337
Str("domain", spec.Domain).
342338
Msgf("Added grounding spec with %d action uses cases", len(spec.UseCases))
343339

340+
p.TelemetrySvc.TrackEvent(EventProjectGroundingApplied, map[string]any{
341+
"version": Version,
342+
"project_id": HashUUID(projectID),
343+
})
344+
344345
return nil
345346
}
346347

@@ -415,6 +416,11 @@ func (p *PlanEngine) RemoveGroundingSpecByName(projectID string, name string) er
415416
Str("name", name).
416417
Msg("Removed grounding spec")
417418

419+
p.TelemetrySvc.TrackEvent(EventProjectGroundingRemoved, map[string]any{
420+
"version": Version,
421+
"project_id": HashUUID(projectID),
422+
})
423+
418424
return nil
419425
}
420426

@@ -468,6 +474,7 @@ func (p *PlanEngine) AddProject(project *Project) error {
468474
}
469475

470476
p.projects[project.ID] = project
477+
p.TelemetrySvc.TrackEvent(EventProjectAdded, map[string]any{"version": Version})
471478
return nil
472479
}
473480

planengine/events.go

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
/*
2+
* This Source Code Form is subject to the terms of the Mozilla Public
3+
* License, v. 2.0. If a copy of the MPL was not distributed with this
4+
* file, You can obtain one at https://mozilla.org/MPL/2.0/.
5+
*/
6+
7+
package main
8+
9+
// Tracked telemetry events
10+
11+
const (
12+
EventServerStart = "server_start"
13+
EventServerStop = "server_stop"
14+
EventProjectAdded = "project_added"
15+
EventProjectServiceRegistered = "service_registered"
16+
EventProjectServiceUpdated = "service_updated"
17+
EventExecutionPlanNotActionable = "execution_plan_not_actionable"
18+
EventExecutionPlanFailedCreation = "execution_plan_failed_creation"
19+
EventExecutionPlanFailedValidation = "execution_plan_failed_validation"
20+
EventExecutionPlanAttempted = "execution_plan_attempted"
21+
EventExecutionPlanCompleted = "execution_plan_completed"
22+
EventExecutionPlanAborted = "execution_plan_aborted"
23+
EventExecutionPlanFailed = "execution_plan_failed"
24+
EventCompensationStarted = "execution_plan_compensation_started"
25+
EventCompensationCompleted = "execution_plan_compensation_completed"
26+
EventCompensationFailed = "execution_plan_compensation_failed"
27+
EventCompensationExpired = "execution_plan_compensation_expired"
28+
EventProjectGroundingApplied = "project_grounding_applied"
29+
EventProjectGroundingRemoved = "project_grounding_removed"
30+
)

0 commit comments

Comments
 (0)