Project organization -- version numbers, directory structure, etc.

[tggagne]

Always interested to learn more about packaging best practices (because I haven't found anyone writing about them), I'm curious

• how you decide when to change version
• when is the package version created, before or after the PR merge?
• what concerns or designs went into the directory structure? It's deeper than I expected
• what other open-source packages are worth looking at for structure, versioning, etc.?

[jongpie]

Hi @tggagne I went ahead and converted your issue into this discussion, since it's more about project questions instead of project enhancements/bugs.

And these are some great questions that people don't frequently ask me about, so I'm happy to share some insights! I've added me thoughts below, but please let me know if you have any follow up questions/

How you decide when to change version

In the current directory structure, all of the metadata for the unlocked exists in the nebula-logger/core/ directory. So any time something changes within nebula-logger/core/, I bump the package version number / create a new package version.

When is the package version created, before or after the PR merge?

I personally like to create the package version as part of the PR's build pipeline, and promote it once the PR is merged. My current process is...

• When a PR is created, the pipeline generates a new package version, using the metadata in nebula-logger/core/.
  • The pipeline automatically installs the new package version into an extra developer-edition org that I use specifically for testing Nebula Logger.
  • Any time additional commits are pushed to an open PR, the pipeline will automatically repeat the process of generating a new package version, and installing it into the extra developer-edition org.
  • When the pipeline successfully creates a new package version, the 04t... package version ID is added to README.md and sfdx-project.json
• As part of the PR review process, I manually complete any additional testing that I need to do to ensure the changes are working as expected.
• Once the PR is merged, the pipeline looks for the 04t... package version ID of the unlocked package in README.md, and automatically promotes the package version.

Some people prefer to have their pipeline automatically create & promote a package version after the PR is merged to main, but I like to be able to manually test the changes (as needed), using the exact same package version that will be released to the public.

What concerns or designs went into the directory structure? It's deeper than I expected

Yeah, the directory structure has grown in complexity over time as the project has grown. The current directory structure is based a mix of...

• My own curiosity in experimenting with the ability to setup custom directory structures
  • Using custom directory structures didn't use to be supported in old Salesforce deployment tools/APIs (like the antiquated Force.com Ant Migration Tool). So when this was introduced, I started (and continue to) use Nebula Logger's repo as a place for me to experiment with custom directory structures.
• Nebula Logger is a mono-repo, containing multiple packages (listed below). I've had multiple times over the years where I've debated whether to split the repo into multiple repos, but every time I've decided to keep it as a mono-repo, as it makes things much easier for me to manage (compared to jumping between multiple repos). This has the trade-off of making the repo fairly big, so using a custom directory structure helps bring some order to the chaos.
  • The nebula-logger/core/ directory, containining the main metadata for Nebula Logger. Within this directory, I've broken it down into what I consider the 3 main layers/modules of the project. These layers are also documented in the architecture wiki page. Having these separate directories isn't necessary at all, it could also just be directly in nebula-logger/core/ - but I like to have a little bit of separation between the different layers/modules.
    • nebula-logger/core/configuration/ - as Nebula Logger's features have grown over the years, a lot of people want to be able to customize it, using things like feature flags. Any metadata focused on configuration is stored in this directory.
    • nebula-logger/core/logger-engine/ - this is the core of Nebula Logger that provides the actual logging capabilities for Apex, Flow, Lightning components, and OmniStudio components. It's dependent on the configuration directory.
    • nebula-logger/core/log-management/ - this contains all of the metadata for making Nebula Logger's data available & usable directly within your Salesforce org. It includes the custom objects, custom tabs, custom application, flexipages, custom LWCs, etc. It's dependent on the logger-engine directory and configuration directory.
  • The nebula-logger/recipes/ directory, containing some example Apex classes, Flows, and LWCs. These are used for...
    • Manual testing, when needed, to validate any PR changes
    • Demoing the capabilities of Nebula Logger
    • Providing functional examples of how to use Nebula Logger in real-world scenarios
  • The nebula-logger/plugins/ directory, containing multiple plugins for Nebula Logger, each of which is a separate package that's dependent on the core package.
    • Nebula Logger's plugin framework is a feature in the unlocked package (not currently available in the managed package) that provides a way to extend Nebula Logger, without having to change Nebula Logger's core metdata directly. This provides a way for people to enhance Nebula Logger, while still being able to upgrade.
    • The plugins directory contains the "official" plugins for Nebula Logger that I maintain. They each provide some targeted functionality that either isn't needed for everyone (so I don't want to add it to the core package), or is a feature that I don't want to add to the core package yet (so I can make larger changes to the plugin, without having to worry about breaking changes in the core package).

What other open-source packages are worth looking at for structure, versioning, etc.?

From what I've seen, not a lot of open-source packages have custom directory structures - and for small/simple repos, I don't always use custom directory structures either, I'll often use the typical structure of force-app/main/default.

But, I would definitely recommend looking at Apex Rollup, made by @jamessimone. He also uses a custom directory structure, as well as its own set of plugins. @jamessimone and I have worked together for years (on open-source projects, as well as at multiple companies/jobs), so we have borrowed ideas from each other over the years for things like this.

I know at some point, I had a few other projects bookmarked that also use custom directory structures, but I can't find them right now. I'll try to find them & will send them your way if I can find them.

[tggagne]

I'm confused about the package versioning. It looks like only the Core package is versioned. What is the value of the other packages existing in the same repository with dependencies on Core if each of them doesn't have its own package version? Can they be installed and upgraded separately from Core?

[jamessimone]

They are versioned - as all packages must be - and they can be installed and upgraded separately. Nebula Logger is not yet so large that keeping the plugin packages hosted within the same monorepo is a burden, and aids in discoverability, tooling, and refactoring.

[tggagne]

I figured out the versioning and sfdx-project to juggle multiple "default" paths. I don't know if you're doing it with jq scripts to turn them false/true, but that's how we're approaching it to reduce unnecessary commits to sfdx-project.json.

[jongpie]

Hi @tggagne I'm not quite sure what you mean by juggling multiple default paths - there's only 1 default path, and the default path doesn't have any impact on creating package versions, etc., so I don't have anything that tries to flip the values.