[feat] richer directory structure

[zkurtz]

Product

BAML

Problem Statement / Use Case

I wonder if it's too late to refine the default directory structure:

baml_client/
baml_src/
   generators.baml
   clients.baml
   extractor1.baml
   extractor2.baml
   ...

This is really a collection of three problems:

1. nit: repetitive top level naming (baml_*)
2. nit: lack of separation between baml_src config files and extraction scripts
3. forced grouping of baml scripts vs arbitrary use case directories

Proposed Solution

Nit 1: it would be a little less repetitive to have the top level split like

baml/
   client/
   src/

Nit 2: "clients" and "generators" are qualitatively different than all the other source files, so it would be nice to separate them, something like

baml_src/
   config/
      generators.baml
      clients.baml
   extractor1.baml
   extractor2.baml
   ...

and/or allow the extractors to go in a separate subdirectory.

Finally, it would be especially nice to support co-locating extractor.baml scripts with the source code that grows around it to support its development and usage instead of requiring them all to be in a centralized baml directory (i.e. a pythonic path-based name space instead of a global namespace). I imagine something like the following should be doable: have a baml_src/sourcefiles.baml config where users can "register" fullpaths to extractors located at arbitrary relative filepaths; then the client is still able to locate and work with them.

Alternative Solutions

No response

Additional Context

No response

[linear]

BAML-538

[sxlijin]

baml_src/
   config/
      generators.baml
      clients.baml
   extractor1.baml
   extractor2.baml

is perfectly possible! You can put generators and clients anywhere in your .baml files, they don't have to be locked to generators.baml or clients.baml - it's just that way in the example code we bootstrap for you.

[sxlijin]

Re baml_client - generators allow you to control the output directory location using the output_dir field, so if you set output_dir ../../skunkworks/python/llms we'll put the generated python baml client there.

[sxlijin]

On the last point of colocating baml_src with the associated application code - we hear you on that. We're holding off on this one until we tackle namespacing, because we see the two problems (directory structure and modules/namespacing) as intrinsically intertwined.

[sxlijin]

Answered across multiple comments, but hopefully this gives some insight into our thought processes!

[zkurtz]

You can put generators and clients anywhere in your .baml files, they don't have to be locked to generators.baml or clients.baml - it's just that way in the example code we bootstrap for you.

Confirmed, thank you!