Native IFC and asynchronous workflows

[brunopostle]

@theoryshaw suggested that it would be a good idea to bring this discussion here.

First, some background, you can skip if you have heard all this before:

The Bonsai BIM tool is a Native IFC application, ie. instead of working with an application-specific data model, Native IFC tools operate by editing and authoring IFC models directly - rather than relegating IFC to a reference or archiving role.

Specifically, in a Native IFC application such as Bonsai BIM or FreeCAD, the data is serialised in a consistent format - to the extent that an edited and saved IFC file will be identical to the original file (aside from just those entities that have been edited).

IFC-SPF is a line-oriented file format, so this lends itself nicely to storage in a version control system such as Git. We find that because files are only partially updated and that repositories are compressed, a Git repository recording a full history of hundreds of discrete changes can be smaller than the raw IFC file itself.

A unexpected development that was that a combination of this parsimonious approach to file handling, and a quirk of the SPF file format, gives us an asynchronous collaboration workflow practically for free - we can have multiple authors working on the same IFC model without requiring a continuous connection to a central database.

Twenty-first century software development is characterised by this kind of asynchronous working. The reason we use Git for software development is that two authors can make independent changes to the same file, 'forking' it, and then what is called a three-way-merge can resolve these changes into a single combined result - unifying the forked 'branches'.

We can do exactly the same thing with IFC, in fact in many ways the process is more robust because in an IFC-SPF file every entity is fully addressable due to the STEP-IDs (the #12434 numbering you see when you open one in a text editor). Three-way-merging software sourcecode can easily result in invalid and broken files, but we don't have that problem because IFC-SPF files are well-structured and this kind of corruption is hard to achieve. For a live demonstration of this in action, see this video presentation using Bonsai BIM (at the time called BlenderBIM), you can browse a copy of the IFC repository used in the demonstration here.

That was an overview of the situation, hopefully the links above give more comprehensive information.

IFC5 is text based, the JSON encoding and other features have lots of potential. Native IFC editing will clearly continue to work, and storage in version control systems such as Git will be fine. However it would be a major backwards step if this asynchronous collaboration isn't supported (and potentially improved-on).

IFC5 does away with STEP-IDs, the file format appears to be one big anonymous list, but one where the order of entities is significant.

Entities have a 'name' attribute which resembles the SPF GUID, but it isn't unique: the hello-wall sample has six entities with the name N93791d5d5beb437bb8ec2f1f0ba4bf3b. Does each of them override or add-to information on the previous entity with the same name?

How do we address these entities? This is important to be able to track changes. Do we have to refer to the 'fifth N93791d5d5beb437bb8ec2f1f0ba4bf3b entity? What if some software deletes the second? (but then the fifth entity is now the fourth).

Is this intended to be an append-only file format? I can see that this would have some benefits, as the entire history could be reconstructed by trimming later entities? Is multi-user collaboration intended by simply appending all changes from all sources, with the most recent arrival 'winning'?

[yorikvanhavre]

Where can we see the hello-wall example you are referring to, @brunopostle ?

[brunopostle]

It's here: hello-wall.ifcx.

[aothms]

I think an important aspect in this discussion is that IFC5 borrows the concepts of layers from USD. So there is less of a need to work around express's limitations on not supporting relationships outside of file boundaries. Asynchronously created models can remain independent layers and folded into the same 'stage' conflict-free by means of selecting the layer order, while still allowing rich relationships and data integration by means of the central tree.

Of course this is only one possible way of working.

Working towards a single layer by means of explicit conflict resolution for the entire stage is still possible, either line-based or more maybe more object-model based. I think the models proposed in this repo work really well for that because the model is guaranteed to be a flat list, so no arbitrarily deep diffs.

The duplicate names you see are specific 'components' that all are to be superimposed/composed to form the complete definition of that node ({'def': 'over'}). These are namespaced, so the pair (N93791d5d5beb437bb8ec2f1f0ba4bf3b, 'ifc:class') maybe could be unique. But whether that is something that would be enforced is not clear to me.

Is this intended to be an append-only file format?

It could indeed be like that. If you look at USD, they indeed don't have a deletion operation specifically, but with an active=False (or similar) flag they can 'remove' subtrees from the model. So indeed that is akin to something append only. But of course you could also remove data from the model if you 'own' the layer.

The basis in all of this is very flexible and allows for multiple ways of working.

Also see the basic viewer prototype that is online now: https://ifc5.technical.buildingsmart.org/viewer/

[yorikvanhavre]

If I understand correctly, the json format allows things the step/express format wouldn't. But how does that translate in file formats support? There is no plan to "remove" the IFC-STP format, right? Right? 😅

If not, then how will that work? Will there be certain features that cannot be translated to IFC_STP?

[aothms]

I know what you mean. People can get emotionally attached to the craziest things ;) The step file grammar is nice and elegant, but for the majority outside of people like us it's an impediment to use IFC when it doesn't come with a builtin parser in your favourite programming language.

[brunopostle]

@yorikvanhavre the overlay system won't survive translation back to STEP, and I guess that since all attributes are named rather than part of a fixed length list, any custom attributes you add won't survive either.

I find the JSON incredibly hard to read, but once I converted the sample to YAML with yq it was much clearer what is going on as this is much less verbose. Also note that the trailing ']' in the JSON effectively prevents using this as a truly append-only file format - YAML doesn't have this problem.

[aothms]

I think that's an interesting suggestion. YAML also adds support for references (but not to a named path it seems, only to explicit anchors). Maybe TOML is more in favour these days, but I never really bother to understand the differences.

I'm also a fan of https://jsonlines.org/ because of the streaming/appending you mentioned, but I guess it wouldn't work well for this application area.

[brunopostle]

I guess with JSON any appended data could be a new list starting with [, this would allow atomic updates, ie. if you wanted to wind back it would be clear what chunk needed to be removed.

[theoryshaw]

How about GUIDs for all entities?

[aothms]

So, regarding terminology. The entities in ECS exist only by components referencing them by means of their ID. Whether components also need to have a surrogate identity was not really concluded actually. I'm personally not convinced that the majority of authoring tools would be able to maintain these identities, but probably an optional provision for it should still be there. It would probably reassure quite some people's minds. Sent from a mobile device, excuse my brevity. Kind regards, Thomas Op vr 22 nov 2024 17:22 schreef Ryan Schultz ***@***.***>:

…

How about GUIDs for all entities <https://chatgpt.com/share/6740ae19-4afc-8013-ace2-5fc20e44c867>? — Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAILWV3LV2TEQ2RN4O3TZDL2B5K4RAVCNFSM6AAAAABR4F2MSGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJUGE2DEOJQHE> . You are receiving this because you commented.Message ID: ***@***.***>

[brunopostle]

@aothms what does ECS refer-to here? I'm not good with all these acronyms.

If a project is defined by a cloud of linked files providing overlays, how is this configured? Does each file refer to its parent internally, or is there a separate configuration 'file' that defines the inheritance relationships?

[aothms]

Sorry, ECS is Entity-Component-System here.

That's a very good and relevant question though. In USD it's as you describe (but reversed) layers have explicit sublayers so in the end you load only one file. Currently the working hypothesis is that AEC is a bit more fragmented (anarchaic?) and that it's more flexible to just let every end-user individually compose their view of the state of the project adhoc locally by determining their own (non-hierarhical) layer order.

[brunopostle]

How would this system deal with changes in a parent that break children?

eg:

1. A parent file creates and uses a material (or a type etc..)
2. A child file uses this material
3. The parent file has no more need for the material and removes it

The child file now has a broken reference to a nonexistent material.

(ifcmerge will break under the same circumstances, but Bonsai just ignores the error)

[aothms]

I would say it's up to the software. There's quite some possible scenarios:

• Both authors used a def, for example in case of a building storey. Author 1 deleting their def essentially makes no difference as the def of author 2 still exists.
• Author 2 used an over. in this case author 2's opinions are essentially not applied onto the tree, but this is detectable.
• The relationship is not a matter of defs and overs, but it's a schema datatype (the spaceboundaries in the hello wall example for ex).

[tomi-p]

Is there also some kind of json-zip solution to be developed? We already have a problem with IFC files as the amount of data they contain has grown (and is still growing). IFC files larger than 1 Gb are by no means exceptional these days. The json format is going to increase the size of raw files many times over. Although network speeds have increased, files of several gigabytes in size will be a problem. And while the vision is probably API-based data transfer, files will certainly be used well into the future. What is the plan?