Skip to content

Closes: #31 - Polymorphic object fields#442

Merged
arthanson merged 60 commits into
featurefrom
31-polymorphic-object-fields
May 7, 2026
Merged

Closes: #31 - Polymorphic object fields#442
arthanson merged 60 commits into
featurefrom
31-polymorphic-object-fields

Conversation

@bctiemann

@bctiemann bctiemann commented Apr 2, 2026

Copy link
Copy Markdown
Contributor

Closes: #31

This PR addresses a major feature addition: polymorphic object/multi-object fields (using a GenericForeignKey pattern). This means that the related object type for these fields can be any of a selected list, rather than just a single related object type.

Screenshot 2026-04-01 at 8 43 41 PM

Editing a custom object with such a field shows an individual field in the edit form for each object type supported by the object/multiobject field, allowing the selection lists to work with existing object-type-based selection lists:

Screenshot 2026-04-01 at 8 44 37 PM

Unit tests have been added covering all UI and API routes, including validating that only the selected related object types can be assigned to the field.

NOTE:

        # KNOWN LIMITATION: polymorphic fields (is_polymorphic=True) store their
        # allowed target types on the related_object_types M2M, not on the
        # related_object_type FK.  This DFS therefore does not traverse edges
        # introduced by polymorphic fields.  A cycle that passes entirely through
        # polymorphic legs (e.g. A →(poly) B →(poly) A) will go undetected.
        #
        # Fixing this requires also iterating field.related_object_types.filter(
        # app_label=APP_LABEL) and recursing into each.  The check_polymorphic_recursion
        # signal already guards the direct A→B assignment, but cannot see multi-hop
        # cycles that depend on polymorphic fields already on intermediate types.

Comment thread netbox_custom_objects/api/serializers.py Fixed
@bctiemann

Copy link
Copy Markdown
Contributor Author

@claude /review

@claude

This comment was marked as outdated.

@bctiemann

Copy link
Copy Markdown
Contributor Author

@claude /review

@claude

This comment was marked as outdated.

@bctiemann

Copy link
Copy Markdown
Contributor Author

@claude /review

@claude

This comment was marked as outdated.

@bctiemann

Copy link
Copy Markdown
Contributor Author

@claude /review

@claude

This comment was marked as outdated.

@bctiemann

Copy link
Copy Markdown
Contributor Author

@claude /review

@claude

This comment was marked as outdated.

@bctiemann

Copy link
Copy Markdown
Contributor Author

@claude /review

@claude

This comment was marked as outdated.

@bctiemann bctiemann requested a review from arthanson April 30, 2026 11:27

@arthanson arthanson left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes per mark - need to list the type of field next to each item on the detail view.

bctiemann and others added 2 commits May 4, 2026 14:46
…ds in detail view

For MultiObject fields with is_polymorphic=True, the linked-objects card on
the custom object detail view now includes a Type column header and shows
each linked object's model verbose_name alongside the link. Non-polymorphic
multi-object fields are unchanged (single-column layout).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Resolves import conflict in views.py: combined ProtectedError/RestrictedError
from feature with redirect/restrict_form_fields/form utilities from HEAD.

Migration linearization:
  - Feature's 0010_backfill_base_columns and 0011_non_deferrable_fk_constraints
    keep their numbers (depend on 0009)
  - 31 branch's 0010_polymorphic_object_fields renamed to 0012 (depends on 0011)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
bctiemann and others added 9 commits May 4, 2026 17:22
Conflict resolutions:
- serializers.py: add to_representation() (suppress on_delete_behavior for
  non-Object types) and create() from feature
- forms.py: use get_field_value() for TYPE_MULTIOBJECT check (feature's
  more robust HTMX-aware helper)
- models.py: keep both _original_is_polymorphic (31 branch) and
  _original_on_delete_behavior (feature); add on_delete_behavior reset
  before polymorphic-aware recursion check; keep MultiObject through-table
  renaming logic from 31 branch
- exception handler in _ensure_field_fk_constraint: adopt feature's
  logger.error with named exception

Migration linearization:
  0009 → 0010_backfill_base_columns → 0011_non_deferrable_fk_constraints
       → 0012_fix_fk_and_on_delete_behavior → 0013_polymorphic_object_fields

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
clear_graphql_cache.js, docs/graphql.md, graphql_cache_clear.html, and
management/commands/refresh_graphql.py are unrelated to the polymorphic
object fields feature and were picked up during branch merges.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The trivial super() stub introduced during the feature merge conflict
resolution shadowed the real atomic create() below it.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
is_polymorphic and related_object_types are intentionally omitted from
the portable schema pipeline for now; full support (list encoding/decoding,
M2M apply logic) is deferred to a follow-up issue.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wires is_polymorphic and related_object_types into the full export/
diff/apply cycle:

format.py
- Add is_polymorphic and related_object_types to FIELD_TYPE_ATTRS for
  object and multiobject types
- Add is_polymorphic: False to FIELD_DEFAULTS for omit-on-default export

exporter.py
- Emit is_polymorphic: true and related_object_types list for polymorphic
  fields; omit singular related_object_type for those fields
- Prefetch related_object_types M2M on the field queryset

comparator.py
- Compare is_polymorphic bool and related_object_types sorted list
- Add prefetch_related for M2M on the field queryset in diff_cot
- Expand the COT slug cache pre-fetch to cover M2M targets too

executor.py
- Add is_polymorphic to _schema_def_to_field_kwargs scalar kwargs
- Set related_object_types M2M after field.save() in _apply_field_add
- Handle related_object_types M2M in _apply_field_alter (deferred until
  after save() with a pending_m2m dict)
- Walk related_object_types list in _build_dep_order for polymorphic
  cross-COT dependency ordering
- Add _resolve_related_object_types() list helper

cot_schema_v1.json
- Update object_field and multiobject_field defs to support both
  polymorphic (is_polymorphic + related_object_types required) and
  non-polymorphic (related_object_type required) variants via if/then/else

tests
- Add create_polymorphic_field() and get_prefix_object_type() to base
- Add ExporterPolymorphicTestCase (export, elision, round-trip)
- Add ComparatorPolymorphicTestCase (clean round-trip, change detection)
- Add ExecutorPolymorphicFieldTestCase (ADD, ALTER, dep ordering)
- Add polymorphic dep-order pure-logic test to ExecutorBuildDepOrderTestCase
- Add JSON Schema validation tests for polymorphic fields
- Remove is_polymorphic / related_object_types from _SCHEMA_EXCLUDED_ATTRS

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ry entries

Two bugs were causing test failures when running the full suite:

1. test_stale_registry_entry used manual savepoint/savepoint_rollback which
   raised TransactionManagementError in Django 5.2 because needs_rollback was
   True when validate_no_broken_transaction() ran.  Fix: use transaction.atomic()
   whose __exit__ clears needs_rollback before issuing ROLLBACK TO SAVEPOINT.

2. CustomObjectsTestCase.tearDown() fell back to {} when _plugin_model_snapshot
   was absent (test classes that override setUp() without super()).  This caused
   all models — including static ones like CustomObjectType — to be deleted from
   apps.all_models, corrupting the registry for every subsequent test.  Fix:
   fall back to None and early-return to skip cleanup when setUp() was bypassed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- models.py clean(): reset on_delete_behavior to SET_NULL for polymorphic
  TYPE_OBJECT fields. GFK columns use a hardcoded SET_NULL content_type FK
  with no real DB constraint; storing cascade/protect created a false
  impression those semantics would be enforced.

- models.py _ensure_all_fk_constraints(): add is_polymorphic=False to the
  queryset filter so polymorphic GFK fields are not visited. Previously the
  loop called _ensure_field_fk_constraint() for each polymorphic field, which
  silently no-oped (GFK has no remote_field) but was wasteful and misleading.

- api/serializers.py validate(): remove the try/except that swallowed
  ValidationError during the related_object_types_input immutability pre-check.
  An unresolvable type is invalid regardless of immutability; re-raise
  immediately so the user gets the correct error message rather than a
  deferred, differently-worded one from the normal validation path.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Polymorphic fields allow references to objects of multiple types.  A single
ModelChoiceFilter/queryset cannot cover heterogeneous targets, so one filter
is generated per allowed content type, named {field}_{app_label}_{model}
(e.g. location_dcim_device, location_dcim_site).

filtersets.py
- Add PolymorphicObjectFilter(ModelChoiceFilter): filters by matching both
  {name}_content_type_id and {name}_object_id columns on the GFK.
- Add PolymorphicMultiObjectFilter(ModelMultipleChoiceFilter): queries the
  polymorphic through table for matching (content_type_id, object_id__in)
  rows and returns the union of their source_ids (no duplicates).
- Add _build_polymorphic_filters(): produces the per-type filter dict.
- Change build_filter_for_field() return type from Optional[Filter] to
  dict[str, Filter]; non-polymorphic fields still return {field.name: filter}.
- Update get_filterset_class() to attrs.update(build_filter_for_field(field)).

field_types.py
- ObjectFieldType.get_filterform_field(): return a dict of per-type
  DynamicModelChoiceField instances for polymorphic fields instead of None.
- MultiObjectFieldType.get_filterform_field(): same, using
  DynamicModelMultipleChoiceField.

dynamic_forms.py
- build_filterset_form_class(): handle dict returns from get_filterform_field
  by updating attrs rather than assigning to field.name.

tests/test_filtersets.py
- PolymorphicObjectFilterFormFieldTestCase: form field shape assertions.
- PolymorphicObjectFiltersetTestCase: queryset filtering by device/site,
  including cross-type mismatch returning no results.
- PolymorphicMultiObjectFiltersetTestCase: queryset filtering with OR
  semantics, multi-value union, no-duplicate guarantee.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
bctiemann and others added 2 commits May 5, 2026 08:28
PolymorphicObjectFilter and PolymorphicMultiObjectFilter previously
inherited from ModelChoiceFilter/ModelMultipleChoiceFilter. NetBox's
get_additional_lookups() isinstance-checks for those base classes and
then validates the filter name against real model fields, raising
ValueError for virtual names like targets_dcim_device.

Switching to django_filters.Filter with explicit field_class bypasses
that isinstance check so get_additional_lookups returns {} without
attempting field validation.

Also tighten BuildFilterForFieldGuardsTestCase assertions (returns {}
not None) and relax test_wrong_type_filter_returns_nothing to
assertNotIn rather than assertEqual([], ...) since coincidental PK
matches between Device and Site can legitimately surface obj_site.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ld validation failures

ModelChoiceField validates submitted PKs against its queryset; when the
PK does not match any object of the filter's type (e.g. submitting a
device PK to a site filter) the field raises ValidationError, the key
is absent from cleaned_data, and the filter is silently skipped —
returning the full queryset instead of an empty one.

Switch PolymorphicObjectFilter to IntegerField and PolymorphicMultiObjectFilter
to a custom _IntegerListField that accepts a list of raw integer PKs.
The content_type_id is baked in at construction time, so the DB-level
filter (content_type_id=X AND object_id=value) already enforces type
isolation correctly without any model-queryset lookup.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@bctiemann

Copy link
Copy Markdown
Contributor Author

@claude /review all changes since the last review

@claude

claude Bot commented May 5, 2026

Copy link
Copy Markdown

Claude finished @bctiemann's task in 2m 14s —— View job


Code Review: Changes Since arthanson's Last Approval (2026-05-04)

  • Gather git log to identify changes since last review
  • Review changed files
  • Compile final review

Overview

10 commits landed after arthanson's approval, spanning four functional areas:

Commit Area
381ea32 Serializer bug fix — duplicate create() stub removal
b546660 on_delete_behavior correctness for polymorphic GFK fields
f47780c Test infrastructure — CI isolation for --keepdb runs
5898cc7 Schema pipeline — full polymorphic export/diff/apply support
ef568f7 Ruff lint fixes
746cbb0 Filterset — per-type filters for polymorphic fields
1595ed2 Filterset — inherit from Filter not ModelChoiceFilter
233eb77 Filterset — use IntegerField to prevent ModelChoiceField validation bypass

Bugs / Issues Found

1. _apply_field_alter does not skip related_object_types immutability guard — executor.py:348-350

The executor calls field.save() to apply pending_m2m["related_object_types"], which triggers CustomObjectTypeField.clean(). The model-level clean calls check_polymorphic_recursion. However, unlike _apply_field_add (which is a fresh save), _apply_field_alter sets related_object_types after field.save(). If the model's clean() or signal guard rejects the M2M change, the field row is left in a half-altered state (scalar attrs saved, M2M not updated). This is a pre-existing edge case from the main PR, and the executor's _apply_field_alter approach actually handles it as well as could be expected without wrapping in atomic(). Low severity, worth a comment.

2. _apply_field_add in executor.py is not atomic — executor.py:297-304

field = CustomObjectTypeField(**kwargs)
field.save()
# Wire up M2M related_object_types for polymorphic fields (must be after save).
if fc.schema_def.get("is_polymorphic"):
    rot_list = fc.schema_def.get("related_object_types", [])
    if rot_list:
        field.related_object_types.set(_resolve_related_object_types(rot_list))

If _resolve_related_object_types() or field.related_object_types.set() raises (unknown type, signal rejection), the field row exists in DB with is_polymorphic=True but zero related_object_types. This duplicates the pre-existing concern flagged by arthanson. Given the immutability guards on related_object_types, this is a data-integrity risk. Consider wrapping in transaction.atomic() and deleting the field on exception. Fix this →

3. PolymorphicMultiObjectFilter.__init__ does not call kwargs.setdefault("required", False)filtersets.py:102-105

PolymorphicObjectFilter correctly calls kwargs.setdefault("required", False) at line 76. PolymorphicMultiObjectFilter does not, so if a caller doesn't pass required=False explicitly the field inherits the django_filters.Filter default (required=False is actually the default, so this is low-risk, but it's inconsistent and could bite on a NetBox FilterSet subclass override). Minor.


Code Quality — Positives

Filterset design (746cbb0, 1595ed2, 233eb77) is well-reasoned:

The three-commit progression from ModelChoiceFilterFilterIntegerField as the base shows good iteration. The final design is correct: baking content_type_id at construction time and doing two-column filtering ({name}_content_type_id AND {name}_object_id) is the right approach for GFK filtering. The explanatory docstring in PolymorphicObjectFilter about why IntegerField is used over ModelChoiceField is excellent.

_IntegerListField (filtersets.py:24-48):
Clean implementation. The validate() override explicitly allowing empty lists prevents false positive required-field errors on inactive filters.

Schema pipeline (5898cc7) is thorough:

  • if/then/else in JSON Schema correctly encodes the branching requirement (polymorphic → related_object_types required; non-polymorphic → related_object_type required).
  • _build_dep_order correctly walks both the singular related_object_type string and the list related_object_types — dependency ordering for polymorphic cross-COT references is properly handled.
  • pending_m2m pattern in _apply_field_alter cleanly separates save-time vs post-save operations.
  • Test coverage is comprehensive: export round-trips, comparator change detection, executor ADD/ALTER, dep ordering.

b546660 correctness:
Adding is_polymorphic=False to _ensure_all_fk_constraints is the right fix — previously iterating over polymorphic fields and silently no-op'ing on them was wasteful and misleading. The on_delete_behavior normalization to SET_NULL for polymorphic fields with the explanatory comment is exactly the right approach.

Test isolation fix (f47780c):
Replacing transaction.savepoint() / savepoint_rollback() with transaction.atomic() in the stale-table test is the correct fix. The comment in test_models.py explaining why raw savepoint rollback failed (validate_no_broken_transaction() while needs_rollback=True) is exemplary.

Serializer cleanup (381ea32):
Removing the trivial create() stub that shadowed the real atomic create() below it is a correct and important fix.


Minor

exporter.py:133related_object_types.all() in _export_field:

export_cot() already calls prefetch_related("related_object_types") on the field queryset (line 197), so calling .all() inside _export_field hits the prefetch cache rather than the DB. This is correct. No issue.

PolymorphicMultiObjectFilter.filter() deferred imports (filtersets.py:110-111):

The from django.apps import apps as django_apps and from netbox_custom_objects.constants import APP_LABEL are inside filter() to avoid circular imports. This is the established pattern in the codebase and is fine, but consider moving them to module level if the circular import concern is resolved.


Summary

The post-approval commits represent high-quality, well-tested completion work. The filterset support is correctly implemented and addresses the remaining open feature gap. The schema pipeline integration is thorough. The main actionable item is wrapping the executor _apply_field_add M2M assignment in transaction.atomic() to avoid orphaned polymorphic fields on failure.

executor.py: wrap _apply_field_add's field.save() + M2M assignment in
transaction.atomic() so a failure during _resolve_related_object_types
or .set() rolls back the field row, preventing orphaned polymorphic
fields with is_polymorphic=True but empty related_object_types.

filtersets.py: add kwargs.setdefault('required', False) to
PolymorphicMultiObjectFilter.__init__ for consistency with
PolymorphicObjectFilter.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@bctiemann

Copy link
Copy Markdown
Contributor Author

Issues 2 and 3 have been addressed. The rest are low-impact and left as-is.

All # noqa: PLC0415 deferred imports in filtersets.py, executor.py,
comparator.py, and api/views.py were precautionary rather than
addressing a real circular dependency or apps-registry timing issue.
models.py imports nothing from the schema subpackage, so there is no
import cycle.

Moved to module level:
- filtersets.py: django.apps.apps, netbox_custom_objects.constants.APP_LABEL
- executor.py: core.models.ObjectType, extras.models.CustomFieldChoiceSet,
  netbox_custom_objects.models.{CustomObjectType,CustomObjectTypeField},
  netbox_custom_objects.schema.comparator.{FieldOp,diff_document}
  (also removed the duplicate 'from django.db import transaction' that was
  already present at module level)
- comparator.py: netbox_custom_objects.models.CustomObjectType
- api/views.py: schema.comparator.diff_document, schema.executor.*

The deferred import in exporter.py (CustomObjectType inside
_encode_related_object_type) is intentionally kept so the module can be
loaded independently of the Django app stack in standalone unit tests.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@bctiemann bctiemann requested a review from arthanson May 5, 2026 14:41

@arthanson arthanson left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you create a poly field, link it to an object then go to that object it doesn't show that it is linked.

Note I have Butler Communications linked in this custom object:
Image

But Butler Communications is showing now custom objects linked:
Image

bctiemann and others added 2 commits May 6, 2026 19:19
Both the UI panel (template_content.py CustomObjectLink.left_page) and
the API endpoint (LinkedObjectsView) previously only queried
related_object_type (singular FK), missing polymorphic fields entirely.

Polymorphic fields use related_object_types (M2M) and store values via:
- TYPE_OBJECT: {name}_content_type_id / {name}_object_id columns
- TYPE_MULTIOBJECT: through table with (source_id, content_type_id,
  object_id) instead of non-polymorphic (source_id, target_id)

Both locations now also query related_object_types=content_type and
dispatch to the correct lookup path based on field.is_polymorphic.

Tests: LinkedObjectsAPITest gains two new cases for polymorphic GFK
and multiobject fields. CustomObjectLinkPanelTest is a new class
exercising the UI panel data-gathering logic directly, covering
non-polymorphic FK, polymorphic GFK, polymorphic M2M, and the
unrelated-object-not-returned case.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@arthanson arthanson merged commit 69afc21 into feature May 7, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow GenericForeignKey Custom Object Type Fields

3 participants