Add support for Dynamic / arbitrary groups of objects

[jedelman8]

Overview

There is often a need to group objects together within a single model or content-type. For example, I may want to group all "primary" devices within pairs, all sites that have a circuit from a given carrier, all devices that terminate a circuit, etc. and the list can go on. The point is there is many reasons it makes sense to group objects together.

Once this feature is complete, it will enable applying config contexts to these groups, custom and computed fields to these groups, webhooks, etc. Anywhere a content-type is being selected, a group could also be applied.

User Story

As Nelly the Network Engineer,
I want to be able to create dynamic groups,
So that I can easily see which devices match a specific query further enabling network automation use cases on those "groups" of devices that often map back to "network services."

I know this is done when:

• I can go to a page called "Dynamic Groups" and view all Groups
• I can click on a Group and view all objects within that dynamic group
• I can create a new dynamic group based on a queryset (this could be expanded in the future)

This is required for #768

TODO

• Implement Dynamic Group Model, Forward, Reverse Lookup (done via: Dynamic Groups feature (phase 1) #1425, Dynamic Groups Feature (phase 2) #1502)
• Add Regex lookup expression to magic FilterSet generator (see Add Regex lookup expression FilterSet fields to autogenerated list #1525)
• Housekeeping: Groom existing FilterSets which are missing all model attributes (see: Assert that FilterSet filter fields are law and derive forms from them automatically #1483)
• Decorate Additional Models for Dynamic Groups Support #1822
• Enhancement: To support advanced query types, review Extensibility to FilterSet and Filter Forms #1470
• Spike: Dynamic Groups Enhancement - Groups of groups, allowing inclusion/exclusion rules of sub-group members #1614

[itdependsnetworks]

I was thinking the UI can be extended for how RBAC creates attribute based access control (ABAC) and used here as well.

[glennmatthews]

I think there's room for some clarification here - in some past conversations "dynamic" groups have implied the ability to define a set of group membership criteria and have the members of the group automatically (dynamically) determined based on those criteria, as opposed to manually defined groups wherein the user creates a group and manually assigns each member into the group. Or to put it another way, once I have dynamic groups defined (based on querysets or other filtering criteria), and I create a new object/record that matches those criteria, is it expected to automatically become a member of all applicable groups, or is it on me as the user to assign it into those groups myself?

[itdependsnetworks]

To clarify what I had in mind, I created the included wireframe.

Some notes:

• Introspection will be non-trivial for
  • What I labeled as FK Model (admittingly not the best of names)
  • What check type can be used
  • Which models can and cannot be aggregated
• I only included the relevant parts of the RBAC (or ABAC) model

[jedelman8]

I think there's room for some clarification here - in some past conversations "dynamic" groups have implied the ability to define a set of group membership criteria and have the members of the group automatically (dynamically) determined based on those criteria, as opposed to manually defined groups wherein the user creates a group and manually assigns each member into the group. Or to put it another way, once I have dynamic groups defined (based on querysets or other filtering criteria), and I create a new object/record that matches those criteria, is it expected to automatically become a member of all applicable groups, or is it on me as the user to assign it into those groups myself?

It should be "dynamic" for this feature. Given #892, that FR may end up being the "static" part of this feature.

@itdependsnetworks with what you are showing, are you just showing that as a way to build the groups as an option vs. quersyet vs. regex, etc. ?

[itdependsnetworks]

This is a potential UI design for queryset

[glennmatthews]

Some initial thoughts/ideas/brainstorming:

• Initial scope might only be single-table within a group (not a group spanning multiple models/tables)
• Potential for a "group of groups" model to allow aggregating groups across tables for convenience of reporting/visualization/API access
• Using dynamic groups for dynamically rendered fields (config context, computed field) is easier/simpler than using them for custom fields since we'd need to create/delete custom field data as group membership changes.

[dgarros]

I've been thinking A LOT about this and I've tried different options in the last few months that I think are interesting for this discussion.
Sorry for the long post but hopefully it will be useful for this discussion

When we are talking about Dynamic Groups, for me it primarily comes down to :

• Capture & Store a query for a given object type.
• Dynamically calculate the association to/from this group when needed.

Based on these requirements, there are multiple aspects to consider:

• Language of the query
• Compatibility with the UI/API
• Performance
• Extensibility / Maintainability

Language of the Query

As far as I know in Nautobot we have 2 types of Query languages :

• Queryset, the language of the ORM
• Filterset, a more user friendly query language that we are using in the UI/API. (from django-filter)

A Filterset can be converted into a Queryset but there is no easy way to automatically convert a Queryset into a Filterset.

Compatibility with the UI & API

There are few situations to consider:

• How will the user define the query in the UI & API
• How can we represent the result of the query in the UI & API

For the second part, the API only supports Filterset so it means that we must convert our query to a Filterset at some point.

Queryset

We are using Queryset type query in few places today:

• Config Context : The query is limited to some criteria exposed via a dedicated form. In the background, the query is stored with M2M relationships in the database.
• Permission System : The query is described in JSON. It allows more options than the config context but the user is still limited to Q arguments.

Filterset

Filterset are used in few places today:

• REST API: Parameters for the rest API to limit the scope of a query are automatically generated from a Filterset
• GraphQL: Query Parameters for GraphQL are automatically generated from a Filterset
• Table Filtering : All table filters are internally using Filterset. A FilterForm is used to map a FilterSet into a user friendly list of dropdown
• Relationship : The filter to limit the scope of a Relationship is described in JSON and map directly to a Filterset.

At the time, I decided to use a Filterset for the relationships because most models have a Filterset already defined and since it's the language of the API it was easy to integrate with the APISelect widget

Performance

The dynamic calculation is interesting because there is really 2 ways to process that :
A. Which objects are part of a given group (Group > objects)
B. Which groups am I part of, from an object point of view (Object > Groups)

A is very important when we want to execute a workflow on a given list of devices and B is important when an object need to retrieve all the properties of the groups it's part of (config Context)

• Queryset can do both A and B but for B the query is pretty advanced, meaning we can't capture it easily from a user input.
• Filterset are really good at A but do not support B. As far as know, if we have a list of N Filterset, we need to execute N queries to understand which groups a given object is part of.

Conclusion

To conclude this long brain dump

Both Filterset and Queryset have some pros and cons to create a dynamic group and I think the idea solution should be a mix of both.
We need a solution that can :

• Be defined via a friendly user interface in the UI
• Be converted into a Filterset if needed
• Be easily extended from a plugin
• Dynamically calculate the associations between objects and groups in both direction with good performance.

I have some ideas that I would like to try that I think would match all these criteria

Please me know if I'm missing an important part or if I got something wrong
Thanks for reading

[mzbroch]

To add to @glennmatthews 's brainstorming point about group of groups, as the plugin author I would like for following:

1. "Group Expressions"

Allow for "group expressions". Maintaining the relationships between groups quickly become cumbersome, thus I would like to allow my plugin for following queries:

• if device IN Group1 AND IN Group2
• if device IN Group1 AND NOT IN Group2
• if device IN Group1 AND NOT (device in Group2 and device in Group3)
• etc

2. "Static Groups"

Allow to statically group instances of objects. Statically groups objects are to fill the use case of exceptions. Once most of the objects would be handled via dynamic groups, we should consider how to handle exceptions too.

3. "Interface for plugins"

I think we are missing the point what kind of interface should be provided for plugin authors. The goal is to simplify the developer's experience, thus developer's should not be re-implementing the object groupping / exceptions etc. As of today, some of the plugins are already reported for the improvements in terms of how objects are selected.

[jedelman8]

@mzbroch regarding static groups, the thought was we would at a minimum have RegEx to build dynamic groups and thus, could have multiple RegEx's using exact matches yielding what would be a static group.

Group expressions is an interesting idea. I'm sure there are multiple ways to do this, but will remind us of Ansible inventory patterns. Maybe this is an add-on or power-use method for expressions in the future.

@dgarros Thanks for the PR on the prototype (#1047 ). The initial thought was to have a UI similar to what @itdependsnetworks proposed above with a row per filter criteria. The filter criteria could be a Queryset or RegEx (or more precise using the interface you have in the screen shots that is similar to config contexts). Is this inline with what you had in mind as well?

Will defer to @glennmatthews @jathanism @lampwins on the implementation of your prototype!

Thanks @dgarros !

[glennmatthews]

Another place where dynamic groups would be useful would be in adding a feature to allow webhooks to be triggered only for objects in a specific dynamic group.

[glennmatthews]

An attempt at consolidating the requirements described in this issue thus far together with some additional requirements identified through internal discussions and documentation:

• A dynamic group must automatically reflect changes in membership, i.e. object creation/change/deletion should automatically be reflected in its group membership.
• A dynamic group can be defined based on something resembling a filterset and/or queryset, specifically:
  • Must be able to define a dynamic group based on "include"/"match" constraints
    • These constraints could potentially include exact match, substring match, regex match, greater-than/less-than match, etc.
  • Must be able to define a dynamic group based on "exclude"/"no-match" constraints
    • As above, potentially many different kinds of such constraints
  • Must be able to define a dynamic group based on multiple constraints (match A and match B, match A and exclude C, exclude C and exclude D)
  • Must be able to define a dynamic group based on membership in other groups ("belonging to group A or group B", "belonging to group A but not group C", etc.)
• Performance of "given an object, find all dynamic groups it belongs to" must be reasonable
• Performance of "given a dynamic group, find all objects that belong to it" must be reasonable
• UI/UX for defining (creating and editing) dynamic groups must be reasonably user-friendly
• UI must present a listing of all defined dynamic groups
  • Filtering this listing by content-type (i.e., only show groups of Devices, or of Sites) must be supported
• UI must present a consistent way of showing all members of a given dynamic group
• UI must present a consistent way of showing all dynamic groups an object is a member of
• REST API must present a consistent way to query for members of dynamic groups and group membership of an object
• REST API must be capable of filtering retrieved objects based on their group membership
• Dynamic groups of plugin-defined model instances must be possible/supported

Non-requirements at this time:

• Definition of dynamic groups containing multiple different content-types (the biggest driver for this would be Devices/Virtual Machines, and that's better addressed by Unify Device and VirtualMachine in a single table. Differentiate between the 2 using is_virtual flag. #1178)

[itdependsnetworks]

@bryanculver I understood "Dynamic / arbitrary groups of objects" was not making it into 1.3 but my understanding was that all functionality was going to be in 1.4, is that not the case?

As an example, this refers to #1483 which seems to have been added to 1.4 and removed.

[bryanculver]

@bryanculver I understood "Dynamic / arbitrary groups of objects" was not making it into 1.3 but my understanding was that all functionality was going to be in 1.4, is that not the case?

As an example, this refers to #1483 which seems to have been added to 1.4 and removed.

Dynamic groups were added in 1.3.

Dynamic groups of groups is currently being considered for 1.4. Currently there is a spike and prototype ongoing as to how to implement this, see: #1614.

Please be mindful that Epics are likely not going to ever be marked for a milestone in entirety because they often contain more stories and work, which may even include breaking changes, than can fit in a reasonable release window.

Paging @lampwins.

[bryanculver]

Dynamic Groups was added in 1.3. Groups of Dynamic Groups was added in 1.4. Future follow-ons can be followed in #3268