Support for embedded subgraphs

[chadrik]

When thinking about the ability for one graph to load another graph within it (i.e. subgraphs), it seems to me that the storage of such graphs can take 3 forms:

1. graphs published and consumed as components: often defined as code, and thus not editable within a graph editor UI.
2. external graph references: graphs saved externally to the parent graph (i.e. a separate "project"). the same graph may be referenced by many other graphs. unlike #1, these are the product of a graph editor (stored as .json or .fbp), rather than code. it may or may not be editable from the parent graph editor, depending on the abilities of the UI
3. embedded graphs: saved within the parent graph. can only be referenced indirectly by other graphs via its parent. editable from the parent graph editor.

fbp-protocol already supports #1 (in reality it requires no additional support as these subgraphs can simply be treated as components). This ticket is for #3. EDIT: and #2

Below is a bare-bones example of a command sequence that creates a graph within a graph:

protocol: graph
command: clear
payload:
    id: myGraph

---
protocol: graph
command: addnode
payload:
    id: myNode
    graph: myGraph

---
protocol: graph
command: clear
payload:
    id: mySubGraph
    parent: myGraph

---
protocol: graph
command: addnode
payload:
    id: mySubNode
    graph: myGraph/mySubGraph

---
protocol: graph
command: addinport
payload:
    public: IN
    node: mySubNode
    port: IN
    graph: myGraph/mySubGraph

This adds 2 new concepts:

• sub-graphs may be identified within any graph attribute using /, as in parentGraph/childGraph
• graph clear gets an optional parent attribute

I chose to add the parent attribute because I wanted this action to be explicit. An alternative would be to use the parent/child notation for the graph clear, like this:

protocol: graph
command: clear
payload:
    id: myGraph/mySubGraph

I'm open to suggestions and opinions.

[jonnor]

What is the benefit over modelling these subgraphs as separate components/graphs - which just happen to have instances inside another graph? (it's parent)

[chadrik]

In brief, not all graphs are reusable, and forcing them to be exposed as components is more complex than what's proposed here and has additional downsides. For graphs not exposed as components, we need something like what is proposed here.

Also, I realize now that my proposal covers uses #3 and #2, because the only difference between them is where the graph is stored, which is irrelevant from the perspective of the protocol.

Below is a more in depth explanation.

Imagine this sequence of events in a UI:

• user authors a graph named "sub" and saves it
• user authors another graph named "master", and adds a reference to "sub" within it

How will the runtime know the definition of "sub"?

One approach would be to expose "sub" to the runtime's component loading mechanism, so that it can be presented back to the UI as a normal component (e.g. using component list). This approach is more complex than it seems because a runtime might have a very specific way of looking for collections of components on disk (e.g. a python-based runtime might import modules with a particular naming convention) whereas a UI might store a graph in a database, or in the browser's local cache, which are inaccessible to the runtime. Making this work will require first sending the graph data to the runtime and then instructing the runtime to save the graph as a new component (would this be done using setsource?), which the runtime will presumably want to expose back to the UI via component list and serialize to disk so that they are accessible when the runtime restarts.

Aside from the complexity (my runtime does not yet support setsource because there are a lot of things to consider to make this work, especially for subgraphs), the next biggest downside of this is that not all graphs are meant to be reusable the way that components are. In many cases forcing a graph to be exposed as a component might simply add noise to component search results in other projects. At some point the user must draw the line between what is a reusable component and what is the graph document.

[jonnor]

In which cases aren't graphs reusable? Trying to understand the details/background of the usecase...

[chadrik]

Hey @jonnor, I'm still mulling this issue over so that I can come up with a good proposal. It would help me if you could clarify the workflow for creating and using subgraphs in noflo. After I've created a graph and added exports, how do I then reference that graph in another graph?

[jonnor]

You add a process/node with that graphs name as the component. So basically each graph immediately becomes available as a component. This is all done in memory, there is no change to things on disk.
This implies also that there can, in general, be multiple instances of a (sub)graph.

Afair, the (subgraphs) are built with the regular graph protocol. Then emit component:component messages on changes which affect the "outside" / component view of the things, like changing exported in/outports.

@chadrik As mentioned in previous comment, I do not understand what cases one would have graphs which cannot be reused. Can you give some examples?

[chadrik]

You add a process/node with that graphs name as the component. So basically each graph immediately becomes available as a component.

What is the exact mechanism through which this happens? I'm not seeing this behavior with my runtime in noflo-ui, so I'm assuming it is something that we've failed to account for on the runtime side. For example, if I create a new project, which creates a new graph named 'main' by default, then I create a second graph called 'sub', then return to 'main' and search for a component called 'sub', it is not found.

When the client requests the list of components via component:list is the runtime supposed to emit a component:component message for each graph that has been sent to the runtime via graph:clear, etc? Otherwise, if it's handled transparently in memory by noflo-ui (basically merging the list of components sent from the runtime with subgraph defined on the client), then I thought the feature would already work with my runtime.

As mentioned in previous comment, I do not understand what cases one would have graphs which cannot be reused. Can you give some examples?

It's not that they cannot be reused, it's that they are not reusable enough to become a permanent part of a collection the way that a component is. That said, before I go further with this idea, I'm trying to go back and understand more about how subgraphs currently work in noflo-ui, because I suspect that some of my assumptions are wrong, and this is causing a disconnect in communication on this topic.