-
Notifications
You must be signed in to change notification settings - Fork 16
Project Management "Whitepaper"
During the past year of 2022, the primary focus of development on lively.next was focused on refactoring and redesigning the underlying technology stack. This involved things like migrating the renderer away from a VDOM rendering approach, decoupling the master component architecture from morph objects or making the build process of lively.next app more UNIX compliant (be able to bundle from the shell).
The goal of the next coming months will now be to implement a system in lively.next that allows for streamlined workflows with "Projects". What that means is that there should be a uniform interface for importing/opening projects in lively, saving and updating these projects and finally being able to version these changes and collaborate together on a project in a Team. A particular focus here should be to satisfy different types of users with varying degrees of technical proficiency. For instance a designer using the system will most likely not be able to understand the basics of GIT versioning systems but still needs to be able to share their changes with the team.
The reason why lively.next up till now did not support proper project management was that "Projects" where essentially worlds that were stored away as snapshots in a databases or into the file system of the host machine. Snapshots are cyclic graphs with substantial complexity, so to this date no viable approach of merging or rebasing was implemented.
The redesign covers roughly 2 major areas.
- Firstly a revised landing page for lively.next that now lists Projects that can be managed by the GIT versioning system.
- Secondly, a direct manipulation interface that allows Designers to create and modify UIs in within the project in a way that does not require them to write code. We took heavy inspiration for this UI from the recently popularized prototyping tools such as Figma or Framer.
Making lively.next interface more seamlessly with existing versioning systems such as GIT requires us to integrate more direclty with the file system that the lively.next system is running on. At the same time, we do not want to be an entirely file system driven tool (such as VS Code or VIM). This would make it a prerequisiste for any user to fully understand the details behind file system structure and versioning systems. Rather we want to present lively.next in a way to the user that is familiar with tools like Pages or Figma who work with files or projects but UI wise allow to stay mostly entirely within the App. This is achieved by displaying files or projects via tiles that allow the user to easily access all of the previous. Also these Apps make the mechanism by which projects are versioned and saved very transparent. That means, users who do not bother are not bothered by versioning their changes explicitly.
The project browser is a UI that can be either opened during an active development session or is presented upon launch of lively.next. It gives an overview the projects that are available to the user. The tiles in the project browser are of 2 different kinds that can be toggled between: Playgrounds and Projects.
A Project is a folder that resides somewhere on the file system the lively.next server is running and is versioned by GIT. It contains the following by default:
- A
package.jsonfile. Among the default NPM meta information, thepackage.jsoncontains a version binding which defaults to the current commit hash of the lively.next repo. For more information on lively.next version management see Version Binding. - A
./testfolder that contains all test files related to that project. - A
./uifolder which is auto populated by component files generated by the designers interaction. - A default build script to bundle the project and deploy.
- A
./workspacesfolder which can be populated with snapshots of workspaces that open the project with a particular instantiation of the tools and components. - A
./assetsfolder that includes images, fonts etc. used in the project.
The lively.next server maintains a list of previously imported projects and also all the projects that are stored inside the dedicated projects folder. This is persisted on the server side and also maintained once the server restarts.
Upon first launch of a lively.next landing page, the server checks for the presence of a project directory residing at the current working directory the server is running at. It it is not found, it is created by default. A project then can be created by clicking the New Project button via one of three ways:
- By opening a folder from the file system the
lively.nextserver is running on. - By cloning a git repository which is then placed into the default projects folder configured for
lively.next. - By creating a new folder that is automatically populated and setup with the files and folder defined earlier in this chapter.
Opening a button happens via the open button that is attached to each project tile. The button is split into two parts.
- Clicking the part that says Open will do the following things:
- Ensure that the latest changes form remote have been pulled.
- If there are uncommited changes, commit those and automatically create a merge commit if needed.
- After that we open the project by loading the default workspace.
- Clicking the dropdown part of the button will allow you to:
- load the project in a custom workspace while also leaving the state of the GIT repository untouched. This is the way experts would usually open a project.
In both ways the system will ensure that the project is imported into the system as a package upon load (naturally).
With a project openend and the workspace loaded, we continously allow the user to save the changes. Again there is a save button in the top right corner of the system, attached to the top bar (basically where we currently have the user info stats which will be soon gone).
The button is used as follows:
- Clicking the Save part, will
- Serialize the current workspace and store it to the file system.
- Stage and commit and push all changes to the files in the project.
- Clicking on the drop down will prompt us with the following choices:
- Just storing the workspace in isolation, leaving the repository and files untouched.
- Store the workspace under an alternative name, essentially creating a different working copy of the workspace.
- Open up a git console which allows to perform manual interaction with the git console.
The idea behind playgrounds is that projects demand a lot of "dicipline" from the developer, since any change that is to be used productively needs to end up in a file. This in particular discourages the use of state for evolving the app. Utilizing state evolution in order to develop an app, used to be the hallmark of the old parts based development approach. We like to refer to these two approaches also as the Engineering Workflow (Projects) vs. the Tinkering Workflow as in the case of the parts based development. The issue with Tinkering is that it lacks sufficient scalability with respect to both collaboration among members of a team ("How do I merge the conflicting changes in Rick's world with my local one?") and complexity management ("Which morph did I put the following behavior in again? And what state does it depend on? How can I reset everything to be sure?"). While we do not deem the above problems unsolvable they are still unsolved to a sufficient degree as of now. In order to still keep the door open to the desirable properties of tinkering workflow which is great for quick prototyping and experimantion we offer this in the form of playgrounds.
Clicking the New Playground button prompts for a name to be given for the new world, and then launches a session.
Loads the snapshot version associated with the selected playground. Unlike with projects, there is no submenu available on the open button.
Clicking the Save button in the top right, stores the serialized version of the playground as a snapshot in the ObjectDB. Unlike with projects there is no submenu available.
A project as well as the playgrounds built in lively.next usually depend on lively.next at a particular version. So in theory what we would want is a dependency inside the package.json that lets us point to a particular version of lively.next as it was published on NPM. However there are 2 issues with this:
- In its current form lively.next is not yet ready to be shipped as a standalone package that can just be imported by NPM. This would require to decouple all parts of the system from the flatn custom module resolver as well as precisely definining which files require a custom lively source transform and which can be left unattached. (Can be solved by introducing a custom lively file type).
- As of now we do not have the time and resources to adhere all changes to particular semver versioning scheme which are accompanied by a respective changelog.
While we are striving to reach that target in the long run, we will need a temporary solution wich prevents us from running into version conflicts with projects and playgrounds that were created in earlier versions of lively.next. For that we will use the current commit hash of the deployed lively.next repo as the version that is in turn used to tag projects or playgrounds accordingly. The commit hash is entered in the package.json under the custom "lively" section which supports a "version" attributed pointing to the commit hash. Alternatively (and in tune with where we would want to be in the future), we could also represent this dependency as a 'lively.next' dep inside the usual dependencies: property of the package.json. This is automatically written upon creation of a project/playground. Upon opening of a project in a workspace or loading a playground the system checks wether the version tag still matches the version of the running lively.next.
- If the version is ahead the current
lively.nextthe load is canceled notifying the user to update the system to be able to load that version. - If the version is behind the current
lively.nextthe load is performed and upon completion the user is prompted to confirm that the version tag of the project/playground be updated. If that is declined, the session is closed again. If confirmed the session continues. - If the version matches the current
lively.nexteverything loads as normal.
With the aforementioned UI we want to reach a viable compromise between different users of the system while also being mindful about the resources we have available for the development of lively. On the one hand we do not want to restricts experts (i.e. programmers) ability to control any part in the project with regards to collaboration. On the other hand we want to provide a basic way for non-programmers (i.e. Designers) to operate with projects that are versioned asynchronously. Asynchronous collaboration can quickly lead to complicated scenarios that novices have a hard time wrapping their head around. We definitely do not want people to run accidentally into conflicts and version mismatches without intention. We therefore attempt to hide these complexities behind the simple "Save" and "Open" buttons for projects in the system. Auto merge and auto push mechanisms combined with separating the areas in the project that Designers and programmers work on (Programmers focus on .js files, Designers "own" their .cp.js files), we strive to make the collaboration in a team 90% time without friction.
It is worth mentioning that the current state of the art for solving any sort of collaboration for non-expert users is to provide them with real-time collaboration. The problem with real time collaboration is that the concrete implementation is only well understood for severely restricted domains like rich text documents, drawing boards or UI design systems. By restricted we mean that all of them have a fixed model that does not change at runtime and can therefore support the real time resolution of any conflict that arises.
For various reason we will not go into detail here, implementing real time collaboration support for lively.next projects and playgrounds is still an uncracked problem. What we can however explore in the near future is to identify several subtasks that are often carried out as a part of working with lively.next and supporting real time collaboration for that. For instance collaborative code editing (excluding the runtime) and evaluation triggering could be easily built and field tested. Another area that is fairly straight forward should be real time collaboration between designers working on component definitions since there is no custom behavior that can be triggered as part of working with the UI composition itself. (Note: Although we support source code reconciliation for designer direct manipulation, this by no means that we can build a real time collaboration feature based on text changes on top of that. Reason being, that sole real time collaboration on text will not honor correct syntax which is fine if programmers directly edit the text, but quick way to crash a collaboration session with designers who are unaware of the code changes they are generating.)
Designers work with projects by opening the default workspace via the landing page. After that the components they define and adjust are all defined in the ui folder of the respective project. Designers are able to organize the visual representations of the components any way they like on the workspace. The visual placement of the components does not affect the way they are stored in the module structure of the project.
- Components are declared via Halo interface.
- At first the component name declared the file name that is created and the component is defined in.
- The halo interface ensures that there are not name conflicts within the project.
- Components without parent components reside in their own file
- All derived components are defined via copy halo interface. Once they are declared to be components, they are placed in the same file as their parent.
- Via this system, the components are partitioned into separate files that contain the entire derivation trees of the components.
- Breaking up derivation relationships is not supported for the time being. What can be done is overridding masters in various ways. This is described in more detail in Sidebar Controls for Components.
- The world should have a zoom feature that allows to zoom all elements that are not Halos/Windows/IDE related.
- The world should also have infinite bounds. This means we can scroll into any direction as far as we want.
- There should be an indicator regarding the current "zoom level", especially for stuff like the Responsive Halo this is important
- Technically not difficult to achieve but tricky to reconcile with the World API
- Comments are related to a Component
- Comments are serialized in a separate comments.json file in a project
- The comments browser loads comments from this file and "matches" them with existing components
- Renaming/Deleting Comments need to be reconciled with the comments.json file
- In Sandbox Mode, Comments and the Comments Browser are disabled
- Global Rich Text Control -> Sidebar Controls
- Rich Text Halo -> Selection Based, entered by double-clicking inside of a Text
The properties panel on the right, contains a dedicated tab that allows us to switch over to the component specific controls for the currently focused morph.
-
Overridden Properties: A list of property names that have been overridden and are therefore excluded from the styling of the master/parent-master. If the property value can be visualized as is the case for
Color,PointorBooleanit is printed next to the overridden property entry. If desired the overridden property can be revoked in order to yield control to the original master policy. - Event State Masters: These include 3 different types: Default, Hover and Click. For the sake of debuggability the user can manually toggle each of these states indpendently on the event state actually being active or not. Default will always point to a component definition, however Hover and Click may also point to a custom set of properties that are supposed to be overridden when the particular event state is activated. This avoids unnessecary verbose and cumbersome component definitions that just take the place of a Default, Hover or Click style.
- Breakpoint Masters: Aside from the state masters, the user is also able to define an arbitrary set of extent breakpoints which toggle different masters based on the width/height of the morph they control. The breakpoint masters are visualized in the side-bar however configuration of them happens via the responsive design halo that can be toggled around the selected morph. The responsive design halo allows to add/remove breakpoints and also apply different masters depending on the breakpoint matching.
- Is a Parts Bin like way to quickly browse all available components in the project and other projects (including the core packages).
- Allows to instantiate components with behavior enabled or disabled. This is controlled via a global flag on the bottom of the components browser.
- The components browser is basically prompt that can be popped out into a window to be freely moved around. This can be convenient in case more frequent access is required.
- Components can be instantiated via drag and drop.
- Components can be applied to override certain state based components by dragging the components on UI elements in the side bar or the responsive design halo.
- Components can be organized by name (alphabetically) or grouped by component files. Also there is always the option for the user to query for a particular component via search.
- Components residing in other (i.e. 3rd party) projects can not be edited. However they can be reused in the currently open project to serve as building blocks for new components.
- Again the problem of versioning arises, which we again resolve by tagging the 3rd party project version of an imported component into the projects
package.json. It is then assumed all of the components from that project stem from that project at that version.- 3rd part projects are entered into the dependencies property of
package.json. - This is esentially analogous to the way we handle tagging of projects dependent on
lively.nextcore. As long as no custom semver version is specified in thepackage.jsonof the outside project, we assume the checked out commit at that point to be the version. - The warning prompts that appear upon version mismatch are the same as the one fore
lively.nextcore: Downgrading not allowed, upgrading on prompt. Canceling the upgrade will cancel the load of the project. - The warning prompt also give an overview of the affected components that may have issues due to the changed project version.
- 3rd part projects are entered into the dependencies property of
:::danger How to open the Asset Manager? -> Properties Panel when the "World" is target? :::
A project comes with an asset folder which contains all images/videos/font files associated with the project. Aside from local files, assets can also be imported via url which avoids placing files into the asset folder. This may be useful if you have a central place where you prefer to fetch your assets from, like google fonts for font files.
Is controlled by the asset manager tool, which is a a little similar to the components editor (UI wise). It allows to browse all assets imported into the project, either via a search query or by filename and category.
Images can be dragged out of the asset manager and get auto instantiated as image morphs. Dragging an image onto an existing image morph assigns the url of that image morph to the one of being dragged.
Fonts imported are all considered relevant by the system and are auto loaded by `lively.next` once the project is loaded or the bundle on load.
New assets can be added to the asset manager by dragging files into the window (similar to how its done now) or by using the import button on the asset manager which allows the user to select a file on the client via the file API.
These assets are such that are not stored in the asset manager and therefore do not get stored in the project folder. Remote fonts can be installed into the project via the Rich Text Control in the right side bar, where we allow to remote install new fonts from the font drop down. Remote fonts are fetched and loaded via an ensureFont() call that gets automatically updated inside the index.js of the project folder.
Remote images can be imported via entering a URL into the image property in the right side bar, or by dragging an image via html5 drag and drop into the world. In that case lively.next instantiates an image morph automatically that has the image url property pointing to the dragged image. This can then in turn be incorporated into component definitions.
The final area where we are revisiting is the tool collection that support the programmers in working with the projects.
- Deleting modules should notify dependants and allow us to resolve issues this causes
- Unsuccessful module load should be more clear from the browser (initialization state of module);