Roadmap for release 3.3.0

[joyfullservice]

Starting a discussion here about how we want to move forward in the goal of finishing some outstanding development work in the dev branch, and getting it merged back into master.

[joyfullservice]

In a nutshell, here is where things are at with the dev branch... Inspired by some great ideas and collaboration with @mmzmusicnotes in #81, we launched into a significant rework of the import process to ultimately support a tighter integration with git. Two important aspects of this were as follows:

1. Improve the fast save functionality to be more accurate and robust than the simple file date comparison being done on the current release. This involved implementing an index file to maintain the status of changes, and implementing a sha1 checksum on database components that did not have a reliable way to determine a modified date. (I.e. VBA code modules, project options, database properties, etc...)

2. Implement a merge build concept where changes in source files can be merged into an existing database project at the component level without needing to build the entire database from scratch. This allows more seamless collaboration with multiple developers working concurrently on the same project.

There were additional ideas also in the works, like a sync process where a two-way synchronization could happen with the source files, bringing the latest changes into the database, and exporting the latest changes to source, all in a single operation. Ultimately this will provide the greatest benefit to development teams because they can then truly collaborate at the source level even with large database projects. (But that might be for a 3.4 release.)

However, after just getting started on the project, due to some external factors outside his control, @mmzmusicnotes was reassigned to some other responsibilities and not really able to contribute as he had hoped. I have been making some slow and steady progress as I am able, but my time is somewhat limited due to some other pressing projects on my plate.

In order to get us through to merging dev back into master for general release as 3.3.0, I think we need to complete the following:

• Finish building out the index file functionality to record the state of the database objects
• Implement the updated import/export process across all the database objects. (See clsDbForm and clsDbModule for examples of what this looks like.)
• Implement the merge functions across the database objects so individual components can be replaced/updated from source files within a database without requiring a full build.

As to the time frame, that depends on how quickly we are able to work through these things. It could be a few weeks or several months, just depending on who all is able to work on it, and how much time they have available to work on these outstanding items.

While the development continues on the dev branch, I am mainly looking at the current master branch as in a bug-fix maintenance state to focus as much of the development efforts towards the next release where we can merge everything into master.

Following the 3.3 release, we can probably resume our primary development work on the master branch since we will be making smaller changes/enhancements/bug fixes vs. the major refactoring we have done in the dev branch to support some of the new functionality.

Well, that was kind of a long-winded nutshell version of where things are at, but hopefully it gives some perspective on where we are heading. 😄

[hecon5]

I like this, the one change I'd suggest is to not include the Merge (bullet 3) and git integration with 3.3.0 (or at least have it be abundantly evident that it's still under test).

There were additional ideas also in the works, like a sync process where a two-way synchronization could happen with the source files, bringing the latest changes into the database, and exporting the latest changes to source, all in a single operation. Ultimately this will provide the greatest benefit to development teams because they can then truly collaborate at the source level even with large database projects. (But that might be for a 3.4 release.)

This is such a huge step, that I think it should be a 4.x release, IMHO. It should have a ton more testing than I think could be possible in the short term; even RD killed off their git integration during one of their recent 2.x releases, so while I think this would be a HUGE improvement, I think it's such an enormous shift that it should be released more slowly.

I'd also suggest that perhaps a version checker / update might be more useful for users (or at least a version check prompt)? In my experience, many users (even devs!) will just keep using the software until it's broken, then go see if there's a fix. Having them periodically reminded to check for new versions would allow more updating in a seamless way, and less disruption to their source files (namely, prtmip will make numerous changes to their source files and add .json files, with a zero-code-change operation).

I think even something as simple as a lastupdated prompt, where every 2-4 weeks it has a button that shows up and says "check for new version?" on the load form; user clicks it, goes to the releases tab, and see if there's a new version could help.

Another mechanism: perhaps an even better way to do it is to do a semi-CI/CD flow, get rid of the dev branch (or relegate it to a stage release), and have two releases: a "stable" and a "development" that's compiled monthly? This way, PRs always get pulled to the correct branch (there's only one branch), and the built/released version is just built off a tag and not a branch.

[hecon5]

I'll also note: doing this will virtually eliminate the issue found with #163, as the base will already be present.

[joyfullservice]

I like this, the one change I'd suggest is to not include the Merge (bullet 3) and git integration with 3.3.0 (or at least have it be abundantly evident that it's still under test).

That sounds reasonable to me. You're right that the merge functionality is technically not needed to move this back into general release, and can be added later.

There were additional ideas also in the works, like a sync process where a two-way synchronization could happen with the source files, bringing the latest changes into the database, and exporting the latest changes to source, all in a single operation. Ultimately this will provide the greatest benefit to development teams because they can then truly collaborate at the source level even with large database projects. (But that might be for a 3.4 release.)

This is such a huge step, that I think it should be a 4.x release, IMHO. It should have a ton more testing than I think could be possible in the short term; even RD killed off their git integration during one of their recent 2.x releases, so while I think this would be a HUGE improvement, I think it's such an enormous shift that it should be released more slowly.

I like the idea of turning that into a v4.x release down the road.

On the topic of testing, I am not sure if you have had a chance to do much with the Testing.accdb database, but the idea behind that was to incorporate as many of the different features as practically possible in the test database, then we could diff exports before and after changes to the code base to make sure that fixing one thing wasn't breaking something else. I never quite got the testing database ready to build from source, but that is the end goal to include it as just source files, and let the developers build and test it on their own machines.

Thanks for the link on the RD discussion on git integration. This encourages me that we are on the right track with integrating but not replacing the actual version control tools that people are using. I would like to keep this add-in focused on the export and import of source code, and let other programs like GitHub Desktop, SourceTree, etc... do what they already do best in managing the source files. I don't see any reason to reinvent the wheel here. 😄

[joyfullservice]

Just an update on Testing.accdb database... As of today, you can now build it from source and add objects/tests as needed. (I finished adding in relative path support for linked tables, which was the main holdup.)

[joyfullservice]

I am wondering if we should go ahead and merge dev back into master... There is still more that could be done with implementing the fast save with other objects, but I have the main ones finished and we can always continue that process on the master branch. I am pretty happy with the Export/Build functionality on the dev branch right now, based on testing with several of my projects, so I am thinking that perhaps it would be more helpful to get this rolled back into the master branch for ongoing development. What do you think? @hecon5, @lk-12, @Tanarri, @A9G-Data-Droid, @Indigo744

[hecon5]

I think merging the current dev into master, updating the build number, and the release file is a good idea. Getting too far apart will lead to integration issues for bugfixes.

As to using master for development work on a software product without CI/CD auto building capabilities I think will lead to issues when others try to bug fix or patch the "current version" as it will be out of date.

Since I don't think the CI/CD nightly build is automated, and is still manual, I would suggest just bringing master forward, updating the release build, and continue dev work on dev, perhaps more frequently update master from dev to ensure it doesn't get out of date would work better for this product.

[A9G-Data-Droid]

I haven't tested the new dev version and I don't think I will have time to in the near term. I like the idea of fast save using hashes of the text as that's a solid way to know when it's time to export.

I agree that it's good to push changes incrementally and it's been a while since there was a release.

I also agree that we should have the master reflect the current release. All work should be is some sort of dev/next/version branch. I like naming branches after what they contain. That way you can see what they are just by looking at the history of what was merged in to master.

[hecon5]

After pondering last night, I have a new process idea to run by you all.

TL/DR:
Keep dev branch release to Major.minor.build. This will make it a lot easier to track master and the current release to the developed version. To update master, simply merge dev onto master and upload the new release. If GitHub lets you hook into the release version API, could even accelerate that and use Deploy to automate it, and then it's simply a PR from dev and you're done.

Details
When making a new branch to work on for a PR, the first action would be to increase the build number via IncrementAppVersion.

This would be a new process, but would help differentiate between test builds when doing PR testing. This would help to ensure the the version you're running tests on is either the current "dev" branch version or the one you're reviewing by the build number, instead of compiling the wrong one, switching branches, and forgetting you haven't compiled it yet (something I've done, and I think was a contributor to the recent need to re-fix the clsDevMode again).

I also thought about Deploy, and to use that, but since that is really just to release the version on your machine, (unless we were to integrate it with GitHub's release channel tools, as I think @joyfullservice mentioned you have integrated with other APIs via VBA), just increasing the version number should suffice for now.

[A9G-Data-Droid]

I am using 3.3.22 with no issues so far.