Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dev/financial#84 Remove sequential credit notes from core #16462

Closed

Conversation

eileenmcnaughton
Copy link
Contributor

@eileenmcnaughton eileenmcnaughton commented Feb 4, 2020

Overview

Moves sequential credit notes from core to an extension that ships with core.

https://lab.civicrm.org/dev/financial/issues/84

Extension is now
https://lab.civicrm.org/extensions/sequentialcreditnotes
https://civicrm.org/extensions/sequential-credit-notes

Review request https://lab.civicrm.org/extensions/extension-review-requests/issues/12

Before

When a contribution is created as refunded a credit note id is calculated. This calculation is slow and is done regardless of whether the site requires it.

After

The above is done in an extension - sites can disable the extension & eventually it will not ship with core

Technical Details

The code that adds sequentialcredtit notes is a hangover from pre-Lexim days when people's various needs just got added to core.
It would not have been added to core post Lexim for the following reasons

  1. architectural - it assumes a one to one relationship between contributions & credit notes which is not reflective of our data model
  1. performance - the code had significant performance impact on large sites
  2. specificity - the code was for a specific localised use case of needing guaranteed sequential credit-notes, appropriate to an extension
  3. maintainability - we've had to put in a lot of work to get the code down to the few lines being removed. Several places were deprecated several months ago - https://github.com/civicrm/civicrm-core/pull/15492/files & we've had no reports of the deprecation warnings being hit. The goal is not
    to allow organisations to transfer maintenance burden onto the community unless there is a clear rationale.

Medium term the intent is that this extension will not ship with core & will simply be available from the extensions
directory as any other extension. In order to reach this point we need a transitional approach. In the first instance sites
should need to take no action at all in order to experience no change of behaviour. In order to achieve this
we continue to ship this functionality with core as an extension. We enable the extension on upgrade (but on new
installs it is opt-in).

At this point sites who do not want this functionality can disable the extension.

The next step will be to get people who want the extension to install it to their extensions directory
so we can remove this from core later. This will involve us fixing the extension load code to prefer the extension
directory over core if the extension is in both places & adding checks & messaging to encourage people to
install the extension if they continue to want it.

At some point we will remove it from the core tarball. Ongoing maintenance will be provided by whoever wants to maintain it - if
someone wants to be added as a maintainer of the extension repo the answer is yes.

Comments

Previous attempts to fix performance issues
#13511
#15232 - makes it possible to opt out by extension but logically it should be opt in by extension
#15235

@civibot civibot bot added the master label Feb 4, 2020
@civibot
Copy link

civibot bot commented Feb 4, 2020

(Standard links)

@eileenmcnaughton eileenmcnaughton changed the title Creditnote gone dev/financial#84 Remove sequential credit notes from core Feb 4, 2020
@eileenmcnaughton eileenmcnaughton force-pushed the creditnote_gone branch 2 times, most recently from a504ac1 to ad00405 Compare February 4, 2020 07:31
This moves the code that allows settings to be added to forms by metadata from the Generic setting form to the setting trait.

This means many more forms will support injecting settings by metadata. Specifically we switch across the Contribution preferences form,
making credit_note_prefix a metadata-added field. This is preliminary to moving that to an extension & allows us to inject it
in by simply defining the metadata in a hook
…sion that ships with core.

https://lab.civicrm.org/dev/financial/issues/84

The code that adds  sequentialcredtit notes is a hangover from pre-Lexim days when people's various needs just got added to core.
It would not have been added to core post Lexim for the following reasons
1) architectural - it assumes a one to one relationship between contributions & credit notes which is not reflective of our data model
2) performance - the code had significant performance impact on large sites
3) specificity - the code was for a specific localised use case of needing guaranteed sequential credit-notes
4) maintainability - we've had to put in a lot of work to get the code down to the few lines being removed. The goal is not
to allow organisations to transfer maintenance burden onto the community unless there is a clear rationale.

Medium term the intent is that this extension will not ship with core & will simply be available from the extensions
directory as any other extension. In order to reach this point we need a transitional approach. In the first instance sites
should need to take no action at all in order to experience no change of behaviour. In order to achieve this
we continue to ship this functionality with core as an extension. We enable the extension on upgrade (but on new
installs it is opt-in).

At this point sites who do not want this functionality can disable the extension.

The next step will be to get people who want the extension to install it to their extensions directory
so we can remove this from core later. This will involve us fixing the extension load code to prefer the extension
directory over core if the extension is in both places & adding checks & messaging to encourage people to
install the extension if they continue to want it.

At some point we will remove it from the core tarball. Ongoing maintenance will be provided by whoever wants to maintain  it - if
someone  wants to be added as a maintainer of the extension repo the answer is yes.
@eileenmcnaughton
Copy link
Contributor Author

@bjendres @sluc23

eileenmcnaughton added a commit to eileenmcnaughton/civicrm-core that referenced this pull request Feb 5, 2020
In the course of civicrm#16462
I concluded that there might be (untested scenarios) where this would still be hit.

In civicrm#16462 I adopted a cleaner (tested) approach. However, I think it's unlikely that
will be merged before 5.23 is cut so I propose reverting this change & re-reverting it
once the rc is cut as a more conservative approach. We can focus on getting civicrm#16462
wrapped up for 5.24

This reverts commit d7943ca.
'deferred_revenue_enabled' => CRM_Core_BAO_Setting::CONTRIBUTE_PREFERENCES_NAME,
'default_invoice_page' => CRM_Core_BAO_Setting::CONTRIBUTE_PREFERENCES_NAME,
'invoicing' => CRM_Core_BAO_Setting::CONTRIBUTE_PREFERENCES_NAME,
];
Copy link
Member

@totten totten Feb 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a tangential cleanup? (Not trying to criticize - just so that one can read/interpret better.)

* @throws \CiviCRM_API3_Exception
*/
public static function installCreditNotes(CRM_Queue_TaskContext $ctx) {
civicrm_api3('Extension', 'install', ['sequentialcreditnotes']);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if one is accepting the premise/goal and focused on technical aspects of implementation - would it be fair to say that this task (automatically enabling the extension during an upgrade) is perhaps the main "new" thing going on? e.g. for reviewing, it's important to do some r-run on this aspect?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I separated it out #16513

@totten
Copy link
Member

totten commented Feb 5, 2020

On the Gitlab issue, I've pinged about the policy/conceptual level.

Regarding review of the implementation... I think this code passes a basic skimming, and let me try to verbalize a bit about what a reviewer would be looking for (apologies if this is stating the obvious):

  1. If somebody currently uses the functionality and goes through an upgrade, will they still get a drop-in replacement (ie with work-a-like functionality)? (This is sort of an implicit review of the extracted extension...)
  2. If there is existing data (which are based on this functionality) or settings/configuration (which drive this functionality), will they transition ownership smoothly?
  3. If somebody currently does not use the functionality, do they get any kind of weird side-effect?

@eileenmcnaughton
Copy link
Contributor Author

On gitlab there was push back - and to be honest i think the same push back would apply to all the code in core that we might ideally consider moving into non-core extensions (eg. event cart, financial type acls, Surveys, USPS parsing)

  1. bags you got it no returns. ie. the maintenance argument - who will maintain this code going forwards if you don't and
  2. this is desperately vital because here is my use case.

I have concluded it's in the too hard basket for the next few years to actually move things out of core (and that we should focus on only letting code in if it improves the maintainability of the code).

However, in looking at how to deal with this code it was apparent

  1. The code was implemented as a spaghetti source over the top of convoluted Contribution & Financial accounts BAO code for no functional benefit
  2. it is easily implemented as hooks (especially since I have done a couple of rounds of cleanup on this code to get to the point)
  3. We should continue to move the implementation of this code - and other code that we are not realistically going to every get out of core to a more hook-like implementation.

In discussion with @colemanw we realised that
a) we have a framework for hook-like-implementation - it's the extension system
b) we can utilise that at a technical level without addressing 1 & 2 above - ie -

  1. maintenance we structure the code as an extension but still within the core codebase, rather than a separate sucked in repo and more no judgement as to where the code will be long-term
  2. we make the extension-ness invisible to the user by quashing these 'CoreExtensions' from the extension UI (it's OK for devs to be able to see them as extensions).

By doing this we provide a path forwards for other cleanup - ie. it's always been tough to find a path towards tidying up code features that are badly implemented all over the place. I tried to extensionise some of the dedupe code at one point to deal with maintainability issues (there is a hook in the middle of it that makes it uncleanupable because the surface areas is 'everything' & the idea was to have a version 1 extension & then later a version 2 extension with a sensible hook structure) - but because it had to be neck or nothing I had to throw in the towel. If there was a path towards refactoring-towards-code-being-a-concise-feature then I would have been chipping away at that.

Likewise the invoicing code uses around about 100 lines of code across 20 forms (although I have cleaned that up a bit) to do what could be done in a one-line in a preProcess hook (assign a variable as to whether the setting for invoicing is enabled)- having a way to register hooks by features would help.

While I don't think we should be totally flip about designating every checkbox as a CoreExtension - the process of separating the features out is also the process of cleaning up the code & if we have a way to structure those features & use hooks (without being committed to making a maintenance decision of any sort) it unblocks us a lot in cleaning up some of the code that is plaguing us and encourages us to mature our hook infrastructure.

@eileenmcnaughton
Copy link
Contributor Author

Closing - this is still part of core - although it's restructured within core

@eileenmcnaughton eileenmcnaughton deleted the creditnote_gone branch July 3, 2020 04:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants