Skip to content
This repository has been archived by the owner on Aug 1, 2021. It is now read-only.

Simplify SQL and translation pipeline #1

Open
totten opened this issue Jan 21, 2018 · 3 comments
Open

Simplify SQL and translation pipeline #1

totten opened this issue Jan 21, 2018 · 3 comments
Labels

Comments

@totten
Copy link
Member

totten commented Jan 21, 2018

The current version of civicrm-setup still relies on the same SQL files as the old installer -- which means:

  • All the SQL files need to be pre-generated and pre-translated -- so, eg, developers need to understand th workflow of both setup.sh and regen.sh.
  • Updates for the schema and l10n data cannot be simply dropped in to place. Generally, people have to download a carefully build tarball.
  • We need more bits of infrastructure providing the above.
  • You can only get sample data on en_US.

To simplify the workflow:

  • Rewrite plugins/installDatabase/InstallSchema.civi-setup.php...
  • Don't read specific SQL files.
  • Instead, use CRM/Core/CodeGen/*, xml/schema/*, and l10n/* to produce the SQL on-demand.

Eventually, this should allow us to remove regen.sh and remove the various civicrm_data.xx_XX.mysql files.

@dsnopek
Copy link

dsnopek commented Jan 30, 2018

I like the sound of this:

(without needing setup.sh / regen.sh / GenCode)

Does this mean that civicrm-setup could be used to power the Drupal 8 installer, allowing us to remove the lines from the process that copy the SQL files?

Ie., this line: https://gist.github.com/dsnopek/56311dbea347874e75180883efabb620#file-installing-civicrm-on-drupal-8-via-composer-sh-L54

That would be amazing!

@totten
Copy link
Member Author

totten commented Jan 31, 2018

Does this mean that civicrm-setup could be used to power the Drupal 8 installer, allowing us to remove the lines from the process that copy the SQL files?

@dsnopek - Spot-on for both counts:

  1. The civicrm-setup API should be available now for use in the Civi-D8 integration. (Specifically, anything based on master and the current revision of core's composer.json.)
  2. The SQL files would become unnecessary if we addressed this ticket -- making it easier to install from source.

@monishdeb
Copy link
Member

I have submitted a PR #9 in this direction, although my changes aren't complete and marked with WIP. In that PR I have added a new event listener civi.setup.generateSchema to generate SQL and DAO files, likewise in regen.sh The new API would be $setup->generateSchema() that need to be called before $setup->installDatabase()

totten added a commit to totten/civicrm-core that referenced this issue Feb 15, 2018
For civicrm/civicrm-setup#1, this change allows
`civicrm-setup` to read the XML metadata in the same way as GenCode -- which
helps us to install the current schema without any intermediate files.
totten added a commit to totten/civicrm-core that referenced this issue Feb 16, 2018
For civicrm/civicrm-setup#1, the general goal is to allow installing the
database schema without needing to run `GenCode`.

The current draft is crashing because the SQL does translation using `ts()`.
But if you try to use `ts()` in a pre-boot environment, it will attempt to
boot automatically so that it can read `$config->customTranslateFunction.

This is a chicken-egg situation.  We haven't yet reached the phase where the
installer can boot up Civi...  because we don't have the SQL...  but the SQL
can't be generated (translated) because Civi hasn't been booted.

The aim of this patch is to loosen the coupling between `ts()`
and `CRM_Core_Config` so that `ts()` can be used on its own.

> Aside: You might ask how this works today -- basically, `GenCode` does a
> database-less-boot, which placates `ts()`.  However, the `civicrm-setup`
> will eventually need to do full-boot, and AFAIK we don't have any
> situations where one transitions from database-less-boot to full-boot; I
> have a gut fear that such a transition would be its own slipper slope.
totten added a commit to totten/civicrm-core that referenced this issue Feb 20, 2018
Overview
--------

The `ts()`/`{ts}` feature from `CRM_Core_I18n` has an option to escape output
(e.g.  `{ts escape="sql"}Activities{/ts}`).  However, SQL encoding is subjective
(depending on the connection details). For `{ts}` to support this feature, it
must have a dependency on the DB subsystem (e.g. it eventually calls `CRM_Core_DAO`).

For civicrm/civicrm-setup#1, we have an issue
where expressions like `{ts escape="sql"}Activities{/ts}` fail during
installation because `CRM_Core_DAO` is not fully available.

This change loosens the coupling, so that we can use `{ts escape="sql"}Activities{/ts}`
without needing `CRM_Core_DAO` per se.

Before
------

`ts`/`CRM_Core_I18n` is tightly coupled to `CRM_Core_DAO`. There is no way
to use `{ts escape=sql}Activities{/ts}` without spinning-up `CRM_Core_DAO`.

After
-----

`ts`/`CRM_Core_I18n` *defaults* to calling `CRM_Core_DAO`.  However, this can
be overriden by manipulating a property.

Comments
--------

* I feel a little dirty keeping any coupling between i18n and the DB.
  However, changing this would mean removing support for the
  `{ts escape=sql}` option, and that would be a clear compatibility-break.
* Arguably, there may be a microsecond-level penalty in using
  `call_user_func($SQL_ESCAPER)` rather than a specific class/function.
  However, it's only incurred if you actually call `{ts escape=sql}`
  while setting `$SQL_ESCAPER`, and that's pretty rare circumstance.
  The typical runtime use-cases for `ts()` are unaffected.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

3 participants