Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Plone 6: migrate ulines properties to lines, make lines contain strings #3305

Closed
mauritsvanrees opened this issue Aug 19, 2021 · 4 comments
Closed

Plone 6: migrate ulines properties to lines, make lines contain strings #3305

mauritsvanrees opened this issue Aug 19, 2021 · 4 comments
Labels
02 type: regression 12 prio: high 24 status: ready

Comments

@mauritsvanrees
Copy link
Sponsor Member

See zopefoundation/Zope#987

Since Zope 5.3, ulines is deprecated and you should use a lines property. Same for a few others: utext, utoken, ustring. The unicode variants are planned to be removed in Zope 6.

Also since Zope 5.3, a lines property stores strings instead of bytes. But there is no migration yet.
Result is that getProperty on an already created lines property will return the old value with bytes, but a newly created lines property will return strings. This could lead to problems, especially when you insert a new value into the list. GenericSetup had problems with that, and I have created a fix which is merged. It remains to be seen how many problems will arise in other code. I guess it will mostly be okay, but it is not ideal.

We should create some migration code:

  • Go over all lines properties and make sure they contain a tuple of strings, probably by calling the converter. This should happen in Plone 6.0 alpha or beta.
  • Turn all ulines properties into lines, so they remain working in Zope 6. Could be done later, but it makes sense to do this at the same time.

The best place for this code is probably in Zope itself. plone.app.upgrade should call it, probably with zopeFindAndApply.

Also, we should look for spots in our own code where we create deprecated unicode properties, or where we add bytes in a lines field.
mauritsvanrees added a commit to zopefoundation/Zope that referenced this issue Dec 2, 2021
@mauritsvanrees
Add function ZPublisher.utils.fix_properties.
cbaff58 
You can call this to fix lines properties to only contain strings, not bytes.
It also replaces the deprecated property types ulines, utext, utoken, and ustring with their non-unicode variants.
See #987 for why this is needed.

Note: the code is not called by Zope itself.
The idea is that integrators who think they need this in their database, can call it.
Sample use:

    app.ZopeFindAndApply(app, apply_func=fix_properties)

I intend to use this (or a temporary copy) in the Plone 6 upgrade code.
See plone/Products.CMFPlone#3305
mauritsvanrees added a commit to zopefoundation/Zope that referenced this issue Dec 2, 2021
@mauritsvanrees
Add function ZPublisher.utils.fix_properties.
05f947b 
You can call this to fix lines properties to only contain strings, not bytes.
It also replaces the deprecated property types ulines, utext, utoken, and ustring with their non-unicode variants.
See #987 for why this is needed.

Note: the code is not called by Zope itself.
The idea is that integrators who think they need this in their database, can call it.
Sample use:

    app.ZopeFindAndApply(app, apply_func=fix_properties)

I intend to use this (or a temporary copy) in the Plone 6 upgrade code.
See plone/Products.CMFPlone#3305
mauritsvanrees added a commit to zopefoundation/Zope that referenced this issue Dec 2, 2021
@mauritsvanrees
Add function ZPublisher.utils.fix_properties.
967f2bd 
You can call this to fix lines properties to only contain strings, not bytes.
It also replaces the deprecated property types ulines, utext, utoken, and ustring with their non-unicode variants.
See #987 for why this is needed.

Note: the code is not called by Zope itself.
The idea is that integrators who think they need this in their database, can call it.
Sample use:

    app.ZopeFindAndApply(app, apply_func=fix_properties)

I intend to use this (or a temporary copy) in the Plone 6 upgrade code.
See plone/Products.CMFPlone#3305
@mauritsvanrees mauritsvanrees mentioned this issue Dec 2, 2021
Merged
mauritsvanrees added a commit to zopefoundation/Zope that referenced this issue Dec 2, 2021
@mauritsvanrees @dataflake
Add function ZPublisher.utils.fix_properties. (#993)
49e9574 
* Add function ``ZPublisher.utils.fix_properties``.

You can call this to fix lines properties to only contain strings, not bytes.
It also replaces the deprecated property types ulines, utext, utoken, and ustring with their non-unicode variants.
See #987 for why this is needed.

Note: the code is not called by Zope itself.
The idea is that integrators who think they need this in their database, can call it.
Sample use:

    app.ZopeFindAndApply(app, apply_func=fix_properties)

I intend to use this (or a temporary copy) in the Plone 6 upgrade code.
See plone/Products.CMFPlone#3305

* Update src/ZPublisher/utils.py

Co-authored-by: Jens Vagelpohl <jens@plyp.com>

* Update src/ZPublisher/utils.py

Co-authored-by: Jens Vagelpohl <jens@plyp.com>

* Fix unicode properties: mark object as changed when _properties is changed.

Co-authored-by: Jens Vagelpohl <jens@plyp.com>
mauritsvanrees added a commit to plone/plone.app.upgrade that referenced this issue Dec 2, 2021
@mauritsvanrees
Fix unicode properties.
9facae1 
See plone/Products.CMFPlone#3305.

This goes through all objects in the site and does two things:

1. Make sure lines contain only strings, instead of bytes,  or worse: a combination of strings and bytes.
2. Replace deprecated ulines, utext, utoken, and ustring properties with their non-unicode variant, using native strings.

See plone/Products.CMFPlone#3305

I have just created and merged a PR that adds a utility function in Zope that we use:
zopefoundation/Zope#993
It will be included in Zope 5.4, but this is not released yet, so for now we use our own copy.
mauritsvanrees added a commit to plone/plone.app.upgrade that referenced this issue Dec 2, 2021
@mauritsvanrees
Fix unicode properties.
565dd21 
This goes through all objects in the site and does two things:

1. Make sure lines contain only strings, instead of bytes,  or worse: a combination of strings and bytes.
2. Replace deprecated ulines, utext, utoken, and ustring properties with their non-unicode variant, using native strings.

See plone/Products.CMFPlone#3305

I have just created and merged a PR that adds a utility function in Zope that we use:
zopefoundation/Zope#993
It will be included in Zope 5.4, but this is not released yet, so for now we use our own copy.
@mauritsvanrees mauritsvanrees mentioned this issue Dec 2, 2021
Merged
mauritsvanrees added a commit to plone/plone.app.upgrade that referenced this issue Dec 3, 2021
@mauritsvanrees
Fix unicode properties.
9be379b 
This goes through all objects in the site and does two things:

1. Make sure lines contain only strings, instead of bytes,  or worse: a combination of strings and bytes.
2. Replace deprecated ulines, utext, utoken, and ustring properties with their non-unicode variant, using native strings.

See plone/Products.CMFPlone#3305

I have just created and merged a PR that adds a utility function in Zope that we use:
zopefoundation/Zope#993
It will be included in Zope 5.4, but this is not released yet, so for now we use our own copy.
mister-roboto pushed a commit to plone/buildout.coredev that referenced this issue Dec 20, 2021
@jensens
[fc] Repository: plone.app.upgrade
16b66b2 
Branch: refs/heads/master
Date: 2021-12-04T00:38:09+01:00
Author: Maurits van Rees (mauritsvanrees) <maurits@vanrees.org>
Commit: plone/plone.app.upgrade@9be379b

Fix unicode properties.

This goes through all objects in the site and does two things:

1. Make sure lines contain only strings, instead of bytes,  or worse: a combination of strings and bytes.
2. Replace deprecated ulines, utext, utoken, and ustring properties with their non-unicode variant, using native strings.

See plone/Products.CMFPlone#3305

I have just created and merged a PR that adds a utility function in Zope that we use:
zopefoundation/Zope#993
It will be included in Zope 5.4, but this is not released yet, so for now we use our own copy.

Files changed:
A news/3305.bugfix
M plone/app/upgrade/v60/alphas.py
M plone/app/upgrade/v60/configure.zcml
Repository: plone.app.upgrade

Branch: refs/heads/master
Date: 2021-12-15T00:35:41+01:00
Author: Maurits van Rees (mauritsvanrees) <maurits@vanrees.org>
Commit: plone/plone.app.upgrade@8ada20f

Fix unicode properties: fix sub objects as well.

I thought this was the default.

Files changed:
M plone/app/upgrade/v60/alphas.py
Repository: plone.app.upgrade

Branch: refs/heads/master
Date: 2021-12-20T11:53:36+01:00
Author: Jens W. Klein (jensens) <jk@kleinundpartner.at>
Commit: plone/plone.app.upgrade@74eff47

Merge pull request #265 from plone/maurits-fix-unicode-properties

Fix unicode properties.

Files changed:
A news/3305.bugfix
M plone/app/upgrade/v60/alphas.py
M plone/app/upgrade/v60/configure.zcml
mauritsvanrees added a commit to plone/plone.dexterity that referenced this issue Dec 20, 2021
@mauritsvanrees
Plone 6: Always use a lines property for behaviors, no longer the dep…
d1e0495 
…recated ulines.

Part of plone/Products.CMFPlone#3305.
Note: I have switched coredev 5.2 to use `plone.dexterity` branch 2.x.
@mauritsvanrees mauritsvanrees mentioned this issue Dec 20, 2021
Merged
@mauritsvanrees
Copy link
Sponsor Member Author

mauritsvanrees commented Dec 20, 2021 •
edited
Loading

In plone.app.upgrade we now have an upgrade step, which was merged today.
Further steps are that we should prevent creating deprecated ulines and other properties, at least in Plone. Probably in Zope as well.

Looking for ulines in the plone organization:

@mauritsvanrees
Copy link
Sponsor Member Author

mauritsvanrees commented Dec 20, 2021 •
edited
Loading

For ustring I only found it in plone.app.users and then seemingly only in doctests for Python 2.

mauritsvanrees added a commit to plone/plone.app.users that referenced this issue Dec 20, 2021
@mauritsvanrees
Test fix: remove deprecated ustring handling, which is only needed on…
b361328 
… Python 2.

Part of plone/Products.CMFPlone#3305
@mauritsvanrees mauritsvanrees mentioned this issue Dec 20, 2021
Merged
@mauritsvanrees
Copy link
Sponsor Member Author

mauritsvanrees commented Dec 20, 2021 •
edited
Loading

mauritsvanrees added a commit to plone/Products.CMFDynamicViewFTI that referenced this issue Dec 20, 2021
@mauritsvanrees
Plone 6: always use lines for the view_methods property.
3451b42 
Part of plone/Products.CMFPlone#3305
@mauritsvanrees mauritsvanrees mentioned this issue Dec 20, 2021
Merged
mauritsvanrees added a commit that referenced this issue Dec 20, 2021
@mauritsvanrees
Removed management_page_charset support from usergroup-groupdetails p…
8097d18 
…age.

This is related to deprecated unicode property types, like ustring.
Core Plone should not need them, but there could be custom groups that use them,
so we keep the compatibility checks in here.
Part of #3305
@mauritsvanrees mauritsvanrees mentioned this issue Dec 20, 2021
Merged
mister-roboto pushed a commit to plone/buildout.coredev that referenced this issue Dec 21, 2021
@jensens
[fc] Repository: plone.dexterity
b091e59 
Branch: refs/heads/master
Date: 2021-12-20T12:42:18+01:00
Author: Maurits van Rees (mauritsvanrees) <maurits@vanrees.org>
Commit: plone/plone.dexterity@d1e0495

Plone 6: Always use a lines property for behaviors, no longer the deprecated ulines.

Part of plone/Products.CMFPlone#3305.
Note: I have switched coredev 5.2 to use `plone.dexterity` branch 2.x.

Files changed:
A news/3305.bugfix
M plone/dexterity/fti.py
Repository: plone.dexterity

Branch: refs/heads/master
Date: 2021-12-21T17:26:23+01:00
Author: Jens W. Klein (jensens) <jk@kleinundpartner.at>
Commit: plone/plone.dexterity@e43f08b

Merge pull request #161 from plone/no-ulines

Plone 6: Always use lines property for behaviors, no longer ulines

Files changed:
A news/3305.bugfix
M plone/dexterity/fti.py
mister-roboto pushed a commit to plone/buildout.coredev that referenced this issue Dec 21, 2021
@jensens
[fc] Repository: plone.dexterity
fc58cb0 
Branch: refs/heads/master
Date: 2021-12-20T12:42:18+01:00
Author: Maurits van Rees (mauritsvanrees) <maurits@vanrees.org>
Commit: plone/plone.dexterity@d1e0495

Plone 6: Always use a lines property for behaviors, no longer the deprecated ulines.

Part of plone/Products.CMFPlone#3305.
Note: I have switched coredev 5.2 to use `plone.dexterity` branch 2.x.

Files changed:
A news/3305.bugfix
M plone/dexterity/fti.py
Repository: plone.dexterity

Branch: refs/heads/master
Date: 2021-12-21T17:26:23+01:00
Author: Jens W. Klein (jensens) <jk@kleinundpartner.at>
Commit: plone/plone.dexterity@e43f08b

Merge pull request #161 from plone/no-ulines

Plone 6: Always use lines property for behaviors, no longer ulines

Files changed:
A news/3305.bugfix
M plone/dexterity/fti.py
mister-roboto pushed a commit to plone/buildout.coredev that referenced this issue Dec 22, 2021
@jensens
[fc] Repository: plone.app.users
f093c27 
Branch: refs/heads/master
Date: 2021-12-20T23:14:59+01:00
Author: Maurits van Rees (mauritsvanrees) <maurits@vanrees.org>
Commit: plone/plone.app.users@b361328

Test fix: remove deprecated ustring handling, which is only needed on Python 2.

Part of plone/Products.CMFPlone#3305

Files changed:
A news/3305.bugfix
M plone/app/users/tests/personal_preferences_prefs_user_details.rst
M plone/app/users/tests/test_doctests.py
M plone/app/users/tests/userdata_prefs_user_details.rst
Repository: plone.app.users

Branch: refs/heads/master
Date: 2021-12-22T11:56:39+01:00
Author: Jens W. Klein (jensens) <jk@kleinundpartner.at>
Commit: plone/plone.app.users@cbc6474

Merge pull request #105 from plone/maurits-cleanup-ustring

Test fix: remove deprecated ustring handling, which is only needed on Py2

Files changed:
A news/3305.bugfix
M plone/app/users/tests/personal_preferences_prefs_user_details.rst
M plone/app/users/tests/test_doctests.py
M plone/app/users/tests/userdata_prefs_user_details.rst
mister-roboto pushed a commit to plone/buildout.coredev that referenced this issue Dec 22, 2021
@jensens
[fc] Repository: Products.CMFPlone
c45e3bf 
Branch: refs/heads/master
Date: 2021-12-21T00:12:41+01:00
Author: Maurits van Rees (mauritsvanrees) <maurits@vanrees.org>
Commit: plone/Products.CMFPlone@8097d18

Removed management_page_charset support from usergroup-groupdetails page.

This is related to deprecated unicode property types, like ustring.
Core Plone should not need them, but there could be custom groups that use them,
so we keep the compatibility checks in here.
Part of plone/Products.CMFPlone#3305

Files changed:
A news/3305.bugfix
M Products/CMFPlone/controlpanel/browser/usergroups_groupdetails.pt
Repository: Products.CMFPlone

Branch: refs/heads/master
Date: 2021-12-22T11:57:29+01:00
Author: Jens W. Klein (jensens) <jk@kleinundpartner.at>
Commit: plone/Products.CMFPlone@09c3364

Merge pull request #3385 from plone/maurits-management-page-charset-remove-ustring

Removed management_page_charset support from usergroup-groupdetails

Files changed:
A news/3305.bugfix
M Products/CMFPlone/controlpanel/browser/usergroups_groupdetails.pt
mister-roboto pushed a commit to plone/buildout.coredev that referenced this issue Jan 10, 2022
@mauritsvanrees
[fc] Repository: Products.CMFDynamicViewFTI
be62276 
Branch: refs/heads/master
Date: 2021-12-20T23:25:23+01:00
Author: Maurits van Rees (mauritsvanrees) <maurits@vanrees.org>
Commit: plone/Products.CMFDynamicViewFTI@627f0a8

Get rid of Python 2 code.

pyupgrade --py37-plus

Files changed:
M Products/CMFDynamicViewFTI/__init__.py
M Products/CMFDynamicViewFTI/browser/__init__.py
M Products/CMFDynamicViewFTI/browser/typeinfo.py
M Products/CMFDynamicViewFTI/browserdefault.py
M Products/CMFDynamicViewFTI/content_for_tests.py
M Products/CMFDynamicViewFTI/fti.py
M Products/CMFDynamicViewFTI/interface.py
M Products/CMFDynamicViewFTI/interfaces.py
M Products/CMFDynamicViewFTI/permissions.py
M Products/CMFDynamicViewFTI/tests/CMFDVFTITestCase.py
M Products/CMFDynamicViewFTI/tests/__init__.py
M Products/CMFDynamicViewFTI/tests/test_browserdefault.py
M Products/CMFDynamicViewFTI/tests/test_fti.py
M setup.py
Repository: Products.CMFDynamicViewFTI

Branch: refs/heads/master
Date: 2021-12-20T23:28:54+01:00
Author: Maurits van Rees (mauritsvanrees) <maurits@vanrees.org>
Commit: plone/Products.CMFDynamicViewFTI@3451b42

Plone 6: always use lines for the view_methods property.

Part of plone/Products.CMFPlone#3305

Files changed:
A news/3305.bugfix
M Products/CMFDynamicViewFTI/fti.py
Repository: Products.CMFDynamicViewFTI

Branch: refs/heads/master
Date: 2021-12-20T23:44:02+01:00
Author: Maurits van Rees (mauritsvanrees) <maurits@vanrees.org>
Commit: plone/Products.CMFDynamicViewFTI@f9f3692

Removed Py2-only test from Products/CMFDynamicViewFTI/tests/test_fti.py

And remove six from the rest of the code.

According to a comment, originally on Py 3 it yielded a TypeError: 'ReplaceableWrapper' object is not callable.
On Plone 6 I do not see this.  But the two tests in here still fail with a NotFound.
When I try to recreate the situation of this test setup manually in Plone 6, it works fine:
create a folder and document, give them both an empty layout property, and try to view them.
So at least for a Dexterity FTI it seems to go fine.

Files changed:
M Products/CMFDynamicViewFTI/browserdefault.py
M Products/CMFDynamicViewFTI/tests/test_fti.py
M setup.py
Repository: Products.CMFDynamicViewFTI

Branch: refs/heads/master
Date: 2021-12-20T23:46:44+01:00
Author: Maurits van Rees (mauritsvanrees) <maurits@vanrees.org>
Commit: plone/Products.CMFDynamicViewFTI@6fb010d

Bumped version for breaking release.

Files changed:
M setup.py
Repository: Products.CMFDynamicViewFTI

Branch: refs/heads/master
Date: 2021-12-20T23:49:21+01:00
Author: Maurits van Rees (mauritsvanrees) <maurits@vanrees.org>
Commit: plone/Products.CMFDynamicViewFTI@510e417

Updated classifiers

Files changed:
M setup.py
Repository: Products.CMFDynamicViewFTI

Branch: refs/heads/master
Date: 2022-01-10T12:32:42+01:00
Author: Maurits van Rees (mauritsvanrees) <m.van.rees@zestsoftware.nl>
Commit: plone/Products.CMFDynamicViewFTI@345be9b

Merge pull request #23 from plone/maurits-plone6-only

Get rid of Python 2 code

Files changed:
A news/3305.bugfix
M Products/CMFDynamicViewFTI/__init__.py
M Products/CMFDynamicViewFTI/browser/__init__.py
M Products/CMFDynamicViewFTI/browser/typeinfo.py
M Products/CMFDynamicViewFTI/browserdefault.py
M Products/CMFDynamicViewFTI/content_for_tests.py
M Products/CMFDynamicViewFTI/fti.py
M Products/CMFDynamicViewFTI/interface.py
M Products/CMFDynamicViewFTI/interfaces.py
M Products/CMFDynamicViewFTI/permissions.py
M Products/CMFDynamicViewFTI/tests/CMFDVFTITestCase.py
M Products/CMFDynamicViewFTI/tests/__init__.py
M Products/CMFDynamicViewFTI/tests/test_browserdefault.py
M Products/CMFDynamicViewFTI/tests/test_fti.py
M setup.py
@mauritsvanrees
Copy link
Sponsor Member Author

All PRs have been merged.

mauritsvanrees added a commit to plone/plone.dexterity that referenced this issue Jan 19, 2022
@mauritsvanrees
Really always use a lines property for behaviors, no longer the depre…
bbf336e 
…cated ulines.

This improves the fix from the previous release.
Part of plone/Products.CMFPlone#3305.
@mauritsvanrees mauritsvanrees mentioned this issue Jan 19, 2022
Merged
mister-roboto pushed a commit to plone/buildout.coredev that referenced this issue Jan 21, 2022
@mauritsvanrees
[fc] Repository: plone.dexterity
c4e341e 
Branch: refs/heads/master
Date: 2022-01-19T21:45:30+01:00
Author: Maurits van Rees (mauritsvanrees) <maurits@vanrees.org>
Commit: plone/plone.dexterity@bbf336e

Really always use a lines property for behaviors, no longer the deprecated ulines.

This improves the fix from the previous release.
Part of plone/Products.CMFPlone#3305.

Files changed:
A news/3305.bugfix
M plone/dexterity/fti.py
Repository: plone.dexterity

Branch: refs/heads/master
Date: 2022-01-21T01:27:28+01:00
Author: Maurits van Rees (mauritsvanrees) <m.van.rees@zestsoftware.nl>
Commit: plone/plone.dexterity@c2e702b

Merge pull request #164 from plone/fix-ulines-lines

Really always use a lines property for behaviors

Files changed:
A news/3305.bugfix
M plone/dexterity/fti.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
02 type: regression 12 prio: high 24 status: ready
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mauritsvanrees