Merge testing branch into master #2529 #2563
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In a number of trials with btrfs, especially during raid changes, it was found that a practical minimum to allow for basic experimentation was 5GiB. Currently we ignore all devices below 1 GiB. This change increases our 'ignore devices below size' to 5 GiB. Includes: - Moving this definition away from build-time Django settings to plain run-time: allowing for user modification if need be. - Explanatory code comments and upstream btrfs wiki references. - Note/reference on recommended non mixed mode minimum size as 16 GiB+.
…_recognised_disk_size_to_5GB_and_define_at_run-time Increase minimum disk size to 5 GiB and define at run-time rockstor#2308
With reference to "move to python 3" GitHub rockstor#1877 it is proposed that we begin our Django update 'train' from our just prior to LTS version of (1.8.16) to the last Python 2 supported version of (1.11.29). These changes are intended to address the majority of updates required prior to our Python 2 -> 3 move as newer versions of Django than (1.11.29) will have to wait until our base code is transitioned to Python 3. N.B. these changes are not sufficient for continued function across the board but are intended as a stop-off point to encourage further developer interest and help to promote co-operation in this address of our existing technical dept. ## Includes - Remove defunct patterns module: deprecated (1.8) removed (1.10). - Url updates. -- Move to using plain list for urlpatters construct. -- Update appliances url to list format. -- Remove defunct patterns use on storageadmin/smart_manager urls. -- Remove defunct leading empty string in urlpatterns lists. -- Move remaining urlpattern list elements to url() type. Django 2.0 introduces path() to remove/replace regex requirement, with re-path() as a direct drop-in replacement. But we are not there yet. -- Normalise on url formatting, at least for now pre path() re_path(). Moved all root api page url definitions to top url file. If only one url is then left in any include, remove that include. This allows for common leading "/" prior to include without breaking the top api call, and reduces the number of files to view in order to manage api reformatting. -- Remove redundant leading "/" from all url elements. This tends to numerous (urls.W002) warnings. -- add now missing trailing '/' to include sub url lists. These now replace the prior leading '/' within each include. But they were removed due to the following warnings: ""?: (urls.W002) Your URL pattern '^/(?P<did>\d+)$' has a regex beginning with a '/'." "If this pattern is targeted in an include(), ensure the include() pattern has a trailing '/'." - Update Django requires/version specification. - Move db_router.py allow_migrate to post 1.10 format. Our prior syntax was deprecated in 1.8 and removed in 1.10. - Upgrade django-rest-framework - 3.1.1 to 3.9.3 for Django compat. This is the last version of django-rest-framework to support Python 2 - Upgrade django-oauth-toolkit and it's dependency 'Requests'. - Requests in turn required a chardet update. Moved to latest and last to support Python 2.7. - Update import for newer django-oauth-toolkit. - Highlight/todo django-ztask to Huey transition points rockstor#2276. These have now been merged out by the subsequent Huey transition work. - Django migrate --list deprecated (1.8), use showmigrations --list. - Update to last release of djangorecipe from 1.9. Includes removal of now redundant/deprecated "projectegg" option. - Move to render() from removed render_to_response() in home.py. - Remove now defunct TEMPLATE_* settings - set TEMPLATES equivalent. - Apply remaining non-hard-coded oauth2_provider migrations. Our prior move to Django 1.8 incorporated all migrations to that date, including those for our prior django-oauth-toolkit == 0.9.0. This ended up causing a column/field conflict when applying the full migrations post updating to django-oauth-toolkit == 1.1.2. Resolve by fake applying those we already have pre-applied. - Update paths in django-hack for updated eggs. - Update django-hack template to account for djangorecipe updates. Djangorecipe no longer contains the manage unit so we revert to using a mashup of path pre-definition and an upstream python 2 template for django command line management. - Update test-settings.conf.in re Django update - TEMPLATE settings. Includes some outstanding huey migrations that were missed in the django-ztask to huey move. Only affects the testing environment. - Initial modification to serializers.py - djangorestframework update. As from djangorestframework error we have: "Creating a ModelSerializer without either the 'fields' attribute or the 'exclude' attribute has been deprecated since 3.3.0, and is now disallowed." Add required fields param serializers.py - djangorestframework update. - Fix Django 1.11.29 errors re insufficient Disk Obj save() calls. From ""./bin/test -v 3 -p test_commands.py" we previously received: "ValueError: <Disk: Disk object> instance isn't saved. Use bulk=False or save the object first." in subtests of: test_bootstrap_command, and test_refresh_disk_state. Both subtests related to _update_disk_state() N.B. we may now over save in the earlier stages of _update_disk_state(). - Fix newer Django-rest-framwork warning re unordered object list. test_get (storageadmin.tests.test_appliances.AppliancesTests): "UnorderedObjectListWarning: Pagination may yield inconsistent results with an unordered object_list: <class 'storageadmin.models.appliance.Appliance'> QuerySet. - Fix another disk instance isn't saved test failure. test_compression (storageadmin.tests.test_pools.PoolTests): ValueError: <Disk: Disk object> instance isn\'t saved. Use bulk=False or save the object first.\n'] In this case we already proceed to save each disk object in turn but this newer version of Django (1.11.29) responded to bulk=False option which in turn calls each associated objects save() method. - Another Django-rest-framwork warning re unordered object list. unordered object_list: <class 'storageadmin.models.update_subscription. UpdateSubscription'> QuerySet. paginator = self.django_paginator_class(queryset, page_size) Both reported within supervisord_gunicorn_stderr.log - Add default ordering to some more models rockstor#2254 Addresses more test provoked warnings re: "UnorderedObjectListWarning: Pagination may yield inconsistent results with an unordered object_list" - Restore prior default of APPEND_SLASH = False. It is suspected that a newer Django began enabling this option if not found. - Remove CommonMiddleware and APPEND_SLASH setting for now rockstor#2254 - Add TODO's to appliances.py re exception handling. - Order pool and network device model outputs by default. - Add development logging for ongoing concerns. - Modify imports re Django parse_header and rest_framework media. - Add follow=true and APIClient to test_appliances.py. - In progress modifications to explore some ongoing test failures. - Add auto generated migration file related to Django update. It seems we have a new requirement in newer django to explicitly state our disk attached manager config.
Our django-ouath-toolkit 1.1.2 has only >= 2.0.3 requirement. However as of oauthlib 3.1.1 (2021-05-31) Python 2.7 compatibility was dropped. Pinning to 3.1.0 for now as last known Python 2.7 compatible while also having Python 3.7 compatibility. Thanks to FroggyFlox on GitHub for some assistance in this matter.
…thin_current_Python_2_constrains Update Django within current Python 2 constrains rockstor#2254
…r#2366 Add to_boolean() osi helper to address some short falls in our docker network state information extraction when transposing "docker network" commands to import state to our Django models. It is assumed that we previously had silent conversion from string 'true' via an older version of Django.
…tor#2366 We now expect a boolean: True False and previously we tested for the behaviour at the time which was a string equivalent.
…ust_be_either_True_or_False."_in_network.py [t] 'true' value must be either True or False." in network.py rockstor#2366
Add in required as of Django 1.11 django.setup() to our data_collector. It may well be that we are still doing things wrong here but adding for now to re-establish some more basic functionality. N.B. django.setup() can only be run once. See included reference.
…or_(Dashboard_backend)_fails_to_start (t) data-collector (Dashboard backend) fails to start rockstor#2368
Prevent creation of a duplicate entry if the local network domain is localdomain in the /etc/postfix/generic
…sn't already applied. rockstor#2376
…_migration Fake apply 0002_08_updates migration for oauth2_provider only if not applied rockstor#2376
…#2370 Remove trailing "/" from JS ServiceCollection, ReplicaCollection, TaskDefCollection (and it's parent TaskDef) in order to re-enable Web-UI access, post Django update. Note that this may be an issue instead within our url definitions, but it appears to fix more than it breaks. Thanks to FroggyFlox on GitHub for diagnostic assistance. Thanks to Hooverdan96 on GitHub for pointing out tasks also affected. ## Includes - Unrelated django-hack fix re typo in settings initialisation. - Further removed js hard coding of trailing "/" in url for POST operations related to scheduled task add and replication add to be compatible with our new url parsing.
…ces_and_Storage-Replication_Send_Receive_pages_fail (t) Services, Replication-Send_Receive, and Tasks pages fail rockstor#2370
Move to newer and non deprecated/removed request.data. Note that a code base wide search failed to turn up any other instances of this older/defunct capitalised variant.
…_NotImplementedError (t) request.DATA NotImplementedError rockstor#2379
Remark out model mocks, that are currently incompatible for our recently updated Django, in test_pool_scrub.py so it can be used as a proof of fix for the outstanding fixtures issue. Includes: - prior model mocks on some unit tests. - minor test fix re inadvertent follow= parameter in test_appliances.py - notes on creating appropriate test_pool_scrub_balance and test_pools fixtures/db-dumps. - minor formatting fixes. - Logging comments in pool_scrub.py _validate_pool() to ascertain current pool objects.
- Modify APITestMixin to - not skip `setUpClass()` - fetch User from fixture - Do not inherit from APITestCase in `PoolScrubTests()` as APITestMixin already inherits from it. - Create new fixture that includes newly-added oauth2_provider migrations.
Keep fixture for test_pool_scrub.py to minimal models needed.
…stor#2382 Remove extraneous oauth2_provider from fixture instance & comment. We are now attempting to minimalise all fixtures. The oauth2_provider entries in test_api.json look to be redundant. Includes - Fix typo in comment on fixture creation. - Remove redundant test_pool_scrub_balance related fixtures. - Remark out debug logging associated with issue. Not removing entirely as we are very likely to need these in related test issues to follow shortly.
…n_functional (t) Test fixtures non functional rockstor#2382
Remove a redundant re-calculation of pool.free in PUT (remove) pool command when the exact same calculation has already been made via an earlier and current Pool object @Property instantiation.
…ulating_pool_free (t) avoid recalculating pool free rockstor#2386
…stor#2388 Avoid literal dumping of the Python list type containing one or more disk names to-be-removed into user facing error message.
…oval_error_message_formatting_re_pool_free_space Improve disk removal error message formatting re pool free space rockstor#2388
Reduces code run under Huey and enables easier testing of balance command construction procedure which will be needed as we expand btrfs-raid support to for example raid1c3 and raid1c4. Also aligns with format used in pool resize regarding abstracting the command generation and doing only logging and exception handling in btrfs Huey tasks.
…fs_Huey_task_procedures_having_minimum_content Abstract pool balance command construction rockstor#2390
Remove all possible mocks and create a minimal new fixture to allow for all current passing tests to *still* be passing. Delete seemingly unused samba.json fixture.
Update scrub.py model Add django migration file
…e_calculation_re_new_raid_levels Enhance Pool size calculation re new raid levels rockstor#2525
Includes improved instructions on Stable Updates activation. We have an existing automation with our shop for activation code generation, but we are now an Open Collective making the shop redundant, bar our existing automation. We also: - Switch out 'Commercial' for 'Paid' support. - Switch out PayPal donations for Open Collective contribute options. - Switch to https for all links in code area.
Thanks to @Hooverdan96 on GitHub for their review/suggestion.
…ith_'OPEN_COLLECTIVE'_in_header_menu Replace 'SHOP' with 'OPEN COLLECTIVE' in header menu rockstor#2538
We are keeping our jslibs version in sync, and need to increase our pyproject.toml definition in line with our pending git tag.
…_a_4.5.9_base_(RC6) Bump versions to a 4.5.9 base (RC6) rockstor#2541
Our prior poetry.lock file had greenlet version 2.0.1, as a dependency of gevent. This version fails to compile under more recent/stricter versions of gcc-13. "poetry.lock" was refresh for its floating package dependencies via: "poetry update".
…64_build_failure Update indirect greenlet dependency to latest version rockstor#2543
Previously we used lsblk for this purpose, but sporadically lsblk would return an empty uuid for our just-created LUKS container. Thought to be due to it not having yet been updated re system state. Hence the use of the more specific LUKS command. ## Include - New get_luks_container_uuid(disk_byid) in luks.py. - Fresh black formatting. - Minor comment improvements. - Leaving in-place the prior general procedure lsblk based mechanism as this may server elsewhere.
…_via_keyfile_missing_uuid use cryptsetup to retrieve container uuid rockstor#2545
…kstor#2549 The rockon-delete currently only supports the "normal" case of a single RockOn model entry. In case one of these entries is duplicated, however, it currently fails. This commit thus adapt the current logic to get all entries (0, 1, or more) with a given name. Further detail script's usage (@Hooverdan96's idea). Black formatting.
continuation of: pr rockstor#2452 issue rockstor#2451
Newer systems now establish default sshd config in /usr/etc/ssh/sshd_config; and include overrides via drop-in files in /etc/ssh/sshd_config.d/*.conf. Accommodate these changes: initially instantiated in Tumbleweed. ## Includes -Establish dict of named tuple to abstract distro aware sshd config files and their paths. - Account for distro 1.7.0 onwards re sshd config. - Add PermitRootLogin mechanism by way of flag file. We use AllowUsers from initrock onwards. Enable the exclusion of the 'root' user via a flag file deletion mechanism. - Establish system constants.py file. Helps to avoid circular dependencies and normalises our system variables. - Re-factor sshd code to be centralised in ssh.py. Removes sshd knowledge requirements from services.py and initrock.py. - Improve readability via minor renaming refactoring. - Fix bug blocking creation of non existent sshd config files. Required as for Tumbleweed, on first build, we establish drop-in files rather than just edit the default os sshd files. - Abstract Subsystem sftp /path/sftp-server disablement. - Add TODO re abstraction of AllowRootLogin configuration. - Transition existing test_sftp.py to properly use fixtures. - Provide standardised instructions to create new fixtures, and run tests (test_sftp.py). - Black formatted. - Add additional mocks to avoid fs/sshd interaction. The prior sftp API tests had insufficient mocking casing unintended sshd config interactions. Fix by adding missing mocks. Facilitates the re-enablement of previously remarked out API test entries.
…anagement_deletion_script_enhancement Make rockon-delete compatible with duplicate RockOn model entries rockstor#2549
Cleanup json2.js reference rockstor#2550
During sshd sftp initialisation use os.open rather than build-in to enable permissions on edited/created sshd sftp files.
…shd_config_changes Account for Tumbleweed sshd config changes rockstor#2501
Following our transition to Poetry, some local files generated by Rockstor can still include now-legacy path to Rockstor binaries. This commit ensures these local files include the new paths to the binaries in the Poetry venv or correct when necessary at Rockstor's start time. Explicitly set file mask to defined setting or copied from target file. Check for status of associated systemd service (if defined) and restart if currently active.
Our crontabs are currently created in-place with a 644 mask. This commit instead writes the crontabs to a tempfile that is then set with a 600 mask before being moved to its final destination.
Fix CommandException handling to ignore report of already unmounted pool. ## Includes: - Improved user facing warning re Pool Delete dialog. - Avoid filesystem interaction during pool delete when: -- pool is unmounted. -- pool has exceeded its raid level redundancy. -- remove tangentially related exception handler on long removed code.
…red_config_paths_re_Poetry_build migrate hard wired config paths re poetry build rockstor#2540
…ission_on_crontabs_created_by_Rockstor Improve permissions on crontabs created by rockstor rockstor#2556
…un-mounted/un-mountable_pool Allow delete of un-mounted/un-mountable pool rockstor#2395
We are keeping our jslibs version in sync, and need to increase our pyproject.toml definition in line with our pending git tag.
…_a_4.6.0_base_(RC7) Bump versions to a 4.6.0 base (RC7) rockstor#2561
|
@FroggyFlox & @Hooverdan96 I'm merging to enable an end-to-end test of our proposed "First Stable Poetry build" rpm. Which will still be published in the testing channel repos first (but build from the master channel post this pull requests merge), and then published to the stable updates repository shortly there-after if we have no show-stoppers. |
Hooverdan96
added a commit
to Hooverdan96/rockstor-core
that referenced
this pull request
Oct 3, 2024
commit 4d60a2c Merge: 7051ade 6e08e10 Author: Philip Guyton <phillxnet@users.noreply.github.com> Date: Wed Jul 12 13:06:48 2023 +0100 Merge pull request rockstor#2617 from phillxnet/2615_Bump_versions_to_a_4.6.1_base_(Stable)_-_master_branch Bump versions to a 4.6.1 base (Stable) - master branch rockstor#2615 commit 6e08e10 Author: Philip Guyton <philip@yewtreeapps.com> Date: Wed Jul 12 13:04:14 2023 +0100 Bump versions to a 4.6.1 base (Stable) - master branch rockstor#2615 pyproject.toml build.sh commit 7051ade Merge: 9fdeff3 e3648a8 Author: Philip Guyton <phillxnet@users.noreply.github.com> Date: Tue Jul 11 17:18:10 2023 +0100 Merge pull request rockstor#2614 from phillxnet/2613_Cherry-pick_-_Update_unit_tests_re_recent_SSL_Cert_update_changes Update unit tests re recent SSL Cert update changes rockstor#2613 commit e3648a8 Author: Philip Guyton <philip@yewtreeapps.com> Date: Tue Jul 11 16:45:55 2023 +0100 Update unit tests re recent SSL Cert update changes rockstor#2611 Move test mocking from superctl to systemctl accordingly. Includes - Commented instructions on running these tests. commit 9fdeff3 Merge: c11c912 6284eb7 Author: Philip Guyton <phillxnet@users.noreply.github.com> Date: Mon Jul 10 15:34:26 2023 +0100 Merge pull request rockstor#2605 from phillxnet/2566_SSL_Certificate_update_doesn't_restart_nginx SSL Certificate update doesn't reload nginx rockstor#2566 commit 6284eb7 Author: Philip Guyton <philip@yewtreeapps.com> Date: Mon Jul 10 15:17:34 2023 +0100 SSL Certificate update doesn't reload nginx rockstor#2566 Use systemctl wrapper to reload nginx post SSL cert reconfig. From Rockstor v4.5.4-0 onwards nginx is no longer managed by supervisord. # Includes - Nginx reload not restart to avoid Web-UI service interruption and enable confirmation dialog display. commit c11c912 Merge: 65df1a6 ae3f342 Author: Philip Guyton <phillxnet@users.noreply.github.com> Date: Sun Jul 9 18:28:36 2023 +0100 Merge pull request rockstor#2601 from phillxnet/#2599_Revise_Stable_Updates_activation_instructions_re_legacy_shop_removal Revise Stable Updates activation re legacy shop removal rockstor#2599 commit ae3f342 Author: Philip Guyton <philip@yewtreeapps.com> Date: Sun Jul 9 18:08:49 2023 +0100 Revise Stable Updates activation instructions re legacy shop removal rockstor#2599 Modify our "Activate Stable updates" dialog instructions in accordance with our removal of the now legacy/redundant shop that has now been replaced by Appman's initial integration with our newly established Open Collective. commit 65df1a6 Merge: 1d4b9d5 b496902 Author: Philip Guyton <phillxnet@users.noreply.github.com> Date: Sun Jul 9 17:46:04 2023 +0100 Merge pull request rockstor#2597 from FroggyFlox/2596_Release_process_changes_in_accompanying_repositories Add GitHub Action to trigger post-release updates across repositories rockstor#2596 commit b496902 Author: FroggyFlox <flox2b@gmail.com> Date: Mon Jul 3 11:24:18 2023 -0400 Add GitHub Action to trigger post-release updates across repositories After each release, we are currently manually performing the required changes and udpates across our repositories. This commit adds a GitHub Action workflow to trigger these changes automatically. commit 1d4b9d5 Merge: 0249f52 46f8cb4 Author: Philip Guyton <phillxnet@users.noreply.github.com> Date: Tue May 30 17:05:42 2023 +0100 Merge pull request rockstor#2563 from phillxnet/master Merge testing branch into master rockstor#2529 commit 0249f52 Merge: 81a0d5f 15ae137 Author: Philip Guyton <phillxnet@users.noreply.github.com> Date: Wed Nov 2 17:18:34 2022 +0000 Merge pull request rockstor#2413 from rockstor/2412_configure_Dependabot_against_testing_branch configure Dependabot against testing branch rockstor#2412 commit 15ae137 Author: Philip Guyton <phillxnet@users.noreply.github.com> Date: Mon Oct 31 15:12:36 2022 +0000 configure Dependabot against testing branch rockstor#2412 Begin explicit configuration of GitHub Dependabot, adding a target-branch directive of "testing".
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Bring master branch up to testing 4.6.0-0 Stable Release Candidate 7 (RC7).
Fixes #2529
We can then prove our back-end build systems ready for our "First Stable Poetry build" milestone:
https://github.com/rockstor/rockstor-core/milestone/19