Re-integrate next_infra into develop #5616

joshmoore · 2018-01-15T07:19:40Z

What this PR does

Re-integrates the next_infra branch back into develop. Once merged, the next_infra integration branch will be deleted. The following PRs will then be included for release in 5.4.2:

Users

CLI

have import fail fast if into illegal container #5607: bin/omero import --target fix
allow SciJava native lib loader loglevel to be configured #5610: bin/omero import --debug fix
extend CLI admin email timeout #5608: bin/omero admin mail timeout improvements (*)
add a "log" command to RawAccessRequest #5538: bin/omero admin log added command (*)

Devs

Python

Bump omero-marshal to 0.5.2 #5595: new omero-marshal (0.5.2) with masks support
Rename tools/test_setup.py to OmeroPy/src/omero_setup.py #5585: Ship omero_setup.py for downstream build systems (*)
Test scripts #5573: Refactor test classes (scripts) for downstream build systems
Test cli #5548: Refactor test classes (CLI) for downstream build systems
Note: the build system changes were designed to permit integration tests in downstream projects, see:

Java

Add update tables method #5490: add methods for updating OMERO.tables (*)
Java gateway bugfixes #5513: fixes for the Java Gateway (sessions) (*)
Gateway methods #5545: fixes for the Java Gateway (rendering)

(*): includes new API that should be signed off on

A new "tracked service" internal API has been added. All calls to `_create_func` in `ProxyObjectWrapper` now registers the string representation of the created service as well as the current call stack to the owning `BlitzGateway` instance. If on deletion, services are left open, then an ERROR message including the call stack will be printed.

Avoids potential for confusion with messages genuinely from the RawAccessRequest class' logger.

define in Constants.ice rather than Repositories.ice

Intended only for internal use, testing and diagnostics. Good candidate for early microservice.

add a "log" command to RawAccessRequest

This release adds support for Masks - see https://github.com/openmicroscopy/omero-marshal/blob/v0.5.2/CHANGELOG.md

extend CLI admin email timeout

have import fail fast if into illegal container

allow SciJava native lib loader loglevel to be configured

Bump omero-marshal to 0.5.2

joshmoore · 2018-01-16T08:17:04Z

History of the PRs included here:

0fabfb0e36 (origin/next_infra) Merge pull request #5595 from sbesson/omero_marshal_0.5.2 (19 hours ago) <Josh Moore>
2b7b7fe03e Merge pull request #5610 from mtbc/fix-loglevel (25 hours ago) <Josh Moore>
1c9a8d97d1 Merge pull request #5607 from mtbc/import-target (25 hours ago) <Josh Moore>
622b27da71 Merge pull request #5608 from mtbc/extend-email-timeout (4 days ago) <Josh Moore>
077384b7e4 Merge pull request #5538 from mtbc/log-cmd (6 weeks ago) <Josh Moore>
8b8be4ea7d (origin/clisplit) Merge pull request #5585 from joshmoore/test_omero_setup (7 weeks ago) <Josh Moore>
3a9dedaef1 Merge pull request #5490 from dominikl/add_update_tables (8 weeks ago) <Josh Moore>
e4a1a63afa Merge pull request #5573 from jburel/test-scripts (8 weeks ago) <Josh Moore>
54d6f3bdb3 Merge pull request #5548 from jburel/test-cli (8 weeks ago) <Josh Moore>
48d5860ebc Merge pull request #5513 from dominikl/java_gateway_fixes (8 weeks ago) <Josh Moore>
b476a51f22 Merge pull request #5545 from jburel/gateway-methods (8 weeks ago) <Josh Moore>

jburel · 2018-01-16T09:54:30Z

components/blitz/src/omero/gateway/Gateway.java

@@ -1220,7 +1234,9 @@ private void keepSessionAlive() throws DSOutOfServiceException {
            c = i.next();
            if (c.needsKeepAlive()) {
                if (!c.keepSessionAlive()) {
-                    throw new DSOutOfServiceException("Network not available");
+                    // Session has died, e. g. due to server restart.


@dominikl how does insight handle the fact that no error is thrown now?

Should work as before (show connection warning and reconnect again, when possible), tested on the original PR #5513 (point 3).

As far as I remember the Exception was the problem, because this reconnect mechanism only worked for a stale network connection, but not when the server was restarted. In that case even when the server connection is back, keepSessionAlive fails, and the exception crashed Insight. With this change the stale connector is quietly removed and a new connector will be created.

jburel · 2018-01-16T09:56:49Z

components/blitz/src/omero/gateway/facility/AdminFacility.java

@@ -392,6 +404,9 @@ public void setAdminPrivileges(SecurityContext ctx, ExperimenterData user,
    public void addAdminPrivileges(SecurityContext ctx, ExperimenterData user,
            Collection<String> privileges) throws DSOutOfServiceException,
            DSAccessException {
+        if(!Pojos.hasID(user) || CollectionUtils.isEmpty(privileges))


space between if (

jburel · 2018-01-16T09:57:09Z

components/blitz/src/omero/gateway/facility/AdminFacility.java

@@ -421,6 +436,8 @@ public void addAdminPrivileges(SecurityContext ctx, ExperimenterData user,
    public void removeAdminPrivileges(SecurityContext ctx,
            ExperimenterData user, Collection<String> privileges)
            throws DSOutOfServiceException, DSAccessException {
+        if(!Pojos.hasID(user) || CollectionUtils.isEmpty(privileges))


jburel · 2018-01-16T09:57:48Z

components/blitz/src/omero/gateway/facility/BrowseFacility.java

@@ -108,23 +113,32 @@
    }

    /**
-     * Retrieves hierarchy trees rooted by a given node.
-     * i.e. the requested node as root and all of its descendants.
+     * Retrieves hierarchy trees rooted by a given node. i.e. the requested node


extra . before i.e.

jburel · 2018-01-16T10:00:08Z

components/blitz/src/omero/gateway/facility/MetadataFacility.java

@@ -83,6 +87,9 @@
     */
    public ImageAcquisitionData getImageAcquisitionData(SecurityContext ctx,
            long imageId) throws DSOutOfServiceException, DSAccessException {
+        if(imageId < 0)


space after if

jburel · 2018-01-16T10:34:29Z

components/blitz/src/omero/gateway/facility/TablesFacilityHelper.java

@@ -225,6 +226,8 @@ void parseData(Data data, TableDataColumn[] header) {
                for (int j = 0; j < nRows; j++) {
                    MaskData md = new MaskData(mc.x[j], mc.y[j], mc.w[j],
                            mc.h[j], mc.bytes[j]);
+                    md.setZ(mc.theZ[j]);


Will theZ and theT always be filled?

I'd expect that both are always arrays of nRows size, but you're right, I guess the values could be -1 if z and t are not explicitly set. Which is actually a bug in ShapeData (getZ and getT should return 0 not -1 in that case). Follow-up PR maybe?

Or does -1 for z/t mean "all" z/t ? As channel can also be -1.
The getters and setters in ShapeData are all inconsistent, e. g. if z is not set, getZ() will return -1, but if you setZ(-1) actually 0 is set!

I think I leave it as it is for now.

don't modify in this PR
Changing the getter and setter should be done in another PR

jburel · 2018-01-16T10:35:11Z

components/blitz/src/omero/gateway/facility/TablesFacilityHelper.java

+        if (toUpdate.rowNumbers.length != data.getData()[0].length)
+            throw new IllegalArgumentException("Row size is different!");
+
+        for (int c = 0; c < data.getColumns().length; c++) {


same cols = data.getColumns()
and for data.getData()

jburel · 2018-01-16T10:49:49Z

components/tools/OmeroJava/test/integration/gateway/TablesFacilityTest.java

@@ -276,5 +281,46 @@ public void testThreshold() throws Exception {
        Assert.assertEquals(td.getData()[0].length,
                TablesFacility.DEFAULT_MAX_ROWS_TO_FETCH);
    }
+
+    @Test(dependsOnMethods = { "testThreshold" })
+    public void testUpdateTable() throws Exception {


cols= td.getColumns() same for data
since the various will have an effect on performance

I'll leave @dominikl to respond, but I think these are all local invocations since it's omero.gateway.model.TableData and not omero.api.TablesPrx.

in that case no need to make changes
I got lost in the PR review

Yes, just accesses the columns array in the TableData object.

jburel · 2018-01-16T10:50:30Z

components/tools/OmeroPy/src/omero/gateway/__init__.py

+
+    def _unregister_service(self, service_string):
+        """
+        Called on close of a service.


jburel · 2018-01-16T10:53:32Z

components/tools/OmeroPy/test/integration/clitest/cli.py

@@ -94,6 +104,9 @@ class ArgumentFixture(object):
    """
    Used to test the user/group argument
    """
+    warnings.warn("Deprecated in 5.4.1."


message should be 5.4.2 now

Next infra fixes

Without the exclusion, python files appear under the OMERO scripts menu which lead to exceptions ("No params found").

joshmoore · 2018-01-18T12:17:35Z

Based on today's greenedness, merging to prepare for release. This is a last call for objections to the API additions (*) noted in the description.

joshmoore and others added 30 commits October 7, 2017 17:56

render/gateway: close as many rendering engines as possible

9d7f974

Fix flake8

5b4fd30

Remove usage of guest session

e9efd2b

Also throw DSOutOfServiceException on connection loss

4fe85ae

Add join session 'LoginCredentials' and test

287771d

Fix: Don't swallow Exception for Gateway.getSessionId()

3e636bc

Add accidentaly commented out tests again

acc7426

Use Java's UUID method instead of regex

9b8797d

Catch exception

88abb1e

Make Gateway AutoClosable

80342c1

Remove stale connectors

8c58265

Check arguments

a3915ee

Added update method

5c8b424

javadoc: add note that one can't change table size

6b99631

add a "log" command to RawAccessRequest

0d2d29f

differently name manual RawAccessRequest logger

6702988

Avoids potential for confusion with messages genuinely from the RawAccessRequest class' logger.

getHierarchy without ids is a valid request (~ 'get all')

edbf0cd

Just clarified doc for getHierarchy method

c094f0a

Removed printStackTrace from integration test

a1f1a73

Adjusted javadoc comment

84a45bf

Add new method

7d49f1c

Add documentation

8c273ba

Add tests

3e060ca

flake8

622a217

Add tests for register/unregister method

9b189ab

add test for _closeRE method

92fde04

Add doc

14bfedd

Make sure rendering engines are closed

4f62e3e

Introduce new file in testlib and deprecate one

c776cc6

mtbc and others added 13 commits December 6, 2017 10:12

move omero.grid.LogLevel to omero.constants.LogLevel

98da1d6

define in Constants.ice rather than Repositories.ice

deprecate RawAccessRequest

4d287f0

Intended only for internal use, testing and diagnostics. Good candidate for early microservice.

Merge pull request #5538 from mtbc/log-cmd

077384b

add a "log" command to RawAccessRequest

Bump omero-marshal to 0.5.2

9dcf737

This release adds support for Masks - see https://github.com/openmicroscopy/omero-marshal/blob/v0.5.2/CHANGELOG.md

Add Mask shape to the fixture of the JSON API ROI integration tests

78131ff

prevent import targeting a container where canLink is false

321e6b4

extend timeout for sending e-mail from CLI

625c0f8

allow CLI admin email to have --wait argument set

63595a5

apply selected log level to SciJava native library loader

9fb3908

tidy class header

c341f69

Merge pull request #5608 from mtbc/extend-email-timeout

622b27d

extend CLI admin email timeout

Merge pull request #5607 from mtbc/import-target

1c9a8d9

have import fail fast if into illegal container

Merge pull request #5610 from mtbc/fix-loglevel

2b7b7fe

allow SciJava native lib loader loglevel to be configured

joshmoore added this to the 5.4.2 milestone Jan 15, 2018

joshmoore mentioned this pull request Jan 15, 2018

Bump omero-marshal to 0.5.2 #5595

Merged

Merge pull request #5595 from sbesson/omero_marshal_0.5.2

0fabfb0

Bump omero-marshal to 0.5.2

joshmoore changed the title ~~Merge next_infra branch into develop for 5.4.2~~ Re-integration next_infra into develop Jan 16, 2018

joshmoore changed the title ~~Re-integration next_infra into develop~~ Re-integrate next_infra into develop Jan 16, 2018

jburel reviewed Jan 16, 2018

View reviewed changes

joshmoore added 2 commits January 16, 2018 16:07

End trailing whitespace

98be4f4

Fix suggestions from jburel

34b2050

joshmoore mentioned this pull request Jan 16, 2018

Next infra fixes #5618

Merged

joshmoore added 2 commits January 17, 2018 09:20

Merge pull request #5618 from joshmoore/next_infra_fixes

00a8913

Next infra fixes

Correct more 'deprecated in 5.4.1' lines

c2dc7c0

hflynn mentioned this pull request Jan 17, 2018

5.4.2 version history #5623

Merged

5 tasks

scripts: Exclude setup.py and test/**

2bfba62

Without the exclusion, python files appear under the OMERO scripts menu which lead to exceptions ("No params found").

joshmoore merged commit 0901b07 into develop Jan 18, 2018

joshmoore deleted the next_infra branch January 31, 2018 10:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Re-integrate next_infra into develop #5616

Re-integrate next_infra into develop #5616

joshmoore commented Jan 15, 2018 •

edited

Loading

joshmoore commented Jan 16, 2018

jburel Jan 16, 2018

dominikl Jan 16, 2018 •

edited

Loading

jburel Jan 16, 2018

jburel Jan 16, 2018

jburel Jan 16, 2018

jburel Jan 16, 2018

jburel Jan 16, 2018

dominikl Jan 16, 2018

dominikl Jan 16, 2018

jburel Jan 16, 2018

jburel Jan 16, 2018

jburel Jan 16, 2018

joshmoore Jan 16, 2018

jburel Jan 16, 2018

dominikl Jan 16, 2018

jburel Jan 16, 2018

jburel Jan 16, 2018

joshmoore commented Jan 18, 2018

Re-integrate next_infra into develop #5616

Re-integrate next_infra into develop #5616

Conversation

joshmoore commented Jan 15, 2018 • edited Loading

What this PR does

Users

CLI

Devs

Python

Java

joshmoore commented Jan 16, 2018

Choose a reason for hiding this comment

dominikl Jan 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joshmoore commented Jan 18, 2018

joshmoore commented Jan 15, 2018 •

edited

Loading

dominikl Jan 16, 2018 •

edited

Loading