Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python: referencing imported nested types #703

Closed
GreyCat opened this issue Feb 17, 2020 · 3 comments
Closed

Python: referencing imported nested types #703

GreyCat opened this issue Feb 17, 2020 · 3 comments
Assignees

Comments

@GreyCat
Copy link
Member

GreyCat commented Feb 17, 2020

Hi, I wonder if this is already implemented, or I'm using it the wrong way.
I'm trying to consolidate various individual ksy files into one file, and then calling those as subtypes, but I'm not getting the expected results at least under python. However using opaques, It works.
How should I use it, if it is implemented?

Follow is the example

File test1.ksy

meta:
  id: test1
  imports:
    - test2

seq:
  - id: test3
    type: test2::test3

I got the following python, no import test2, and no test2.Test3 neither

class Test1(KaitaiStruct):
    def __init__(self, _io, _parent=None, _root=None):
        self._io = _io
        self._parent = _parent
        self._root = _root if _root else self
        self._read()

    def _read(self):
        self.test3 = self._root.Test3(self._io, self, self._root)

However, if I use opaque-types as follows:

File test1.ksy

meta:
  id: test1
  ks-opaque-types: true

seq:
  - id: test3
    type: test2::test3

Then python code is:

import test2
class Test1(KaitaiStruct):
    def __init__(self, _io, _parent=None, _root=None):
        self._io = _io
        self._parent = _parent
        self._root = _root if _root else self
        self._read()

    def _read(self):
        self.test3 = test2.self._root.Test3(self._io)

Originally posted by @LmpOSX in #35 (comment)

@GreyCat
Copy link
Member Author

GreyCat commented Feb 17, 2020

I confirm that (1) the syntax is correct, (2) this seems to be broken now and needs fixing.

It looks like we do have a::b::c syntax test in nested_types3, but we don't have the relevant test for that + imported types.

@generalmimon generalmimon self-assigned this Jun 21, 2020
@kjczarne
Copy link

Facing the same issue here. Trying to put a bunch of enums in a separate file and import and reference them. Doesn't work in Python.

@Mimickal
Copy link

I did some fiddling with this. Top-level imports work, but anything under types or enums from an imported file doesn't get code-generated properly. Kaitai will accept that as valid ksy, but the output code will be missing the imports.

If you use a top-level import dummy (as some comments have suggested), code-gen will put an import your_module_here statement, however, using individual types and enums still does not work. I also found that ks-opaque-types: true has no impact on this generation.

Given this ksy

meta:
  id: example
  endian: le
  imports:
    - shared
seq:
  - id: field1
    type: shared::thing
types:
  dummy:
    seq:
      - id: dummy
        type: shared

We get this output

from pkg_resources import parse_version
import kaitaistruct
from kaitaistruct import KaitaiStruct, KaitaiStream, BytesIO


if parse_version(kaitaistruct.__version__) < parse_version('0.9'):
    raise Exception("Incompatible Kaitai Struct Python API: 0.9 or later is required, but you have %s" % (kaitaistruct.__version__))

# The import we want to see!
import shared 
class Example(KaitaiStruct):
    def __init__(self, _io, _parent=None, _root=None):
        self._io = _io
        self._parent = _parent
        self._root = _root if _root else self
        self._read()

    def _read(self):
        # This still doesn't work because we need shared.Shared	
        self.field1 = Shared.Thing(self._io, self, self._root)

    class Dummy(KaitaiStruct):
        def __init__(self, _io, _parent=None, _root=None):
            self._io = _io
            self._parent = _parent
            self._root = _root if _root else self
            self._read()

        def _read(self):
            self.dummy = shared.Shared(self._io)

So I suppose this means we can import a single type (provided there is only one per-file as the top-level seq), however we cannot use imported enums currently.

Also worth pointing out no import statements are generated for the Construct target.

CEnnis91 added a commit to smash64-dev/krec-struct-format that referenced this issue Feb 26, 2022
generalmimon added a commit to kaitai-io/kaitai_struct_tests that referenced this issue Mar 18, 2024
generalmimon added a commit to kaitai-io/kaitai_struct_compiler that referenced this issue Mar 20, 2024
Related:
* kaitai-io/kaitai_struct#703
* kaitai-io/kaitai_struct#963

Until now, using a nested type of an imported spec could affect the
derived parent type of that nested type. For example:

`main.ksy`

```ksy
meta:
  id: main
  imports:
    - imported

seq:
  - id: foo
    type: imported::nested_type
```

`imported.ksy`

```ksy
meta:
  id: imported
types:
  nested_type: {}
```

If you compile `main.ksy` for Java (`kaitai-struct-compiler -t java
main.ksy`), `Imported.java` looks like this:

```java
public class Imported extends KaitaiStruct {
    // ...
    public static class NestedType extends KaitaiStruct {
        public NestedType(KaitaiStream _io, Main _parent, Imported _root) { /* ... */ }
        //                                  ^^^^^^^^^^^^
    }
    // ...
}
```

Notice the `Main _parent` parameter - the compiler saw that
`imported::nested_type` is only used from the `main` type, so it decided
that the `_parent` type of `nested_type` will be `main`.

However, this means that the `_parent` crosses KSY spec boundaries and
the generated `Imported.java` code will be different depending on
whether you compile it as imported from `main.ksy` or standalone.
Furthermore, you could even access fields of `main` in
`imported::nested_type` via `_parent`, but that would mean that
`imported.ksy` would only work when imported and compiled via
`main.ksy`.

From <kaitai-io/kaitai_struct#71 (comment)>,
I suppose none of this should be possible:

> It would be a huge mess if `_root` relied on this particular ksy being
> imported from some other ksy, and only work in that case.

I agree that `_root` and `_parent` arguably shouldn't cross spec
boundaries at all, they should only be passed locally within one .ksy
spec, and therefore also parent types should only be derived from local
type usages.

This commit only adjusts the parent type derivation, not invocations of
imported nested types with `_parent` and `_root` still being passed (see
also kaitai-io/kaitai_struct#963) - they will
be fixed later.
generalmimon added a commit to kaitai-io/kaitai_struct_compiler that referenced this issue Mar 23, 2024
Related:
* kaitai-io/kaitai_struct#703
* kaitai-io/kaitai_struct#963

Until now, using a nested type of an imported spec could affect the
derived parent type of that nested type. For example:

`main.ksy`

```ksy
meta:
  id: main
  imports:
    - imported

seq:
  - id: foo
    type: imported::nested_type
```

`imported.ksy`

```ksy
meta:
  id: imported
types:
  nested_type: {}
```

If you compile `main.ksy` for Java (`kaitai-struct-compiler -t java
main.ksy`), `Imported.java` looks like this:

```java
public class Imported extends KaitaiStruct {
    // ...
    public static class NestedType extends KaitaiStruct {
        public NestedType(KaitaiStream _io, Main _parent, Imported _root) { /* ... */ }
        //                                  ^^^^^^^^^^^^
    }
    // ...
}
```

Notice the `Main _parent` parameter - the compiler saw that
`imported::nested_type` is only used from the `main` type, so it decided
that the `_parent` type of `nested_type` will be `main`.

However, this means that the `_parent` crosses KSY spec boundaries and
the generated `Imported.java` code will be different depending on
whether you compile it as imported from `main.ksy` or standalone.
Furthermore, you could even access fields of `main` in
`imported::nested_type` via `_parent`, but that would mean that
`imported.ksy` would only work when imported and compiled via
`main.ksy`.

From <kaitai-io/kaitai_struct#71 (comment)>,
I suppose none of this should be possible:

> It would be a huge mess if `_root` relied on this particular ksy being
> imported from some other ksy, and only work in that case.

I agree that `_root` and `_parent` arguably shouldn't cross spec
boundaries at all, they should only be passed locally within one .ksy
spec, and therefore also parent types should only be derived from local
type usages.

This commit only adjusts the parent type derivation, not invocations of
imported nested types with `_parent` and `_root` still being passed (see
also kaitai-io/kaitai_struct#963) - they will
be fixed later.
generalmimon added a commit to kaitai-io/kaitai_struct_compiler that referenced this issue Mar 30, 2024
Related:
* kaitai-io/kaitai_struct#703
* kaitai-io/kaitai_struct#963

Until now, using a nested type of an imported spec could affect the
derived parent type of that nested type. For example:

`main.ksy`

```ksy
meta:
  id: main
  imports:
    - imported

seq:
  - id: foo
    type: imported::nested_type
```

`imported.ksy`

```ksy
meta:
  id: imported
types:
  nested_type: {}
```

If you compile `main.ksy` for Java (`kaitai-struct-compiler -t java
main.ksy`), `Imported.java` looks like this:

```java
public class Imported extends KaitaiStruct {
    // ...
    public static class NestedType extends KaitaiStruct {
        public NestedType(KaitaiStream _io, Main _parent, Imported _root) { /* ... */ }
        //                                  ^^^^^^^^^^^^
    }
    // ...
}
```

Notice the `Main _parent` parameter - the compiler saw that
`imported::nested_type` is only used from the `main` type, so it decided
that the `_parent` type of `nested_type` will be `main`.

However, this means that the `_parent` crosses KSY spec boundaries and
the generated `Imported.java` code will be different depending on
whether you compile it as imported from `main.ksy` or standalone.
Furthermore, you could even access fields of `main` in
`imported::nested_type` via `_parent`, but that would mean that
`imported.ksy` would only work when imported and compiled via
`main.ksy`.

From <kaitai-io/kaitai_struct#71 (comment)>,
I suppose none of this should be possible:

> It would be a huge mess if `_root` relied on this particular ksy being
> imported from some other ksy, and only work in that case.

I agree that `_root` and `_parent` arguably shouldn't cross spec
boundaries at all, they should only be passed locally within one .ksy
spec, and therefore also parent types should only be derived from local
type usages.

This commit only adjusts the parent type derivation, not invocations of
imported nested types with `_parent` and `_root` still being passed (see
also kaitai-io/kaitai_struct#963) - they will
be fixed later.
generalmimon added a commit to kaitai-io/kaitai_struct_compiler that referenced this issue Apr 5, 2024
See the same change in Lua: 1a0b2c0

This would have fixed the `NestedTypesImport` test for JavaScript if it
weren't for the missing import of `NestedTypes3` it uses (in other
words, this test still fails only because of
kaitai-io/kaitai_struct#703).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants