Add framework for extensions in API spec

[oddhack]

Now integrates all khr extensions. Build with all extensions via

$ python3 makeSpec -clean -spec khr apihtml chtml (other 'make' arguments if needed)

or build with no extensions included via

$ python3 makeSpec -clean -spec core

Note: 'makeAll' script has been replaced by 'makeSpec' which has slightly different syntax.

[bashbaug]

This is looking really nice! A few comments from a brief review:

1. It looks like some of the heading levels are slightly off for the extensions so the table of contents looks really bad. I think the extension name should be at the current level, but all other heading should be increased one level.
2. The extension APIs don't seem to be linking at all, see for example clCreateCommandQueueWithPropertiesKHR.
3. Most of the extension types don't seem to be linking at all either, though for some of these it's not clear what they would be linking to. The one exception I found is the new structure for cl_device_integer_dot_product_acceleration_properties_khr, which is linking nicely.
4. The extension enums I tried are links, but they don't seem to be linking to anything. Not sure what's going on here.
5. Editorial comment: some of the column widths seem to be "off" now, especially the clGetDeviceInfo table, where the leftmost column is very wide now. I suspect this is because it's sizing the column to fit the very long enum CL_DEVICE_INTEGER_DOT_PRODUCT_ACCELERATION_PROPERTIES_4x8BIT_PACKED_KHR text, which doesn't have <wbr> tags like the asciidoctor attributes do? One "solution" to fix this would be to omit the duplicated enum name from the text "CL_DEVICE_INTEGER_DOT_PRODUCT_ACCELERATION_PROPERTIES_4x8BIT_PACKED_KHR requires cl_khr_integer_dot_product", and just say something like "Requires cl_khr_integer_dot_product", which would be consistent with our other version text like "Missing before version 1.1."
6. It'd be nice if there were some indicator tying an extension structure (like cl_device_integer_dot_product_acceleration_properties_khr) to the extension name, similar to the extension enums.

[oddhack]

This is looking really nice! A few comments from a brief review:

1. It looks like some of the heading levels are slightly off for the extensions so the table of contents looks really bad. I think the extension name should be at the current level, but all other heading should be increased one level.

Yes. I'm holding off on adjusting that sort of stuff in order to do it in one pass. Also there will be some increased autogeneration in the extension appendices before I'm done, for the interface summaries - the existing markup is largely a placeholder from the separate documents.

2. The extension APIs don't seem to be linking at all, see for example clCreateCommandQueueWithPropertiesKHR.
3. Most of the extension types don't seem to be linking at all either, though for some of these it's not clear what they would be linking to. The one exception I found is the new structure for cl_device_integer_dot_product_acceleration_properties_khr, which is linking nicely.
4. The extension enums I tried are links, but they don't seem to be linking to anything. Not sure what's going on here.

All the extension appendices in the API spec have been done now, but the actual integration of changes / additions to the spec body remains TBD for a lot of them, which is what I'm focusing on now. Also, I'm not entirely clear on the conventions for the various asciidoctor attributes in the dictionaries so would appreciate some thoughts on (for example) these attributes extracted from the generated api-dictionary.asciidoc:

• cl_device_integer_dot_product_capabilities_khr_TYPE - formatted text with breaks
• cl_mutable_dispatch_exec_info_khr_TYPE_label - formatted text with breaks
• cl_mutable_dispatch_exec_info_khr_TYPE - xref
• clGetPlatformIDs_label - formatted text (perhaps with breaks?)
• clGetPlatformIDs - xref
• CL_CHAR_BIT_label - formatted text with breaks
• CL_CHAR_BIT - xref
• CL_CHAR_BIT_anchor - anchor

This is somewhat inconsistent:

• why are the type names suffixed with '_TYPE' but the command / enum names are not?
• cl_device_integer_dot_product_capabilities_khr (enumerated type) and cl_mutable_dispatch_exec_info_khr (struct) are treated differently even though they're both types:
  • the former gets only the formatted text attribute
  • the latter gets both formatted text and xref attributes

ISTM it would be more consistent to have

• {API_label} -> text (always)
• {API} -> xref (always),
• {API_anchor} -> anchor (only needed for enum names, I think)

and no _TYPE suffixes at all (except for API names that actually contain _TYPE, as some enums do). That would cause some markup changes, though most of the 400-odd uses are just basic scalar (cl_uint etc.) names.

5. Editorial comment: some of the column widths seem to be "off" now, especially the clGetDeviceInfo table, where the leftmost column is very wide now. I suspect this is because it's sizing the column to fit the very long enum CL_DEVICE_INTEGER_DOT_PRODUCT_ACCELERATION_PROPERTIES_4x8BIT_PACKED_KHR text, which doesn't have <wbr> tags like the asciidoctor attributes do? One "solution" to fix this would be to omit the duplicated enum name from the text "CL_DEVICE_INTEGER_DOT_PRODUCT_ACCELERATION_PROPERTIES_4x8BIT_PACKED_KHR requires cl_khr_integer_dot_product", and just say something like "Requires cl_khr_integer_dot_product", which would be consistent with our other version text like "Missing before version 1.1."

Will do.

6. It'd be nice if there were some indicator tying an extension structure (like cl_device_integer_dot_product_acceleration_properties_khr) to the extension name, similar to the extension enums.

Also will do. Thanks for the feedback!

[bashbaug]

This is somewhat inconsistent:

• why are the type names suffixed with '_TYPE' but the command / enum names are not?

• cl_device_integer_dot_product_capabilities_khr (enumerated type) and cl_mutable_dispatch_exec_info_khr (struct) are treated differently even though they're both types:

  • the former gets only the formatted text attribute
  • the latter gets both formatted text and xref attributes

Yes, I agree this is a little confusing.

The reason the type names are suffixed with _TYPE is because asciidoctor attributes are not case-sensitive so we cannot otherwise differentiate between cl_float (the type) and CL_FLOAT (the enum). More info here: #444

We could have added a suffix to everything but that hasn't been needed so far.

IIRC the struct types get handled differently because the struct description gets generated from the XML and included in the spec so we have something to link to. The non-struct types do not, so we do not include a link for them. If we added a description for the other types (say, in an appendix) then we could link to them also, but we do not do this currently, and I'm not sure how much value it would add.

[oddhack]

@neiltrevett @bashbaug this is ready now. If possible, I recommend merging #1081 to the branch underlying #950 before merging #950. If the WG is not willing to take that step (removing the separate khr extension documents) yet, it can be retargeted to main and deferred. Any outstanding khr extension work in branches should be re-done as part of the API and C specs ASAP, to avoid divergences and rebasing pain.

I am pretty confident in how this looks now, but definitely review needed before publishing. I'm sure there will be some minor cleanup to deal with, and it would be an excellent idea to run a link-checker over the C and API specs prior to publication - there are many more internal links and a fair number of external links from the API spec extension appendices into the C spec.

I strongly recommend squash-on-merge when accepting #950. There is a huge amount of churn in the commits as this took many months to do incrementally, none of which is of particular interest to downstream extension work.

[oddhack]

Another good consistency check will be a careful comparison of 'makeSpec -spec core chtml apihtml' with the current HTML from main branch. I have not done this for a while and while the differences should not be big, there are unavoidable diffs. N.b. I find 'wdiff -3 new.html old.html' to be quite useful when changes are small. There are some HTML comparison tools we use in Vulkan but they are... not great. I wish I knew of a great one.

[bashbaug]

Discussed in the March 12th teleconference and agreed to merge these changes in principle. We'll wait for the CI builds to complete and give one more quick review now that #1081 is merged, but assuming all goes well we'll merge these changes ASAP.

[bashbaug]

@outofcontrol can you please check what's happening with the CLA bot for this PR? Thanks!

[KhronosWebservices]

My bad. Used the wrong account for something which ate up the API rate limit for the account we use for cla-assisstant. :(

Will be up and running again within the hour.

[oddhack]

@bashbaug if you want, I can just go ahead and merge this - not sure if you have those permissions on the repo or not? You probably should.

Since I'm a Khronos member I'm already covered for my contributions.

[KhronosWebservices]

CLA-Assistant is responding again. Might be good to close and re-open this PR, have the reviewer sign-off, and the CLA check should then be available.

[bashbaug]

Merging as discussed in the March 12th and March 19th teleconferences.

[SunSerega]

Can someone please explain what exactly being ratified means here? Is it something similar to being an ARB extension in OpenGL? Does it mean an extension is planned to be added to the future version of the core spec?

[oddhack]

Can someone please explain what exactly being ratified means here? Is it something similar to being an ARB extension in OpenGL? Does it mean an extension is planned to be added to the future version of the core spec?

Yes, it is similar. Ratification has no effect on developers or users, but is of concern to implementers who want to be sure they do not implement functionality that is not covered by the reciprocal IP licensing obligations of the Khronos IP agreements. It means that Khronos has agreed that a core version or extension is covered by those agreements. There is no obligation or expectation that a ratified extension will become part of a future core API version, and many will not. Conversely non-ratified vendor or multivendor extensions are sometimes promoted to the core API, at which time they are ratified.

[SunSerega]

Thanks. Also, in registry.rnc:

#   depends - boolean expression of API and/or extension names
#       upon which this extension depends.

I would intuitively expect and/& as operators since this value is said to be a boolean expression, but currently I only see a + operator.
Well, it can be seen as a boolean operator too (but maybe or, not and?), but how about specifying this in the registry.rnc in more detail?
And does this mean the value can be something like (a+b) * (c+d) in the future?

And another thing:

        <extension name="cl_arm_get_core_id" condition="defined(CL_VERSION_1_2)" supported="opencl">
        <extension name="cl_ext_image_requirements_info" condition="defined(CL_VERSION_1_2)" supported="opencl">
        <extension name="cl_ext_image_from_buffer" condition="defined(CL_VERSION_1_2)" supported="opencl">

These were probably supposed to use this depends attribute instead of condition, since it now supports both extension and feature dependencies?