[Texture support][Part 0] Device API and runtime support #7711

csullivan · 2021-03-19T23:21:47Z

This PR introduces 2d texture memory support to the OpenCL Device API runtime.

Device runtime

The device runtime supports allocating texture memory both as a temporal workspace and as a runtime data space. In the latter case, special invocation of AllocDataSpace with a memory_scope == "texture(:weight)" is required. Special memory scopes were added to the runtime in [Runtime] Special Memory Scope Support #7488.
Workspace allocations are handled via a set of idle texture pools which are grown to match the requested sizes. The strategy employed is to first pick the pool which requires the least amount of extra space beyond, and then to minimize the amount of wasted space that growing a two dimensional pool may incur. A similar approach is taken for the ahead of time graph runtime memory planner for data space allocations (see: [Texture support][Part 5] Graph runtime and memory planning support for 2d allocations #7690).
CopyFromTo support is expanded to handle the case of directly reading from / writing to image buffers from host.

See RFC here: https://discuss.tvm.apache.org/t/rfc-texture-memory-support/9467

include/tvm/runtime/device_api.h

src/runtime/c_runtime_api.cc

src/runtime/texture.h

tqchen · 2021-03-20T00:20:55Z

Thanks @csullivan some quick comments, will read more carefully in the incoming week.

csullivan · 2021-05-04T22:46:03Z

@tqchen, @ZihengJiang, would you kindly consider reviewing once more?

The main change is to remove texture specific device apis and rely on tir.tvm_call_packed (cf #7932) for texture workspace allocations and AllocDataSpace (w/ scope).

I also introduced an opencl buffer descriptor that tracks the allocation layout. With the layout and the DLTensor CopyDataFromTo overload I've verified that a sub-texture allocation can be correctly copied out of a 2d texture pool of larger extent. This solves an issue I raised in Part 4.

I appreciate any additional feedback you have.

ZihengJiang · 2021-05-11T12:36:55Z

LGTM. @tqchen, could you check whether the pr look good to you?

src/runtime/opencl/opencl_common.h

src/runtime/opencl/opencl_device_api.cc

src/runtime/texture_pool.cc

tqchen · 2021-05-11T13:23:02Z

src/runtime/opencl/opencl_device_api.cc

+      case cl::BufferDescriptor::MemoryLayout::IMAGE_2D_ACTIVATION:
+      case cl::BufferDescriptor::MemoryLayout::IMAGE_2D_WEIGHT:
+        auto image_info = GetImageInfo(from_desc, from);
+        // TODO(csullivan): Support calculating row_pitch correctly in the case of reuse.


Would be great to add a few testcases in python that demonstrates the copy into image where image size is bigger than the normal one. Perhaps the easiest way is to construct an NDArray then write a PackedFunc that takes a smaller view from it.

Thank you for the good suggestion, I added a test to demonstrate writing to a subview of a texture and check that the data from larger allocation contains the expected two dimensional strides when copied back to host.

This should be replaced with AOT memory planning when the relay/tir/compile engine refactor lands.

allocations and copying to/from host/image2d directly. Allocation employs a lowering convention to 2d images for activations and weights.

utilities that can be shared by codegen and the runtime.

git@github.com:ZihengJiang/tvm.git:52822c5bd [RUNTIME] OpenCL texture memory.

memory layout through OpenCL Device API.

overload and tensor shapes to calculate image extent when copying date directly to or from texture cache.

of storage allocs of texture scope.

csullivan · 2021-05-29T22:08:17Z

@tqchen Thanks for the great feedback, could you take a look again?

feedback addressed

tqchen · 2021-06-04T12:52:50Z

Thanks @csullivan . will let @ZihengJiang manage the PR

ZihengJiang · 2021-06-05T05:06:56Z

Merged now. Thanks @csullivan for the hard working.

@ZihengJiang

* Add TVMBackendAllocTexture and support in OpenCL device API. * Add runtime optimized caching allocator. This should be replaced with AOT memory planning when the relay/tir/compile engine refactor lands. * Few bug fixes for runtime texture allocator. * Add OpenCL device api support for image2d<float16> textures. * Update OpenCL DeviceAPI to support Image2D data space allocations and copying to/from host/image2d directly. Allocation employs a lowering convention to 2d images for activations and weights. * Fix to follow OpenCL spec. for indexing. * Rename texture_pool.h -> texture.h * Move Nd to 2d lowering convention code into runtime texture utilities that can be shared by codegen and the runtime. * Update texture lowering utilities * Add TODO comment about pitch support * Remove FreeTexture * Fix ICHECK comment * Partial cherry pick from @ZihengJiang git@github.com:ZihengJiang/tvm.git:52822c5bd [RUNTIME] OpenCL texture memory. * Remove runtime and device texture APIs. * Add OpenCL packed functions for texture workspace (de)allocations. * Add OpenCLBuffer structure to track memory layout through OpenCL Device API. * Rebase: TVMContext -> Device * Implement DLTensor* overload of CopyDataToFrom in OpenCL DeviceAPI. * Implement OpenCL CopyDataFromTo(DLTensor*...) overload and tensor shapes to calculate image extent when copying date directly to or from texture cache. * Update format (cpp-lint) * Update format (clang) * Buffer descriptor name change and formatting. * Add texture pool documentation. * Update runtime to use new global.texture scope. * Move texture_pool.cc into opencl impl. * Add test coverage for copying in and out of storage allocs of texture scope. * Documented APIs and structures, renamed buffer descriptor layout tags. Co-authored-by: ZihengJiang <ziheng@apache.org>

@ZihengJiang

* Add TVMBackendAllocTexture and support in OpenCL device API. * Add runtime optimized caching allocator. This should be replaced with AOT memory planning when the relay/tir/compile engine refactor lands. * Few bug fixes for runtime texture allocator. * Add OpenCL device api support for image2d<float16> textures. * Update OpenCL DeviceAPI to support Image2D data space allocations and copying to/from host/image2d directly. Allocation employs a lowering convention to 2d images for activations and weights. * Fix to follow OpenCL spec. for indexing. * Rename texture_pool.h -> texture.h * Move Nd to 2d lowering convention code into runtime texture utilities that can be shared by codegen and the runtime. * Update texture lowering utilities * Add TODO comment about pitch support * Remove FreeTexture * Fix ICHECK comment * Partial cherry pick from @ZihengJiang git@github.com:ZihengJiang/tvm.git:52822c5bd [RUNTIME] OpenCL texture memory. * Remove runtime and device texture APIs. * Add OpenCL packed functions for texture workspace (de)allocations. * Add OpenCLBuffer structure to track memory layout through OpenCL Device API. * Rebase: TVMContext -> Device * Implement DLTensor* overload of CopyDataToFrom in OpenCL DeviceAPI. * Implement OpenCL CopyDataFromTo(DLTensor*...) overload and tensor shapes to calculate image extent when copying date directly to or from texture cache. * Update format (cpp-lint) * Update format (clang) * Buffer descriptor name change and formatting. * Add texture pool documentation. * Update runtime to use new global.texture scope. * Move texture_pool.cc into opencl impl. * Add test coverage for copying in and out of storage allocs of texture scope. * Documented APIs and structures, renamed buffer descriptor layout tags. Co-authored-by: ZihengJiang <ziheng@apache.org>

csullivan mentioned this pull request Mar 20, 2021

[Texture support][Part 1] TIR lowering and OpenCL support #7686

Merged

tqchen requested changes Mar 20, 2021

View reviewed changes

include/tvm/runtime/device_api.h Outdated Show resolved Hide resolved

src/runtime/c_runtime_api.cc Outdated Show resolved Hide resolved

src/runtime/texture.h Show resolved Hide resolved

tqchen assigned ZihengJiang Mar 20, 2021

csullivan marked this pull request as ready for review May 4, 2021 18:59

ZihengJiang approved these changes May 11, 2021

View reviewed changes

tqchen previously requested changes May 11, 2021

View reviewed changes

tqchen added the status: need update need update based on feedbacks label May 14, 2021

csullivan and others added 20 commits May 18, 2021 16:18

Add TVMBackendAllocTexture and support in OpenCL device API.

7c517db

Add runtime optimized caching allocator.

2932a30

This should be replaced with AOT memory planning when the relay/tir/compile engine refactor lands.

Few bug fixes for runtime texture allocator.

523688f

Add OpenCL device api support for image2d<float16> textures.

c5fdc97

Update OpenCL DeviceAPI to support Image2D data space

6134525

allocations and copying to/from host/image2d directly. Allocation employs a lowering convention to 2d images for activations and weights.

Fix to follow OpenCL spec. for indexing.

862d52a

Rename texture_pool.h -> texture.h

d641f80

Move Nd to 2d lowering convention code into runtime texture

99d0740

utilities that can be shared by codegen and the runtime.

Update texture lowering utilities

3f2c5b2

Add TODO comment about pitch support

a5c9f8e

Remove FreeTexture

fa771d2

Fix ICHECK comment

f6ca66c

Partial cherry pick from @ZihengJiang

3c38800

git@github.com:ZihengJiang/tvm.git:52822c5bd [RUNTIME] OpenCL texture memory.

Remove runtime and device texture APIs.

b6b8ec8

Add OpenCL packed functions for texture workspace (de)allocations.

83cd18d

Add OpenCLBuffer structure to track

1cb2606

memory layout through OpenCL Device API.

Rebase: TVMContext -> Device

03c2cac

Implement DLTensor* overload of CopyDataToFrom in OpenCL DeviceAPI.

32bd0eb

Implement OpenCL CopyDataFromTo(DLTensor*...)

c1f993b

overload and tensor shapes to calculate image extent when copying date directly to or from texture cache.

Update format (cpp-lint)

938c1ea

csullivan added 7 commits May 18, 2021 16:21

Update format (clang)

1087b9e

Buffer descriptor name change and formatting.

515b2aa

Add texture pool documentation.

238ef4a

Update runtime to use new global.texture scope.

d402cd8

Move texture_pool.cc into opencl impl.

89c9de9

Add test coverage for copying in and out

d12c4f1

of storage allocs of texture scope.

Documented APIs and structures, renamed buffer descriptor layout tags.

fe9a836

csullivan requested a review from tqchen May 29, 2021 23:36

tqchen approved these changes Jun 4, 2021

View reviewed changes

ZihengJiang merged commit 010d11b into apache:main Jun 5, 2021

tmoreau89 mentioned this pull request Jan 10, 2022

[RFC] Support Codegen Using Texture Memory #2250

Closed

masahi mentioned this pull request Jan 10, 2022

[Tracking Issue] Texture codegen remaining tasks #9891

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Texture support][Part 0] Device API and runtime support #7711

[Texture support][Part 0] Device API and runtime support #7711

csullivan commented Mar 19, 2021 •

edited

Loading

tqchen commented Mar 20, 2021

csullivan commented May 4, 2021

ZihengJiang commented May 11, 2021

tqchen May 11, 2021

csullivan May 29, 2021

csullivan commented May 29, 2021

tqchen commented Jun 4, 2021

ZihengJiang commented Jun 5, 2021 •

edited

Loading

[Texture support][Part 0] Device API and runtime support #7711

[Texture support][Part 0] Device API and runtime support #7711

Conversation

csullivan commented Mar 19, 2021 • edited Loading

tqchen commented Mar 20, 2021

csullivan commented May 4, 2021

ZihengJiang commented May 11, 2021

tqchen May 11, 2021

Choose a reason for hiding this comment

csullivan May 29, 2021

Choose a reason for hiding this comment

csullivan commented May 29, 2021

tqchen commented Jun 4, 2021

ZihengJiang commented Jun 5, 2021 • edited Loading

csullivan commented Mar 19, 2021 •

edited

Loading

ZihengJiang commented Jun 5, 2021 •

edited

Loading