[QNN] Lookup operations for hard to implement operators #10053
Conversation
AndrewZhaoLuo
requested review from
anijain2305,
areusch,
comaniac,
Huyuwei,
icemelon,
jcf94,
jroesch,
junrushao,
jwfromm,
kevinthesun,
Laurawly,
MarisaKirisame,
masahi,
mbrookhart,
merrymercy,
slyubomirsky,
tqchen,
vinx13,
wweic,
yzhliu,
zhiics and
ZihengJiang
as code owners
|
Thanks, @AndrewZhaoLuo, this is a really interesting PR! I'm not 100% sure what the long term correct approach for running these kind of ops. We can talk about Look Up Tables like this, we could do polynomial approximations over the estimated range, or we could talk about casting back to FP32. I imagine that the most performant solution is going to depend on the hardware target, some ops will be fast for lookup tables, others will do better with polynomial approximations, and others might be faster just casting back to FP32. Perhaps we should make this part of the device-dependent legalization pass, instead of canonicalization? https://github.com/apache/tvm/blob/main/python/tvm/relay/qnn/op/legalizations.py @masahi @anijain2305 Thoughts? |
|
I think this is an interesting idea. I think moving the lowering to legalization is probably the right call once we have alternative methods. Need to read more into how legalization works. |
|
So I don't understand the functional difference between legalization and canonicalization. They seem to be able to be combined into one step? |
AndrewZhaoLuo
changed the title
|
This is now ready for review cc @mbrookhart @masahi |
AndrewZhaoLuo
force-pushed
the
aluo/fq2i/elemwise-lookup-ops-qnn-flavor
branch
from
4e7b96a to
b609d63
Compare
…:AndrewZhaoLuo/tvm into aluo/fq2i/elemwise-lookup-ops-qnn-flavor
* initial tanh impl * smalls error * support uint and int lookup into tables * reinterpret cast, working tanh tests * refactor relay func creation * basic casting tests * explicitly say do not handle multi-channel lookups * add example funcs * fix silent fail * fix some bugs with floating point funcs not working * add TODO * add tood * canonicalizations * refactor integer lookup ops into own folder * fq2i stuff * clean up existing tests * flesh out todo * more tests * test on keeping shape good * lookup table fix * replace canonicalization for rsqrt * remove canonicalization of rsqrt * add asf headers * topi tests * gather supports unsigned integer tests * fix things * move to legalization * jostle ci * linting * use take instead of gather * remove gather changes * undo changes * undo changes * undo changes * move thing in range * initial tanh impl * smalls error * support uint and int lookup into tables * reinterpret cast, working tanh tests * refactor relay func creation * basic casting tests * explicitly say do not handle multi-channel lookups * add example funcs * fix silent fail * fix some bugs with floating point funcs not working * add TODO * add tood * canonicalizations * refactor integer lookup ops into own folder * fq2i stuff * clean up existing tests * flesh out todo * more tests * test on keeping shape good * lookup table fix * replace canonicalization for rsqrt * remove canonicalization of rsqrt * add asf headers * gather supports unsigned integer tests * fix things * move to legalization * jostle ci * linting * use take instead of gather * remove gather changes * undo changes * undo changes * undo changes * move thing in range * lint * remove unneeded line * jostle Co-authored-by: andrewzhaoluo (generated by with_the_same_user script) <andrewzhaoluo@system76-pc.localdomain>
This PR adds a way to implement difficult and arbitrary elemwise ops in q-space via a table lookup operation via gather. This is intended to be used as the "canonicalization" of some qnn functions.
This is all in python RN so legalizations of relevant functions has to be done via python. An example with
qnn.rsqrtis given too.Things this PRs adds:
qnn.rsqrtop.If reviewers want I can split into separate PRs.
TODO:
take(separate PR / issue): Indexing ops should support unsigned integer types #10064