Perley Polyhedron Gridder literal interaction fails on Numba 0.51

[sjperkins]

• Codex Africanus version: 0.2.5, master
• Python version: 3.6
• Operating System: 18.04

Description

In discussion with @bennahugo numba was pinned in #202 to <= 0.49.0 to avoid these failures.

This issue serves to document the issues on an upgrade to numba 0.51.0. The Perley Polyhedron Gridding code fails to compile.

What I Did

`$ py.test -s -vvv -k test_gridder_nondask` and output

$ py.test -s -vvv -k test_gridder_nondask
============================= test session starts ==============================
platform linux -- Python 3.6.8, pytest-5.4.3, py-1.8.1, pluggy-0.13.1 -- /home/sperkins/venv/afr/bin/python3
cachedir: .pytest_cache
benchmark: 3.2.3 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/sperkins/work/ska/code/codex-africanus, inifile: setup.cfg
plugins: flaky-3.6.1, benchmark-3.2.3, flake8-1.0.6
collecting ... collected 16 items / 15 deselected / 1 selected

africanus/gridding/perleypolyhedron/tests/test_daskintrf.py::test_gridder_nondask FAILED

=================================== FAILURES ===================================
_____________________________ test_gridder_nondask _____________________________

self = CPUDispatcher(<function gridder at 0x7fe0df5de2f0>)
sig = (array(float64, 2d, C), array(complex64, 3d, C), array(float64, 1d, C), array(int64, 1d, C), int64, float64, ...)

    @global_compiler_lock
    def compile(self, sig):
        if not self._can_compile:
            raise RuntimeError("compilation disabled")
        # Use counter to track recursion compilation depth
        with self._compiling_counter:
            args, return_type = sigutils.normalize_signature(sig)
            # Don't recompile if signature already exists
            existing = self.overloads.get(tuple(args))
            if existing is not None:
                return existing.entry_point
            # Try to load from disk cache
            cres = self._cache.load_overload(sig, self.targetctx)
            if cres is not None:
                self._cache_hits[sig] += 1
                # XXX fold this in add_overload()? (also see compiler.py)
                if not cres.objectmode and not cres.interpmode:
                    self.targetctx.insert_user_function(cres.entry_point,
                                                        cres.fndesc, [cres.library])
                self.add_overload(cres)
                return cres.entry_point
    
            self._cache_misses[sig] += 1
            try:
>               cres = self._compiler.compile(args, return_type)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/dispatcher.py:819: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.dispatcher._FunctionCompiler object at 0x7fe0df5dc4e0>
args = (array(float64, 2d, C), array(complex64, 3d, C), array(float64, 1d, C), array(int64, 1d, C), int64, float64, ...)
return_type = None

    def compile(self, args, return_type):
>       status, retval = self._compile_cached(args, return_type)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/dispatcher.py:78: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.dispatcher._FunctionCompiler object at 0x7fe0df5dc4e0>
args = (array(float64, 2d, C), array(complex64, 3d, C), array(float64, 1d, C), array(int64, 1d, C), int64, float64, ...)
return_type = None

    def _compile_cached(self, args, return_type):
        key = tuple(args), return_type
        try:
            return False, self._failed_cache[key]
        except KeyError:
            pass
    
        try:
>           retval = self._compile_core(args, return_type)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/dispatcher.py:92: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.dispatcher._FunctionCompiler object at 0x7fe0df5dc4e0>
args = (array(float64, 2d, C), array(complex64, 3d, C), array(float64, 1d, C), array(int64, 1d, C), int64, float64, ...)
return_type = None

    def _compile_core(self, args, return_type):
        flags = compiler.Flags()
        self.targetdescr.options.parse_as_flags(flags, self.targetoptions)
        flags = self._customize_flags(flags)
    
        impl = self._get_implementation(args, {})
        cres = compiler.compile_extra(self.targetdescr.typing_context,
                                      self.targetdescr.target_context,
                                      impl,
                                      args=args, return_type=return_type,
                                      flags=flags, locals=self.locals,
>                                     pipeline_class=self.pipeline_class)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/dispatcher.py:110: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

typingctx = <numba.core.typing.context.Context object at 0x7fe0df7bf6a0>
targetctx = <numba.core.cpu.CPUContext object at 0x7fe0df7bf748>
func = <function gridder at 0x7fe0df5de2f0>
args = (array(float64, 2d, C), array(complex64, 3d, C), array(float64, 1d, C), array(int64, 1d, C), int64, float64, ...)
return_type = None
flags = Flags(enable_looplift=True, enable_pyobject_looplift=True, enable_ssa=True, release_gil=True, auto_parallel=<numba.cor...ions.FastMathOptions object at 0x7fe0d9eb0828>, inline=<numba.core.cpu_options.InlineOptions object at 0x7fe1069d3b00>)
locals = {}, library = None
pipeline_class = <class 'numba.core.compiler.Compiler'>

    def compile_extra(typingctx, targetctx, func, args, return_type, flags,
                      locals, library=None, pipeline_class=Compiler):
        """Compiler entry point
    
        Parameter
        ---------
        typingctx :
            typing context
        targetctx :
            target context
        func : function
            the python function to be compiled
        args : tuple, list
            argument types
        return_type :
            Use ``None`` to indicate void return
        flags : numba.compiler.Flags
            compiler flags
        library : numba.codegen.CodeLibrary
            Used to store the compiled code.
            If it is ``None``, a new CodeLibrary is used.
        pipeline_class : type like numba.compiler.CompilerBase
            compiler pipeline
        """
        pipeline = pipeline_class(typingctx, targetctx, library,
                                  args, return_type, flags, locals)
>       return pipeline.compile_extra(func)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/compiler.py:625: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.compiler.Compiler object at 0x7fe1074f9668>
func = <function gridder at 0x7fe0df5de2f0>

    def compile_extra(self, func):
        self.state.func_id = bytecode.FunctionIdentity.from_function(func)
        try:
            ExtractByteCode().run_pass(self.state)
        except Exception as e:
            if self.state.status.can_giveup:
                CompileInterpMode().run_pass(self.state)
                return self.state.cr
            else:
                raise e
    
        self.state.lifted = ()
        self.state.lifted_from = None
>       return self._compile_bytecode()

../../../../venv/afr/lib/python3.6/site-packages/numba/core/compiler.py:361: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.compiler.Compiler object at 0x7fe1074f9668>

    def _compile_bytecode(self):
        """
        Populate and run pipeline for bytecode input
        """
        assert self.state.func_ir is None
>       return self._compile_core()

../../../../venv/afr/lib/python3.6/site-packages/numba/core/compiler.py:423: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.compiler.Compiler object at 0x7fe1074f9668>

    def _compile_core(self):
        """
        Populate and run compiler pipeline
        """
        pms = self.define_pipelines()
        for pm in pms:
            pipeline_name = pm.pipeline_name
            func_name = "%s.%s" % (self.state.func_id.modname,
                                   self.state.func_id.func_qualname)
    
            event("Pipeline: %s for %s" % (pipeline_name, func_name))
            self.state.metadata['pipeline_times'] = {pipeline_name:
                                                     pm.exec_times}
            is_final_pipeline = pm == pms[-1]
            res = None
            try:
                pm.run(self.state)
                if self.state.cr is not None:
                    break
            except _EarlyPipelineCompletion as e:
                res = e.result
                break
            except Exception as e:
                self.state.status.fail_reason = e
                if is_final_pipeline:
>                   raise e

../../../../venv/afr/lib/python3.6/site-packages/numba/core/compiler.py:403: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.compiler.Compiler object at 0x7fe1074f9668>

    def _compile_core(self):
        """
        Populate and run compiler pipeline
        """
        pms = self.define_pipelines()
        for pm in pms:
            pipeline_name = pm.pipeline_name
            func_name = "%s.%s" % (self.state.func_id.modname,
                                   self.state.func_id.func_qualname)
    
            event("Pipeline: %s for %s" % (pipeline_name, func_name))
            self.state.metadata['pipeline_times'] = {pipeline_name:
                                                     pm.exec_times}
            is_final_pipeline = pm == pms[-1]
            res = None
            try:
>               pm.run(self.state)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/compiler.py:394: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.compiler_machinery.PassManager object at 0x7fe0d9eb0588>
state = {'typingctx': <numba.core.typing.context.Context object at 0x7fe0df7bf6a0>, 'targetctx': <numba.core.cpu.CPUContext ob...            literally(baseline_transform_policy))
                   ^
, can_fallback=False, can_giveup=0, 'nargs': 17}

    def run(self, state):
        """
        Run the defined pipelines on the state.
        """
        from numba.core.compiler import _EarlyPipelineCompletion
        if not self.finalized:
            raise RuntimeError("Cannot run non-finalised pipeline")
    
        # walk the passes and run them
        for idx, (pss, pass_desc) in enumerate(self.passes):
            try:
                event("-- %s" % pass_desc)
                pass_inst = _pass_registry.get(pss).pass_inst
                if isinstance(pass_inst, CompilerPass):
                    self._runPass(idx, pass_inst, state)
                else:
                    raise BaseException("Legacy pass in use")
            except _EarlyPipelineCompletion as e:
                raise e
            except Exception as e:
                msg = "Failed in %s mode pipeline (step: %s)" % \
                    (self.pipeline_name, pass_desc)
                patched_exception = self._patch_error(msg, e)
>               raise patched_exception

../../../../venv/afr/lib/python3.6/site-packages/numba/core/compiler_machinery.py:341: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.compiler_machinery.PassManager object at 0x7fe0d9eb0588>
state = {'typingctx': <numba.core.typing.context.Context object at 0x7fe0df7bf6a0>, 'targetctx': <numba.core.cpu.CPUContext ob...            literally(baseline_transform_policy))
                   ^
, can_fallback=False, can_giveup=0, 'nargs': 17}

    def run(self, state):
        """
        Run the defined pipelines on the state.
        """
        from numba.core.compiler import _EarlyPipelineCompletion
        if not self.finalized:
            raise RuntimeError("Cannot run non-finalised pipeline")
    
        # walk the passes and run them
        for idx, (pss, pass_desc) in enumerate(self.passes):
            try:
                event("-- %s" % pass_desc)
                pass_inst = _pass_registry.get(pss).pass_inst
                if isinstance(pass_inst, CompilerPass):
>                   self._runPass(idx, pass_inst, state)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/compiler_machinery.py:332: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = (<numba.core.compiler_machinery.PassManager object at 0x7fe0d9eb0588>, 11, <numba.core.untyped_passes.FindLiterallyCal...           literally(baseline_transform_policy))
                   ^
, can_fallback=False, can_giveup=0, 'nargs': 17})
kwargs = {}

    @functools.wraps(func)
    def _acquire_compile_lock(*args, **kwargs):
        with self:
>           return func(*args, **kwargs)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/compiler_lock.py:32: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.compiler_machinery.PassManager object at 0x7fe0d9eb0588>
index = 11
pss = <numba.core.untyped_passes.FindLiterallyCalls object at 0x7fe0f3f349e8>
internal_state = {'typingctx': <numba.core.typing.context.Context object at 0x7fe0df7bf6a0>, 'targetctx': <numba.core.cpu.CPUContext ob...            literally(baseline_transform_policy))
                   ^
, can_fallback=False, can_giveup=0, 'nargs': 17}

    @global_compiler_lock  # this need a lock, likely calls LLVM
    def _runPass(self, index, pss, internal_state):
        mutated = False
    
        def check(func, compiler_state):
            mangled = func(compiler_state)
            if mangled not in (True, False):
                msg = ("CompilerPass implementations should return True/False. "
                       "CompilerPass with name '%s' did not.")
                raise ValueError(msg % pss.name())
            return mangled
    
        def debug_print(pass_name, print_condition, printable_condition):
            if pass_name in print_condition:
                fid = internal_state.func_id
                args = (fid.modname, fid.func_qualname, self.pipeline_name,
                        printable_condition, pass_name)
                print(("%s.%s: %s: %s %s" % args).center(120, '-'))
                if internal_state.func_ir is not None:
                    internal_state.func_ir.dump()
                else:
                    print("func_ir is None")
    
        # debug print before this pass?
        debug_print(pss.name(), self._print_before + self._print_wrap, "BEFORE")
    
        # wire in the analysis info so it's accessible
        pss.analysis = self._analysis
    
        with SimpleTimer() as init_time:
            mutated |= check(pss.run_initialization, internal_state)
        with SimpleTimer() as pass_time:
>           mutated |= check(pss.run_pass, internal_state)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/compiler_machinery.py:291: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

func = <bound method FindLiterallyCalls.run_pass of <numba.core.untyped_passes.FindLiterallyCalls object at 0x7fe0f3f349e8>>
compiler_state = {'typingctx': <numba.core.typing.context.Context object at 0x7fe0df7bf6a0>, 'targetctx': <numba.core.cpu.CPUContext ob...            literally(baseline_transform_policy))
                   ^
, can_fallback=False, can_giveup=0, 'nargs': 17}

    def check(func, compiler_state):
>       mangled = func(compiler_state)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/compiler_machinery.py:264: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.untyped_passes.FindLiterallyCalls object at 0x7fe0f3f349e8>
state = {'typingctx': <numba.core.typing.context.Context object at 0x7fe0df7bf6a0>, 'targetctx': <numba.core.cpu.CPUContext ob...            literally(baseline_transform_policy))
                   ^
, can_fallback=False, can_giveup=0, 'nargs': 17}

    def run_pass(self, state):
>       find_literally_calls(state.func_ir, state.args)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/untyped_passes.py:427: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

func_ir = <numba.core.ir.FunctionIR object at 0x7fe05fd09278>
argtypes = (array(float64, 2d, C), array(complex64, 3d, C), array(float64, 1d, C), array(int64, 1d, C), int64, float64, ...)

    def find_literally_calls(func_ir, argtypes):
        """An analysis to find `numba.literally` call inside the given IR.
        When an unsatisfied literal typing request is found, a `ForceLiteralArg`
        exception is raised.
    
        Parameters
        ----------
    
        func_ir : numba.ir.FunctionIR
    
        argtypes : Sequence[numba.types.Type]
            The argument types.
        """
        from numba.core import ir_utils
    
        marked_args = set()
        first_loc = {}
        # Scan for literally calls
        for blk in func_ir.blocks.values():
            for assign in blk.find_exprs(op='call'):
                var = ir_utils.guard(ir_utils.get_definition, func_ir, assign.func)
                if isinstance(var, (ir.Global, ir.FreeVar)):
                    fnobj = var.value
                else:
                    fnobj = ir_utils.guard(ir_utils.resolve_func_from_module,
                                           func_ir, var)
                if fnobj is special.literally:
                    # Found
                    [arg] = assign.args
                    defarg = func_ir.get_definition(arg)
                    if isinstance(defarg, ir.Arg):
                        argindex = defarg.index
                        marked_args.add(argindex)
                        first_loc.setdefault(argindex, assign.loc)
        # Signal the dispatcher to force literal typing
        for pos in marked_args:
            query_arg = argtypes[pos]
            do_raise = (isinstance(query_arg, types.InitialValue) and
                        query_arg.initial_value is None)
            if do_raise:
                loc = first_loc[pos]
                raise errors.ForceLiteralArg(marked_args, loc=loc)
    
            if not isinstance(query_arg, (types.Literal, types.InitialValue)):
                loc = first_loc[pos]
>               raise errors.ForceLiteralArg(marked_args, loc=loc)
E               numba.core.errors.ForceLiteralArg: Failed in nopython mode pipeline (step: find literally calls)
E               Pseudo-exception to force literal arguments in the dispatcher
E               
E               File "africanus/gridding/perleypolyhedron/gridder.py", line 93:
E               def gridder(uvw,
E                   <source elided>
E                       btp.policy(uvw[r, :], ra0, dec0, ra, dec,
E                                  literally(baseline_transform_policy))
E                                  ^

../../../../venv/afr/lib/python3.6/site-packages/numba/core/analysis.py:620: ForceLiteralArg

During handling of the above exception, another exception occurred:

self = CPUDispatcher(<function gridder at 0x7fe0df5de2f0>)
args = [array([[ 4846.47527682, -1116.41261315,  9655.38023952],
       [ 4846.60821778, -1116.35279883,  9655.3204252 ],
   ...0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), 110, 1.1406824779202709, ...]
kws = {}
error_rewrite = <function _DispatcherBase._compile_for_args.<locals>.error_rewrite at 0x7fe10cd599d8>
argtypes = [array(float64, 2d, C), array(complex64, 3d, C), array(float64, 1d, C), array(int64, 1d, C), int64, float64, ...]
a = True, already_lit_pos = []

    def _compile_for_args(self, *args, **kws):
        """
        For internal use.  Compile a specialized version of the function
        for the given *args* and *kws*, and return the resulting callable.
        """
        assert not kws
        # call any initialisation required for the compilation chain (e.g.
        # extension point registration).
        self._compilation_chain_init_hook()
    
        def error_rewrite(e, issue_type):
            """
            Rewrite and raise Exception `e` with help supplied based on the
            specified issue_type.
            """
            if config.SHOW_HELP:
                help_msg = errors.error_extras[issue_type]
                e.patch_message('\n'.join((str(e).rstrip(), help_msg)))
            if config.FULL_TRACEBACKS:
                raise e
            else:
                reraise(type(e), e, None)
    
        argtypes = []
        for a in args:
            if isinstance(a, OmittedArg):
                argtypes.append(types.Omitted(a.value))
            else:
                argtypes.append(self.typeof_pyval(a))
        try:
>           return self.compile(tuple(argtypes))

../../../../venv/afr/lib/python3.6/site-packages/numba/core/dispatcher.py:367: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = (CPUDispatcher(<function gridder at 0x7fe0df5de2f0>), (array(float64, 2d, C), array(complex64, 3d, C), array(float64, 1d, C), array(int64, 1d, C), int64, float64, ...))
kwargs = {}

    @functools.wraps(func)
    def _acquire_compile_lock(*args, **kwargs):
        with self:
>           return func(*args, **kwargs)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/compiler_lock.py:32: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = CPUDispatcher(<function gridder at 0x7fe0df5de2f0>)
sig = (array(float64, 2d, C), array(complex64, 3d, C), array(float64, 1d, C), array(int64, 1d, C), int64, float64, ...)

    @global_compiler_lock
    def compile(self, sig):
        if not self._can_compile:
            raise RuntimeError("compilation disabled")
        # Use counter to track recursion compilation depth
        with self._compiling_counter:
            args, return_type = sigutils.normalize_signature(sig)
            # Don't recompile if signature already exists
            existing = self.overloads.get(tuple(args))
            if existing is not None:
                return existing.entry_point
            # Try to load from disk cache
            cres = self._cache.load_overload(sig, self.targetctx)
            if cres is not None:
                self._cache_hits[sig] += 1
                # XXX fold this in add_overload()? (also see compiler.py)
                if not cres.objectmode and not cres.interpmode:
                    self.targetctx.insert_user_function(cres.entry_point,
                                                        cres.fndesc, [cres.library])
                self.add_overload(cres)
                return cres.entry_point
    
            self._cache_misses[sig] += 1
            try:
                cres = self._compiler.compile(args, return_type)
            except errors.ForceLiteralArg as e:
                def folded(args, kws):
                    return self._compiler.fold_argument_types(args, kws)[1]
>               raise e.bind_fold_arguments(folded)
E               numba.core.errors.ForceLiteralArg: Pseudo-exception to force literal arguments in the dispatcher
E               
E               File "africanus/gridding/perleypolyhedron/gridder.py", line 93:
E               def gridder(uvw,
E                   <source elided>
E                       btp.policy(uvw[r, :], ra0, dec0, ra, dec,
E                                  literally(baseline_transform_policy))
E                                  ^

../../../../venv/afr/lib/python3.6/site-packages/numba/core/dispatcher.py:823: ForceLiteralArg

During handling of the above exception, another exception occurred:

self = CPUDispatcher(<function gridder at 0x7fe0df5de2f0>)
sig = (array(float64, 2d, C), array(complex64, 3d, C), array(float64, 1d, C), array(int64, 1d, C), int64, float64, ...)

    @global_compiler_lock
    def compile(self, sig):
        if not self._can_compile:
            raise RuntimeError("compilation disabled")
        # Use counter to track recursion compilation depth
        with self._compiling_counter:
            args, return_type = sigutils.normalize_signature(sig)
            # Don't recompile if signature already exists
            existing = self.overloads.get(tuple(args))
            if existing is not None:
                return existing.entry_point
            # Try to load from disk cache
            cres = self._cache.load_overload(sig, self.targetctx)
            if cres is not None:
                self._cache_hits[sig] += 1
                # XXX fold this in add_overload()? (also see compiler.py)
                if not cres.objectmode and not cres.interpmode:
                    self.targetctx.insert_user_function(cres.entry_point,
                                                        cres.fndesc, [cres.library])
                self.add_overload(cres)
                return cres.entry_point
    
            self._cache_misses[sig] += 1
            try:
>               cres = self._compiler.compile(args, return_type)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/dispatcher.py:819: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.dispatcher._FunctionCompiler object at 0x7fe0df5dc4e0>
args = (array(float64, 2d, C), array(complex64, 3d, C), array(float64, 1d, C), array(int64, 1d, C), int64, float64, ...)
return_type = None

    def compile(self, args, return_type):
>       status, retval = self._compile_cached(args, return_type)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/dispatcher.py:78: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.dispatcher._FunctionCompiler object at 0x7fe0df5dc4e0>
args = (array(float64, 2d, C), array(complex64, 3d, C), array(float64, 1d, C), array(int64, 1d, C), int64, float64, ...)
return_type = None

    def _compile_cached(self, args, return_type):
        key = tuple(args), return_type
        try:
            return False, self._failed_cache[key]
        except KeyError:
            pass
    
        try:
>           retval = self._compile_core(args, return_type)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/dispatcher.py:92: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.dispatcher._FunctionCompiler object at 0x7fe0df5dc4e0>
args = (array(float64, 2d, C), array(complex64, 3d, C), array(float64, 1d, C), array(int64, 1d, C), int64, float64, ...)
return_type = None

    def _compile_core(self, args, return_type):
        flags = compiler.Flags()
        self.targetdescr.options.parse_as_flags(flags, self.targetoptions)
        flags = self._customize_flags(flags)
    
        impl = self._get_implementation(args, {})
        cres = compiler.compile_extra(self.targetdescr.typing_context,
                                      self.targetdescr.target_context,
                                      impl,
                                      args=args, return_type=return_type,
                                      flags=flags, locals=self.locals,
>                                     pipeline_class=self.pipeline_class)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/dispatcher.py:110: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

typingctx = <numba.core.typing.context.Context object at 0x7fe0df7bf6a0>
targetctx = <numba.core.cpu.CPUContext object at 0x7fe0df7bf748>
func = <function gridder at 0x7fe0df5de2f0>
args = (array(float64, 2d, C), array(complex64, 3d, C), array(float64, 1d, C), array(int64, 1d, C), int64, float64, ...)
return_type = None
flags = Flags(enable_looplift=True, enable_pyobject_looplift=True, enable_ssa=True, release_gil=True, auto_parallel=<numba.cor...ions.FastMathOptions object at 0x7fe05fcf9c50>, inline=<numba.core.cpu_options.InlineOptions object at 0x7fe1069d3b00>)
locals = {}, library = None
pipeline_class = <class 'numba.core.compiler.Compiler'>

    def compile_extra(typingctx, targetctx, func, args, return_type, flags,
                      locals, library=None, pipeline_class=Compiler):
        """Compiler entry point
    
        Parameter
        ---------
        typingctx :
            typing context
        targetctx :
            target context
        func : function
            the python function to be compiled
        args : tuple, list
            argument types
        return_type :
            Use ``None`` to indicate void return
        flags : numba.compiler.Flags
            compiler flags
        library : numba.codegen.CodeLibrary
            Used to store the compiled code.
            If it is ``None``, a new CodeLibrary is used.
        pipeline_class : type like numba.compiler.CompilerBase
            compiler pipeline
        """
        pipeline = pipeline_class(typingctx, targetctx, library,
                                  args, return_type, flags, locals)
>       return pipeline.compile_extra(func)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/compiler.py:625: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.compiler.Compiler object at 0x7fe0d9e4ef28>
func = <function gridder at 0x7fe0df5de2f0>

    def compile_extra(self, func):
        self.state.func_id = bytecode.FunctionIdentity.from_function(func)
        try:
            ExtractByteCode().run_pass(self.state)
        except Exception as e:
            if self.state.status.can_giveup:
                CompileInterpMode().run_pass(self.state)
                return self.state.cr
            else:
                raise e
    
        self.state.lifted = ()
        self.state.lifted_from = None
>       return self._compile_bytecode()

../../../../venv/afr/lib/python3.6/site-packages/numba/core/compiler.py:361: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.compiler.Compiler object at 0x7fe0d9e4ef28>

    def _compile_bytecode(self):
        """
        Populate and run pipeline for bytecode input
        """
        assert self.state.func_ir is None
>       return self._compile_core()

../../../../venv/afr/lib/python3.6/site-packages/numba/core/compiler.py:423: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.compiler.Compiler object at 0x7fe0d9e4ef28>

    def _compile_core(self):
        """
        Populate and run compiler pipeline
        """
        pms = self.define_pipelines()
        for pm in pms:
            pipeline_name = pm.pipeline_name
            func_name = "%s.%s" % (self.state.func_id.modname,
                                   self.state.func_id.func_qualname)
    
            event("Pipeline: %s for %s" % (pipeline_name, func_name))
            self.state.metadata['pipeline_times'] = {pipeline_name:
                                                     pm.exec_times}
            is_final_pipeline = pm == pms[-1]
            res = None
            try:
                pm.run(self.state)
                if self.state.cr is not None:
                    break
            except _EarlyPipelineCompletion as e:
                res = e.result
                break
            except Exception as e:
                self.state.status.fail_reason = e
                if is_final_pipeline:
>                   raise e

../../../../venv/afr/lib/python3.6/site-packages/numba/core/compiler.py:403: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.compiler.Compiler object at 0x7fe0d9e4ef28>

    def _compile_core(self):
        """
        Populate and run compiler pipeline
        """
        pms = self.define_pipelines()
        for pm in pms:
            pipeline_name = pm.pipeline_name
            func_name = "%s.%s" % (self.state.func_id.modname,
                                   self.state.func_id.func_qualname)
    
            event("Pipeline: %s for %s" % (pipeline_name, func_name))
            self.state.metadata['pipeline_times'] = {pipeline_name:
                                                     pm.exec_times}
            is_final_pipeline = pm == pms[-1]
            res = None
            try:
>               pm.run(self.state)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/compiler.py:394: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.compiler_machinery.PassManager object at 0x7fe05fc53e80>
state = {'typingctx': <numba.core.typing.context.Context object at 0x7fe0df7bf6a0>, 'targetctx': <numba.core.cpu.CPUContext ob...e/codex-africanus/africanus/gridding/perleypolyhedron/gridder.py (112)
, can_fallback=False, can_giveup=0, 'nargs': 17}

    def run(self, state):
        """
        Run the defined pipelines on the state.
        """
        from numba.core.compiler import _EarlyPipelineCompletion
        if not self.finalized:
            raise RuntimeError("Cannot run non-finalised pipeline")
    
        # walk the passes and run them
        for idx, (pss, pass_desc) in enumerate(self.passes):
            try:
                event("-- %s" % pass_desc)
                pass_inst = _pass_registry.get(pss).pass_inst
                if isinstance(pass_inst, CompilerPass):
                    self._runPass(idx, pass_inst, state)
                else:
                    raise BaseException("Legacy pass in use")
            except _EarlyPipelineCompletion as e:
                raise e
            except Exception as e:
                msg = "Failed in %s mode pipeline (step: %s)" % \
                    (self.pipeline_name, pass_desc)
                patched_exception = self._patch_error(msg, e)
>               raise patched_exception

../../../../venv/afr/lib/python3.6/site-packages/numba/core/compiler_machinery.py:341: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.compiler_machinery.PassManager object at 0x7fe05fc53e80>
state = {'typingctx': <numba.core.typing.context.Context object at 0x7fe0df7bf6a0>, 'targetctx': <numba.core.cpu.CPUContext ob...e/codex-africanus/africanus/gridding/perleypolyhedron/gridder.py (112)
, can_fallback=False, can_giveup=0, 'nargs': 17}

    def run(self, state):
        """
        Run the defined pipelines on the state.
        """
        from numba.core.compiler import _EarlyPipelineCompletion
        if not self.finalized:
            raise RuntimeError("Cannot run non-finalised pipeline")
    
        # walk the passes and run them
        for idx, (pss, pass_desc) in enumerate(self.passes):
            try:
                event("-- %s" % pass_desc)
                pass_inst = _pass_registry.get(pss).pass_inst
                if isinstance(pass_inst, CompilerPass):
>                   self._runPass(idx, pass_inst, state)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/compiler_machinery.py:332: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = (<numba.core.compiler_machinery.PassManager object at 0x7fe05fc53e80>, 14, <numba.core.typed_passes.NopythonTypeInfere.../codex-africanus/africanus/gridding/perleypolyhedron/gridder.py (112)
, can_fallback=False, can_giveup=0, 'nargs': 17})
kwargs = {}

    @functools.wraps(func)
    def _acquire_compile_lock(*args, **kwargs):
        with self:
>           return func(*args, **kwargs)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/compiler_lock.py:32: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.compiler_machinery.PassManager object at 0x7fe05fc53e80>
index = 14
pss = <numba.core.typed_passes.NopythonTypeInference object at 0x7fe0f3ecf8d0>
internal_state = {'typingctx': <numba.core.typing.context.Context object at 0x7fe0df7bf6a0>, 'targetctx': <numba.core.cpu.CPUContext ob...e/codex-africanus/africanus/gridding/perleypolyhedron/gridder.py (112)
, can_fallback=False, can_giveup=0, 'nargs': 17}

    @global_compiler_lock  # this need a lock, likely calls LLVM
    def _runPass(self, index, pss, internal_state):
        mutated = False
    
        def check(func, compiler_state):
            mangled = func(compiler_state)
            if mangled not in (True, False):
                msg = ("CompilerPass implementations should return True/False. "
                       "CompilerPass with name '%s' did not.")
                raise ValueError(msg % pss.name())
            return mangled
    
        def debug_print(pass_name, print_condition, printable_condition):
            if pass_name in print_condition:
                fid = internal_state.func_id
                args = (fid.modname, fid.func_qualname, self.pipeline_name,
                        printable_condition, pass_name)
                print(("%s.%s: %s: %s %s" % args).center(120, '-'))
                if internal_state.func_ir is not None:
                    internal_state.func_ir.dump()
                else:
                    print("func_ir is None")
    
        # debug print before this pass?
        debug_print(pss.name(), self._print_before + self._print_wrap, "BEFORE")
    
        # wire in the analysis info so it's accessible
        pss.analysis = self._analysis
    
        with SimpleTimer() as init_time:
            mutated |= check(pss.run_initialization, internal_state)
        with SimpleTimer() as pass_time:
>           mutated |= check(pss.run_pass, internal_state)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/compiler_machinery.py:291: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

func = <bound method BaseTypeInference.run_pass of <numba.core.typed_passes.NopythonTypeInference object at 0x7fe0f3ecf8d0>>
compiler_state = {'typingctx': <numba.core.typing.context.Context object at 0x7fe0df7bf6a0>, 'targetctx': <numba.core.cpu.CPUContext ob...e/codex-africanus/africanus/gridding/perleypolyhedron/gridder.py (112)
, can_fallback=False, can_giveup=0, 'nargs': 17}

    def check(func, compiler_state):
>       mangled = func(compiler_state)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/compiler_machinery.py:264: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.typed_passes.NopythonTypeInference object at 0x7fe0f3ecf8d0>
state = {'typingctx': <numba.core.typing.context.Context object at 0x7fe0df7bf6a0>, 'targetctx': <numba.core.cpu.CPUContext ob...e/codex-africanus/africanus/gridding/perleypolyhedron/gridder.py (112)
, can_fallback=False, can_giveup=0, 'nargs': 17}

    def run_pass(self, state):
        """
        Type inference and legalization
        """
        with fallback_context(state, 'Function "%s" failed type inference'
                              % (state.func_id.func_name,)):
            # Type inference
            typemap, return_type, calltypes = type_inference_stage(
                state.typingctx,
                state.func_ir,
                state.args,
                state.return_type,
                state.locals,
>               raise_errors=self._raise_errors)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/typed_passes.py:98: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

typingctx = <numba.core.typing.context.Context object at 0x7fe0df7bf6a0>
interp = <numba.core.ir.FunctionIR object at 0x7fe05fc226a0>
args = (array(float64, 2d, C), array(complex64, 3d, C), array(float64, 1d, C), array(int64, 1d, C), int64, float64, ...)
return_type = None, locals = {}, raise_errors = True

    def type_inference_stage(typingctx, interp, args, return_type, locals={},
                             raise_errors=True):
        if len(args) != interp.arg_count:
            raise TypeError("Mismatch number of argument types")
        warnings = errors.WarningsFixer(errors.NumbaWarning)
        infer = typeinfer.TypeInferer(typingctx, interp, warnings)
        with typingctx.callstack.register(infer, interp.func_id, args):
            # Seed argument types
            for index, (name, ty) in enumerate(zip(interp.arg_names, args)):
                infer.seed_argument(name, index, ty)
    
            # Seed return type
            if return_type is not None:
                infer.seed_return(return_type)
    
            # Seed local types
            for k, v in locals.items():
                infer.seed_type(k, v)
    
            infer.build_constraint()
>           infer.propagate(raise_errors=raise_errors)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/typed_passes.py:70: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.typeinfer.TypeInferer object at 0x7fe05fc3d240>
raise_errors = True

    def propagate(self, raise_errors=True):
        newtoken = self.get_state_token()
        oldtoken = None
        # Since the number of types are finite, the typesets will eventually
        # stop growing.
    
        while newtoken != oldtoken:
            self.debug.propagate_started()
            oldtoken = newtoken
            # Errors can appear when the type set is incomplete; only
            # raise them when there is no progress anymore.
            errors = self.constraints.propagate(self)
            newtoken = self.get_state_token()
            self.debug.propagate_finished()
        if errors:
            if raise_errors:
                force_lit_args = [e for e in errors
                                  if isinstance(e, ForceLiteralArg)]
                if not force_lit_args:
                    raise errors[0]
                else:
>                   raise reduce(operator.or_, force_lit_args)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/typeinfer.py:1073: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.typeinfer.ConstraintNetwork object at 0x7fe05fc38208>
typeinfer = <numba.core.typeinfer.TypeInferer object at 0x7fe05fc3d240>

    def propagate(self, typeinfer):
        """
        Execute all constraints.  Errors are caught and returned as a list.
        This allows progressing even though some constraints may fail
        due to lack of information
        (e.g. imprecise types such as List(undefined)).
        """
        errors = []
        for constraint in self.constraints:
            loc = constraint.loc
            with typeinfer.warnings.catch_warnings(filename=loc.filename,
                                                   lineno=loc.line):
                try:
>                   constraint(typeinfer)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/typeinfer.py:154: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.typeinfer.CallConstraint object at 0x7fe05fbea390>
typeinfer = <numba.core.typeinfer.TypeInferer object at 0x7fe05fc3d240>

    def __call__(self, typeinfer):
        msg = "typing of call at {0}\n".format(self.loc)
        with new_error_context(msg):
            typevars = typeinfer.typevars
            with new_error_context(
                    "resolving caller type: {}".format(self.func)):
                fnty = typevars[self.func].getone()
            with new_error_context("resolving callee type: {0}", fnty):
>               self.resolve(typeinfer, typevars, fnty)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/typeinfer.py:566: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numba.core.typeinfer.CallConstraint object at 0x7fe05fbea390>
typeinfer = <numba.core.typeinfer.TypeInferer object at 0x7fe05fc3d240>
typevars = {'arg.uvw': arg.uvw := array(float64, 2d, C), 'arg.vis': arg.vis := array(complex64, 3d, C), 'arg.lambdas': arg.lambda....20': $512.20 := float64, '$512.21': $512.21 := array(complex128, 2d, A), '$558.2': $558.2 := array(complex128, 3d, C)}
fnty = Function(<function policy at 0x7fe0df5de268>)

    def resolve(self, typeinfer, typevars, fnty):
        assert fnty
        context = typeinfer.context
    
        r = fold_arg_vars(typevars, self.args, self.vararg, self.kws)
        if r is None:
            # Cannot resolve call type until all argument types are known
            return
        pos_args, kw_args = r
    
        # Check argument to be precise
        for a in itertools.chain(pos_args, kw_args.values()):
            # Forbids imprecise type except array of undefined dtype
            if not a.is_precise() and not isinstance(a, types.Array):
                return
    
        # Resolve call type
        try:
            sig = typeinfer.resolve_call(fnty, pos_args, kw_args)
        except ForceLiteralArg as e:
            # Adjust for bound methods
            folding_args = ((fnty.this,) + tuple(self.args)
                            if isinstance(fnty, types.BoundFunction)
                            else self.args)
            folded = e.fold_arguments(folding_args, self.kws)
            requested = set()
            unsatisified = set()
            for idx in e.requested_args:
                maybe_arg = typeinfer.func_ir.get_definition(folded[idx])
                if isinstance(maybe_arg, ir.Arg):
                    requested.add(maybe_arg.index)
                else:
                    unsatisified.add(idx)
            if unsatisified:
                raise TypingError("Cannot request literal type.", loc=self.loc)
            elif requested:
>               raise ForceLiteralArg(requested, loc=self.loc)
E               numba.core.errors.ForceLiteralArg: Failed in nopython mode pipeline (step: nopython frontend)
E               Pseudo-exception to force literal arguments in the dispatcher
E               
E               File "africanus/gridding/perleypolyhedron/gridder.py", line 112:
E               def gridder(uvw,
E                   <source elided>
E                               stokes_conversion_policy,
E                               policy_type=literally(convolution_policy))
E                               ^
E               
E               During: resolving callee type: Function(<function policy at 0x7fe0df5de268>)
E               During: typing of call at /home/sperkins/work/ska/code/codex-africanus/africanus/gridding/perleypolyhedron/gridder.py (112)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/typeinfer.py:604: ForceLiteralArg

During handling of the above exception, another exception occurred:

self = CPUDispatcher(<function gridder at 0x7fe0df5de2f0>)
args = [array([[ 4846.47527682, -1116.41261315,  9655.38023952],
       [ 4846.60821778, -1116.35279883,  9655.3204252 ],
   ...0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), 110, 1.1406824779202709, ...]
kws = {}
error_rewrite = <function _DispatcherBase._compile_for_args.<locals>.error_rewrite at 0x7fe0d9e661e0>
argtypes = [array(float64, 2d, C), array(complex64, 3d, C), array(float64, 1d, C), array(int64, 1d, C), int64, float64, ...]
a = True, already_lit_pos = []

    def _compile_for_args(self, *args, **kws):
        """
        For internal use.  Compile a specialized version of the function
        for the given *args* and *kws*, and return the resulting callable.
        """
        assert not kws
        # call any initialisation required for the compilation chain (e.g.
        # extension point registration).
        self._compilation_chain_init_hook()
    
        def error_rewrite(e, issue_type):
            """
            Rewrite and raise Exception `e` with help supplied based on the
            specified issue_type.
            """
            if config.SHOW_HELP:
                help_msg = errors.error_extras[issue_type]
                e.patch_message('\n'.join((str(e).rstrip(), help_msg)))
            if config.FULL_TRACEBACKS:
                raise e
            else:
                reraise(type(e), e, None)
    
        argtypes = []
        for a in args:
            if isinstance(a, OmittedArg):
                argtypes.append(types.Omitted(a.value))
            else:
                argtypes.append(self.typeof_pyval(a))
        try:
>           return self.compile(tuple(argtypes))

../../../../venv/afr/lib/python3.6/site-packages/numba/core/dispatcher.py:367: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = (CPUDispatcher(<function gridder at 0x7fe0df5de2f0>), (array(float64, 2d, C), array(complex64, 3d, C), array(float64, 1d, C), array(int64, 1d, C), int64, float64, ...))
kwargs = {}

    @functools.wraps(func)
    def _acquire_compile_lock(*args, **kwargs):
        with self:
>           return func(*args, **kwargs)

../../../../venv/afr/lib/python3.6/site-packages/numba/core/compiler_lock.py:32: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = CPUDispatcher(<function gridder at 0x7fe0df5de2f0>)
sig = (array(float64, 2d, C), array(complex64, 3d, C), array(float64, 1d, C), array(int64, 1d, C), int64, float64, ...)

    @global_compiler_lock
    def compile(self, sig):
        if not self._can_compile:
            raise RuntimeError("compilation disabled")
        # Use counter to track recursion compilation depth
        with self._compiling_counter:
            args, return_type = sigutils.normalize_signature(sig)
            # Don't recompile if signature already exists
            existing = self.overloads.get(tuple(args))
            if existing is not None:
                return existing.entry_point
            # Try to load from disk cache
            cres = self._cache.load_overload(sig, self.targetctx)
            if cres is not None:
                self._cache_hits[sig] += 1
                # XXX fold this in add_overload()? (also see compiler.py)
                if not cres.objectmode and not cres.interpmode:
                    self.targetctx.insert_user_function(cres.entry_point,
                                                        cres.fndesc, [cres.library])
                self.add_overload(cres)
                return cres.entry_point
    
            self._cache_misses[sig] += 1
            try:
                cres = self._compiler.compile(args, return_type)
            except errors.ForceLiteralArg as e:
                def folded(args, kws):
                    return self._compiler.fold_argument_types(args, kws)[1]
>               raise e.bind_fold_arguments(folded)
E               numba.core.errors.ForceLiteralArg: Pseudo-exception to force literal arguments in the dispatcher
E               
E               File "africanus/gridding/perleypolyhedron/gridder.py", line 112:
E               def gridder(uvw,
E                   <source elided>
E                               stokes_conversion_policy,
E                               policy_type=literally(convolution_policy))
E                               ^

../../../../venv/afr/lib/python3.6/site-packages/numba/core/dispatcher.py:823: ForceLiteralArg

During handling of the above exception, another exception occurred:

    def test_gridder_nondask():
        with clock("Non-DASK gridding") as tictoc:
            # construct kernel
            W = 5
            OS = 9
            kern = kernels.pack_kernel(kernels.kbsinc(W, oversample=OS), W, OS)
            nrow = int(1e6)
            np.random.seed(0)
            # simulate some ficticious baselines rotated by an hour angle
            uvw = np.zeros((nrow, 3), dtype=np.float64)
            blpos = np.random.uniform(26, 10000, size=(25, 3))
            ntime = int(nrow / 25.0)
            d0 = np.pi / 4.0
            for n in range(25):
                for ih0, h0 in enumerate(
                        np.linspace(np.deg2rad(-20), np.deg2rad(20), ntime)):
                    s = np.sin
                    c = np.cos
                    R = np.array([[s(h0), c(h0), 0],
                                  [-s(d0) * c(h0),
                                   s(d0) * s(h0),
                                   c(d0)],
                                  [c(d0) * c(h0), -c(d0) * s(h0),
                                   s(d0)]])
                    uvw[n * ntime + ih0, :] = np.dot(R, blpos[n, :].T)
            pxacrossbeam = 5
            nchan = 128
            frequency = np.linspace(1.0e9, 1.4e9, nchan)
            wavelength = 299792458.0 / frequency
            cell = np.rad2deg(
                wavelength[0] /
                (max(np.max(np.absolute(uvw[:, 0])),
                     np.max(np.absolute(uvw[:, 1]))) * pxacrossbeam))
            npixfacet = 100
            fftpad = 1.1
    
            image_centres = np.array([[0, d0]])
            chanmap = np.zeros(nchan, dtype=np.int64)
            detaper_facet = kernels.compute_detaper_dft_seperable(
                int(npixfacet * fftpad), kernels.unpack_kernel(kern, W, OS), W,
                OS)
            vis_dft = np.ones((nrow, nchan, 2), dtype=np.complex64)
            vis_grid_facet = gridder.gridder(
                uvw,
                vis_dft,
                wavelength,
                chanmap,
                int(npixfacet * fftpad),
                cell * 3600.0,
                image_centres[0, :], (0, d0),
                kern,
                W,
                OS,
                "None",
                "None",
                "I_FROM_XXYY",
                "conv_1d_axisymmetric_packed_scatter",
>               do_normalize=True)

africanus/gridding/perleypolyhedron/tests/test_daskintrf.py:185: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../venv/afr/lib/python3.6/site-packages/numba/core/dispatcher.py:390: in _compile_for_args
    return self._compile_for_args(*args)
../../../../venv/afr/lib/python3.6/site-packages/numba/core/dispatcher.py:390: in _compile_for_args
    return self._compile_for_args(*args)
../../../../venv/afr/lib/python3.6/site-packages/numba/core/dispatcher.py:415: in _compile_for_args
    error_rewrite(e, 'typing')
../../../../venv/afr/lib/python3.6/site-packages/numba/core/dispatcher.py:358: in error_rewrite
    reraise(type(e), e, None)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

tp = <class 'numba.core.errors.TypingError'>
value = TypingError('Failed in nopython mode pipeline (step: nopython frontend)\nInternal error at <numba.core.typeinfer.CallC...          stokes_conversion_policy,\n                policy_type=literally(convolution_policy))\n                ^\n',)
tb = None

    def reraise(tp, value, tb=None):
        if value is None:
            value = tp()
        if value.__traceback__ is not tb:
>           raise value.with_traceback(tb)
E           numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
E           Internal error at <numba.core.typeinfer.CallConstraint object at 0x7fe05fa98ac8>.
E           'NoneType' object is not callable
E           During: resolving callee type: Function(<function policy at 0x7fe0df5de268>)
E           During: typing of call at /home/sperkins/work/ska/code/codex-africanus/africanus/gridding/perleypolyhedron/gridder.py (112)
E           
E           Enable logging at debug level for details.
E           
E           File "africanus/gridding/perleypolyhedron/gridder.py", line 112:
E           def gridder(uvw,
E               <source elided>
E                           stokes_conversion_policy,
E                           policy_type=literally(convolution_policy))
E                           ^

../../../../venv/afr/lib/python3.6/site-packages/numba/core/utils.py:80: TypingError
=========================== short test summary info ============================
FAILED africanus/gridding/perleypolyhedron/tests/test_daskintrf.py::test_gridder_nondask
====================== 1 failed, 15 deselected in 21.80s =======================

This seems to be the most relevant part of the output.

E               numba.core.errors.ForceLiteralArg: Pseudo-exception to force literal arguments in the dispatcher
E               
E               File "africanus/gridding/perleypolyhedron/gridder.py", line 112:
E               def gridder(uvw,
E                   <source elided>
E                               stokes_conversion_policy,
E                               policy_type=literally(convolution_policy))
E                               ^

@bennahugo, @JSKenyon Would either of you have the bandwidth to make a MRE and take it up with the numba team?

[sjperkins]

There seems to be some discussion around this topic in numba/numba#5411 (comment)

[sjperkins]

It looks as if the strategy for handling this is to use SentryLiteralArgs:

numba/numba#5411 (comment)

there's some documentation around this, but I don't find it obvious:

https://numba.pydata.org/numba-doc/dev/developer/literal.html#inside-extensions

[bennahugo]

I was doing that but I had to stop due to other work. I'm unsure what has changed in the literal type system. There were no deprecation warnings raised so I think this is a bonafide bug in the releases >=0.5.I also don't quite follow the documentation for SentryLiteralArgs.

[sjperkins]

I don't think its a bug -- I think literals are an obscure feature that hasn't been fully fleshed out. If the use of SentryLiteralArgs is the prescribed method of resolving these issues, then they should be used.

[sjperkins]

One possible way of resolving this is to revert the original PR and continue development in #219 until the issue is resolved.

[sjperkins]

I've reverted the original PR in #220. All existing changes are preserved in #219. Sorry @bennahugo, I think some more work is required here.

[sjperkins]

Fixed in #222