Some typing errors

[kevinboulain]

Hi there, I got some typing errors and I'm not sure where the problems came from, could you give me your thoughts about the following examples?
Feel free to close the issue if there is nothing wrong on enforce's side.

Version used, pipe show enforce:

Name: enforce
Version: 0.3.4
...

Example 1

In this case, there is no problem if the callback is not a bound method.

Code:

from typing import Optional, Callable, Any, Dict
import enforce

class Example:
    @enforce.runtime_validation
    def method(self, d: Dict):
        pass

@enforce.runtime_validation
def bound_callback_example(callback: Optional[Callable[[Dict], Any]]=None):
    pass

example = Example()
bound_callback_example(example.method)

This code may raise two different tracebacks, the difference between the two being the type of the function:

Traceback (most recent call last):
  File "enforce_examples.py", line 14, in <module>
    bound_callback_example(example.method)
  File ".../venv/lib/python3.6/site-packages/enforce/decorators.py", line 104, in universal
    _args, _kwargs, _ = enforcer.validate_inputs(parameters)
  File ".../venv/lib/python3.6/site-packages/enforce/enforcers.py", line 86, in validate_inputs
    raise RuntimeTypeError(exception_text)
enforce.exceptions.RuntimeTypeError: 
  The following runtime type errors were encountered:
       Argument 'callback' was not of type typing.Union[typing.Callable[[typing.Dict], typing.Any], NoneType]. Actual type was BoundFunctionWrapper.

Traceback (most recent call last):
  File "enforce_examples.py", line 14, in <module>
    bound_callback_example(example.method)
  File ".../venv/lib/python3.6/site-packages/enforce/decorators.py", line 104, in universal
    _args, _kwargs, _ = enforcer.validate_inputs(parameters)
  File ".../venv/lib/python3.6/site-packages/enforce/enforcers.py", line 86, in validate_inputs
    raise RuntimeTypeError(exception_text)
enforce.exceptions.RuntimeTypeError: 
  The following runtime type errors were encountered:
       Argument 'callback' was not of type typing.Union[typing.Callable[[typing.Dict], typing.Any], NoneType]. Actual type was typing.Callable.

The errors go away when the callback type is less strict:

@enforce.runtime_validation
def bound_callback_example(callback: Optional[Callable]=None):
    pass

Is this the expected behavior?

Example 2

The traceback makes me think this may be an error on enforce's side.

Code:

from typing import Generator
import enforce

@enforce.runtime_validation
def generator() -> Generator[int, None, None]:
    i = 0
    while True:
        yield i
        i += 1

g = generator()
print(next(g))
print(next(g))

Traceback:

Traceback (most recent call last):
  File ".../venv/lib/python3.6/site-packages/enforce/nodes.py", line 539, in preprocess_data
    enforcer = data.__enforcer__
AttributeError: 'generator' object has no attribute '__enforcer__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "enforce_examples.py", line 11, in <module>
    g = generator()
  File ".../venv/lib/python3.6/site-packages/enforce/decorators.py", line 118, in universal
    return enforcer.validate_outputs(result)
  File ".../venv/lib/python3.6/site-packages/enforce/enforcers.py", line 96, in validate_outputs
    if not self.validator.validate(output_data, 'return'):
  File ".../venv/lib/python3.6/site-packages/enforce/validator.py", line 26, in validate
    validation_result = visit(validation_tree)
  File ".../venv/lib/python3.6/site-packages/enforce/utils.py", line 17, in visit
    stack.append(last.send(last_result))
  File ".../venv/lib/python3.6/site-packages/enforce/nodes.py", line 64, in validate
    clean_data = self.preprocess_data(validator, data)
  File ".../venv/lib/python3.6/site-packages/enforce/nodes.py", line 541, in preprocess_data
    return GenericProxy(data)
  File ".../venv/lib/python3.6/site-packages/enforce/enforcers.py", line 130, in __init__
    raise TypeError('Only generics can be wrapped in GenericProxy')
TypeError: Only generics can be wrapped in GenericProxy

There is a similar problem when a function returns a Coroutine.

[kevinboulain]

Forgot this one.

Example 3

Code:

import datetime
from typing import List, NamedTuple
import enforce

class Example(NamedTuple):
    datetime: datetime.datetime

@enforce.runtime_validation
def example() -> List[Example]:
    return [Example(datetime.datetime.now())]

example()

Traceback:

Traceback (most recent call last):
  File "enforce_examples.py", line 9, in <module>
    def example() -> List[Example]:
  File ".../venv/lib/python3.6/site-packages/enforce/decorators.py", line 58, in runtime_validation
    return generate_decorated()
  File ".../venv/lib/python3.6/site-packages/enforce/decorators.py", line 158, in build_wrapper
    return decorate(wrapped, configuration, None)
  File ".../venv/lib/python3.6/site-packages/enforce/decorators.py", line 70, in decorate
    data = apply_enforcer(data, parent_root=parent_root, settings=configuration)
  File ".../venv/lib/python3.6/site-packages/enforce/enforcers.py", line 159, in apply_enforcer
    func.__enforcer__ = generate_new_enforcer(func, generic, parent_root, instance_of, settings)
  File ".../venv/lib/python3.6/site-packages/enforce/enforcers.py", line 227, in generate_new_enforcer
    validator = init_validator(hints, parent_root)
  File ".../venv/lib/python3.6/site-packages/enforce/validator.py", line 71, in init_validator
    syntax_tree = visit(root_parser)
  File ".../venv/lib/python3.6/site-packages/enforce/utils.py", line 17, in visit
    stack.append(last.send(last_result))
  File ".../venv/lib/python3.6/site-packages/enforce/parsers.py", line 54, in _parse_namedtuple
    new_node = yield nodes.NamedTupleNode(hint)
  File ".../venv/lib/python3.6/site-packages/enforce/nodes.py", line 390, in __init__
    super().__init__(runtime_validation(data_type), is_sequence=True, is_container=True, **kwargs)
  File ".../venv/lib/python3.6/site-packages/enforce/decorators.py", line 46, in runtime_validation
    return get_typed_namedtuple(configuration, data, fields, field_types)
  File "...venv/lib/python3.6/site-packages/enforce/decorators.py", line 180, in get_typed_namedtuple
    exec(new_init_template, context)
  File "<string>", line 1, in <module>
NameError: name 'datetime' is not defined

Can't remember the first time I got this one, sorry (it was with a custom type and not datetime but the error was similar).

EDIT: just remembered where I found a similar issue: when using forward references as indicated in the PEP/documentation.

Code:

import enforce

class Example:
    @classmethod
    @enforce.runtime_validation
    def create(cls, *args, **kwargs) -> 'Example':
        return cls(*args, **kwargs)

Example.create()

[RussBaz]

Right . . . I think they are separate issues but I will address them here.

1. I think I know where the problem lies but it is a bit troublesome to pinpoint exactly. Decorating logic issue due to the wrapt library special handling of callables.
2. Yeah, I have neglected generators, coroutines and other async stuff so far. Generators should be the easiest to add. PR are surely welcome, though!
3. Looks like a corner case of NamedTuple special handling (because you inherited from it). Will give it a look.
4. Forward Reference is not yet supported. I need to add some lazy evaluation to it. Tracked in Forward References #45

This is a pretty huge report. It will certainly take time to fix all of them as I am not very free at the moment. Anyway, I will see what I can do given the amount of time I have.

Thanks.

[kevinboulain]

This issue could be split into smaller ones (I can open new ones and close this one), whatever is best for you, I just batched some errors that I encountered while using enforce.

The Callable issue isn't that important: it's just less self-documenting and move the burden of checking the arguments to the callee.

For the generators/iterators/... that's what I thought when I saw the empty tests. :)

For NamedTuple inheritance, it could be nice to handle it as it seems to be the recommended way to do it.

[RussBaz]

You are right. Sigh... More work for me. Hahaha. Alright, I will try fixing some of them on the weekend.

Anyway, I would prefer to have separate issues. If you can, please split them.

Thank you.

[kevinboulain]

Cheer up, I think this is the most advanced library out there for this kind of thing :)
Keep up the good work.

I opened #46 & #47.

By the way, I'm not sure if Python is really suited/designed for strict typing so maybe you shouldn't try to implement everything. But I do really like to have the ability to 'typecheck' some important parts of the program (for me, this is one of the most annoying thing with Python: even if the ease of development is great, there are too much ways to get bitten and I think enforce allow us to have a little more confidence on some critical parts).