Impossible to preserve integer type in variables with complex data structure

[emanation]

ISSUE TYPE

• Bug Report

COMPONENT NAME

variables handling

ANSIBLE VERSION

ansible 2.1.0.0
ansible 2.1.2.0

CONFIGURATION

[defaults]
inventory = /home/ansible/inventory/
gathering = smart
fact_caching = redis
fact_caching_timeout = 11800
display_skipped_hosts = False
ssh_args = -o ControlMaster=no -o ControlPath=none -o ControlPersist=no
hash_behaviour = replace
ansible_managed = Ansible managed: {file} modified by {uid} on {host}
module_name = shell

[ssh_connection]
ssh_args = -o ControlMaster=no

OS / ENVIRONMENT

Amazon Linux, MacOS X

SUMMARY

Some modules and APIs require numbers to be integer or float type. Mandatory syntax with quoted variables in YAML has big practical problem. It's impossible to define integers with that.

STEPS TO REPRODUCE

However it works only with straight simple variables but not with complex data structures.

- name: Testing variables
  connection: local
  vars:
    env: staging
    service_vars:
      production:
        count: 12
        cpu: 1024
        ram: 1300
      staging:
        count: 2
        cpu: 100
        ram: 800
    staging:
        count: 2
        cpu: 100
        ram: 800
    first_level: 2
    service:
      string_ok: "myapp/{{env}}"
      always_string:
      - not_int1: "{{service_vars[env].cpu | int}}"
        not_int2: "{{service_vars[env].ram + 1}}"
      not_working: "{{staging.count | int}}"
      works_here: "{{first_level}}"

  tasks:
  - debug: var=service

EXPECTED RESULTS

ok: [localhost] => {
    "service": {
        "containers": [
            {
                "cpu": 100, 
                "memory": 801, 
                "name": "myapp"
            }
        ], 
        "count": 2, 
        "family": "mytaskdef", 
        "name": "something", 
        "repo": "myapp/staging"
    }
}

ACTUAL RESULTS

ok: [localhost] => {
    "service": {
        "containers": [
            {
                "cpu": "100", 
                "memory": "801", 
                "name": "myapp"
            }
        ], 
        "count": "2", 
        "family": "mytaskdef", 
        "name": "something", 
        "repo": "myapp/staging"
    }
}

Neither {{service_vars[service_env].cpu | int}} filter nor {{service_vars[service_env].ram + 1}} expression help.
However it seems that variable is considered as integer because increment does work but final value is quoted and it's converted to string.
The only way how it may work is to have simple var defined at first level. But then all Environment logic is useless.
Just because of that I have to have similar variable files there just integers are defined explicitly. And that is only difference between var files.

[jctanner]

@emanation Ansible's underlying template engine is jinja. Jinja's origin as a web templating engine has the characteristic of always returning strings. Prior to 2.x, we always templated values to strings, so the fact that 2.x gives you integers sometimes is an improvement. For example, here's what your testcase outputs in 1.9.6 ...

TASK: [debug var=service] *****************************************************
ok: [localhost] => {
    "var": {
        "service": {
            "always_string": [
                {
                    "not_int1": "100",
                    "not_int2": "801"
                }
            ],
            "not_working": "2",
            "string_ok": "myapp/staging",
            "works_here": "2"
        }
    }
}

In 2.x we wrote an "interceptor" of sorts that circumvents sending the variable to jinja if we already know the exact value.
https://github.com/ansible/ansible/blob/devel/lib/ansible/template/vars.py#L32-L35
https://github.com/ansible/ansible/blob/devel/lib/ansible/template/__init__.py#L308-L320

If the variable is not "clean", meaning it has a key reference such as "foo.bar" or "foo['bar'", we have to send it to jinja. In that case, we don't know what the original type is and can not preserve it. That is why your staging.count variable becomes a string.

To address your primary concern in the description:

Some modules and APIs require numbers to be integer or float type. Mandatory syntax with quoted variables in YAML has big practical problem. It's impossible to define integers with that.

That is exactly why parameters to modules should be explicitly declaring the type. We cast expected type in the module argument parser so that the rest of the module code gets what is expected.

I wrote a quick module to demonstrate the concept:

[jtanner@fedmac AP-17992]$ cat library/varcheck.py
#!/usr/bin/python

def main():
    module = AnsibleModule(
        argument_spec = dict(
            xint=dict(default=None, type='int'),
            xfloat=dict(default=None, type='float'),
            xstr=dict(default=None, type='str'),
        ),
        supports_check_mode=True
    )
    xint = module.params["xint"]
    xfloat = module.params["xfloat"]
    xstr = module.params["xstr"]
    data = {}
    data['xint'] = {}
    data['xint']['type'] = str(type(xint))
    data['xint']['value'] = str(xint)
    data['xfloat'] = {}
    data['xfloat']['type'] = str(type(xfloat))
    data['xfloat']['value'] = str(xfloat)
    data['xstr'] = {}
    data['xstr']['type'] = str(type(xstr))
    data['xstr']['value'] = str(xstr)

    module.exit_json(changed=False, data=data)

# import module snippets
from ansible.module_utils.basic import *

if __name__ == '__main__':
    main()

I added this task to the end of your example playbook ...

    - varcheck:
        xint: "{{ service.not_working }}"
        xfloat: "{{ service.not_working }}"
        xstr: "{{ service.not_working }}"

This is the result ...

Using module file /home/jtanner/workspace/issues/AP-17992/library/varcheck.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: jtanner
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1476459601.69-265247724299299 `" && echo ansible-tmp-1476459601.69-265247724299299="` echo $HOME/.ansible/tmp/ansible-tmp-1476459601.69-265247724299299 `" ) && sleep 0'
<127.0.0.1> PUT /tmp/tmpeWR3EU TO /home/jtanner/.ansible/tmp/ansible-tmp-1476459601.69-265247724299299/varcheck.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/jtanner/.ansible/tmp/ansible-tmp-1476459601.69-265247724299299/ /home/jtanner/.ansible/tmp/ansible-tmp-1476459601.69-265247724299299/varcheck.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python /home/jtanner/.ansible/tmp/ansible-tmp-1476459601.69-265247724299299/varcheck.py; rm -rf "/home/jtanner/.ansible/tmp/ansible-tmp-1476459601.69-265247724299299/" > /dev/null 2>&1 && sleep 0'
ok: [localhost] => {
    "changed": false,
    "data": {
        "xfloat": {
            "type": "<type 'float'>",
            "value": "2.0"
        },
        "xint": {
            "type": "<type 'int'>",
            "value": "2"
        },
        "xstr": {
            "type": "<type 'str'>",
            "value": "2"
        }
    },
    "invocation": {
        "module_args": {
            "xfloat": 2.0,
            "xint": 2,
            "xstr": "2"
        },
        "module_name": "varcheck"
    }
}

Having data as a string and passing it to modules that expect other types is -not- a problem, if the data can be cast to the expected type. If a module does not declare the type appropriately and misbehaves, that is a bug with that particular module and should be filed accordingly.

[dagwieers]

There is a light at the end of the tunnel. We made a change to Jinja2 so we don't see all variable types changed into strings. See: pallets/jinja#708