[test infrastructure] allow overriding recover system actioin (#480)

Automatical recover test environment is great to let test to run in a healthy environment.

There are scenarios that running a test was to repeat an issue. In this case, we want the
test to fail when prolem happens and leave the system in the state for investigation.

Adding opiton to prevent test to recover system. Default behavior is still recoverying
unhealthy system.

ansible/roles/test/tasks/base_sanity.yml

@@ -36,7 +36,7 @@
 
      when:
        - ({{ ps_out.stdout_lines | length }} <= 0) or ({{ orch_out.stdout_lines | length }} <= 0)
-       - recover is defined
+       - recover == "true"
 
    - name: Get syslog error information
      shell: cat /var/log/syslog |tail -n 5000 |grep -i error

ansible/roles/test/tasks/interface.yml

@@ -24,7 +24,7 @@
 
   when:
     - ansible_interface_link_down_ports | length > 0
-    - recover is defined
+    - recover == "true"
 
 - debug: msg="Found link down ports {{ansible_interface_link_down_ports}}"
   when: ansible_interface_link_down_ports | length > 0

ansible/roles/test/tasks/test_sonic_by_testname.yml

@@ -4,15 +4,19 @@
 - debug: msg="!!!!!!!!!!!!!!!!!!!! start to run test {{ testcase_name }} !!!!!!!!!!!!!!!!!!!!"
 - debug: msg="!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
 
+- set_fact:
+    allow_recover: true
+  when: allow_recover is not defined
+
 - name: do basic sanity check before each test
   include: base_sanity.yml
   vars: 
-     recover: true
+     recover: "{{ allow_recover }}"
 
 - name: validate all interfaces is up
   include: interface.yml
   vars:
-     recover: true
+     recover: "{{ allow_recover }}"
 
 ### by default, when calling a test case name, we pass 'testbed_type', 'ptf_host, 'dut_name(ansible_hoatname)' down to test playbook.
 ### if your test playbook requires more extra vars then default, please make sure you handled them correctly within test playbook.