Wish_List_for_xCAT_FVT

Stabilize the automation environment

• Resolve unstable problem of VMs in automation environment. (one possible solution is using RHEV). (p1) DONE
• Modify unstable test cases. (such like nodeset_cmdline and so on). (p1) DONE
• Make sure the same build run 7 times with the same regression result. (redhat+ppc64le has the highest priority, then SLES, ubuntu) (p1) Almost done, now the timing issue focus on xCAT product itself
• Make sure all automation hardware are in good condition and can be monitored. DONE
  • Check the status (CPU/memory/disk space) of hypervisor by script. (p1)

Increase test cases

• Work out a rule about how to write a test case, such like must clean up environment and how to handle multiple platforms. (p1) DONE
• Add test case depending on customer’s feedback. (p1) in progress
  • Go through github or mail list, add cases for the issues have been resolved.
• Add test case for new feature. (p1) in progress
  • How to plan test case set.
  • Work out a rule to measure the case automated rate and pass rate.
• Summarize the number of new case and its pass rate in the end of each sprint. (p1) in progress
• How to manage the manual test cases. (one option is github). (p2) DONE
• How to involve developer to contribute test cases. (p2) in progress
• Go through existed cases and refine (p3)
  • Offer a ideal test scope by developer
    • Command line : *def , hardware control command, nodeset, makedns , makehosts, makedhcp, updatenode, xdsh , xdcp, pping, ppping, prsync…
    • OS provision process : copycds, rinstall, postscript, postbootscript, confignics, confignetwork, genimage,packimage
    • DB operation : tab*, DB backup/restore.
    • xCAT installation : go_xcat, xcatconfig, xcat deamon control, lsxcatd, migration
    • Hardware discovery : mknb, bmcsetup,
    • REST api
    • KIT
  • Analyse how many existed cases are valid.
  • Compare the valid cases and the ideal test scope to work out how many cases need to add.

Enhance release process

• Figure out the coverage of release cycle.
  • Daily regression DONE
  • Manual test : against specific hardware, os provision against physical machine, discovery. (p1)
  • Release process DONE
• Work out a check list for 3.1, including acceptance criteria. (p1) DONE
• Automated the 3.1.3 release process (p3)

Enhance automation framework

• Work out a step by step to debug problem in automation environment. (p1) DONE
• How to verify a new test case in non-production environment. (p1) DONE
• Reduce the running time of regression project. (p1)
• Make it easier to prepare a xcattest configuration file (p1) DONE
  • Make it easier to find out the error caused by xcattest configuration file or new build in automation environment.
• Make it easier for developer to do unit test (p1) in progress
• Using Jenkins API to add /delete/switch projects. (p1) DONE
• Refine test report to including more useful information to debug. (p2)
  • To highlight the people who checked in code during last day. DONE
• How to maintain and categorize bundle files. Make it easier to add new test case and to invoke bundles. (p2)
  • Using “#INCLUDE” label. DONE
  • Support bundle priority.
  • Manage the case not existed in bundle.
  • Generate bundle for specific purpose. Such like for specific hardware device or for specific business project. in progress
• Enhance automation infrastructure to support more test requirement, including hardware and network. (p2)
  • To support discovery test.
  • To support specific hardware test.
• Make it easier to add/delete server into/from automation platform (p2) in progress
• Make it easier to reset up a new automation platform (p2) in progress
• Find the root cause of provision failure quickly (p3)
• Report auto test result to slack quickly. (p3)
• Learning hot automation tools (p3)

Leverage CI to do quick verification for pull request (p3)

• To verify if a pull request follow expected process. DONE
• To check syntax DONE
• To do build DONE
• To check xcat basic function, such like checking xcatd daemon, xcat DB and so on. DONE
• Set up a build service to build xCAT package depending on specific requirement, such like for specific repo.