Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Failing utils spec tests #1880

Closed
macaktom opened this issue Feb 7, 2024 · 7 comments
Closed

[BUG] Failing utils spec tests #1880

macaktom opened this issue Feb 7, 2024 · 7 comments
Assignees
@agentpoyo
Labels
bug Something isn't working

Comments

@macaktom
Copy link
Contributor

macaktom commented Feb 7, 2024

Describe the bug
Utils spec tests are failing, when running them (individually or via command crystal spec).

To Reproduce
crystal spec
or
crystal spec ./spec/utils/utils_spec.cr

Expected behavior
A clear and concise description of what you expected to happen.

Device (please complete the following information):

  • OS Linux
  • Distro Ubuntu
  • Version 22.04
  • Architecture x86
  • Crystal 1.11.1
  • Kubernetes v1.28.3

How will this be tested? aka Acceptance Criteria (optional)
For verification successfully running crystal spec ./spec/utils/utils_spec.cr should be enough.

Additional context

macaktom@WPF2LRVSF:~/devel/cnf-testsuite$ crystal spec spec/utils/utils_spec.cr
current_branch during compile: "main"
current_tag during compile: 
I, [2024-02-07 14:35:02 +01:00 #1209231]  INFO -- cnf-testsuite: Building ./cnf-testsuite
I, [2024-02-07 14:35:25 +01:00 #1209231]  INFO -- cnf-testsuite: Build Success!
.....✖️  FAILED: IP addresses found
.Successfully setup coredns-1609263557
✔️  PASSED: No privileged containers
Successfully cleaned up coredns-1609263557
..✔️  PASSED: No privileged containers
✖️  FAILED: Found 1 privileged containers: ["privileged-coredns"]
Successfully cleaned up coredns-1609263557
Successfully cleaned up privileged-coredns
.FF.....I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite-helm_deploy: Running helm_deploy
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite-helm_deploy: helm_deploy args: #<Sam::Args:0x7fcd34c2b500 @arr=["verbose"], @named_args={}>
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: check_cnf_config args: #<Sam::Args:0x7fcd34c2b500 @arr=["verbose"], @named_args={}>
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: check_cnf_config cnf: 
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: cnf_config_list
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: find: find cnfs/* -name "cnf-testsuite.yml"
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: find response: ["cnfs/coredns/cnf-testsuite.yml"]
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: task_runner args: #<Sam::Args:0x7fcd34c2b500 @arr=["verbose"], @named_args={}>
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: cnf_config_list
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: find: find cnfs/* -name "cnf-testsuite.yml"
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: find response: ["cnfs/coredns/cnf-testsuite.yml"]
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: ensure_cnf_installed?  true
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: check_cnf_config args: #<Sam::Args:0x7fcd34c2b500 @arr=["verbose"], @named_args={}>
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: check_cnf_config cnf: 
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: cnf_config_list
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: find: find cnfs/* -name "cnf-testsuite.yml"
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: find response: ["cnfs/coredns/cnf-testsuite.yml"]
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: CNF configs found: 1
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: airgapped: false
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: generate_tar_mode: false
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: ensure_cnf_testsuite_yml_path
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: generate_and_set_release_name
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: generate_and_set_release_name config_yml_path: cnfs/coredns/cnf-testsuite.yml
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: airgapped mode: false
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: generate_tar_mode: false
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: ensure_cnf_testsuite_yml_path
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: ensure_cnf_testsuite_yml_dir
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: parsed_config_file: cnfs/coredns/cnf-testsuite.yml
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: src_helm_directory: 
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: parsed_config_file: cnfs/coredns/cnf-testsuite.yml
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: cnf_installation_method
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: cnf_installation_method config: #<Totem::Config:0x7fcd33377f00>
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: cnf_installation_method config: cnfs/coredns/cnf-testsuite.yml
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: directory_parameter_split : 
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: directory_parameter_split : 
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: directory :  parameters: 
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: release_name: coredns
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: helm_directory: 
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: manifest_directory: 
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: Building helm_directory and manifest_directory full paths
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: full_helm_directory: /home/macaktom/devel/cnf-testsuite/cnfs/coredns/ exists? true
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: full_manifest_directory: /home/macaktom/devel/cnf-testsuite/cnfs/coredns/ exists? true
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: cnf_destination_dir config_file: cnfs/coredns/cnf-testsuite.yml
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: parsed_config_file: cnfs/coredns/cnf-testsuite.yml
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: release_name: coredns
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: cnf destination dir: /home/macaktom/devel/cnf-testsuite/cnfs/coredns
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: ensure_cnf_testsuite_yml_dir
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: USING EXPORTED CHART PATH
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite-helm_deploy: Starting test
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: KubectlClient::Get.configmap command: kubectl get configmap cnf-testsuite-coredns-startup-information -o json
✔️  PASSED: Helm deploy successful ⚙️🛠️⬆☁️
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: task_points: task: helm_deploy is worth: 5 points
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: cmd: /home/macaktom/devel/cnf-testsuite/cnf-testsuite helm_deploy verbose
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite-helm_deploy: task_runtime=263; start_time=2024-02-07 13:37:47 UTC; end_time:2024-02-07 13:37:47 UTC
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: task_type_by_task
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: points: {"name" => "helm_deploy", "tags" => "compatibility, dynamic, workload, cert, normal"}
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: resp: ["compatibility", "dynamic", "workload", "cert", "normal"]
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: task_type x: compatibility acc: 
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: task_type x: dynamic acc: 
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: task_type x: workload acc: 
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: task_type x: cert acc: 
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: task_type x: normal acc: cert
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: task_type: normal
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: upsert_task: task: helm_deploy has status: passed and is awarded: 5 points. Runtime: 263 seconds
I, [2024-02-07 14:37:47 +01:00 #1219658]  INFO -- cnf-testsuite: results yaml: {"name" => "cnf testsuite", "testsuite_version" => "main-2024-02-07-143747-4115cbe1", "status" => nil, "command" => "/home/macaktom/devel/cnf-testsuite/cnf-testsuite helm_deploy verbose", "points" => nil, "exit_code" => 0, "items" => [{"name" => "helm_deploy", "status" => "passed", "type" => "normal", "points" => 5}]}
Successfully cleaned up coredns
...

Failures:

  1) Utils 'task_runner' should run a test against a single cnf if passed a cnf-config argument even if there are multiple cnfs installed
     Failure/Error: (resp).includes?("✖️  FAILED: Found 1 privileged containers").should be_true

       Expected: true
            got: false

     # spec/utils/utils_spec.cr:199

  2) Utils 'logger' command line logger level setting via config.yml
     Failure/Error: (/DEBUG -- cnf-testsuite: debug test/ =~ response_s).should be_nil

       Expected: 46 to be nil

     # spec/utils/utils_spec.cr:214

Finished in 2:46 minutes
19 examples, 2 failures, 0 errors, 0 pending

Failed examples:

crystal spec spec/utils/utils_spec.cr:193 # Utils 'task_runner' should run a test against a single cnf if passed a cnf-config argument even if there are multiple cnfs installed
crystal spec spec/utils/utils_spec.cr:209 # Utils 'logger' command line logger level setting via config.yml
@macaktom macaktom added the bug Something isn't working label Feb 7, 2024
@agentpoyo
Copy link
Collaborator

Can you provide DEBUG output? There was a recent PR we pushed and merged because it was passing spec tests in the github runners but found that it was likely passing using cached data as they were now failing after we destroyed and recreated new runners after the github repo URL change. I want to verify if this is one of the causes as a fix is underway with a revert of the contants.cr file that was a cause of the found issue.

@agentpoyo agentpoyo self-assigned this Feb 7, 2024
@agentpoyo agentpoyo mentioned this issue Feb 7, 2024
Closed
@macaktom
Copy link
Contributor Author

macaktom commented Feb 7, 2024

I already run that test as debug (configured log level in config.yaml). Or I need to run it in different way?

FYI: Spec tests also failed for 5G RAN (ran_spec) and 5G core (core_spec) but I didn't have time to create tickets for them. Maybe it will be same issue as with this one.

@agentpoyo
Copy link
Collaborator

agentpoyo commented Feb 7, 2024 via email

@denverwilliams denverwilliams mentioned this issue Feb 12, 2024
Merged
20 tasks
wvwatson added a commit that referenced this issue Feb 12, 2024
@wvwatson
Merge pull request #1883 from cnti-testcatalog/fix_ci
3f944c8 
Fix CI #1880 #1879 #1878 #1874 #1806
@denverwilliams
Copy link
Collaborator

denverwilliams commented Feb 14, 2024 •
edited
Loading

@macaktom The spec tests were only running correctly when LOG_LEVEL=info mode was set because of the below commands:

    Log.info {`./cnf-testsuite cnf_setup cnf-config=sample-cnfs/sample-generic-cnf/cnf-testsuite.yml`}
    Log.info {`./cnf-testsuite cnf_setup cnf-config=sample-cnfs/sample_privileged_cnf/cnf-testsuite.yml`}

When not running in 'info' mode those commands didn't execute at all. Which was causing the test failure.

This is now fixed by this commit: 3f138cc

@denverwilliams denverwilliams mentioned this issue Feb 14, 2024
Merged
20 tasks
agentpoyo added a commit that referenced this issue Feb 15, 2024
@agentpoyo
Merge pull request #1892 from cnti-testcatalog/fix_utils_spec_1880
3ed5d54 
Fix bug with utils spec #1880
collivier pushed a commit to collivier/testsuite that referenced this issue Feb 16, 2024
@agentpoyo @collivier
Merge pull request lfn-cnti#1892 from cnti-testcatalog/fix_utils_spec…
b575a38 
…_1880

Fix bug with utils spec lfn-cnti#1880
@horecoli
Copy link
Contributor

@denverwilliams But fixing this by switching LOG_LEVEL to 'info' is more like a workaround than a real solution of this issue. I think it has to be possible to execute spec tests with different log level and have green results.

@horecoli
Copy link
Contributor

Oh, I was just alerted that a ticket has already been created for that issue -> #1495

@denverwilliams
Copy link
Collaborator

@horecoli This fix wasn't simply switching 'LOG_LEVEL to 'info'. The fix here makes it so spec tests will now execute regardless of the log level.

See: 3f138cc#diff-e46af2840c1b39bfc5a893dfe850e3748f90f0339f72436b344a789747e9646cR194

The spec will now execute tasks regardless of the log level and assign the output to a variable.
It will then print the output, after execution, based on the log level set.

@lixuna lixuna closed this as completed Mar 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@agentpoyo agentpoyo

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@lixuna @agentpoyo @denverwilliams @macaktom @horecoli