Unable to configure 'audit' cookbook for sending compliance data to Chef Automate 2

[sahilsehgal81]

Hi, I've recently setup Chef Automate v2 in my organization and integrated it with Chef Server and I see the data feeds/metrics from chef server, but while I try to setup compliance data reporting in chef automate, I try to add the recommended settings under audit cookbook attributes > default.rb, I see this error message after execution of cookbook:

--------------------------------------------------------------------
Connecting to 10.188.128.88
10.188.128.88 -----> Existing Chef installation detected
10.188.128.88 Starting the first Chef Client run...
10.188.128.88 Starting Chef Client, version 14.7.17
10.188.128.88 resolving cookbooks for run list: ["audit::default"]
10.188.128.88 Synchronizing Cookbooks:
10.188.128.88   - audit (7.3.0)
10.188.128.88 Installing Cookbook Gems:
10.188.128.88 Compiling Cookbooks...
10.188.128.88
10.188.128.88 ================================================================================
10.188.128.88 Recipe Compile Error in /var/chef/cache/cookbooks/audit/attributes/default.rb
10.188.128.88 ================================================================================
10.188.128.88
10.188.128.88 NoMethodError
10.188.128.88 -------------
10.188.128.88 undefined method `push' for {}:Chef::Node::VividMash
10.188.128.88
10.188.128.88 Cookbook Trace:
10.188.128.88 ---------------
10.188.128.88   /var/chef/cache/cookbooks/audit/attributes/default.rb:89:in `from_file'
10.188.128.88
10.188.128.88 Relevant File Content:
10.188.128.88 ----------------------
10.188.128.88 /var/chef/cache/cookbooks/audit/attributes/default.rb:
10.188.128.88
10.188.128.88  82:
10.188.128.88  83:  # controls whether or not existing profile is overwritten when using upload recipe
10.188.128.88  84:  default['audit']['overwrite'] = true
10.188.128.88  85:
10.188.128.88  86:  # Chef Inspec Compliance profiles to be used for scan of node
10.188.128.88  87:  # See README.md for details
10.188.128.88  88:  #default['audit']['profiles'] = []
10.188.128.88  89>> default['audit']['profiles'].push(
10.188.128.88  90:    'name': 'cis-centos7-level2',
10.188.128.88  91:    'compliance': 'admin/cis-centos7-level2' # in the ui for automate, this value is the identifier for the profile
10.188.128.88  92:  )
10.188.128.88  93:
10.188.128.88  94:  # Attributes used to run the given profiles
10.188.128.88  95:  default['audit']['attributes'] = {}
10.188.128.88  96:
10.188.128.88  97:  # If enabled, a hash of the Chef "node" object will be sent to InSpec in an attribute
10.188.128.88  98:  # named `chef_node`
10.188.128.88
10.188.128.88 System Info:
10.188.128.88 ------------
10.188.128.88 chef_version=14.7.17
10.188.128.88 platform=centos
10.188.128.88 platform_version=7.5.1804
10.188.128.88 ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
10.188.128.88 program_name=/bin/chef-client
10.188.128.88 executable=/opt/chefdk/bin/chef-client
10.188.128.88
10.188.128.88
10.188.128.88 Running handlers:
10.188.128.88 [2019-01-17T08:51:43+00:00] ERROR: Running exception handlers
10.188.128.88 Running handlers complete
10.188.128.88 [2019-01-17T08:51:43+00:00] ERROR: Exception handlers complete
10.188.128.88 Chef Client failed. 0 resources updated in 01 seconds
10.188.128.88 [2019-01-17T08:51:43+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
10.188.128.88 [2019-01-17T08:51:43+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
10.188.128.88 [2019-01-17T08:51:43+00:00] FATAL: NoMethodError: undefined method `push' for {}:Chef::Node::VividMash

---

In order to prevent this, I also tried to adjust the 'profiles' value under attributes as follows:

default['audit']['profiles']['cis-centos7-level2'] = { 'compliance': 'admin/cis-centos7-level2' }

after which the cookbook does execute fine, but leaves behind the error messages, as given below:

-----------------------------------------------
[2019-01-17T10:01:43+00:00] INFO: *** Chef 14.7.17 ***
[2019-01-17T10:01:43+00:00] INFO: Platform: x86_64-linux
[2019-01-17T10:01:43+00:00] INFO: Chef-client pid: 28557
[2019-01-17T10:01:43+00:00] DEBUG: Chef-client request_id: 56b9111d-0586-4996-b289-c97034a54035
[2019-01-17T10:01:43+00:00] DEBUG: Running Ohai with the following configuration: {:logger=>Chef::Log}
[2019-01-17T10:01:43+00:00] INFO: The plugin path /etc/chef/ohai/plugins does not exist. Skipping...
[2019-01-17T10:01:44+00:00] DEBUG: Plugin Zpools: Could not shell_out "zpool list -H -o name,size,alloc,free,cap,dedup,health,version". Skipping plugin.
[2019-01-17T10:02:05+00:00] WARN: Data collector token authentication is not recommended for client-server modePlease upgrade Chef Server to 12.11.0 and remove the token from your config file to use key based authentication instead
[2019-01-17T10:02:05+00:00] DEBUG: Extracting run list from JSON attributes provided on command line
[2019-01-17T10:02:05+00:00] DEBUG: Applying attributes from json file
[2019-01-17T10:02:05+00:00] DEBUG: Platform is centos version 7.5.1804
[2019-01-17T10:02:05+00:00] INFO: Run List is [recipe[audit::default]]
[2019-01-17T10:02:05+00:00] INFO: Run List expands to [audit::default]
[2019-01-17T10:02:05+00:00] INFO: Starting Chef Run for localnode
[2019-01-17T10:02:05+00:00] INFO: Running start handlers
[2019-01-17T10:02:05+00:00] INFO: Start handlers complete.
[2019-01-17T10:02:05+00:00] INFO: Error while reporting run start to Data Collector. URL: https://i-024187b4faaecc4a5.tst.nibr.novartis.net/organizations/uch-test/data-collector Exception: 400 -- 400 "Bad Request"  (This is normal if you do not have Chef Automate)
[2019-01-17T10:02:05+00:00] INFO: Loading cookbooks [audit@7.3.0]
[2019-01-17T10:02:05+00:00] DEBUG: Cookbooks to compile: [:audit]
[2019-01-17T10:02:05+00:00] DEBUG: Loading cookbook audit's library file: /var/chef/cache/cookbooks/audit/libraries/compliance.rb
[2019-01-17T10:02:05+00:00] DEBUG: Loading cookbook audit's library file: /var/chef/cache/cookbooks/audit/libraries/helper.rb
[2019-01-17T10:02:05+00:00] DEBUG: Loading cookbook audit's library file: /var/chef/cache/cookbooks/audit/libraries/matchers.rb
[2019-01-17T10:02:05+00:00] DEBUG: Loading cookbook audit's library file: /var/chef/cache/cookbooks/audit/libraries/reporters/automate.rb
[2019-01-17T10:02:05+00:00] DEBUG: Loading cookbook audit's library file: /var/chef/cache/cookbooks/audit/libraries/reporters/compliance.rb
[2019-01-17T10:02:05+00:00] DEBUG: Loading cookbook audit's library file: /var/chef/cache/cookbooks/audit/libraries/reporters/cs_automate.rb
[2019-01-17T10:02:05+00:00] DEBUG: Loading cookbook audit's library file: /var/chef/cache/cookbooks/audit/libraries/reporters/cs_compliance.rb
[2019-01-17T10:02:05+00:00] DEBUG: Loading cookbook audit's library file: /var/chef/cache/cookbooks/audit/libraries/reporters/json_file.rb
[2019-01-17T10:02:05+00:00] DEBUG: Node localnode loading cookbook audit's attribute file /var/chef/cache/cookbooks/audit/attributes/default.rb
[2019-01-17T10:02:05+00:00] DEBUG: Loading cookbook audit's resources from /var/chef/cache/cookbooks/audit/resources/compliance_upload.rb
[2019-01-17T10:02:05+00:00] DEBUG: Loading cookbook audit's resources from /var/chef/cache/cookbooks/audit/resources/inspec_gem.rb
[2019-01-17T10:02:05+00:00] INFO: Processing inspec_gem[inspec] action install (audit::inspec line 20)
[2019-01-17T10:02:05+00:00] DEBUG: Installed InSpec version: 3.1.3
[2019-01-17T10:02:05+00:00] INFO: Installation of InSpec required: false
[2019-01-17T10:02:05+00:00] INFO: inspec_gem: not installing InSpec. It's already installed or an explicit version was not supplied.
[2019-01-17T10:02:05+00:00] INFO: loading handler from /var/chef/cache/cookbooks/audit/files/default/handler/audit_report
[2019-01-17T10:02:05+00:00] DEBUG: Converging node localnode
[2019-01-17T10:02:05+00:00] INFO: Processing inspec_gem[inspec] action nothing (audit::inspec line 20)
[2019-01-17T10:02:05+00:00] DEBUG: Skipping inspec_gem[inspec] due to action :nothing
[2019-01-17T10:02:05+00:00] DEBUG: Saving the current state of node localnode
[2019-01-17T10:02:05+00:00] INFO: Chef Run complete in 0.397461528 seconds
[2019-01-17T10:02:05+00:00] INFO: Running report handlers
[2019-01-17T10:02:07+00:00] DEBUG: Load Chef Server fetcher from: /var/chef/cache/cookbooks/audit/files/default/vendor
[2019-01-17T10:02:07+00:00] DEBUG: Load Chef Automate fetcher from: /var/chef/cache/cookbooks/audit/files/default/vendor
[2019-01-17T10:02:07+00:00] DEBUG: Reporter is [json-automate]
[2019-01-17T10:02:07+00:00] INFO: Using InSpec 3.0.52
[2019-01-17T10:02:07+00:00] DEBUG: Options are set to: {"report"=>true, "format"=>"json-automate", "reporter"=>["json-automate"], "output"=>"/dev/null", "logger"=>Chef::Log, :backend_cache=>true, :attributes=>{}}
[2019-01-17T10:02:07+00:00] ERROR: Report handler Chef::Handler::AuditReport raised #<Net::HTTPFatalError: 503 "Service Unavailable">
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/2.5.0/net/http/response.rb:122:in `error!'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/2.5.0/net/http/response.rb:131:in `value'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/2.5.0/net/http.rb:958:in `connect'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/2.5.0/net/http.rb:920:in `do_start'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/2.5.0/net/http.rb:909:in `start'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/2.5.0/open-uri.rb:337:in `open_http'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/2.5.0/open-uri.rb:755:in `buffer_open'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/2.5.0/open-uri.rb:226:in `block in open_loop'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/2.5.0/open-uri.rb:224:in `catch'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/2.5.0/open-uri.rb:224:in `open_loop'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/2.5.0/open-uri.rb:165:in `open_uri'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/2.5.0/open-uri.rb:735:in `open'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/2.5.0/open-uri.rb:35:in `open'
[2019-01-17T10:02:07+00:00] ERROR: /root/.chefdk/gem/ruby/2.5.0/gems/inspec-3.0.52/lib/fetchers/url.rb:214:in `open_via_uri'
[2019-01-17T10:02:07+00:00] ERROR: /root/.chefdk/gem/ruby/2.5.0/gems/inspec-3.0.52/lib/fetchers/url.rb:196:in `download_archive_to_temp'
[2019-01-17T10:02:07+00:00] ERROR: /root/.chefdk/gem/ruby/2.5.0/gems/inspec-3.0.52/lib/fetchers/url.rb:150:in `temp_archive_path'
[2019-01-17T10:02:07+00:00] ERROR: /root/.chefdk/gem/ruby/2.5.0/gems/inspec-3.0.52/lib/fetchers/url.rb:130:in `sha256'
[2019-01-17T10:02:07+00:00] ERROR: /root/.chefdk/gem/ruby/2.5.0/gems/inspec-3.0.52/lib/plugins/inspec-compliance/lib/inspec-compliance/target.rb:29:in `sha256'
[2019-01-17T10:02:07+00:00] ERROR: /root/.chefdk/gem/ruby/2.5.0/gems/inspec-3.0.52/lib/fetchers/url.rb:115:in `cache_key'
[2019-01-17T10:02:07+00:00] ERROR: /root/.chefdk/gem/ruby/2.5.0/gems/inspec-3.0.52/lib/inspec/cached_fetcher.rb:32:in `cache_key'
[2019-01-17T10:02:07+00:00] ERROR: /root/.chefdk/gem/ruby/2.5.0/gems/inspec-3.0.52/lib/inspec/cached_fetcher.rb:39:in `fetch'
[2019-01-17T10:02:07+00:00] ERROR: /root/.chefdk/gem/ruby/2.5.0/gems/inspec-3.0.52/lib/inspec/profile.rb:72:in `for_fetcher'
[2019-01-17T10:02:07+00:00] ERROR: /root/.chefdk/gem/ruby/2.5.0/gems/inspec-3.0.52/lib/inspec/profile.rb:79:in `for_target'
[2019-01-17T10:02:07+00:00] ERROR: /root/.chefdk/gem/ruby/2.5.0/gems/inspec-3.0.52/lib/inspec/runner.rb:197:in `add_target'
[2019-01-17T10:02:07+00:00] ERROR: /var/chef/cache/cookbooks/audit/files/default/handler/audit_report.rb:170:in `block in call'
[2019-01-17T10:02:07+00:00] ERROR: /var/chef/cache/cookbooks/audit/files/default/handler/audit_report.rb:170:in `each'
[2019-01-17T10:02:07+00:00] ERROR: /var/chef/cache/cookbooks/audit/files/default/handler/audit_report.rb:170:in `call'
[2019-01-17T10:02:07+00:00] ERROR: /var/chef/cache/cookbooks/audit/files/default/handler/audit_report.rb:81:in `report'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/handler.rb:257:in `run_report_unsafe'
[2019-01-17T10:02:07+00:00] ERROR: /var/chef/cache/cookbooks/audit/files/default/handler/audit_report.rb:99:in `run_report_safely'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/handler.rb:123:in `block in run_report_handlers'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/handler.rb:121:in `each'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/handler.rb:121:in `run_report_handlers'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/handler.rb:133:in `block in <class:Handler>'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/client.rb:445:in `block in run_completed_successfully'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/client.rb:444:in `each'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/client.rb:444:in `run_completed_successfully'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/client.rb:301:in `run'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/application.rb:303:in `run_with_graceful_exit_option'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/application.rb:279:in `block in run_chef_client'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/local_mode.rb:44:in `with_server_connectivity'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/application.rb:261:in `run_chef_client'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/application/client.rb:440:in `run_application'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/application.rb:66:in `run'
[2019-01-17T10:02:07+00:00] ERROR: /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/bin/chef-client:25:in `<top (required)>'
[2019-01-17T10:02:07+00:00] ERROR: /bin/chef-client:296:in `load'
[2019-01-17T10:02:07+00:00] ERROR: /bin/chef-client:296:in `<main>'
[2019-01-17T10:02:07+00:00] INFO: Report handlers complete
----------------------------------------------------

Any help would be greatly appreciated.

[welcomebot]

Hey There
It looks like this is the first issue you've filed against the chef-cookbooks project. I'm here to offer you a bit of extra help to make sure we can quickly get back to you.
Make sure you've filled out all the fields in our issue template. Make sure you've provided us with the version of chef-client you're running, your operating system and the version of the cookbook. If you're not using the most up to date version of the cookbook then please make sure to update first. Lots of things change between versions even if you're issue isn't listed in the changelog. Finally please give us a detailed description of the issue you're having. The more we know about what you're trying to do, what actually happens, and how you can reproduce the problem, the better.

If you're looking for more immediate troubleshooting help make sure to check out #general on the Chef Community Slack. There's plenty of folks there willing to lend a helping hand. Thanks for the first issue. We hope we can get back to you soon with a solution.

[kvivek1115]

@sahilsehgal81

1. In the first scenario where you set up your profiles, there is no need to change default['audit']['attributes'] = {} keep it [] and use the array of hashes e.g. .push to set up your profiles(until fixes provided).

2. In order to resolve Report handler Chef::Handler::AuditReport raised #<Net::HTTPFatalError: 503 "Service Unavailable">
   Update inspec version from default to latest in attributes/default.rb.
   e.g.
   default['audit']['inspec_version'] = 'latest'

hope it will address your issue?

[wagostin]

I have the exact same issue. This is the relevant part of my default attributes file.

default['audit']['profiles'].push(
'name': 'cis-windows2016rtm-release1607-level1-memberserver',
'compliance': 'admin/cis-windows2016rtm-release1607-level1-memberserver'
)

What am I doing wrong?

[zenspider]

This feels like a support issue. Assigning to keka