WIP SOC8 support #112

bmwiedemann · 2018-09-22T05:57:00Z

No description provided.

houndci-bot · 2018-09-22T05:57:11Z

chef/cookbooks/ceph/recipes/mon.rb

+  subscribes :restart, resources(template: "/etc/ceph/ceph.conf")
+end
+
+


Layout/EmptyLines: Extra blank line detected. (https://github.com/bbatsov/ruby-style-guide#two-or-more-empty-lines)

houndci-bot · 2018-09-22T05:57:11Z

chef/cookbooks/ceph/recipes/mon.rb

+unless File.exist?("/var/lib/ceph/mgr/ceph-#{mon_name}/done")
+  keyring = "#{Chef::Config[:file_cache_path]}/#{cluster}-#{mon_name}.mgr.keyring"
+  execute "create mgr keyring" do
+    command "ceph-authtool '#{keyring}' --create-keyring --name=mgr. --add-key='#{node["ceph"]["monitor-secret"]}' --cap mgr 'allow *'"


Metrics/LineLength: Line is too long. [135/100] (https://github.com/SUSE/style-guides/blob/master/Ruby.md#metricslinelength)

houndci-bot · 2018-09-22T05:57:11Z

chef/cookbooks/ceph/recipes/mon.rb

 # TODO cluster name
 cluster = "ceph"

+


Layout/EmptyLines: Extra blank line detected. (https://github.com/bbatsov/ruby-style-guide#two-or-more-empty-lines)

houndci-bot · 2018-09-22T05:57:11Z

chef/cookbooks/ceph/recipes/mon.rb

+  action :create
+end
+
+


Layout/EmptyLines: Extra blank line detected. (https://github.com/bbatsov/ruby-style-guide#two-or-more-empty-lines)

rsalevsky · 2018-09-22T22:08:22Z

Please note master is in this case Cloud 8 and we actually disabled support for it.

tserong

Aside from the keyring change, this looks like it will work to deploy ceph-mgr.

Note though that SES5 doesn't include Calamari, so if SOC8 ends up supporting deploying SES5, the Calamari role will need to be removed. This leaves the Ceph cluster without a GUI, unless we were to add an openATTIC role, to deploy openATTIC instead, but that gets more complicated, as parts of openATTIC rely on DeepSea, which won't be present if SES is deployed via Crowbar.

tserong · 2018-09-27T09:36:16Z

chef/cookbooks/ceph/recipes/mon.rb

+unless File.exist?("/var/lib/ceph/mgr/ceph-#{mon_name}/done")
+  keyring = "#{Chef::Config[:file_cache_path]}/#{cluster}-#{mon_name}.mgr.keyring"
+  execute "create mgr keyring" do
+    command "ceph-authtool '#{keyring}' --create-keyring --name=mgr. --add-key='#{node["ceph"]["monitor-secret"]}' --cap mgr 'allow *'"


The mgr keyring needs to be written to /var/lib/ceph/mgr/ceph-#{mon_name}/keyring and needs caps mon 'allow profile mgr' osd 'allow *' mds 'allow *', something like the "create mds keyring" block in chef/cookbooks/ceph/recipes/mds.rb.

bmwiedemann · 2018-10-01T13:53:34Z

Added that, but somehow there is no admin keyring.
Maybe something else needs to be done first.
Addding include_recipe "ceph::keyring" also does not help

ceph auth get-or-create mgr.public.d52-54-77-77-01-03              mon 'allow profile mgr' osd 'allow *' mds 'allow *'              -o /var/lib/ceph/mgr/ceph-public.d52-54-77-77-01-03/keyring
2018-10-01 13:44:28.538622 7f52022dc700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2018-10-01 13:44:28.538650 7f52022dc700 -1 monclient: ERROR: missing keyring, cannot use cephx for authentication
2018-10-01 13:44:28.538653 7f52022dc700  0 librados: client.admin initialization error (2) No such file or directory
[errno 2] error connecting to the cluster

tserong · 2018-10-03T09:44:00Z

Try moving all the mgr creation stuff down to the end of the file; ceph.client.admin.keyring isn't created yet until about line 200.

houndci-bot · 2018-10-04T11:40:23Z

chef/cookbooks/ceph/recipes/radosgw_keystone.rb

@@ -36,7 +37,7 @@
  port keystone_settings["admin_port"]
  auth register_auth_hash
  user_name keystone_settings["service_user"]
-  tenant_name keystone_settings["service_tenant"]
+  project_name keystone_settings['service_tenant']


Style/StringLiterals: Prefer double-quoted strings unless you need single quotes to avoid extra backslashes for escaping. (https://github.com/SUSE/style-guides/blob/master/Ruby.md#stylestringliterals)

houndci-bot · 2018-10-04T11:40:23Z

chef/cookbooks/ceph/recipes/radosgw_keystone.rb

@@ -25,7 +26,7 @@
  auth register_auth_hash
  user_name keystone_settings["service_user"]
  user_password keystone_settings["service_password"]
-  tenant_name keystone_settings["service_tenant"]
+  project_name keystone_settings['service_tenant']


Style/StringLiterals: Prefer double-quoted strings unless you need single quotes to avoid extra backslashes for escaping. (https://github.com/SUSE/style-guides/blob/master/Ruby.md#stylestringliterals)

Needed after crowbar-openstack commit 8774f1a509 Use keystone v3 in config auth urls (SCRD-781)

SES5 only supports this

houndci-bot · 2018-10-04T12:25:06Z

chef/cookbooks/ceph/recipes/mon.rb

@@ -187,3 +200,37 @@
    end
  end
 end
+
+


Layout/EmptyLines: Extra blank line detected. (https://github.com/bbatsov/ruby-style-guide#two-or-more-empty-lines)

bmwiedemann · 2018-10-04T12:46:15Z

got a bit further now. I was able to adapt radosgw recipes to keystonev3
I never used a ceph UI so far, so I guess, dropping calamares can be OK for now.

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of chown ceph:ceph /var/chef/cache/ceph-public.d52-54-77-77-01-01.mon.keyring ; ceph-mon --mkfs -i public.d52-54-77-77-01-01 --keyring '/var/chef/cache/ceph-public.d52-54-77-77-01-01.mon.keyring' --setuser ceph --setgroup ceph ----
STDOUT: 
STDERR: chown: cannot access '/var/chef/cache/ceph-public.d52-54-77-77-01-01.mon.keyring': No such file or directory
2018-10-04 12:24:53.035565 7f0168094f40 -1 mon.public.d52-54-77-77-01-01@-1(probing) e0 unable to find a keyring file on /var/chef/cache/ceph-public.d52-54-77-77-01-01.mon.keyring: (2) No such file or directory
2018-10-04 12:24:53.035590 7f0168094f40 -1 ceph-mon: error creating monfs: (2) No such file or directory
---- End output of chown ceph:ceph /var/chef/cache/ceph-public.d52-54-77-77-01-01.mon.keyring ; ceph-mon --mkfs -i public.d52-54-77-77-01-01 --keyring '/var/chef/cache/ceph-public.d52-54-77-77-01-01.mon.keyring' --setuser ceph --setgroup ceph ----
Ran chown ceph:ceph /var/chef/cache/ceph-public.d52-54-77-77-01-01.mon.keyring ; ceph-mon --mkfs -i public.d52-54-77-77-01-01 --keyring '/var/chef/cache/ceph-public.d52-54-77-77-01-01.mon.keyring' --setuser ceph --setgroup ceph returned 1



Resource Declaration:
---------------------
# In /var/chef/cache/cookbooks/ceph/recipes/mon.rb


109:   execute "ceph-mon mkfs" do
110:     command "chown ceph:ceph #{keyring} ; ceph-mon --mkfs -i #{mon_name} " \
111:             "--keyring '#{keyring}' --setuser ceph --setgroup ceph"
112:     action :nothing
113:   end
114: 


Compiled Resource:
------------------
# Declared in /var/chef/cache/cookbooks/ceph/recipes/mon.rb:109:in `from_file'


execute("ceph-mon mkfs") do
  action [:nothing]
  retries 0
  retry_delay 2
  command "chown ceph:ceph /var/chef/cache/ceph-public.d52-54-77-77-01-01.mon.keyring ; ceph-mon --mkfs -i public.d52-54-77-77-01-01 --keyring '/var/chef/cache/ceph-public.d52-54-77-77-01-01.mon.keyring' --setuser ceph --setgroup ceph"
  backup 5
  returns 0
  cookbook_name "ceph"
  recipe_name "mon"
end


[2018-10-04T12:24:53+00:00] ERROR: Running exception handlers
[2018-10-04T12:24:53+00:00] FATAL: Saving node information to /var/chef/cache/failed-run-data.json
[2018-10-04T12:24:53+00:00] ERROR: Exception handlers complete
[2018-10-04T12:24:53+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2018-10-04T12:24:53+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: execute[ceph-mon mkfs] (ceph::mon line 109) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of chown ceph:ceph /var/chef/cache/ceph-public.d52-54-77-77-01-01.mon.keyring ; ceph-mon --mkfs -i public.d52-54-77-77-01-01 --keyring '/var/chef/cache/ceph-public.d52-54-77-77-01-01.mon.keyring' --setuser ceph --setgroup ceph ----
STDOUT: 
STDERR: chown: cannot access '/var/chef/cache/ceph-public.d52-54-77-77-01-01.mon.keyring': No such file or directory
2018-10-04 12:24:53.035565 7f0168094f40 -1 mon.public.d52-54-77-77-01-01@-1(probing) e0 unable to find a keyring file on /var/chef/cache/ceph-public.d52-54-77-77-01-01.mon.keyring: (2) No such file or directory
2018-10-04 12:24:53.035590 7f0168094f40 -1 ceph-mon: error creating monfs: (2) No such file or directory
---- End output of chown ceph:ceph /var/chef/cache/ceph-public.d52-54-77-77-01-01.mon.keyring ; ceph-mon --mkfs -i public.d52-54-77-77-01-01 --keyring '/var/chef/cache/ceph-public.d52-54-77-77-01-01.mon.keyring' --setuser ceph --setgroup ceph ----
Ran chown ceph:ceph /var/chef/cache/ceph-public.d52-54-77-77-01-01.mon.keyring ; ceph-mon --mkfs -i public.d52-54-77-77-01-01 --keyring '/var/chef/cache/ceph-public.d52-54-77-77-01-01.mon.keyring' --setuser ceph --setgroup ceph returned 1

tserong · 2018-10-05T12:51:08Z

OK, that's weird. You didn't change anything related to mon keyring creation AFAICT, so why are those keyring files inaccessible?

bmwiedemann · 2018-10-05T13:46:22Z

maybe the added package lines did something unexpected or something between SES4 and 5 packaging changed.

tserong · 2018-10-08T08:05:49Z

I don't think this failure can be due to packaging changes -- that keyring file is created by the ceph barclamp during initial mon creation:

crowbar-ceph/chef/cookbooks/ceph/recipes/mon.rb

Lines 46 to 122 in 0a96245

    
           unless File.exist?("/var/lib/ceph/mon/ceph-#{mon_name}/done") 
        
             keyring = "#{Chef::Config[:file_cache_path]}/#{cluster}-#{mon_name}.mon.keyring" 
        
             execute "create monitor keyring" do 
        
               command "ceph-authtool '#{keyring}' --create-keyring --name=mon. --add-key='#{node["ceph"]["monitor-secret"]}' --cap mon 'allow *'" 
        
               not_if { node["ceph"]["monitor-secret"].empty? } 
        
               notifies :run, "execute[ceph-mon mkfs]", :immediately 
        
             end 
        
             ruby_block "generate monitor-secret" do 
        
               block do 
        
                 gen_key = Mixlib::ShellOut.new("ceph-authtool --gen-print-key") 
        
                 monitor_key = gen_key.run_command.stdout.strip 
        
                 gen_key.error! 
        
                 add_key = Mixlib::ShellOut.new("ceph-authtool '#{keyring}' --create-keyring --name=mon. --add-key='#{monitor_key}' --cap mon 'allow *'") 
        
                 add_key.run_command 
        
                 add_key.error! 
        
                 # no need to check if the attribute is already set: it's part of the 
        
                 # only_if 
        
                 node.set["ceph"]["monitor-secret"] = monitor_key 
        
                 node.save 
        
               end 
        
               only_if { node["ceph"]["monitor-secret"].empty? && node[:ceph][:master] } 
        
               notifies :run, "execute[ceph-mon mkfs]", :immediately 
        
             end 
        
             ruby_block "get monitor-secret" do 
        
               block do 
        
                 require "timeout" 
        
                 monitor_key = "" 
        
                 begin 
        
                   Timeout.timeout(600) do 
        
                     while monitor_key.empty? 
        
                       mon_nodes = get_mon_nodes 
        
                       mon_nodes.each do |mon| 
        
                         if mon[:ceph][:master] && !mon["ceph"]["monitor-secret"].empty? 
        
                           monitor_key = mon["ceph"]["monitor-secret"] 
        
                         end 
        
                       end 
        
                       sleep 1 
        
                     end 
        
                   end 
        
                 rescue Timeout::Error 
        
                   message = "Cannot fetch monitor secret from master!" 
        
                   Chef::Log.fatal(message) 
        
                   raise message 
        
                 end 
        
                 add_key = Mixlib::ShellOut.new("ceph-authtool '#{keyring}' --create-keyring --name=mon. --add-key='#{monitor_key}' --cap mon 'allow *'") 
        
                 add_key.run_command 
        
                 add_key.error! 
        
                 # no need to check if the attribute is already set: it's part of the 
        
                 # only_if 
        
                 node.set["ceph"]["monitor-secret"] = monitor_key 
        
                 node.save 
        
               end 
        
               only_if { node["ceph"]["monitor-secret"].empty? && !node[:ceph][:master] } 
        
               notifies :run, "execute[ceph-mon mkfs]", :immediately 
        
             end 
        
             execute "ceph-mon mkfs" do 
        
               command "chown ceph:ceph #{keyring} ; ceph-mon --mkfs -i #{mon_name} " \ 
        
                       "--keyring '#{keyring}' --setuser ceph --setgroup ceph" 
        
               action :nothing 
        
             end 
        
             ruby_block "finalise" do 
        
               block do 
        
                 ["done", service_type].each do |ack| 
        
                   File.open("/var/lib/ceph/mon/ceph-#{mon_name}/#{ack}", "w").close 
        
                 end 
        
               end 
        
             end 
        
           end

It should have been created by one of the three ceph-authtool invocations. Is there any indication earlier in the logs of ceph-authtool failing somehow?

Deploy it after the admin keyring creation.

jsuchome · 2018-11-05T10:26:38Z

Hi @bmwiedemann, any update on this one?

bmwiedemann · 2018-11-05T12:30:31Z

No, I was busy working on other parts and dont know how to continue here. Feel free to play around and improve this PR.

bmwiedemann · 2018-11-20T15:56:33Z

ceph playground is open again: http://crowbar.vi16.cloud.suse.de/crowbar/ceph/1.0/proposals/default

bmwiedemann added the help wanted label Sep 22, 2018

bmwiedemann requested a review from tserong September 22, 2018 05:57

houndci-bot reviewed Sep 22, 2018

View reviewed changes

rsalevsky added the do not merge yet label Sep 22, 2018

tserong suggested changes Sep 27, 2018

View reviewed changes

bmwiedemann force-pushed the soc8 branch from 32c25e2 to 287e935 Compare October 4, 2018 11:39

houndci-bot reviewed Oct 4, 2018

View reviewed changes

bmwiedemann force-pushed the soc8 branch from 287e935 to 2960dcd Compare October 4, 2018 11:43

bmwiedemann added 2 commits October 4, 2018 14:24

Adapt radosgw to keystone v3

e62ea4f

Needed after crowbar-openstack commit 8774f1a509 Use keystone v3 in config auth urls (SCRD-781)

Update to SLE-12-SP3

be58b46

SES5 only supports this

bmwiedemann force-pushed the soc8 branch from 2960dcd to 0a96245 Compare October 4, 2018 12:24

houndci-bot reviewed Oct 4, 2018

View reviewed changes

dirkmueller requested a review from rsalevsky October 18, 2018 16:18

dirkmueller force-pushed the soc8 branch from 0a96245 to c32174a Compare October 18, 2018 16:39

ceph: deploy ceph-mgr

b965ec1

Deploy it after the admin keyring creation.

dirkmueller force-pushed the soc8 branch from c32174a to b965ec1 Compare October 29, 2018 10:25

jsuchome mentioned this pull request Nov 29, 2018

[5.0] Adapt radosgw to keystone v3 #113

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP SOC8 support #112

WIP SOC8 support #112

bmwiedemann commented Sep 22, 2018

houndci-bot Sep 22, 2018

houndci-bot Sep 22, 2018

houndci-bot Sep 22, 2018

houndci-bot Sep 22, 2018

rsalevsky commented Sep 22, 2018

tserong left a comment

tserong Sep 27, 2018

bmwiedemann commented Oct 1, 2018

tserong commented Oct 3, 2018 •

edited

Loading

houndci-bot Oct 4, 2018

houndci-bot Oct 4, 2018

houndci-bot Oct 4, 2018

bmwiedemann commented Oct 4, 2018

tserong commented Oct 5, 2018

bmwiedemann commented Oct 5, 2018

tserong commented Oct 8, 2018 •

edited

Loading

jsuchome commented Nov 5, 2018

bmwiedemann commented Nov 5, 2018

bmwiedemann commented Nov 20, 2018

		subscribes :restart, resources(template: "/etc/ceph/ceph.conf")
		end

@@ @@ -187,3 +200,37 @@ @@
                   end
                 end
               end

WIP SOC8 support #112

Are you sure you want to change the base?

WIP SOC8 support #112

Conversation

bmwiedemann commented Sep 22, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rsalevsky commented Sep 22, 2018

tserong left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bmwiedemann commented Oct 1, 2018

tserong commented Oct 3, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bmwiedemann commented Oct 4, 2018

tserong commented Oct 5, 2018

bmwiedemann commented Oct 5, 2018

tserong commented Oct 8, 2018 • edited Loading

jsuchome commented Nov 5, 2018

bmwiedemann commented Nov 5, 2018

bmwiedemann commented Nov 20, 2018

tserong commented Oct 3, 2018 •

edited

Loading

tserong commented Oct 8, 2018 •

edited

Loading