Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Gangams/multi tenancy add advanced_mode_enabled option in the configmap #1328

Merged
merged 11 commits into from
Oct 7, 2024
Merged

Gangams/multi tenancy add advanced_mode_enabled option in the configmap #1328

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
bdca65a
Contianer insights 3.1.24 release charts update and release notes
wanlonghenry Sep 27, 2024
3780c5f
Merge branch 'ci_prod' into longw/3.1.24
ganga1980 Oct 2, 2024
ab69bce
add azmon multi-tenancy advanced mode option
ganga1980 Oct 3, 2024
27149d8
revert release related updates
ganga1980 Oct 3, 2024
028462c
revert release related changes
ganga1980 Oct 3, 2024
a8c6311
improve comments
ganga1980 Oct 4, 2024
a639733
improve logging
ganga1980 Oct 4, 2024
a15bfac
Merge branch 'ci_prod' into gangams/multi-tenancy-and-release-updates
ganga1980 Oct 4, 2024
6766cb8
get multitenancy config only in msi mode
ganga1980 Oct 6, 2024
cf28311
comment multi-tenancy yaml
ganga1980 Oct 6, 2024
4de2f5e
cleanup unnecessary logging
ganga1980 Oct 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
158 changes: 88 additions & 70 deletions build/common/installer/scripts/tomlparser.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
@annotationBasedLogFiltering = false
@allowed_system_namespaces = ['kube-system', 'gatekeeper-system', 'calico-system', 'azure-arc', 'kube-public', 'kube-node-lease']
@isAzMonMultiTenancyLogCollectionEnabled = false
@isAzMonMultiTenancyLogCollectionAdvancedMode = false
@azMonMultiTenantNamespaces = []
@azMonMultiTenancyMaxStorageChunksUp = 500
@namespace_to_settings = {}
Expand Down Expand Up @@ -462,81 +463,95 @@ def populateSettingValuesFromConfigMap(parsedConfig)

#Get Multi-tenancy log collection settings
begin
if !parsedConfig[:log_collection_settings][:multi_tenancy].nil? && !parsedConfig[:log_collection_settings][:multi_tenancy][:enabled].nil?
multi_tenancy_enabled = parsedConfig[:log_collection_settings][:multi_tenancy][:enabled]
if multi_tenancy_enabled
@isAzMonMultiTenancyLogCollectionEnabled = multi_tenancy_enabled
namespaces = parsedConfig[:log_collection_settings][:multi_tenancy][:namespaces]
puts "config::INFO:multi_tenancy namespaces provided in the configmap: #{namespaces}"
if !namespaces.nil? && !namespaces.empty? &&
namespaces.kind_of?(Array) && namespaces.length > 0 &&
namespaces[0].kind_of?(String) # Checking only for the first element to be string because toml enforces the arrays to contain elements of same type
@azMonMultiTenantNamespaces = namespaces.map(&:strip).map(&:downcase).uniq
end
puts "config::INFO:multi_tenancy unique namespaces provided: #{@azMonMultiTenantNamespaces}"
# max storage chunks
storage_max_chunks_up = parsedConfig[:log_collection_settings][:multi_tenancy][:storage_max_chunks_up]
if is_valid_number?(storage_max_chunks_up)
@azMonMultiTenancyMaxStorageChunksUp = storage_max_chunks_up.to_i
end
puts "config::INFO:multi_tenancy storage_max_chunks_up: #{@azMonMultiTenancyMaxStorageChunksUp}"

# default settings
storage_type = parsedConfig[:log_collection_settings][:multi_tenancy][:storage_type]
updateDefaultConfigSetting("storage_type", storage_type)
mem_buf_limit = parsedConfig[:log_collection_settings][:multi_tenancy][:mem_buf_limit]
updateDefaultConfigSetting("mem_buf_limit", mem_buf_limit)
buffer_chunk_size = parsedConfig[:log_collection_settings][:multi_tenancy][:buffer_chunk_size]
updateDefaultConfigSetting("buffer_chunk_size", buffer_chunk_size)
buffer_max_size = parsedConfig[:log_collection_settings][:multi_tenancy][:buffer_max_size]
updateDefaultConfigSetting("buffer_max_size", buffer_max_size)
throttle_rate = parsedConfig[:log_collection_settings][:multi_tenancy][:throttle_rate]
updateDefaultConfigSetting("throttle_rate", throttle_rate)
throttle_window = parsedConfig[:log_collection_settings][:multi_tenancy][:throttle_window]
updateDefaultConfigSetting("throttle_window", throttle_window)
out_forward_worker_count = parsedConfig[:log_collection_settings][:multi_tenancy][:out_forward_worker_count]
updateDefaultConfigSetting("out_forward_worker_count", out_forward_worker_count)
out_forward_retry_limit = parsedConfig[:log_collection_settings][:multi_tenancy][:out_forward_retry_limit]
updateDefaultConfigSetting("out_forward_retry_limit", out_forward_retry_limit)
out_forward_storage_total_limit_size = parsedConfig[:log_collection_settings][:multi_tenancy][:out_forward_storage_total_limit_size]
updateDefaultConfigSetting("out_forward_storage_total_limit_size", out_forward_storage_total_limit_size)
out_forward_require_ack_response = parsedConfig[:log_collection_settings][:multi_tenancy][:out_forward_require_ack_response]
updateDefaultConfigSetting("out_forward_require_ack_response", out_forward_require_ack_response)
disable_throttle = parsedConfig[:log_collection_settings][:multi_tenancy][:disable_throttle]
updateDefaultConfigSetting("disable_throttle", disable_throttle)

# namepsace to settings
namespace_settings = parsedConfig[:log_collection_settings][:multi_tenancy][:namespace_settings]
puts "config::INFO:multi_tenancy namespace_settings: #{namespace_settings}"
if @azMonMultiTenantNamespaces.length > 0
if !namespace_settings.nil? && !namespace_settings.empty? &&
namespace_settings.kind_of?(Array) && namespace_settings.length > 0 &&
namespace_settings[0].kind_of?(String) # Checking only for the first element to be string because toml enforces the arrays to contain elements of same type
namespace_settings.each do |entry|
namespace, settings_str = entry.split(':')
settings = settings_str.split(';').map { |s| s.split('=') }.to_h
namespace = namespace.strip.downcase
if @namespace_to_settings.key?(namespace)
puts "config::WARN: Duplicate namespace settings found for namespace: #{namespace}, using the previous settings"
else
@namespace_to_settings[namespace] = settings
if !parsedConfig[:log_collection_settings][:multi_tenancy].nil?
if !parsedConfig[:log_collection_settings][:multi_tenancy][:enabled].nil?
multi_tenancy_enabled = parsedConfig[:log_collection_settings][:multi_tenancy][:enabled]
if multi_tenancy_enabled
@isAzMonMultiTenancyLogCollectionEnabled = multi_tenancy_enabled
# multi-tenancy advanced mode
if !parsedConfig[:log_collection_settings][:multi_tenancy][:advanced_mode_enabled].nil?
advanced_mode_enabled = parsedConfig[:log_collection_settings][:multi_tenancy][:advanced_mode_enabled]
end

if advanced_mode_enabled
@isAzMonMultiTenancyLogCollectionAdvancedMode = advanced_mode_enabled
namespaces = parsedConfig[:log_collection_settings][:multi_tenancy][:namespaces]
puts "config::INFO:multi_tenancy namespaces provided in the configmap: #{namespaces}"
if !namespaces.nil? && !namespaces.empty? &&
namespaces.kind_of?(Array) && namespaces.length > 0 &&
namespaces[0].kind_of?(String) # Checking only for the first element to be string because toml enforces the arrays to contain elements of same type
@azMonMultiTenantNamespaces = namespaces.map(&:strip).map(&:downcase).uniq
end
puts "config::INFO:multi_tenancy unique namespaces provided: #{@azMonMultiTenantNamespaces}"
if @azMonMultiTenantNamespaces.length > 0
# max storage chunks
storage_max_chunks_up = parsedConfig[:log_collection_settings][:multi_tenancy][:storage_max_chunks_up]
if is_valid_number?(storage_max_chunks_up)
@azMonMultiTenancyMaxStorageChunksUp = storage_max_chunks_up.to_i
end
puts "config::INFO:multi_tenancy storage_max_chunks_up: #{@azMonMultiTenancyMaxStorageChunksUp}"

# default settings
storage_type = parsedConfig[:log_collection_settings][:multi_tenancy][:storage_type]
updateDefaultConfigSetting("storage_type", storage_type)
mem_buf_limit = parsedConfig[:log_collection_settings][:multi_tenancy][:mem_buf_limit]
updateDefaultConfigSetting("mem_buf_limit", mem_buf_limit)
buffer_chunk_size = parsedConfig[:log_collection_settings][:multi_tenancy][:buffer_chunk_size]
updateDefaultConfigSetting("buffer_chunk_size", buffer_chunk_size)
buffer_max_size = parsedConfig[:log_collection_settings][:multi_tenancy][:buffer_max_size]
updateDefaultConfigSetting("buffer_max_size", buffer_max_size)
throttle_rate = parsedConfig[:log_collection_settings][:multi_tenancy][:throttle_rate]
updateDefaultConfigSetting("throttle_rate", throttle_rate)
throttle_window = parsedConfig[:log_collection_settings][:multi_tenancy][:throttle_window]
updateDefaultConfigSetting("throttle_window", throttle_window)
out_forward_worker_count = parsedConfig[:log_collection_settings][:multi_tenancy][:out_forward_worker_count]
updateDefaultConfigSetting("out_forward_worker_count", out_forward_worker_count)
out_forward_retry_limit = parsedConfig[:log_collection_settings][:multi_tenancy][:out_forward_retry_limit]
updateDefaultConfigSetting("out_forward_retry_limit", out_forward_retry_limit)
out_forward_storage_total_limit_size = parsedConfig[:log_collection_settings][:multi_tenancy][:out_forward_storage_total_limit_size]
updateDefaultConfigSetting("out_forward_storage_total_limit_size", out_forward_storage_total_limit_size)
out_forward_require_ack_response = parsedConfig[:log_collection_settings][:multi_tenancy][:out_forward_require_ack_response]
updateDefaultConfigSetting("out_forward_require_ack_response", out_forward_require_ack_response)
disable_throttle = parsedConfig[:log_collection_settings][:multi_tenancy][:disable_throttle]
updateDefaultConfigSetting("disable_throttle", disable_throttle)

# namepsace to settings
namespace_settings = parsedConfig[:log_collection_settings][:multi_tenancy][:namespace_settings]
puts "config::INFO:multi_tenancy namespace_settings: #{namespace_settings}"
if !namespace_settings.nil? && !namespace_settings.empty? &&
namespace_settings.kind_of?(Array) && namespace_settings.length > 0 &&
namespace_settings[0].kind_of?(String) # Checking only for the first element to be string because toml enforces the arrays to contain elements of same type
namespace_settings.each do |entry|
namespace, settings_str = entry.split(':')
settings = settings_str.split(';').map { |s| s.split('=') }.to_h
namespace = namespace.strip.downcase
if @namespace_to_settings.key?(namespace)
puts "config::WARN: Duplicate namespace settings found for namespace: #{namespace}, using the previous settings"
else
@namespace_to_settings[namespace] = settings
end
end
end
generateAzMonMultiTenantNamespaceConfig()


# azmon multi-tenancy service buffer settings
service_buffer_chunk_size = parsedConfig[:log_collection_settings][:multi_tenancy][:service_buffer_chunk_size]
if !service_buffer_chunk_size.nil? && !service_buffer_chunk_size.empty?
@azMonMultiTenancyServiceBufferChunkSize = service_buffer_chunk_size
end
service_buffer_max_size = parsedConfig[:log_collection_settings][:multi_tenancy][:service_buffer_max_size]
if !service_buffer_max_size.nil? && !service_buffer_max_size.empty?
@azMonMultiTenancyServiceBufferMaxSize = service_buffer_max_size
end
else
puts "config::WARN: Multi-tenancy log collection is enabled with advanced_mode_enabled but no namespaces provided, disabling advanced_mode_enabled"
@isAzMonMultiTenancyLogCollectionAdvancedMode = false
end
end
generateAzMonMultiTenantNamespaceConfig()
end

# azmon multi-tenancy service buffer settings
service_buffer_chunk_size = parsedConfig[:log_collection_settings][:multi_tenancy][:service_buffer_chunk_size]
if !service_buffer_chunk_size.nil? && !service_buffer_chunk_size.empty?
@azMonMultiTenancyServiceBufferChunkSize = service_buffer_chunk_size
end
service_buffer_max_size = parsedConfig[:log_collection_settings][:multi_tenancy][:service_buffer_max_size]
if !service_buffer_max_size.nil? && !service_buffer_max_size.empty?
@azMonMultiTenancyServiceBufferMaxSize = service_buffer_max_size
end
puts "config::INFO: Using config map setting multi_tenancy enabled: #{@isAzMonMultiTenancyLogCollectionEnabled}, advanced_mode_enabled: #{@isAzMonMultiTenancyLogCollectionAdvancedMode} and namespaces: #{@azMonMultiTenantNamespaces} for Multi-tenancy log collection"
end
puts "config::INFO: Using config map setting enabled: #{@isAzMonMultiTenancyLogCollectionEnabled} and namespaces: #{@azMonMultiTenantNamespaces} for Multi-tenancy log collection"
end
rescue => errorStr
ConfigParseErrorLogger.logError("config::error: Exception while reading config map settings for Multi-tenancy log collection - #{errorStr}, please check config map for errors")
Expand Down Expand Up @@ -595,6 +610,7 @@ def populateSettingValuesFromConfigMap(parsedConfig)
file.write("export AZMON_ANNOTATION_BASED_LOG_FILTERING=#{@annotationBasedLogFiltering}\n")
if @isAzMonMultiTenancyLogCollectionEnabled
file.write("export AZMON_MULTI_TENANCY_LOG_COLLECTION=#{@isAzMonMultiTenancyLogCollectionEnabled}\n")
file.write("export AZMON_MULTI_TENANCY_LOG_COLLECTION_ADVANCED_MODE=#{@isAzMonMultiTenancyLogCollectionAdvancedMode}\n")
azMonMultiTenantNamespacesString = @azMonMultiTenantNamespaces.join(",")
file.write("export AZMON_MULTI_TENANCY_NAMESPACES=#{azMonMultiTenantNamespacesString}\n")
file.write("export AZMON_MULTI_TENANCY_STORAGE_MAX_CHUNKS_UP=#{@azMonMultiTenancyMaxStorageChunksUp}\n")
Expand Down Expand Up @@ -679,6 +695,8 @@ def get_command_windows(env_variable_name, env_variable_value)
if @isAzMonMultiTenancyLogCollectionEnabled
commands = get_command_windows("AZMON_MULTI_TENANCY_LOG_COLLECTION", @isAzMonMultiTenancyLogCollectionEnabled)
file.write(commands)
commands = get_command_windows("AZMON_MULTI_TENANCY_LOG_COLLECTION_ADVANCED_MODE", @isAzMonMultiTenancyLogCollectionAdvancedMode)
file.write(commands)
azMonMultiTenantNamespacesString = @azMonMultiTenantNamespaces.join(",")
commands = get_command_windows("AZMON_MULTI_TENANCY_NAMESPACES", azMonMultiTenantNamespacesString)
file.write(commands)
Expand Down
Loading
Loading