[BUG] censys.rb "key not found" Error

[jonathanbrooke]

Describe the bug

I have seen historical bugs here mentioning Censys key errors, which have been fixed in the past.

A rule that has been untouched for months has just stopped working, specifically some of the censys rules.
Commenting out some of the censys rules within the Mihari rule removes the error.
Double-Checked the searches on the Censys platform and they return results correctly.

Broken Rules that cause the stack trace further below:

- analyzer: censys
  query: 'services.software.product="Cobalt Strike"'
- analyzer: censys
  query: 'services.cobalt_strike.x64.post_ex.x86:"exe"'

Stack Trace below:

/usr/local/bundle/gems/rack-3.0.8/lib/rack/auth/digest.rb:8: warning: Rack::Auth::Digest is deprecated and will be removed in Rack 3.1
/usr/local/bundle/gems/mihari-5.4.4/lib/mihari/structs/censys.rb:193:in `fetch': key not found: "autonomous_system" (KeyError)
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/structs/censys.rb:193:in `from_dynamic!'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/structs/censys.rb:287:in `block in from_dynamic!'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/structs/censys.rb:287:in `map'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/structs/censys.rb:287:in `from_dynamic!'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/structs/censys.rb:331:in `from_dynamic!'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/clients/censys.rb:39:in `search'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/clients/censys.rb:54:in `block (2 levels) in search_with_pagination'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/clients/censys.rb:53:in `times'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/clients/censys.rb:53:in `block in search_with_pagination'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/analyzers/censys.rb:30:in `each'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/analyzers/censys.rb:30:in `each'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/analyzers/censys.rb:30:in `map'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/analyzers/censys.rb:30:in `artifacts'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/analyzers/base.rb:78:in `block in normalized_artifacts'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/mixins/retriable.rb:28:in `retry_on_error'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/analyzers/base.rb:77:in `normalized_artifacts'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/analyzers/base.rb:92:in `block in result'
	from /usr/local/bundle/gems/dry-monads-1.6.0/lib/dry/monads/try.rb:29:in `run'
	from /usr/local/bundle/gems/dry-monads-1.6.0/lib/dry/monads/try.rb:71:in `[]'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/analyzers/base.rb:92:in `result'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/analyzers/rule.rb:64:in `block in artifacts'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/analyzers/rule.rb:63:in `each'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/analyzers/rule.rb:63:in `flat_map'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/analyzers/rule.rb:63:in `artifacts'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/analyzers/rule.rb:86:in `normalized_artifacts'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/analyzers/rule.rb:97:in `unique_artifacts'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/analyzers/rule.rb:108:in `enriched_artifacts'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/analyzers/rule.rb:120:in `bulk_emit'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/analyzers/rule.rb:138:in `run'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/services/rule_runner.rb:43:in `run'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/commands/search.rb:81:in `block in update_and_run'
	from /usr/local/bundle/gems/dry-monads-1.6.0/lib/dry/monads/try.rb:29:in `run'
	from /usr/local/bundle/gems/dry-monads-1.6.0/lib/dry/monads/try.rb:71:in `[]'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/commands/search.rb:79:in `update_and_run'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/commands/search.rb:23:in `block (2 levels) in search'
	from /usr/local/bundle/gems/dry-monads-1.6.0/lib/dry/monads/right_biased.rb:55:in `bind'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/commands/search.rb:26:in `block in search'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/database.rb:170:in `with_db_connection'
	from /usr/local/bundle/gems/mihari-5.4.4/lib/mihari/commands/search.rb:19:in `search'
	from /usr/local/bundle/gems/thor-1.2.2/lib/thor/command.rb:27:in `run'
	from /usr/local/bundle/gems/thor-1.2.2/lib/thor/invocation.rb:127:in `invoke_command'
	from /usr/local/bundle/gems/thor-1.2.2/lib/thor.rb:392:in `dispatch'
	from /usr/local/bundle/gems/thor-1.2.2/lib/thor/base.rb:485:in `start'
	from /usr/local/bundle/gems/mihari-5.4.4/exe/mihari:8:in `<top (required)>'
	from /usr/local/bundle/bin/mihari:25:in `load'
	from /usr/local/bundle/bin/mihari:25:in `<main>'

System Information:

• OS: Linux Ubuntu Server, Docker Container
• Ruby version: ruby:3.2.2-alpine3.18
• Mihari version: 5.4.4

Additional context

• Problem started happening around 31 Dec 2024.

Kind Regards & Thanks

[r0ny123]

Don't know what's happening.But you should use the latest version of mihari.

[jonathanbrooke]

Don't know what's happening.But you should use the latest version of mihari.

Unfortunately we cant as there is no database upgrade path to the newer versions.

[ninoseki]

Thanks for reporting. This issue is fixed in the latest version.
But I don't have any motivations towards back porting the fix to v5.

[jonathanbrooke]

Thanks for fixing, ill have to find a way to the latest version. The v6 to v7 is problematic as a new db is required

[ninoseki]

Don't recommend much but you can cherry-pick this commit and apply it in v5.4.4.

181b178

Ruby is not a compiled langugae so you can apply a change on the fly.