Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Libhtp rs v12 #11381

Closed
wants to merge 5 commits into from
Closed

Libhtp rs v12 #11381

wants to merge 5 commits into from

Conversation

catenacyber
Copy link
Contributor

Link to redmine ticket: 2696

Describe changes:

  • Use libhtp-rs.

Rebased from #10996 with libhtp-rs living in the repo
cc @jasonish
First commit good in #11377

Draft to see CI feedback with latest libhtp-rs changes, and not have too many conflicts

TODOs :
Still some oss-fuzz report for libhtp-rs to consider

catenacyber and others added 2 commits June 27, 2024 22:53
lgtm has been superseded by codeql github action
docker and travis are obsolete

Preparatory work for ticket 2696 libhtp rust conversion
Ticket: OISF#2696

There are a lot of changes here, which are described below.

In general these changes are renaming constants to conform to the
libhtp-rs versions (which are generated by cbindgen); making all htp
types opaque and changing struct->member references to
htp_struct_member() function calls; and a handful of changes to offload
functionality onto libhtp-rs from suricata, such as URI normalization
and transaction cleanup.

Constants renamed to correspond to cbindgen generated names:

HTP_OK => HTP_STATUS_OK
HTP_ERROR => HTP_STATUS_ERROR

HTP_SERVER_* => HTP_SERVER_PERSONALITY_*

HTP_REQUEST_INVALID_T_E => HTP_FLAGS_REQUEST_INVALID_T_E
HTP_REQUEST_INVALID_C_L => HTP_FLAGS_REQUEST_INVALID_C_L
HTP_HOST_MISSING => HTP_FLAGS_HOST_MISSING
HTP_HOST_AMBIGUOUS => HTP_FLAGS_HOST_AMBIGUOUS
HTP_HOSTU_INVALID => HTP_FLAGS_HOSTU_INVALID
HTP_HOSTH_INVALID => HTP_FLAGS_HOSTH_INVALID

HTP_AUTH_UNRECOGNIZED => HTP_AUTH_TYPE_UNRECOGNIZED

HTP_M_UNKNOWN => HTP_METHOD_UNKNOWN
HTP_M_GET => HTP_METHOD_GET
HTP_M_POST => HTP_METHOD_POST
HTP_M_PUT => HTP_METHOD_PUT
HTP_M_CONNECT => HTP_METHOD_CONNECT

HTP_STREAM_ERROR => HTP_STREAM_STATE_ERROR
HTP_STREAM_TUNNEL => HTP_STREAM_STATE_TUNNEL

HTP_REQUEST_LINE => HTP_REQUEST_PROGRESS_LINE
HTP_REQUEST_HEADERS => HTP_REQUEST_PROGRESS_HEADERS
HTP_REQUEST_BODY => HTP_REQUEST_PROGRESS_BODY
HTP_REQUEST_TRAILER => HTP_REQUEST_PROGRESS_TRAILER
HTP_REQUEST_COMPLETE => HTP_REQUEST_PROGRESS_COMPLETE
HTP_RESPONSE_LINE => HTP_RESPONSE_PROGRESS_LINE
HTP_RESPONSE_HEADERS => HTP_RESPONSE_PROGRESS_HEADERS
HTP_RESPONSE_BODY => HTP_RESPONSE_PROGRESS_BODY
HTP_RESPONSE_TRAILER => HTP_RESPONSE_PROGRESS_TRAILER
HTP_RESPONSE_COMPLETE => HTP_RESPONSE_PROGRESS_COMPLETE

HTP_PROTOCOL_1_1 => HTP_PROTOCOL_V1_1
HTP_PROTOCOL_1_0 => HTP_PROTOCOL_V1_0
HTP_PROTOCOL_0_9 => HTP_PROTOCOL_V0_9

Functions introduced to handle opaque htp_tx_t:
- tx->cfg => htp_tx_cfg(tx)
- tx->flags => htp_tx_flags(tx)
- tx->is_protocol_0_9 => htp_tx_is_protocol_0_9(tx)
- tx->parsed_uri => htp_tx_parsed_uri(tx)
- tx->parsed_uri->path => htp_uri_path(htp_tx_parsed_uri(tx)
- tx->parsed_uri->hostname => htp_uri_hostname(htp_tx_parsed_uri(tx))
- tx->request_auth_type => htp_tx_request_auth_type(tx)
- tx->request_headers => htp_tx_request_headers(tx)
- tx->request_hostname => htp_tx_request_hostname(tx)
- tx->request_line => htp_tx_request_line(tx)
- tx->request_message_len => htp_tx_request_message_len(tx)
- tx->request_method => htp_tx_request_method(tx)
- tx->request_method_number => htp_tx_request_method_number(tx)
- tx->request_port_number => htp_tx_request_port_number(tx)
- tx->request_progress => htp_tx_request_progress(tx)
- tx->request_protocol => htp_tx_request_protocol(tx)
- tx->request_protocol_number => htp_tx_request_protocol_number(tx)
- tx->request_uri => htp_tx_request_uri(tx)
- tx->response_headers => htp_tx_response_headers(tx)
- tx->response_line => htp_tx_response_line(tx)
- tx->response_message => htp_tx_response_message(tx)
- tx->response_message_len => htp_tx_response_message_len(tx)
- tx->response_status => htp_tx_response_status(tx)
- tx->response_status_number => htp_tx_response_status_number(tx)
- tx->response_progress => htp_tx_response_progress(tx)
- tx->response_protocol_number => htp_tx_response_protocol_number(tx)
- htp_tx_get_user_data() => htp_tx_user_data(tx)
- htp_table_get_c(tx->request_headers, header) => htp_tx_request_header(tx, header)
- htp_table_get_c(tx->response_headers, header) => htp_tx_response_header(tx, header)
- htp_table_get_index(tx->request_headers, index) => htp_tx_request_header_index(tx, index)
- htp_table_size(tx->request_headers) => htp_tx_request_headers_size(tx)
- htp_tx_is_http_2_upgrade(tx) convenience function introduced to detect response status 101
  and “Upgrade: h2c" header.

Functions introduced to handle opaque htp_header_t:
- header->name => htp_header_name(header)
- bstr_ptr(header->name) => htp_header_name_ptr(header)
- bstr_len(header->name) => htp_header_name_len(header)
- header->value => htp_header_value(header)
- bstr_len(header->value) => htp_header_value_len(header)
- bstr_ptr(header->value) => htp_header_value_ptr(header)

Functions introduced to handle opaque htp_headers_t:
- htp_table_size(headers) => htp_headers_size(headers)
- htp_table_get_index(headers, index) => htp_headers_get_index(headers, index)

Functions introduced to handle opaque htp_tx_data_t:
- d->len => htp_tx_data_len()
- d->data => htp_tx_data_data()
- htp_tx_data_tx(data) function to get the htp_tx_t from the htp_tx_data_t
- htp_tx_data_is_empty(data) convenience function introduced to test if the data is empty.

Functions introduced to handle opaque htp_connp_t:
- htp_list_get(connp->transactions, txid) => htp_connp_tx(connp, txid)
- htp_list_size(http_state->conn->transactions) => htp_connp_tx_size(connp)
- htp_connp_get_connection(connp) => htp_connp_connection(connp)
- htp_connp_req_data(connp) => htp_connp_request_data(connp)
- htp_connp_req_close(connp) => htp_connp_request_close(connp)
- htp_connp_res_data(connp) => htp_connp_response_data(connp)
- htp_connp_get_jn_tx(connp) => htp_connp_get_request_tx(connp)
- htp_connp_get_out_tx(connp) => htp_connp_get_response_tx(connp)
- htp_connp_req_data_consumed(connp) => htp_connp_request_data_consumed(connp)
- htp_connp_res_data_consumed(connp) => htp_connp_response_data_consumed(connp)
- htp_connp_get_user_data(connp) => htp_connp_user_data(connp)

Functions introduced to handle opaque htp_conn_t:
- conn->in_data_counter => htp_conn_request_data_counter(conn)
- conn->out_data_counter => htp_conn_response_data_counter(conn)

Other changes:

Build libhtp-rs as a crate inside rust. Update autoconf to no longer
use libhtp as an external dependency. Remove HAVE_HTP feature defines
since they are no longer needed.

Make function arguments and return values const where possible

htp_tx_destroy(tx) will now free an incomplete transaction

htp_time_t replaced with standard struct timeval

Callbacks from libhtp now provide the htp_connp_t and the htp_tx_data_t
as separate arguments. This means the connection parser is no longer
fetched from the transaction inside callbacks.

SCHTPGenerateNormalizedUri() functionality moved inside libhtp-rs, which
now provides normalized URI values.
The normalized URI is available with accessor function: htp_tx_normalized_uri()
Configuration settings added to control the behaviour of the URI normalization:
- htp_config_set_normalized_uri_include_all()
- htp_config_set_plusspace_decode()
- htp_config_set_convert_lowercase()
- htp_config_set_double_decode_normalized_query()
- htp_config_set_double_decode_normalized_path()
- htp_config_set_backslash_convert_slashes()
- htp_config_set_bestfit_replacement_byte()
- htp_config_set_convert_lowercase()
- htp_config_set_nul_encoded_terminates()
- htp_config_set_nul_raw_terminates()
- htp_config_set_path_separators_compress()
- htp_config_set_path_separators_decode()
- htp_config_set_u_encoding_decode()
- htp_config_set_url_encoding_invalid_handling()
- htp_config_set_utf8_convert_bestfit()
- htp_config_set_normalized_uri_include_all()
- htp_config_set_plusspace_decode()
Constants related to configuring uri normalization:
- HTP_URL_DECODE_PRESERVE_PERCENT => HTP_URL_ENCODING_HANDLING_PRESERVE_PERCENT
- HTP_URL_DECODE_REMOVE_PERCENT => HTP_URL_ENCODING_HANDLING_REMOVE_PERCENT
- HTP_URL_DECODE_PROCESS_INVALID => HTP_URL_ENCODING_HANDLING_PROCESS_INVALID

htp_config_set_field_limits(soft_limit, hard_limit) changed to
htp_config_set_field_limit(limit) because libhtp didn't implement soft
limits.

libhtp logging API updated to provide HTP_LOG_CODE constants along with
the message. This eliminates the need to perform string matching on
message text to map log messages to HTTP_DECODER_EVENT values, and the
HTP_LOG_CODE values can be used directly. In support of this,
HTP_DECODER_EVENT values are mapped to their corresponding HTP_LOG_CODE
values:

HTTP_DECODER_EVENT_UNKNOWN_ERROR => HTP_LOG_CODE_UNKNOWN
HTTP_DECODER_EVENT_GZIP_DECOMPRESSION_FAILED => HTP_LOG_CODE_GZIP_DECOMPRESSION_FAILED
HTTP_DECODER_EVENT_REQUEST_FIELD_MISSING_COLON => HTP_LOG_CODE_REQUEST_FIELD_MISSING_COLON
HTTP_DECODER_EVENT_RESPONSE_FIELD_MISSING_COLON => HTP_LOG_CODE_RESPONSE_FIELD_MISSING_COLON
HTTP_DECODER_EVENT_INVALID_REQUEST_CHUNK_LEN => HTP_LOG_CODE_INVALID_REQUEST_CHUNK_LEN
HTTP_DECODER_EVENT_INVALID_RESPONSE_CHUNK_LEN => HTP_LOG_CODE_INVALID_RESPONSE_CHUNK_LEN
HTTP_DECODER_EVENT_INVALID_TRANSFER_ENCODING_VALUE_IN_REQUEST => HTP_LOG_CODE_INVALID_TRANSFER_ENCODING_VALUE_IN_REQUEST
HTTP_DECODER_EVENT_INVALID_TRANSFER_ENCODING_VALUE_IN_RESPONSE => HTP_LOG_CODE_INVALID_TRANSFER_ENCODING_VALUE_IN_RESPONSE
HTTP_DECODER_EVENT_INVALID_CONTENT_LENGTH_FIELD_IN_REQUEST => HTP_LOG_CODE_INVALID_CONTENT_LENGTH_FIELD_IN_REQUEST
HTTP_DECODER_EVENT_INVALID_CONTENT_LENGTH_FIELD_IN_RESPONSE => HTP_LOG_CODE_INVALID_CONTENT_LENGTH_FIELD_IN_RESPONSE
HTTP_DECODER_EVENT_DUPLICATE_CONTENT_LENGTH_FIELD_IN_REQUEST => HTP_LOG_CODE_DUPLICATE_CONTENT_LENGTH_FIELD_IN_REQUEST
HTTP_DECODER_EVENT_DUPLICATE_CONTENT_LENGTH_FIELD_IN_RESPONSE => HTP_LOG_CODE_DUPLICATE_CONTENT_LENGTH_FIELD_IN_RESPONSE
HTTP_DECODER_EVENT_100_CONTINUE_ALREADY_SEEN => HTP_LOG_CODE_CONTINUE_ALREADY_SEEN
HTTP_DECODER_EVENT_UNABLE_TO_MATCH_RESPONSE_TO_REQUEST => HTP_LOG_CODE_UNABLE_TO_MATCH_RESPONSE_TO_REQUEST
HTTP_DECODER_EVENT_INVALID_SERVER_PORT_IN_REQUEST => HTP_LOG_CODE_INVALID_SERVER_PORT_IN_REQUEST
HTTP_DECODER_EVENT_INVALID_AUTHORITY_PORT => HTP_LOG_CODE_INVALID_AUTHORITY_PORT
HTTP_DECODER_EVENT_REQUEST_HEADER_INVALID => HTP_LOG_CODE_REQUEST_HEADER_INVALID
HTTP_DECODER_EVENT_RESPONSE_HEADER_INVALID => HTP_LOG_CODE_RESPONSE_HEADER_INVALID
HTTP_DECODER_EVENT_MISSING_HOST_HEADER => HTP_LOG_CODE_MISSING_HOST_HEADER
HTTP_DECODER_EVENT_HOST_HEADER_AMBIGUOUS => HTP_LOG_CODE_HOST_HEADER_AMBIGUOUS
HTTP_DECODER_EVENT_INVALID_REQUEST_FIELD_FOLDING => HTP_LOG_CODE_INVALID_REQUEST_FIELD_FOLDING
HTTP_DECODER_EVENT_INVALID_RESPONSE_FIELD_FOLDING => HTP_LOG_CODE_INVALID_RESPONSE_FIELD_FOLDING
HTTP_DECODER_EVENT_REQUEST_FIELD_TOO_LONG => HTP_LOG_CODE_REQUEST_FIELD_TOO_LONG
HTTP_DECODER_EVENT_RESPONSE_FIELD_TOO_LONG => HTP_LOG_CODE_RESPONSE_FIELD_TOO_LONG
HTTP_DECODER_EVENT_FILE_NAME_TOO_LONG => HTP_LOG_CODE_REQUEST_LINE_INVALID
HTTP_DECODER_EVENT_REQUEST_LINE_INVALID => HTP_LOG_CODE_REQUEST_BODY_UNEXPECTED
HTTP_DECODER_EVENT_REQUEST_BODY_UNEXPECTED => HTP_LOG_CODE_RESPONSE_BODY_UNEXPECTED
HTTP_DECODER_EVENT_REQUEST_SERVER_PORT_TCP_PORT_MISMATCH => HTP_LOG_CODE_REQUEST_SERVER_PORT_TCP_PORT_MISMATCH
HTTP_DECODER_EVENT_URI_HOST_INVALID => HTP_LOG_CODE_URI_HOST_INVALID
HTTP_DECODER_EVENT_HEADER_HOST_INVALID => HTP_LOG_CODE_HEADER_HOST_INVALID
HTTP_DECODER_EVENT_AUTH_UNRECOGNIZED => HTP_LOG_CODE_AUTH_UNRECOGNIZED
HTTP_DECODER_EVENT_REQUEST_HEADER_REPETITION => HTP_LOG_CODE_REQUEST_HEADER_REPETITION
HTTP_DECODER_EVENT_RESPONSE_HEADER_REPETITION => HTP_LOG_CODE_RESPONSE_HEADER_REPETITION
HTTP_DECODER_EVENT_DOUBLE_ENCODED_URI => HTP_LOG_CODE_DOUBLE_ENCODED_URI
HTTP_DECODER_EVENT_URI_DELIM_NON_COMPLIANT => HTP_LOG_CODE_URI_DELIM_NON_COMPLIANT
HTTP_DECODER_EVENT_METHOD_DELIM_NON_COMPLIANT => HTP_LOG_CODE_METHOD_DELIM_NON_COMPLIANT
HTTP_DECODER_EVENT_REQUEST_LINE_LEADING_WHITESPACE => HTP_LOG_CODE_REQUEST_LINE_LEADING_WHITESPACE
HTTP_DECODER_EVENT_TOO_MANY_ENCODING_LAYERS => HTP_LOG_CODE_TOO_MANY_ENCODING_LAYERS
HTTP_DECODER_EVENT_ABNORMAL_CE_HEADER => HTP_LOG_CODE_ABNORMAL_CE_HEADER
HTTP_DECODER_EVENT_RESPONSE_MULTIPART_BYTERANGES => HTP_LOG_CODE_RESPONSE_MULTIPART_BYTERANGES
HTTP_DECODER_EVENT_RESPONSE_ABNORMAL_TRANSFER_ENCODING => HTP_LOG_CODE_RESPONSE_ABNORMAL_TRANSFER_ENCODING
HTTP_DECODER_EVENT_RESPONSE_CHUNKED_OLD_PROTO => HTP_LOG_CODE_RESPONSE_CHUNKED_OLD_PROTO
HTTP_DECODER_EVENT_RESPONSE_INVALID_PROTOCOL => HTP_LOG_CODE_RESPONSE_INVALID_PROTOCOL
HTTP_DECODER_EVENT_RESPONSE_INVALID_STATUS => HTP_LOG_CODE_RESPONSE_INVALID_STATUS
HTTP_DECODER_EVENT_REQUEST_LINE_INCOMPLETE => HTP_LOG_CODE_REQUEST_LINE_INCOMPLETE
HTTP_DECODER_EVENT_LZMA_MEMLIMIT_REACHED => HTP_LOG_CODE_LZMA_MEMLIMIT_REACHED
HTTP_DECODER_EVENT_COMPRESSION_BOMB => HTP_LOG_CODE_COMPRESSION_BOMB

New log events to describe additional anomalies:
HTP_LOG_CODE_REQUEST_TOO_MANY_LZMA_LAYERS
HTP_LOG_CODE_RESPONSE_TOO_MANY_LZMA_LAYERS
HTP_LOG_CODE_PROTOCOL_CONTAINS_EXTRA_DATA
HTP_LOG_CODE_CONTENT_LENGTH_EXTRA_DATA_START
HTP_LOG_CODE_CONTENT_LENGTH_EXTRA_DATA_END
HTP_LOG_CODE_SWITCHING_PROTO_WITH_CONTENT_LENGTH
HTP_LOG_CODE_DEFORMED_EOL
HTP_LOG_CODE_PARSER_STATE_ERROR
HTP_LOG_CODE_MISSING_OUTBOUND_TRANSACTION_DATA
HTP_LOG_CODE_MISSING_INBOUND_TRANSACTION_DATA
HTP_LOG_CODE_ZERO_LENGTH_DATA_CHUNKS
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD_NO_PROTOCOL
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD_INVALID_PROTOCOL
HTP_LOG_CODE_REQUEST_LINE_NO_PROTOCOL
HTP_LOG_CODE_RESPONSE_LINE_INVALID_PROTOCOL
HTP_LOG_CODE_RESPONSE_LINE_INVALID_RESPONSE_STATUS
HTP_LOG_CODE_RESPONSE_BODY_INTERNAL_ERROR
HTP_LOG_CODE_REQUEST_BODY_DATA_CALLBACK_ERROR
HTP_LOG_CODE_RESPONSE_INVALID_EMPTY_NAME
HTP_LOG_CODE_REQUEST_INVALID_EMPTY_NAME
HTP_LOG_CODE_RESPONSE_INVALID_LWS_AFTER_NAME
HTP_LOG_CODE_RESPONSE_HEADER_NAME_NOT_TOKEN
HTP_LOG_CODE_REQUEST_INVALID_LWS_AFTER_NAME
HTP_LOG_CODE_LZMA_DECOMPRESSION_DISABLED
HTP_LOG_CODE_CONNECTION_ALREADY_OPEN
HTP_LOG_CODE_COMPRESSION_BOMB_DOUBLE_LZMA
HTP_LOG_CODE_INVALID_CONTENT_ENCODING
HTP_LOG_CODE_INVALID_GAP
HTP_LOG_CODE_ERROR

The new htp_log API supports consuming log messages more easily than
walking a list and tracking the current offset. Internally, libhtp-rs
now provides log messages as a queue of htp_log_t, which means the
application can simply call htp_conn_next_log() to fetch the next log
message until the queue is empty. Once the application is done with a
log message, they can call htp_log_free() to dispose of it.

Functions supporting htp_log_t:
htp_conn_next_log(conn) - Get the next log message
htp_log_message(log) - To get the text of the message
htp_log_code(log) - To get the HTP_LOG_CODE value
htp_log_free(log) - To free the htp_log_t
@suricata-qa
Copy link

Information:

ERROR: QA failed on SURI_TLPW2_single_suri_time.

ERROR: QA failed on SURI_TLPW2_single_alerts_cmp.

ERROR: QA failed on SURI_TLPW2_autofp_alerts_cmp.

ERROR: QA failed on SURI_TLPW1_suri_time.

ERROR: QA failed on SURI_TLPW1_files_sha256.

ERROR: QA failed on SURI_TLPR1_alerts_cmp.

field baseline test %
SURI_TLPW2_single_stats_chk
.uptime 776 821 105.8%
SURI_TLPW1_stats_chk
.uptime 178 184 103.37%
.app_layer.error.http.parser 47 2 4.26%
SURI_TLPR1_stats_chk
.app_layer.error.http.parser 700 369 52.71%

Pipeline 21342

@jasonish
Copy link
Member

jasonish commented Jul 3, 2024

Rebased from #10996 with libhtp-rs living in the repo

Given the tight coupling Suricata has on libhtp versions, I do think this makes sense.

@catenacyber
Copy link
Contributor Author

Continued in #11427

@catenacyber catenacyber closed this Jul 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs rebase Needs rebase to master
Development

Successfully merging this pull request may close these issues.

4 participants