Ticket: OISF#2696

There are a lot of changes here, which are described below.

In general these changes are renaming constants to conform to the
libhtp-rs versions (which are generated by cbindgen); making all htp
types opaque and changing struct->member references to
htp_struct_member() function calls; and a handful of changes to offload
functionality onto libhtp-rs from suricata, such as URI normalization
and transaction cleanup.

Constants renamed to correspond to cbindgen generated names:

HTP_OK => HTP_STATUS_OK
HTP_ERROR => HTP_STATUS_ERROR

HTP_SERVER_* => HTP_SERVER_PERSONALITY_*

HTP_REQUEST_INVALID_T_E => HTP_FLAGS_REQUEST_INVALID_T_E
HTP_REQUEST_INVALID_C_L => HTP_FLAGS_REQUEST_INVALID_C_L
HTP_HOST_MISSING => HTP_FLAGS_HOST_MISSING
HTP_HOST_AMBIGUOUS => HTP_FLAGS_HOST_AMBIGUOUS
HTP_HOSTU_INVALID => HTP_FLAGS_HOSTU_INVALID
HTP_HOSTH_INVALID => HTP_FLAGS_HOSTH_INVALID

HTP_AUTH_UNRECOGNIZED => HTP_AUTH_TYPE_UNRECOGNIZED

HTP_M_UNKNOWN => HTP_METHOD_UNKNOWN
HTP_M_GET => HTP_METHOD_GET
HTP_M_POST => HTP_METHOD_POST
HTP_M_PUT => HTP_METHOD_PUT
HTP_M_CONNECT => HTP_METHOD_CONNECT

HTP_STREAM_ERROR => HTP_STREAM_STATE_ERROR
HTP_STREAM_TUNNEL => HTP_STREAM_STATE_TUNNEL

HTP_REQUEST_LINE => HTP_REQUEST_PROGRESS_LINE
HTP_REQUEST_HEADERS => HTP_REQUEST_PROGRESS_HEADERS
HTP_REQUEST_BODY => HTP_REQUEST_PROGRESS_BODY
HTP_REQUEST_TRAILER => HTP_REQUEST_PROGRESS_TRAILER
HTP_REQUEST_COMPLETE => HTP_REQUEST_PROGRESS_COMPLETE
HTP_RESPONSE_LINE => HTP_RESPONSE_PROGRESS_LINE
HTP_RESPONSE_HEADERS => HTP_RESPONSE_PROGRESS_HEADERS
HTP_RESPONSE_BODY => HTP_RESPONSE_PROGRESS_BODY
HTP_RESPONSE_TRAILER => HTP_RESPONSE_PROGRESS_TRAILER
HTP_RESPONSE_COMPLETE => HTP_RESPONSE_PROGRESS_COMPLETE

HTP_PROTOCOL_1_1 => HTP_PROTOCOL_V1_1
HTP_PROTOCOL_1_0 => HTP_PROTOCOL_V1_0
HTP_PROTOCOL_0_9 => HTP_PROTOCOL_V0_9

Functions introduced to handle opaque htp_tx_t:
- tx->cfg => htp_tx_cfg(tx)
- tx->flags => htp_tx_flags(tx)
- tx->is_protocol_0_9 => htp_tx_is_protocol_0_9(tx)
- tx->parsed_uri => htp_tx_parsed_uri(tx)
- tx->parsed_uri->path => htp_uri_path(htp_tx_parsed_uri(tx)
- tx->parsed_uri->hostname => htp_uri_hostname(htp_tx_parsed_uri(tx))
- tx->request_auth_type => htp_tx_request_auth_type(tx)
- tx->request_headers => htp_tx_request_headers(tx)
- tx->request_hostname => htp_tx_request_hostname(tx)
- tx->request_line => htp_tx_request_line(tx)
- tx->request_message_len => htp_tx_request_message_len(tx)
- tx->request_method => htp_tx_request_method(tx)
- tx->request_method_number => htp_tx_request_method_number(tx)
- tx->request_port_number => htp_tx_request_port_number(tx)
- tx->request_progress => htp_tx_request_progress(tx)
- tx->request_protocol => htp_tx_request_protocol(tx)
- tx->request_protocol_number => htp_tx_request_protocol_number(tx)
- tx->request_uri => htp_tx_request_uri(tx)
- tx->response_headers => htp_tx_response_headers(tx)
- tx->response_line => htp_tx_response_line(tx)
- tx->response_message => htp_tx_response_message(tx)
- tx->response_message_len => htp_tx_response_message_len(tx)
- tx->response_status => htp_tx_response_status(tx)
- tx->response_status_number => htp_tx_response_status_number(tx)
- tx->response_progress => htp_tx_response_progress(tx)
- tx->response_protocol_number => htp_tx_response_protocol_number(tx)
- htp_tx_get_user_data() => htp_tx_user_data(tx)
- htp_table_get_c(tx->request_headers, header) => htp_tx_request_header(tx, header)
- htp_table_get_c(tx->response_headers, header) => htp_tx_response_header(tx, header)
- htp_table_get_index(tx->request_headers, index) => htp_tx_request_header_index(tx, index)
- htp_table_size(tx->request_headers) => htp_tx_request_headers_size(tx)
- htp_tx_is_http_2_upgrade(tx) convenience function introduced to detect response status 101
  and “Upgrade: h2c" header.

Functions introduced to handle opaque htp_header_t:
- header->name => htp_header_name(header)
- bstr_ptr(header->name) => htp_header_name_ptr(header)
- bstr_len(header->name) => htp_header_name_len(header)
- header->value => htp_header_value(header)
- bstr_len(header->value) => htp_header_value_len(header)
- bstr_ptr(header->value) => htp_header_value_ptr(header)

Functions introduced to handle opaque htp_headers_t:
- htp_table_size(headers) => htp_headers_size(headers)
- htp_table_get_index(headers, index) => htp_headers_get_index(headers, index)

Functions introduced to handle opaque htp_tx_data_t:
- d->len => htp_tx_data_len()
- d->data => htp_tx_data_data()
- htp_tx_data_tx(data) function to get the htp_tx_t from the htp_tx_data_t
- htp_tx_data_is_empty(data) convenience function introduced to test if the data is empty.

Functions introduced to handle opaque htp_connp_t:
- htp_list_get(connp->transactions, txid) => htp_connp_tx(connp, txid)
- htp_list_size(http_state->conn->transactions) => htp_connp_tx_size(connp)
- htp_connp_get_connection(connp) => htp_connp_connection(connp)
- htp_connp_req_data(connp) => htp_connp_request_data(connp)
- htp_connp_req_close(connp) => htp_connp_request_close(connp)
- htp_connp_res_data(connp) => htp_connp_response_data(connp)
- htp_connp_get_jn_tx(connp) => htp_connp_get_request_tx(connp)
- htp_connp_get_out_tx(connp) => htp_connp_get_response_tx(connp)
- htp_connp_req_data_consumed(connp) => htp_connp_request_data_consumed(connp)
- htp_connp_res_data_consumed(connp) => htp_connp_response_data_consumed(connp)
- htp_connp_get_user_data(connp) => htp_connp_user_data(connp)

Functions introduced to handle opaque htp_conn_t:
- conn->in_data_counter => htp_conn_request_data_counter(conn)
- conn->out_data_counter => htp_conn_response_data_counter(conn)

Other changes:

Build libhtp-rs as a crate inside rust. Update autoconf to no longer
use libhtp as an external dependency. Remove HAVE_HTP feature defines
since they are no longer needed.

Make function arguments and return values const where possible

htp_tx_destroy(tx) will now free an incomplete transaction

htp_time_t replaced with standard struct timeval

Callbacks from libhtp now provide the htp_connp_t and the htp_tx_data_t
as separate arguments. This means the connection parser is no longer
fetched from the transaction inside callbacks.

SCHTPGenerateNormalizedUri() functionality moved inside libhtp-rs, which
now provides normalized URI values.
The normalized URI is available with accessor function: htp_tx_normalized_uri()
Configuration settings added to control the behaviour of the URI normalization:
- htp_config_set_normalized_uri_include_all()
- htp_config_set_plusspace_decode()
- htp_config_set_convert_lowercase()
- htp_config_set_double_decode_normalized_query()
- htp_config_set_double_decode_normalized_path()
- htp_config_set_backslash_convert_slashes()
- htp_config_set_bestfit_replacement_byte()
- htp_config_set_convert_lowercase()
- htp_config_set_nul_encoded_terminates()
- htp_config_set_nul_raw_terminates()
- htp_config_set_path_separators_compress()
- htp_config_set_path_separators_decode()
- htp_config_set_u_encoding_decode()
- htp_config_set_url_encoding_invalid_handling()
- htp_config_set_utf8_convert_bestfit()
- htp_config_set_normalized_uri_include_all()
- htp_config_set_plusspace_decode()
Constants related to configuring uri normalization:
- HTP_URL_DECODE_PRESERVE_PERCENT => HTP_URL_ENCODING_HANDLING_PRESERVE_PERCENT
- HTP_URL_DECODE_REMOVE_PERCENT => HTP_URL_ENCODING_HANDLING_REMOVE_PERCENT
- HTP_URL_DECODE_PROCESS_INVALID => HTP_URL_ENCODING_HANDLING_PROCESS_INVALID

htp_config_set_field_limits(soft_limit, hard_limit) changed to
htp_config_set_field_limit(limit) because libhtp didn't implement soft
limits.

libhtp logging API updated to provide HTP_LOG_CODE constants along with
the message. This eliminates the need to perform string matching on
message text to map log messages to HTTP_DECODER_EVENT values, and the
HTP_LOG_CODE values can be used directly. In support of this,
HTP_DECODER_EVENT values are mapped to their corresponding HTP_LOG_CODE
values:

HTTP_DECODER_EVENT_UNKNOWN_ERROR => HTP_LOG_CODE_UNKNOWN
HTTP_DECODER_EVENT_GZIP_DECOMPRESSION_FAILED => HTP_LOG_CODE_GZIP_DECOMPRESSION_FAILED
HTTP_DECODER_EVENT_REQUEST_FIELD_MISSING_COLON => HTP_LOG_CODE_REQUEST_FIELD_MISSING_COLON
HTTP_DECODER_EVENT_RESPONSE_FIELD_MISSING_COLON => HTP_LOG_CODE_RESPONSE_FIELD_MISSING_COLON
HTTP_DECODER_EVENT_INVALID_REQUEST_CHUNK_LEN => HTP_LOG_CODE_INVALID_REQUEST_CHUNK_LEN
HTTP_DECODER_EVENT_INVALID_RESPONSE_CHUNK_LEN => HTP_LOG_CODE_INVALID_RESPONSE_CHUNK_LEN
HTTP_DECODER_EVENT_INVALID_TRANSFER_ENCODING_VALUE_IN_REQUEST => HTP_LOG_CODE_INVALID_TRANSFER_ENCODING_VALUE_IN_REQUEST
HTTP_DECODER_EVENT_INVALID_TRANSFER_ENCODING_VALUE_IN_RESPONSE => HTP_LOG_CODE_INVALID_TRANSFER_ENCODING_VALUE_IN_RESPONSE
HTTP_DECODER_EVENT_INVALID_CONTENT_LENGTH_FIELD_IN_REQUEST => HTP_LOG_CODE_INVALID_CONTENT_LENGTH_FIELD_IN_REQUEST
HTTP_DECODER_EVENT_INVALID_CONTENT_LENGTH_FIELD_IN_RESPONSE => HTP_LOG_CODE_INVALID_CONTENT_LENGTH_FIELD_IN_RESPONSE
HTTP_DECODER_EVENT_DUPLICATE_CONTENT_LENGTH_FIELD_IN_REQUEST => HTP_LOG_CODE_DUPLICATE_CONTENT_LENGTH_FIELD_IN_REQUEST
HTTP_DECODER_EVENT_DUPLICATE_CONTENT_LENGTH_FIELD_IN_RESPONSE => HTP_LOG_CODE_DUPLICATE_CONTENT_LENGTH_FIELD_IN_RESPONSE
HTTP_DECODER_EVENT_100_CONTINUE_ALREADY_SEEN => HTP_LOG_CODE_CONTINUE_ALREADY_SEEN
HTTP_DECODER_EVENT_UNABLE_TO_MATCH_RESPONSE_TO_REQUEST => HTP_LOG_CODE_UNABLE_TO_MATCH_RESPONSE_TO_REQUEST
HTTP_DECODER_EVENT_INVALID_SERVER_PORT_IN_REQUEST => HTP_LOG_CODE_INVALID_SERVER_PORT_IN_REQUEST
HTTP_DECODER_EVENT_INVALID_AUTHORITY_PORT => HTP_LOG_CODE_INVALID_AUTHORITY_PORT
HTTP_DECODER_EVENT_REQUEST_HEADER_INVALID => HTP_LOG_CODE_REQUEST_HEADER_INVALID
HTTP_DECODER_EVENT_RESPONSE_HEADER_INVALID => HTP_LOG_CODE_RESPONSE_HEADER_INVALID
HTTP_DECODER_EVENT_MISSING_HOST_HEADER => HTP_LOG_CODE_MISSING_HOST_HEADER
HTTP_DECODER_EVENT_HOST_HEADER_AMBIGUOUS => HTP_LOG_CODE_HOST_HEADER_AMBIGUOUS
HTTP_DECODER_EVENT_INVALID_REQUEST_FIELD_FOLDING => HTP_LOG_CODE_INVALID_REQUEST_FIELD_FOLDING
HTTP_DECODER_EVENT_INVALID_RESPONSE_FIELD_FOLDING => HTP_LOG_CODE_INVALID_RESPONSE_FIELD_FOLDING
HTTP_DECODER_EVENT_REQUEST_FIELD_TOO_LONG => HTP_LOG_CODE_REQUEST_FIELD_TOO_LONG
HTTP_DECODER_EVENT_RESPONSE_FIELD_TOO_LONG => HTP_LOG_CODE_RESPONSE_FIELD_TOO_LONG
HTTP_DECODER_EVENT_FILE_NAME_TOO_LONG => HTP_LOG_CODE_REQUEST_LINE_INVALID
HTTP_DECODER_EVENT_REQUEST_LINE_INVALID => HTP_LOG_CODE_REQUEST_BODY_UNEXPECTED
HTTP_DECODER_EVENT_REQUEST_BODY_UNEXPECTED => HTP_LOG_CODE_RESPONSE_BODY_UNEXPECTED
HTTP_DECODER_EVENT_REQUEST_SERVER_PORT_TCP_PORT_MISMATCH => HTP_LOG_CODE_REQUEST_SERVER_PORT_TCP_PORT_MISMATCH
HTTP_DECODER_EVENT_URI_HOST_INVALID => HTP_LOG_CODE_URI_HOST_INVALID
HTTP_DECODER_EVENT_HEADER_HOST_INVALID => HTP_LOG_CODE_HEADER_HOST_INVALID
HTTP_DECODER_EVENT_AUTH_UNRECOGNIZED => HTP_LOG_CODE_AUTH_UNRECOGNIZED
HTTP_DECODER_EVENT_REQUEST_HEADER_REPETITION => HTP_LOG_CODE_REQUEST_HEADER_REPETITION
HTTP_DECODER_EVENT_RESPONSE_HEADER_REPETITION => HTP_LOG_CODE_RESPONSE_HEADER_REPETITION
HTTP_DECODER_EVENT_DOUBLE_ENCODED_URI => HTP_LOG_CODE_DOUBLE_ENCODED_URI
HTTP_DECODER_EVENT_URI_DELIM_NON_COMPLIANT => HTP_LOG_CODE_URI_DELIM_NON_COMPLIANT
HTTP_DECODER_EVENT_METHOD_DELIM_NON_COMPLIANT => HTP_LOG_CODE_METHOD_DELIM_NON_COMPLIANT
HTTP_DECODER_EVENT_REQUEST_LINE_LEADING_WHITESPACE => HTP_LOG_CODE_REQUEST_LINE_LEADING_WHITESPACE
HTTP_DECODER_EVENT_TOO_MANY_ENCODING_LAYERS => HTP_LOG_CODE_TOO_MANY_ENCODING_LAYERS
HTTP_DECODER_EVENT_ABNORMAL_CE_HEADER => HTP_LOG_CODE_ABNORMAL_CE_HEADER
HTTP_DECODER_EVENT_RESPONSE_MULTIPART_BYTERANGES => HTP_LOG_CODE_RESPONSE_MULTIPART_BYTERANGES
HTTP_DECODER_EVENT_RESPONSE_ABNORMAL_TRANSFER_ENCODING => HTP_LOG_CODE_RESPONSE_ABNORMAL_TRANSFER_ENCODING
HTTP_DECODER_EVENT_RESPONSE_CHUNKED_OLD_PROTO => HTP_LOG_CODE_RESPONSE_CHUNKED_OLD_PROTO
HTTP_DECODER_EVENT_RESPONSE_INVALID_PROTOCOL => HTP_LOG_CODE_RESPONSE_INVALID_PROTOCOL
HTTP_DECODER_EVENT_RESPONSE_INVALID_STATUS => HTP_LOG_CODE_RESPONSE_INVALID_STATUS
HTTP_DECODER_EVENT_REQUEST_LINE_INCOMPLETE => HTP_LOG_CODE_REQUEST_LINE_INCOMPLETE
HTTP_DECODER_EVENT_LZMA_MEMLIMIT_REACHED => HTP_LOG_CODE_LZMA_MEMLIMIT_REACHED
HTTP_DECODER_EVENT_COMPRESSION_BOMB => HTP_LOG_CODE_COMPRESSION_BOMB

New log events to describe additional anomalies:
HTP_LOG_CODE_REQUEST_TOO_MANY_LZMA_LAYERS
HTP_LOG_CODE_RESPONSE_TOO_MANY_LZMA_LAYERS
HTP_LOG_CODE_PROTOCOL_CONTAINS_EXTRA_DATA
HTP_LOG_CODE_CONTENT_LENGTH_EXTRA_DATA_START
HTP_LOG_CODE_CONTENT_LENGTH_EXTRA_DATA_END
HTP_LOG_CODE_SWITCHING_PROTO_WITH_CONTENT_LENGTH
HTP_LOG_CODE_DEFORMED_EOL
HTP_LOG_CODE_PARSER_STATE_ERROR
HTP_LOG_CODE_MISSING_OUTBOUND_TRANSACTION_DATA
HTP_LOG_CODE_MISSING_INBOUND_TRANSACTION_DATA
HTP_LOG_CODE_ZERO_LENGTH_DATA_CHUNKS
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD_NO_PROTOCOL
HTP_LOG_CODE_REQUEST_LINE_UNKNOWN_METHOD_INVALID_PROTOCOL
HTP_LOG_CODE_REQUEST_LINE_NO_PROTOCOL
HTP_LOG_CODE_RESPONSE_LINE_INVALID_PROTOCOL
HTP_LOG_CODE_RESPONSE_LINE_INVALID_RESPONSE_STATUS
HTP_LOG_CODE_RESPONSE_BODY_INTERNAL_ERROR
HTP_LOG_CODE_REQUEST_BODY_DATA_CALLBACK_ERROR
HTP_LOG_CODE_RESPONSE_INVALID_EMPTY_NAME
HTP_LOG_CODE_REQUEST_INVALID_EMPTY_NAME
HTP_LOG_CODE_RESPONSE_INVALID_LWS_AFTER_NAME
HTP_LOG_CODE_RESPONSE_HEADER_NAME_NOT_TOKEN
HTP_LOG_CODE_REQUEST_INVALID_LWS_AFTER_NAME
HTP_LOG_CODE_LZMA_DECOMPRESSION_DISABLED
HTP_LOG_CODE_CONNECTION_ALREADY_OPEN
HTP_LOG_CODE_COMPRESSION_BOMB_DOUBLE_LZMA
HTP_LOG_CODE_INVALID_CONTENT_ENCODING
HTP_LOG_CODE_INVALID_GAP
HTP_LOG_CODE_ERROR

The new htp_log API supports consuming log messages more easily than
walking a list and tracking the current offset. Internally, libhtp-rs
now provides log messages as a queue of htp_log_t, which means the
application can simply call htp_conn_next_log() to fetch the next log
message until the queue is empty. Once the application is done with a
log message, they can call htp_log_free() to dispose of it.

Functions supporting htp_log_t:
htp_conn_next_log(conn) - Get the next log message
htp_log_message(log) - To get the text of the message
htp_log_code(log) - To get the HTP_LOG_CODE value
htp_log_free(log) - To free the htp_log_t