Skip to content
This repository has been archived by the owner on May 27, 2024. It is now read-only.

CSV Uploader performance issues #79

Closed
rkokkelk opened this issue Oct 27, 2022 · 1 comment · Fixed by #80
Closed

CSV Uploader performance issues #79

rkokkelk opened this issue Oct 27, 2022 · 1 comment · Fixed by #80

Comments

@rkokkelk
Copy link
Contributor

rkokkelk commented Oct 27, 2022 •
edited
Loading

When using the CSV uploader we encountered an enormous performance issue when submitting very large files (~80k). Uploading these files took roughly 30 - 45m before being fully processed. This offcourse results in issues with HTTP connections which timeouts.

Delving into this issue, it was quickly appareant that it solely related to the CSV uploader, IntelMQ can efficiently handle the data, and is encountered on the /submit function, /preview and /upload are not as much effected by the file size.

Running a profile for the /submit function for 100 records, the results of which can be seen below, showed the issues related to the intelmq/lib/utils.py:201(load_configuration) function. For the overall running time of 34.816s around 34.543s is spend in that function. This function is trigged by the intelmq/lib/message.py:95(__init__) function. Basically for every event in de upload, an new Event(Message) object is created which in turn loads and parses the harmonization.conf file. The loading of that particular file attributes for every single events results in the massive performance issue.

Mainly an effort has to be made on how we can e.g. use only a single Message object which can be reused or some other way that the harmonization config file is only loaded once.

PATH: '/submit'
         25739027 function calls (25656078 primitive calls) in 34.821 seconds

   Ordered by: internal time, call count

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   201596    3.316    0.000    5.141    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:1409(scan_flow_scalar_non_spaces)
   127058    2.299    0.000    4.143    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:845(scan_to_next_token)
  1023837    2.222    0.000    2.324    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/reader.py:163(forward)
   833048    2.009    0.000    2.792    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:197(need_more_tokens)
    52116    1.680    0.000    9.742    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:1361(scan_flow_scalar)
    63529    1.604    0.000    4.927    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/parser.py:335(parse_node)
   127058    1.484    0.000   19.543    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:214(fetch_more_tokens)
   149480    1.483    0.000    2.042    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:1463(scan_flow_scalar_spaces)
   464095    1.358    0.000   22.902    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:165(check_token)
   705990    1.107    0.000    1.398    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/parser.py:122(scanner)
63529/101    1.104    0.000   33.501    0.332 /usr/lib/python3/dist-packages/ruamel/yaml/composer.py:109(compose_node)
  3510255    0.956    0.000    0.975    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/reader.py:132(peek)
   285932    0.857    0.000    1.063    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/reader.py:178(get_mark)
    41309    0.699    0.000   15.577    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/parser.py:707(parse_flow_mapping_key)
   864257    0.681    0.000    0.702    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:342(stale_possible_simple_keys)
   168468    0.653    0.000    0.884    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:1732(scan_line_break)
   324816    0.589    0.000    1.142    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/composer.py:40(parser)
    31714    0.548    0.000    0.921    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:617(fetch_value)
    53934    0.510    0.000    1.156    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/composer.py:142(compose_scalar_node)
  2695692    0.479    0.000    0.479    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:144(reader)
    63529    0.453    0.000    0.732    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:362(save_possible_simple_key)
 9595/101    0.432    0.000   33.474    0.331 /usr/lib/python3/dist-packages/ruamel/yaml/composer.py:194(compose_mapping_node)
   365822    0.369    0.000    0.572    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/main.py:185(parser)
  1894368    0.367    0.000    0.367    0.000 {built-in method builtins.hasattr}
   190587    0.366    0.000    0.671    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/composer.py:47(resolver)
   187759    0.344    0.000   29.217    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/parser.py:136(check_event)
    31714    0.330    0.000   13.097    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/parser.py:740(parse_flow_mapping_value)
   158873    0.315    0.000    0.898    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:186(get_token)
   396728    0.295    0.000    0.345    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/reader.py:140(prefix)
   737199    0.262    0.000    0.262    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:326(next_possible_simple_key)
    22119    0.245    0.000    0.524    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:549(fetch_flow_entry)
   227856    0.223    0.000    0.358    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/main.py:225(resolver)
  1049565    0.221    0.000    0.253    0.000 {built-in method builtins.isinstance}
2005394/2005276    0.215    0.000    0.219    0.000 {built-in method builtins.len}
   127159    0.201    0.000    0.270    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:404(unwind_indent)
    63529    0.199    0.000    0.479    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/constructor.py:128(construct_object)
    52116    0.192    0.000   10.654    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:730(fetch_flow_scalar)
   705990    0.191    0.000    0.191    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/main.py:176(scanner)
     9595    0.187    0.000    0.792    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/constructor.py:203(construct_mapping)
   498839    0.164    0.000    0.224    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:85(flow_level)
    63428    0.151    0.000    5.059    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/parser.py:323(parse_flow_node)
    83022    0.142    0.000    0.477    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:178(peek_token)
   285932    0.134    0.000    0.134    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/error.py:30(__init__)
   686116    0.130    0.000    0.130    0.000 {built-in method builtins.getattr}
    31714    0.125    0.000    0.527    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:789(check_value)
    33532    0.121    0.000    0.398    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:156(scanner_processing_version)
    53934    0.118    0.000    0.243    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/events.py:125(__init__)
    53934    0.114    0.000    0.132    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/tokens.py:241(__init__)
     9595    0.112    0.000    0.235    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:530(fetch_flow_collection_end)
    53934    0.109    0.000    0.152    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/nodes.py:81(__init__)
     9595    0.107    0.000    0.228    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:508(fetch_flow_collection_start)
    63529    0.105    0.000    0.147    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/events.py:42(__init__)
    63630    0.099    0.000    0.125    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:386(remove_possible_simple_key)
    52116    0.096    0.000   10.750    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:726(fetch_double)
    63529    0.086    0.000    0.133    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/resolver.py:357(resolve)
   609615    0.085    0.000    0.085    0.000 {method 'append' of 'list' objects}
    40905    0.082    0.000    0.234    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/resolver.py:382(processing_version)
    40905    0.079    0.000    0.152    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/resolver.py:124(parser)
    53934    0.077    0.000    0.109    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/constructor.py:344(construct_scalar)
   593880    0.075    0.000    0.075    0.000 {built-in method sys._getframe}
   286740    0.072    0.000    0.072    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/reader.py:101(stream)
    52116    0.068    0.000    0.173    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/constructor.py:687(construct_yaml_str)
     1818    0.060    0.000    0.117    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:1517(scan_plain)
   242907    0.059    0.000    0.059    0.000 {method 'pop' of 'list' objects}
     9595    0.054    0.000    4.010    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/parser.py:701(parse_flow_mapping_first_key)
    63529    0.053    0.000    0.053    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/nodes.py:15(__init__)
   158874    0.051    0.000    0.051    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/tokens.py:16(__init__)
    73528    0.048    0.000    0.048    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/events.py:17(__init__)
   352904    0.047    0.000    0.047    0.000 {method 'extend' of 'list' objects}
19190/9696    0.042    0.000    0.886    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/constructor.py:703(construct_yaml_map)
     9595    0.037    0.000    0.041    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/constructor.py:352(flatten_mapping)
    73528    0.035    0.000    0.103    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/parser.py:158(get_event)
     9595    0.034    0.000    0.868    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/constructor.py:419(construct_mapping)
    73326    0.033    0.000    0.074    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/tokens.py:56(comment)
    31815    0.032    0.000    0.032    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:56(__init__)
    63529    0.026    0.000    0.026    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/resolver.py:218(descend_resolver)
     1818    0.022    0.000    0.059    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/constructor.py:448(construct_yaml_int)
    63529    0.021    0.000    0.021    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/resolver.py:241(ascend_resolver)
     9595    0.021    0.000    0.044    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/events.py:51(__init__)
    53934    0.021    0.000    0.025    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/constructor.py:184(construct_scalar)
     9595    0.021    0.000    0.050    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/nodes.py:117(__init__)
     9595    0.020    0.000    0.247    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:504(fetch_flow_mapping_start)
     2828    0.019    0.000    0.028    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:83(get_token)
     9595    0.019    0.000    0.254    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:526(fetch_flow_mapping_end)
     9595    0.019    0.000    0.029    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/nodes.py:92(__init__)
    63529    0.018    0.000    0.018    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/parser.py:150(peek_event)
      101    0.018    0.000    0.905    0.009 /usr/lib/python3/dist-packages/ruamel/yaml/constructor.py:114(construct_document)
     1818    0.018    0.000    0.023    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:1594(scan_plain_spaces)
        1    0.018    0.018   34.817   34.817 /root/intelmq-webinput-csv/intelmq_webinput_csv/bin/backend.py:391(submit)
      203    0.018    0.000    0.018    0.000 {method 'recv' of '_socket.socket' objects}
    63428    0.018    0.000    0.032    0.000 /usr/lib/python3.8/abc.py:96(__instancecheck__)
    54265    0.016    0.000    0.017    0.000 {method 'join' of 'str' objects}
    63428    0.014    0.000    0.014    0.000 {built-in method _abc._abc_instancecheck}
      707    0.013    0.000    0.016    0.000 {method 'read' of '_io.TextIOWrapper' objects}
      101    0.013    0.000   34.570    0.342 /usr/lib/python3/dist-packages/intelmq/lib/message.py:95(__init__)
9698/9697    0.012    0.000    0.039    0.000 {built-in method builtins.next}
      101    0.012    0.000   34.524    0.342 /usr/lib/python3/dist-packages/ruamel/yaml/main.py:316(load)
     8282    0.012    0.000    0.023    0.000 /usr/lib/python3.8/re.py:188(match)
    12310    0.011    0.000    0.011    0.000 {method 'match' of 're.Pattern' objects}
    31714    0.010    0.000    0.010    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/constructor.py:254(check_mapping_key)
     3636    0.009    0.000    0.015    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/util.py:40(__getattribute__)
      202    0.009    0.000    0.067    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:668(_parse)
      102    0.009    0.000    0.009    0.000 {method 'sendall' of '_socket.socket' objects}
    31719    0.008    0.000    0.008    0.000 {method 'insert' of 'list' objects}
      808    0.008    0.000    0.073    0.000 /usr/lib/python3/dist-packages/intelmq/lib/message.py:192(add)
     1818    0.008    0.000    0.129    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:739(fetch_plain)
     1818    0.008    0.000    0.027    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:808(check_plain)
      404    0.006    0.000    0.014    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:877(_parse_numeric_token)
     3636    0.006    0.000    0.024    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/resolver.py:344(versioned_resolver)
     68/7    0.006    0.000    0.013    0.002 /usr/lib/python3.8/sre_parse.py:493(_parse)
     3636    0.006    0.000    0.013    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/constructor.py:88(resolver)
    19191    0.006    0.000    0.006    0.000 {method 'update' of 'dict' objects}
      909    0.006    0.000    0.030    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/reader.py:258(update)
     2626    0.005    0.000    0.006    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:209(isword)
      202    0.005    0.000    0.014    0.000 /usr/lib/python3.8/_strptime.py:309(_strptime)
      103    0.005    0.000    0.006    0.000 {built-in method io.open}
     8295    0.005    0.000    0.025    0.000 /usr/lib/python3.8/re.py:289(_compile)
      202    0.004    0.000    0.038    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:205(split)
     2828    0.004    0.000    0.033    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:195(__next__)
      101    0.004    0.000    0.014    0.000 /usr/lib/python3/dist-packages/intelmq/lib/harmonization.py:811(sanitize)
      808    0.003    0.000    0.018    0.000 /usr/lib/python3/dist-packages/intelmq/lib/message.py:333(__is_valid_value)
      707    0.003    0.000    0.019    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/reader.py:293(update_raw)
      707    0.003    0.000    0.045    0.000 /usr/lib/python3/dist-packages/intelmq/lib/message.py:357(__sanitize_value)
      606    0.003    0.000    0.003    0.000 {method 'translate' of 'bytes' objects}
      202    0.003    0.000    0.080    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:577(parse)
     9595    0.003    0.000    0.003    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/composer.py:228(check_end_doc_comment)
      101    0.003    0.000   34.534    0.342 /usr/lib/python3/dist-packages/intelmq/lib/utils.py:201(load_configuration)
    103/7    0.003    0.000    0.004    0.001 /usr/lib/python3.8/sre_compile.py:71(_compile)
      202    0.002    0.000    0.004    0.000 {built-in method builtins.sum}
      202    0.002    0.000    0.010    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/parser.py:185(parse_implicit_document_start)
      101    0.002    0.000    0.002    0.000 /usr/lib/python3.8/json/encoder.py:204(iterencode)
      101    0.002    0.000   33.590    0.333 /usr/lib/python3/dist-packages/ruamel/yaml/composer.py:70(get_single_node)
      101    0.002    0.000    0.046    0.000 /usr/lib/python3/dist-packages/redis/client.py:832(execute_command)
      101    0.002    0.000    0.049    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/parser.py:173(parse_stream_start)
      101    0.002    0.000    0.005    0.000 /usr/lib/python3.8/ipaddress.py:1417(__init__)
      106    0.002    0.000    0.002    0.000 {built-in method posix.stat}
      606    0.002    0.000    0.002    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:433(append)
      203    0.002    0.000    0.021    0.000 /usr/lib/python3/dist-packages/redis/connection.py:171(_read_from_socket)
      102    0.002    0.000    0.003    0.000 /usr/lib/python3/dist-packages/redis/connection.py:721(pack_command)
      306    0.002    0.000    0.004    0.000 {built-in method from_bytes}
     1212    0.002    0.000    0.002    0.000 /usr/lib/python3.8/ipaddress.py:1151(_parse_octet)
      506    0.002    0.000    0.002    0.000 {method 'replace' of 'datetime.datetime' objects}
      606    0.002    0.000    0.008    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/reader.py:244(check_printable)
        2    0.002    0.001    0.002    0.001 /usr/lib/python3.8/sre_parse.py:969(parse_template)
     5656    0.002    0.000    0.002    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:214(isnum)
      101    0.001    0.000    0.023    0.000 /usr/lib/python3/dist-packages/redis/client.py:850(parse_response)
      102    0.001    0.000    0.019    0.000 /usr/lib/python3/dist-packages/redis/connection.py:236(readline)
      404    0.001    0.000    0.002    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:1008(_find_hms_idx)
     3030    0.001    0.000    0.001    0.000 /usr/lib/python3/dist-packages/intelmq/lib/message.py:365(__get_type_config)
      101    0.001    0.000    0.019    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/parser.py:236(parse_document_end)
      101    0.001    0.000    0.011    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/main.py:375(get_constructor_parser)
      808    0.001    0.000    0.002    0.000 /usr/lib/python3/dist-packages/intelmq/lib/message.py:324(__is_valid_key)
      606    0.001    0.000    0.006    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/reader.py:236(_get_non_printable)
     7176    0.001    0.000    0.001    0.000 {method 'startswith' of 'str' objects}
      202    0.001    0.000    0.002    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:479(resolve_ymd)
      506    0.001    0.000    0.006    0.000 /usr/lib/python3.8/ipaddress.py:1224(__init__)
      515    0.001    0.000    0.002    0.000 /usr/lib/python3.8/codecs.py:319(decode)
      101    0.001    0.000   33.522    0.332 /usr/lib/python3/dist-packages/ruamel/yaml/composer.py:95(compose_document)
      102    0.001    0.000    0.001    0.000 {method 'isoformat' of 'datetime.datetime' objects}
      101    0.001    0.000    0.003    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/parser.py:203(parse_document_start)
     5588    0.001    0.000    0.001    0.000 {method 'get' of 'dict' objects}
      101    0.001    0.000   34.498    0.342 /usr/lib/python3/dist-packages/ruamel/yaml/constructor.py:106(get_single_data)
      202    0.001    0.000    0.015    0.000 /usr/lib/python3.8/_strptime.py:565(_strptime_datetime)
      102    0.001    0.000    0.009    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/reader.py:109(stream)
      606    0.001    0.000    0.005    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/reader.py:218(_get_non_printable_ascii)
     32/7    0.001    0.000    0.013    0.002 /usr/lib/python3.8/sre_parse.py:435(_parse_sub)
      589    0.001    0.000    0.002    0.000 /usr/lib/python3.8/sre_parse.py:254(get)
     1616    0.001    0.000    0.001    0.000 /usr/lib/python3/dist-packages/intelmq/lib/message.py:505(__contains__)
  202/101    0.001    0.000    0.003    0.000 /usr/lib/python3.8/functools.py:947(__get__)
      606    0.001    0.000    0.001    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:1242(_to_decimal)
      202    0.001    0.000    0.003    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:1169(_build_tzaware)
      101    0.001    0.000    0.006    0.000 /usr/lib/python3/dist-packages/intelmq/lib/message.py:313(serialize)
     1532    0.001    0.000    0.001    0.000 {method 'encode' of 'str' objects}
      202    0.001    0.000    0.028    0.000 /usr/lib/python3/dist-packages/intelmq/lib/harmonization.py:422(__parse)
      202    0.001    0.000    0.002    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:1209(_build_naive)
      101    0.001    0.000    0.007    0.000 /usr/lib/python3/dist-packages/redis/connection.py:1063(get_connection)
      102    0.001    0.000    0.002    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:440(fetch_stream_start)
      515    0.001    0.000    0.001    0.000 {built-in method _codecs.utf_8_decode}
   113/17    0.001    0.000    0.001    0.000 /usr/lib/python3.8/sre_parse.py:174(getwidth)
      102    0.001    0.000    0.020    0.000 /usr/lib/python3/dist-packages/redis/connection.py:309(read_response)
     2626    0.001    0.000    0.001    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:240(<genexpr>)
      101    0.001    0.000    0.001    0.000 {built-in method binascii.a2b_base64}
      202    0.001    0.000    0.001    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:62(__init__)
     3634    0.001    0.000    0.001    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/util.py:35(<lambda>)
      202    0.001    0.000    0.081    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:1258(parse)
       70    0.001    0.000    0.001    0.000 /usr/lib/python3.8/sre_parse.py:355(_escape)
      101    0.001    0.000    0.002    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:449(fetch_stream_end)
      101    0.001    0.000    0.002    0.000 /usr/lib/python3/dist-packages/intelmq/lib/utils.py:172(base64_encode)
       61    0.001    0.000    0.001    0.000 /usr/lib/python3.8/sre_compile.py:276(_optimize_charset)
     7676    0.001    0.000    0.001    0.000 {method 'isdigit' of 'str' objects}
      101    0.001    0.000    0.004    0.000 /usr/lib/python3/dist-packages/intelmq/lib/harmonization.py:788(is_valid)
      101    0.001    0.000    0.008    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/reader.py:187(determine_encoding)
      101    0.001    0.000   34.571    0.342 /usr/lib/python3/dist-packages/intelmq/lib/message.py:513(__init__)
        1    0.001    0.001    0.003    0.003 /usr/lib/python3.8/_strptime.py:49(__init__)
      733    0.001    0.000    0.001    0.000 /usr/lib/python3.8/sre_parse.py:233(__next)
      202    0.001    0.000    0.027    0.000 {built-in method strptime}
      101    0.001    0.000    0.047    0.000 /usr/lib/python3/dist-packages/intelmq/lib/pipeline.py:229(send)
      606    0.001    0.000    0.001    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:342(hms)
       31    0.001    0.000    0.002    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/sansio/multipart.py:145(next_event)
      101    0.001    0.000    0.001    0.000 /usr/lib/python3/dist-packages/dateutil/tz/tz.py:193(__init__)
        8    0.001    0.000    0.003    0.000 /usr/lib/python3.8/re.py:325(_subx)
     1515    0.001    0.000    0.001    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/datastructures.py:477(items)
      202    0.001    0.000    0.001    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:227(__init__)
      102    0.001    0.000    0.013    0.000 /usr/lib/python3/dist-packages/redis/connection.py:684(send_command)
      102    0.001    0.000    0.009    0.000 /usr/lib/python3/dist-packages/redis/connection.py:655(send_packed_command)
      808    0.001    0.000    0.001    0.000 /usr/lib/python3/dist-packages/intelmq/lib/harmonization.py:74(sanitize)
      101    0.001    0.000    0.003    0.000 /usr/lib/python3.8/ipaddress.py:730(num_addresses)
      101    0.001    0.000    0.002    0.000 /usr/lib/python3.8/ipaddress.py:709(broadcast_address)
     4646    0.001    0.000    0.001    0.000 {method 'read' of '_io.StringIO' objects}
      111    0.001    0.000    0.001    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/local.py:308(__get__)
      303    0.001    0.000    0.001    0.000 /usr/lib/python3/dist-packages/intelmq/lib/harmonization.py:97(is_valid)
      204    0.001    0.000    0.002    0.000 /usr/lib/python3.8/locale.py:575(getlocale)
      775    0.001    0.000    0.001    0.000 /usr/lib/python3.8/sre_parse.py:164(__getitem__)
      303    0.001    0.000    0.004    0.000 /usr/lib/python3.8/ipaddress.py:1125(_ip_int_from_string)
      707    0.001    0.000    0.001    0.000 /usr/lib/python3/dist-packages/intelmq/lib/harmonization.py:58(is_valid)
      202    0.001    0.000    0.001    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:1100(_parse_min_sec)
      102    0.001    0.000    0.021    0.000 /usr/lib/python3/dist-packages/redis/connection.py:697(read_response)
      202    0.001    0.000    0.001    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:386(validate)
      202    0.001    0.000    0.001    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:400(__init__)
      101    0.001    0.000    0.001    0.000 {built-in method binascii.b2a_base64}
        1    0.001    0.001    0.001    0.001 {built-in method time.tzset}
      101    0.001    0.000    0.005    0.000 /usr/lib/python3/dist-packages/intelmq/lib/harmonization.py:401(is_valid)
        1    0.001    0.001    0.011    0.011 /usr/local/lib/python3.8/dist-packages/werkzeug/formparser.py:397(parse)
      204    0.001    0.000    0.003    0.000 /usr/lib/python3.8/_strptime.py:26(_getlang)
      202    0.001    0.000    0.001    0.000 {method 'tzname' of 'datetime.datetime' objects}
      101    0.001    0.000    0.024    0.000 /usr/lib/python3/dist-packages/intelmq/lib/harmonization.py:414(sanitize)
      101    0.001    0.000    0.001    0.000 /usr/lib/python3/dist-packages/intelmq/lib/harmonization.py:338(sanitize)
      101    0.001    0.000    0.002    0.000 /usr/lib/python3/dist-packages/intelmq/lib/utils.py:158(base64_decode)
      101    0.001    0.000    0.003    0.000 /usr/lib/python3.8/json/encoder.py:182(encode)
      203    0.000    0.000    0.001    0.000 {built-in method now}
      505    0.000    0.000    0.001    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:335(month)
      101    0.000    0.000    0.005    0.000 /usr/lib/python3/dist-packages/redis/connection.py:214(can_read)
      303    0.000    0.000    0.001    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:1058(_could_be_tzname)
      101    0.000    0.000    0.003    0.000 /usr/lib/python3.8/json/__init__.py:183(dumps)
      505    0.000    0.000    0.001    0.000 /usr/lib/python3/dist-packages/intelmq/lib/utils.py:86(decode)
      101    0.000    0.000    0.019    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/parser.py:319(parse_block_node)
      204    0.000    0.000    0.001    0.000 /usr/lib/python3.8/locale.py:384(normalize)
      101    0.000    0.000    0.001    0.000 /usr/lib/python3/dist-packages/redis/connection.py:1109(release)
      204    0.000    0.000    0.000    0.000 {method 'settimeout' of '_socket.socket' objects}
      204    0.000    0.000    0.001    0.000 /usr/lib/python3.8/locale.py:467(_parse_localename)
        1    0.000    0.000    0.009    0.009 {built-in method builtins.exec}
     2431    0.000    0.000    0.000    0.000 {built-in method builtins.setattr}
     1925    0.000    0.000    0.000    0.000 {method 'replace' of 'str' objects}
      102    0.000    0.000    0.002    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/scanner.py:90(reset_scanner)
      204    0.000    0.000    0.000    0.000 {built-in method _locale.setlocale}
     1818    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/compat.py:58(to_str)
      101    0.000    0.000    0.001    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/events.py:93(__init__)
     2762    0.000    0.000    0.000    0.000 {method 'lower' of 'str' objects}
      205    0.000    0.000    0.000    0.000 {built-in method posix.getpid}
      101    0.000    0.000    0.001    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/events.py:80(__init__)
     1212    0.000    0.000    0.001    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:219(isspace)
      202    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/dateutil/tz/tz.py:224(tzname)
       22    0.000    0.000    0.003    0.000 {method 'sub' of 're.Pattern' objects}
      101    0.000    0.000    0.003    0.000 /usr/lib/python3/dist-packages/intelmq/lib/harmonization.py:121(is_valid)
      101    0.000    0.000    0.046    0.000 /usr/lib/python3/dist-packages/redis/client.py:1677(lpush)
      299    0.000    0.000    0.000    0.000 /usr/lib/python3.8/sre_parse.py:249(match)
      101    0.000    0.000    0.001    0.000 /usr/lib/python3.8/ipaddress.py:714(hostmask)
      101    0.000    0.000    0.001    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/tokens.py:61(move_comment)
      303    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:328(weekday)
      202    0.000    0.000    0.001    0.000 /usr/lib/python3/dist-packages/dateutil/tz/_common.py:19(adjust_encoding)
     1717    0.000    0.000    0.000    0.000 {function Message.__contains__ at 0x7f1f880045e0}
      303    0.000    0.000    0.001    0.000 /usr/lib/python3/dist-packages/intelmq/lib/utils.py:122(encode)
      202    0.000    0.000    0.001    0.000 /usr/lib/python3/dist-packages/redis/connection.py:1054(_checkpid)
      101    0.000    0.000    0.001    0.000 /usr/lib/python3.8/_bootlocale.py:33(getpreferredencoding)
      101    0.000    0.000    0.001    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/constructor.py:75(composer)
     2626    0.000    0.000    0.000    0.000 {method 'isalpha' of 'str' objects}
      101    0.000    0.000    0.001    0.000 /usr/lib/python3.8/ipaddress.py:535(_split_addr_prefix)
      102    0.000    0.000    0.000    0.000 /usr/lib/python3.8/codecs.py:309(__init__)
        1    0.000    0.000    0.001    0.001 /usr/lib/python3/dist-packages/redis/client.py:74(__init__)
      305    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/redis/connection.py:101(encode)
      101    0.000    0.000    0.005    0.000 /usr/lib/python3.8/ipaddress.py:57(ip_network)
        1    0.000    0.000    0.000    0.000 /usr/lib/python3.8/_strptime.py:83(<listcomp>)
      101    0.000    0.000    0.001    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/main.py:215(constructor)
      101    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/intelmq/lib/harmonization.py:322(is_valid)
      202    0.000    0.000    0.027    0.000 /usr/lib/python3/dist-packages/intelmq/lib/harmonization.py:439(parse_utc_isoformat)
  105/102    0.000    0.000    0.013    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/utils.py:97(__get__)
      101    0.000    0.000    0.000    0.000 {built-in method builtins.all}
      202    0.000    0.000    0.003    0.000 /usr/lib/python3.8/ipaddress.py:27(ip_address)
       80    0.000    0.000    0.001    0.000 /usr/lib/python3.8/json/__init__.py:299(loads)
      102    0.000    0.000    0.000    0.000 {method 'write' of '_io.BytesIO' objects}
      202    0.000    0.000    0.004    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:239(__len__)
      101    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/redis/client.py:78(__contains__)
        1    0.000    0.000    0.011    0.011 /usr/local/lib/python3.8/dist-packages/werkzeug/formparser.py:270(_parse_multipart)
       15    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/http.py:377(parse_options_header)
       38    0.000    0.000    0.000    0.000 {method 'strftime' of 'datetime.date' objects}
       80    0.000    0.000    0.000    0.000 /usr/lib/python3.8/json/decoder.py:332(decode)
        1    0.000    0.000    0.008    0.008 /usr/lib/python3.8/_strptime.py:173(__init__)
      203    0.000    0.000    0.018    0.000 /usr/lib/python3/dist-packages/redis/_compat.py:70(recv)
        1    0.000    0.000    0.000    0.000 {built-in method marshal.loads}
      101    0.000    0.000    0.000    0.000 {built-in method _locale.nl_langinfo}
      112    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/local.py:509(_get_current_object)
      202    0.000    0.000    0.000    0.000 {method 'groupdict' of 're.Match' objects}
      101    0.000    0.000    0.002    0.000 /usr/lib/python3/dist-packages/intelmq/lib/harmonization.py:136(sanitize)
      102    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/tokens.py:137(__init__)
        2    0.000    0.000    0.000    0.000 {built-in method builtins.__build_class__}
      303    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:348(ampm)
        1    0.000    0.000   34.817   34.817 /usr/local/lib/python3.8/dist-packages/flask/app.py:1773(dispatch_request)
      101    0.000    0.000    0.001    0.000 /usr/lib/python3.8/base64.py:65(b64decode)
      421    0.000    0.000    0.000    0.000 {method 'join' of 'bytes' objects}
      101    0.000    0.000    0.005    0.000 /usr/lib/python3/dist-packages/redis/connection.py:689(can_read)
      404    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:325(jump)
      101    0.000    0.000    0.002    0.000 /usr/lib/python3.8/genericpath.py:16(exists)
       15    0.000    0.000    0.001    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/datastructures.py:1007(extend)
      505    0.000    0.000    0.000    0.000 /usr/lib/python3.8/codecs.py:331(getstate)
        2    0.000    0.000    0.002    0.001 /usr/lib/python3.8/re.py:315(_compile_repl)
      101    0.000    0.000    0.005    0.000 /usr/lib/python3/dist-packages/redis/connection.py:306(can_read)
      101    0.000    0.000    0.001    0.000 /usr/lib/python3.8/ipaddress.py:609(__str__)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap_external>:586(_validate_timestamp_pyc)
        1    0.000    0.000    0.001    0.001 /usr/local/lib/python3.8/dist-packages/werkzeug/routing/map.py:246(bind_to_environ)
      528    0.000    0.000    0.000    0.000 {method 'split' of 'str' objects}
        7    0.000    0.000    0.019    0.003 /usr/lib/python3.8/sre_compile.py:759(compile)
      101    0.000    0.000    0.006    0.000 /usr/lib/python3/dist-packages/intelmq/lib/message.py:79(serialize)
      204    0.000    0.000    0.000    0.000 {method 'endswith' of 'bytes' objects}
      101    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/events.py:112(__init__)
      101    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/parser.py:129(resolver)
        1    0.000    0.000    0.002    0.002 /usr/local/lib/python3.8/dist-packages/flask/ctx.py:356(push)
        3    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap_external>:1498(find_spec)
      101    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/parser.py:118(dispose)
      202    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:487(<dictcomp>)
      202    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:1063(<genexpr>)
        1    0.000    0.000    0.001    0.001 /root/intelmq-webinput-csv/intelmq_webinput_csv/bin/backend.py:145(<listcomp>)
        1    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/wrappers/request.py:99(__init__)
      299    0.000    0.000    0.000    0.000 /usr/lib/python3.8/sre_parse.py:172(append)
       61    0.000    0.000    0.000    0.000 /usr/lib/python3.8/sre_compile.py:249(_compile_charset)
      101    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/intelmq/lib/harmonization.py:1027(sanitize)
      101    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/main.py:207(composer)
      101    0.000    0.000    0.001    0.000 /usr/lib/python3/dist-packages/intelmq/lib/message.py:133(__setitem__)
      237    0.000    0.000    0.000    0.000 {method 'decode' of 'bytes' objects}
        7    0.000    0.000    0.001    0.000 /usr/lib/python3.8/sre_compile.py:536(_compile_info)
      103    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/redis/connection.py:255(purge)
       30    0.000    0.000    0.000    0.000 {method 'sort' of 'list' objects}
        1    0.000    0.000    0.012    0.012 /usr/local/lib/python3.8/dist-packages/werkzeug/wrappers/request.py:251(_load_form_data)
        1    0.000    0.000    0.002    0.002 /usr/local/lib/python3.8/dist-packages/werkzeug/routing/map.py:488(match)
       14    0.000    0.000    0.001    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/sansio/multipart.py:236(_parse_headers)
      101    0.000    0.000    0.000    0.000 /usr/lib/python3.8/ipaddress.py:1187(_string_from_ip_int)
      102    0.000    0.000    0.001    0.000 /usr/lib/python3/dist-packages/intelmq/lib/utils.py:626(__next__)
      102    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/reader.py:87(reset_reader)
        1    0.000    0.000    0.001    0.001 /root/intelmq-webinput-csv/intelmq_webinput_csv/bin/backend.py:134(handle_parameters)
     1212    0.000    0.000    0.000    0.000 {method 'isascii' of 'str' objects}
        7    0.000    0.000    0.014    0.002 /usr/lib/python3.8/sre_parse.py:937(parse)
      231    0.000    0.000    0.000    0.000 /usr/lib/python3.8/sre_parse.py:160(__len__)
        4    0.000    0.000    0.004    0.001 /usr/lib/python3.8/_strptime.py:238(pattern)
     1212    0.000    0.000    0.000    0.000 {method 'isspace' of 'str' objects}
      102    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/parser.py:108(reset_parser)
      168    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/redis/client.py:90(__setitem__)
      101    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/dateutil/tz/tz.py:206(utcoffset)
      101    0.000    0.000    0.000    0.000 /usr/lib/python3.8/ipaddress.py:158(_split_optional_netmask)
      118    0.000    0.000    0.000    0.000 /usr/lib/python3.8/sre_parse.py:111(__init__)
      851    0.000    0.000    0.000    0.000 {method 'strip' of 'str' objects}
     20/1    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/routing/matcher.py:58(_update_state)
      101    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/redis/client.py:84(__getitem__)
      101    0.000    0.000    0.000    0.000 /usr/lib/python3.8/ipaddress.py:1099(_make_netmask)
        1    0.000    0.000    0.000    0.000 /usr/lib/python3.8/_strptime.py:107(__calc_date_time)
        1    0.000    0.000   34.821   34.821 /usr/local/lib/python3.8/dist-packages/flask/app.py:2495(wsgi_app)
       48    0.000    0.000    0.000    0.000 /usr/lib/python3.8/_strptime.py:234(<genexpr>)
        3    0.000    0.000    0.000    0.000 /usr/lib/python3.8/typing.py:659(__init__)
      202    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:1112(_parsems)
      707    0.000    0.000    0.000    0.000 /usr/lib/python3.8/ipaddress.py:574(__int__)
       17    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/datastructures.py:1090(add)
      510    0.000    0.000    0.000    0.000 {method 'seek' of '_io.BytesIO' objects}
        1    0.000    0.000    0.001    0.001 <frozen importlib._bootstrap_external>:914(get_code)
        1    0.000    0.000    0.001    0.001 /usr/local/lib/python3.8/dist-packages/werkzeug/routing/matcher.py:67(match)
      101    0.000    0.000    0.001    0.000 /usr/lib/python3.8/base64.py:51(b64encode)
       43    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/datastructures.py:890(__getitem__)
      545    0.000    0.000    0.000    0.000 {method 'upper' of 'str' objects}
        1    0.000    0.000    0.000    0.000 /usr/lib/python3.8/_strptime.py:90(<listcomp>)
      267    0.000    0.000    0.000    0.000 {built-in method builtins.min}
        6    0.000    0.000    0.000    0.000 {built-in method builtins.sorted}
      101    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/intelmq/lib/message.py:150(__delitem__)
        7    0.000    0.000    0.005    0.001 /usr/lib/python3.8/sre_compile.py:598(_code)
      102    0.000    0.000    0.009    0.000 /usr/lib/python3/dist-packages/redis/_compat.py:7(sendall)
        1    0.000    0.000    0.001    0.001 /usr/local/lib/python3.8/dist-packages/flask/ctx.py:300(__init__)
        1    0.000    0.000    0.008    0.008 /usr/lib/python3.8/_strptime.py:1(<module>)
        1    0.000    0.000   34.818   34.818 /usr/local/lib/python3.8/dist-packages/flask/app.py:1798(full_dispatch_request)
      235    0.000    0.000    0.000    0.000 {method 'find' of 'bytearray' objects}
      101    0.000    0.000    0.001    0.000 /usr/lib/python3/dist-packages/redis/connection.py:534(connect)
      204    0.000    0.000    0.000    0.000 {method 'readline' of '_io.BytesIO' objects}
      101    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/redis/client.py:503(<lambda>)
      606    0.000    0.000    0.000    0.000 {method 'is_finite' of 'decimal.Decimal' objects}
       17    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/datastructures.py:919(get)
        1    0.000    0.000    0.001    0.001 /usr/lib/python3.8/_strptime.py:88(__calc_month)
       47    0.000    0.000    0.000    0.000 /usr/lib/python3.8/sre_parse.py:432(_uniq)
       22    0.000    0.000    0.000    0.000 /usr/lib/python3.8/typing.py:762(__setattr__)
      137    0.000    0.000    0.000    0.000 /usr/lib/python3.8/sre_parse.py:286(tell)
        1    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/routing/map.py:180(bind)
      101    0.000    0.000    0.000    0.000 /usr/lib/python3.8/base64.py:34(_bytes_from_decode_data)
        1    0.000    0.000    0.001    0.001 /usr/lib/python3.8/_strptime.py:80(__calc_weekday)
      203    0.000    0.000    0.000    0.000 /usr/lib/python3.8/ipaddress.py:417(_check_int_address)
      111    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/local.py:293(bind_f)
        9    0.000    0.000    0.000    0.000 /usr/lib/python3.8/sre_parse.py:224(__init__)
       26    0.000    0.000    0.000    0.000 /usr/lib/python3.8/calendar.py:58(__getitem__)
      406    0.000    0.000    0.000    0.000 {method 'end' of 're.Match' objects}
      101    0.000    0.000    0.001    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:1234(_assign_tzname)
      102    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/redis/_compat.py:169(byte_to_chr)
        6    0.000    0.000    0.000    0.000 /usr/lib/python3.8/typing.py:120(_type_check)
    11/10    0.000    0.000    0.001    0.000 /usr/lib/python3.8/typing.py:255(inner)
       43    0.000    0.000    0.000    0.000 /usr/lib/python3.8/sre_parse.py:168(__setitem__)
        1    0.000    0.000    0.000    0.000 {built-in method io.open_code}
       10    0.000    0.000    0.000    0.000 /usr/lib/python3.8/sre_parse.py:84(opengroup)
       33    0.000    0.000    0.000    0.000 /usr/lib/python3.8/sre_compile.py:423(_simple)
        1    0.000    0.000   34.822   34.822 /usr/local/lib/python3.8/dist-packages/werkzeug/middleware/profiler.py:102(runapp)
        1    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/wrappers/response.py:600(get_wsgi_response)
      304    0.000    0.000    0.000    0.000 {method 'keys' of 'dict' objects}
       10    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/routing/map.py:368(<lambda>)
      9/7    0.000    0.000    0.000    0.000 /usr/lib/python3.8/sre_compile.py:461(_get_literal_prefix)
       48    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/datastructures.py:21(iter_multi_items)
      102    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/ruamel/yaml/main.py:167(reader)
      404    0.000    0.000    0.000    0.000 {method 'toordinal' of 'datetime.date' objects}
      329    0.000    0.000    0.000    0.000 {method 'items' of 'dict' objects}
       16    0.000    0.000    0.000    0.000 /usr/lib/python3.8/enum.py:313(__call__)
        1    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/wsgi.py:136(get_input_stream)
        1    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/redis/connection.py:609(on_connect)
        1    0.000    0.000    0.001    0.001 /usr/local/lib/python3.8/dist-packages/werkzeug/routing/map.py:355(update)
      202    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:366(convertyear)
       19    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/datastructures.py:363(__getitem__)
      303    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/dateutil/tz/tz.py:250(_isdst)
        7    0.000    0.000    0.000    0.000 /usr/lib/python3.8/enum.py:938(__and__)
        1    0.000    0.000    0.000    0.000 /usr/lib/python3.8/_strptime.py:91(<listcomp>)
        1    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/flask/app.py:2289(preprocess_request)
        1    0.000    0.000    0.000    0.000 /usr/lib/python3.8/inspect.py:171(_has_code_flag)
       47    0.000    0.000    0.000    0.000 {built-in method fromkeys}
       16    0.000    0.000    0.001    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/datastructures.py:885(__init__)
       19    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/datastructures.py:238(get)
        2    0.000    0.000    0.001    0.000 /usr/local/lib/python3.8/dist-packages/flask/app.py:2192(create_url_adapter)
       44    0.000    0.000    0.000    0.000 /usr/lib/python3.8/re.py:270(escape)
       15    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/datastructures.py:1075(__contains__)
       29    0.000    0.000    0.000    0.000 {method 'search' of 're.Pattern' objects}
        2    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/sansio/request.py:344(_parse_content_type)
       44    0.000    0.000    0.000    0.000 {method 'translate' of 'str' objects}
      103    0.000    0.000    0.000    0.000 {method 'truncate' of '_io.BytesIO' objects}
        7    0.000    0.000    0.000    0.000 /usr/lib/python3.8/sre_compile.py:492(_get_charset_prefix)
        1    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/routing/map.py:382(__init__)
        1    0.000    0.000    0.012    0.012 /usr/local/lib/python3.8/dist-packages/werkzeug/formparser.py:138(wrapper)
      102    0.000    0.000    0.000    0.000 /usr/lib/python3.8/codecs.py:260(__init__)
        1    0.000    0.000    0.001    0.001 /usr/lib/python3/dist-packages/redis/client.py:641(__init__)
       13    0.000    0.000    0.018    0.001 /usr/lib/python3.8/re.py:250(compile)
        1    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/wrappers/response.py:478(get_wsgi_headers)
       16    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap_external>:121(_path_join)
        4    0.000    0.000    0.000    0.000 /usr/lib/python3.8/sre_compile.py:411(_mk_bitmap)
      101    0.000    0.000    0.000    0.000 {method 'to_bytes' of 'int' objects}
        1    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/sansio/response.py:110(__init__)
        1    0.000    0.000    0.001    0.001 /usr/lib/python3/dist-packages/intelmq/lib/pipeline.py:201(connect)
        1    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/wrappers/request.py:236(make_form_data_parser)
      206    0.000    0.000    0.000    0.000 {built-in method builtins.callable}
       17    0.000    0.000    0.000    0.000 /usr/lib/python3.8/sre_parse.py:295(_class_escape)
      305    0.000    0.000    0.000    0.000 {built-in method builtins.ord}
      202    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:407(has_year)
        1    0.000    0.000    0.001    0.001 <frozen importlib._bootstrap_external>:1367(_get_spec)
       14    0.000    0.000    0.000    0.000 {built-in method builtins.any}
        1    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/flask/ctx.py:242(push)
        8    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/exceptions.py:199(__init__)
        1    0.000    0.000    0.001    0.001 /usr/lib/python3.8/_strptime.py:152(__calc_timezone)
        1    0.000    0.000    0.000    0.000 {method 'connect' of '_socket.socket' objects}
        1    0.000    0.000    0.001    0.001 <frozen importlib._bootstrap>:890(_find_spec)
        1    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/redis/connection.py:627(disconnect)
        2    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap_external>:354(cache_from_source)
        1    0.000    0.000    0.012    0.012 /usr/local/lib/python3.8/dist-packages/werkzeug/formparser.py:230(parse)
       80    0.000    0.000    0.000    0.000 /usr/lib/python3.8/json/decoder.py:343(raw_decode)
        1    0.000    0.000    0.002    0.002 /usr/local/lib/python3.8/dist-packages/flask/ctx.py:346(match_request)
        6    0.000    0.000    0.001    0.000 /usr/lib/python3.8/_strptime.py:219(__seqToRE)
        2    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/wsgi.py:948(read)
        1    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/wrappers/response.py:571(get_app_iter)
        1    0.000    0.000    0.000    0.000 /usr/lib/python3.8/re.py:331(filter)
        1    0.000    0.000    0.012    0.012 /usr/local/lib/python3.8/dist-packages/flask/wrappers.py:111(_load_form_data)
        7    0.000    0.000    0.000    0.000 {built-in method _sre.compile}
        2    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/wsgi.py:671(_make_chunk_iter)
        1    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/wsgi.py:93(get_host)
      113    0.000    0.000    0.000    0.000 {function MultiDict.__getitem__ at 0x7f1f890a9310}
       14    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/formparser.py:369(get_part_charset)
       10    0.000    0.000    0.001    0.000 /usr/lib/python3.8/sre_parse.py:96(closegroup)
      101    0.000    0.000    0.000    0.000 /usr/lib/python3.8/ipaddress.py:577(__eq__)
       30    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/http.py:216(unquote_header_value)
       57    0.000    0.000    0.000    0.000 {built-in method builtins.max}
        3    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/datastructures.py:1373(__getitem__)
        1    0.000    0.000    0.010    0.010 <frozen importlib._bootstrap>:650(_load_unlocked)
        1    0.000    0.000    0.000    0.000 {function UTC.fromutc at 0x7f1f889529d0}
        1    0.000    0.000    0.011    0.011 <frozen importlib._bootstrap>:986(_find_and_load)
       21    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/_internal.py:130(_to_str)
      103    0.000    0.000    0.000    0.000 {method 'split' of 'bytes' objects}
        1    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/redis/connection.py:488(__init__)
      2/1    0.000    0.000    0.000    0.000 /usr/lib/python3.8/typing.py:347(__getitem__)
        1    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/flask/app.py:2402(app_context)
        1    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/flask/helpers.py:163(make_response)
        1    0.000    0.000    0.008    0.008 /usr/local/lib/python3.8/dist-packages/werkzeug/sansio/multipart.py:86(__init__)
        1    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/flask/sessions.py:71(__init__)
      202    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/dateutil/parser/_parser.py:192(__iter__)
        1    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/flask/ctx.py:385(pop)
        1    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap_external>:638(_compile_bytecode)
      101    0.000    0.000    0.000    0.000 {method 'remove' of 'set' objects}
        2    0.000    0.000    0.000    0.000 /usr/lib/python3.8/typing.py:818(__getitem_inner__)
        1    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/intelmq/lib/pipeline.py:225(set_queues)
        1    0.000    0.000    0.001    0.001 /usr/local/lib/python3.8/dist-packages/flask/app.py:2423(request_context)
        1    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/flask/sessions.py:158(make_null_session)
        3    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/flask/wrappers.py:75(blueprint)
        1    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/intelmq/lib/harmonization.py:507(generate_datetime_now)
      101    0.000    0.000    0.000    0.000 /usr/lib/python3/dist-packages/redis/connection.py:167(length)
        6    0.000    0.000    0.000    0.000 /usr/lib/python3.8/sre_parse.py:267(getuntil)
        1    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/routing/matcher.py:53(update)
        3    0.000    0.000    0.000    0.000 {method 'read' of '_io.BufferedReader' objects}
       42    0.000    0.000    0.000    0.000 /usr/local/lib/python3.8/dist-packages/werkzeug/datastructures.py:840(_unicodify_header_value)
rkokkelk added a commit to rkokkelk/intelmq-webinput-csv that referenced this issue Oct 27, 2022
@rkokkelk
fix: extensive loading harmonization config
694badc 
When `Event` is init, it automatically loads the harmonization config
file. This means that for every event uploaded, the entire harmonization
config file is loaded and parsed. This dramatically extends the time
needed to parse a single event.

This PR ensures that the harmonization file is only loaded once and then
used for generating all `Event` objects.

Fixes: certat#79
@rkokkelk rkokkelk mentioned this issue Oct 27, 2022
Merged
@rkokkelk
Copy link
Contributor Author

rkokkelk commented Oct 27, 2022 •
edited
Loading

The PR resulted in decreasing the overall runtime from 34s for 100 records to 120s for 20.000 records.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: extensive loading harmonization config rkokkelk/intelmq-webinput-csv
1 participant
@rkokkelk