import: Optimize importer overall processing speed

[lonng]

What have you changed? (mandatory)

Refactor concurrency mode of ImportJob

Implement the preemptive concurrency mode.

• Before

  parallel for each range of ranges {
      create sst channel
      create import threads
      parallel for each thread of import threads do {
          recv sst from sst channel
          upload sst
          ingest sst
      }
      loop {
          open SSTStreamFile
          for each sst from stream {
              send sst to sst channel
          }
      }
  }
  

• After

  create dispatch range channel
  create dispatch thread do {
      for each range of ranges {
          send range to range channel
      }
  }
  create sst channel
  create split threads
  parallel for each thread of split threads do {
      recv range from range channel
      open SSTStreamFile
      for each sst from stream {
          send sst to sst channel
      }
  }
  create import threads
  parallel for each thread of import threads do {
      recv sst from sst channel
      upload sst
      ingest sst
  }
  
  

Memory optimization

• Avoid reading the entire file into memory.
• Use disk based Env instead of memory based.

Redesign retry range strategy

Retry entire range when any sst import failed.

Implement a speed limit for tikv-importer

Implemented a maximum speed limit for uploading from Importer to TiKV, using Token Bucket algorithm. The speed limit is needed to avoid saturating the network bandwidth which causes PD to assume TiKV nodes went down due to heartbeat not going through.

What are the type of the changes? (mandatory)

The currently defined types are listed below, please pick one of the types for this PR by removing the others:

• New feature (change which adds functionality)
• Improvement (change which is an improvement to an existing feature)
• Bug fix (change which fixes an issue)
• Misc (other changes)

How has this PR been tested? (mandatory)

Manual tested.

Does this PR affect documentation (docs) update? (mandatory)

No.

Does this PR affect tidb-ansible update? (mandatory)

No.

Refer to a related PR or issue link (optional)

Benchmark result if necessary (optional)

Add a few positive/negative examples (optional)

[lonng]

Related unexpected memory usage issue: #4257

[lonng]

@DorianZheng Please remove the S:DNM label, thanks.

[zhouqiang-cl]

/test

[kennytm]

@huachaohuang PTAL

[huachaohuang]

This PR is a combination of multiple optimizations, we should split it into multiple single-purpose PRs.

[kennytm]

@huachaohuang if we split this into 8 PRs, do we also file 8 additional cherrypick-to-2.1 PRs or a single one combining all 8?

[huachaohuang]

@kennytm cherry picks can be combined into one PR.

[kennytm]

@huachaohuang I've split this into 5 PRs (see the references above). There are 2 remaining PRs which rely on #4349, I'll file them after #4349 is merged.

[lonng]

All extracted PRs merged.