Remove distributedLoad

We will not aim to support this feature in the 2.x line, thus removing the user facing details and entrypoint for this feature.
			pr-link: #18569
			change-id: cid-47e4be64e1810fa58869a762d4b3efb160f52d7d

docs/cn/overview/JobService.md

@@ -18,10 +18,6 @@ Job Worker将来自Job Master的任务排列（queue)，并通过管理可配置
 
 ## 不同类型的作业
 
-### 加载 Load
-
-`fs distributedLoad`CLI命令中使用了加载作业，按特定的副本数将文件加载到Alluxio。
-
 ### 迁移 Migrate
 
 `fs distributedCp`CLI命令中使用了迁移作业，使用固定的[写入类型]({{ '/en/overview/Architecture.html#data-flow-write' | relativize_url }})进行数据复制/移动。

docs/en/operation/User-CLI.md

@@ -843,92 +843,6 @@ Please wait for command submission to finish..
 Submitted migrate job successfully, jobControlId = JOB_CONTROL_ID_2
 ```
 
-### distributedLoad
-
-The `distributedLoad` command loads a file or directory from the under storage system into Alluxio storage distributed
-across workers using the job service. The job is a no-op if the file is already loaded into Alluxio.
-By default, the command runs synchronously and the user will get a `JOB_CONTROL_ID` after the command successfully submits the job to be executed.
-The command will wait until the job is complete, at which point the user will see the list of files loaded and statistics on which files completed or failed.
-The command can also run in async mode with the `--async` flag. Similar to before, the user will get a `JOB_CONTROL_ID` after the command successfully submits the job.
-The difference is that the command will not wait for the job to finish.
-Users can use the [`getCmdStatus`](#getCmdStatus) command with the `JOB_CONTROL_ID` as an argument to check detailed status information about the job.
-
-If `distributedLoad` is run on a directory, files in the directory will be recursively loaded and each file will be loaded
-on a random worker.
-
-Options:
-
-* `--replication`: Specifies how many workers to load each file into. The default value is `1`.
-* `--active-jobs`: Limits how many jobs can be submitted to the Alluxio job service at the same time.
-Later jobs must wait until some earlier jobs to finish. The default value is `3000`.
-A lower value means slower execution but also being nicer to the other users of the job service.
-* `--batch-size`: Specifies how many files to be batched into one request. The default value is `20`. Notice that if some task failed in the batched job, the whole batched job would fail with some completed tasks and some failed tasks.
-* `--host-file <host-file>`: Specifies a file contains worker hosts to load target data, each line has a worker host.
-* `--hosts`: Specifies a list of worker hosts separated by comma to load target data.
-* `--excluded-host-file <host-file>`: Specifies a file contains worker hosts which shouldn't load target data, each line has a worker host.
-* `--excluded-hosts`: Specifies a list of worker hosts separated by comma which shouldn't load target data.
-* `--locality-file <locality-file>`: Specifies a file contains worker locality to load target data, each line has a locality.
-* `--locality`: Specifies a list of worker locality separated by comma to load target data.
-* `--excluded-locality-file <locality-file>`: Specifies a file contains worker locality which shouldn't load target data, each line has a worker locality.
-* `--excluded-locality`: Specifies a list of worker locality separated by comma which shouldn't load target data.
-* `--index`: Specifies a file that lists all files to be loaded
-* `--passive-cache`: Specifies using direct cache request or passive cache with read(old implementation)
-* `--async`: Specifies whether to wait for command execution to finish. If not explicitly shown then default to run synchronously.
-
-```console
-$ ./bin/alluxio fs distributedLoad --replication 2 --active-jobs 2000 /data/today
-Sample Output:
-Please wait for command submission to finish..
-Submitted successfully, jobControlId = JOB_CONTROL_ID_3
-Waiting for the command to finish ...
-Get command status information below:
-Successfully loaded path /data/today/$FILE_PATH_1
-Successfully loaded path /data/today/$FILE_PATH_2
-Successfully loaded path /data/today/$FILE_PATH_3
-Total completed file count is 3, failed file count is 0
-Finished running the command, jobControlId = JOB_CONTROL_ID_3
-```
-
-```console
-# Turn on async submission mode. Run this command to get JOB_CONTROL_ID, then use getCmdStatus to check command detailed status.
-$ ./bin/alluxio fs distributedLoad /data/today --async
-Sample Output:
-Entering async submission mode.
-Please wait for command submission to finish..
-Submitted distLoad job successfully, jobControlId = JOB_CONTROL_ID_4
-```
-
-Or you can include some workers or exclude some workers by using options `--host-file <host-file>`, `--hosts`, `--excluded-host-file <host-file>`,
-`--excluded-hosts`, `--locality-file <locality-file>`, `--locality`, `--excluded-host-file <host-file>` and `--excluded-locality`.
-
-Note: Do not use `--host-file <host-file>`, `--hosts`, `--locality-file <locality-file>`, `--locality` with
-`--excluded-host-file <host-file>`, `--excluded-hosts`, `--excluded-host-file <host-file>`, `--excluded-locality` together.
-
-```console
-# Only include host1 and host2
-$ ./bin/alluxio fs distributedLoad /data/today --hosts host1,host2
-# Only include the workset from host file /tmp/hostfile
-$ ./bin/alluxio fs distributedLoad /data/today --host-file /tmp/hostfile
-# Include all workers except host1 and host2 
-$ ./bin/alluxio fs distributedLoad /data/today --excluded-hosts host1,host2
-# Include all workers except the workerset in the excluded host file /tmp/hostfile-exclude
-$ ./bin/alluxio fs distributedLoad /data/today --excluded-file /tmp/hostfile-exclude
-# Include workers which's locality identify belong to ROCK1 or ROCK2
-$ ./bin/alluxio fs distributedLoad /data/today --locality ROCK1,ROCK2
-# Include workers which's locality identify belong to the localities in the locality file
-$ ./bin/alluxio fs distributedLoad /data/today --locality-file /tmp/localityfile
-# Include all workers except which's locality belong to ROCK1 or ROCK2 
-$ ./bin/alluxio fs distributedLoad /data/today --excluded-locality ROCK1,ROCK2
-# Include all workers except which's locality belong to the localities in the excluded locality file
-$ ./bin/alluxio fs distributedLoad /data/today --excluded-locality-file /tmp/localityfile-exclude
-
-# Conflict cases
-# The `--hosts` and `--locality` are `OR` relationship, so host2,host3 and workers in ROCK2,ROCKS3 will be included.
-$ ./bin/alluxio fs distributedLoad /data/today --locality ROCK2,ROCK3 --hosts host2,host3
-# The `--excluded-hosts` and `--excluded-locality` are `OR` relationship, so host2,host3 and workers in ROCK2,ROCKS3 will be excluded.
-$ ./bin/alluxio fs distributedLoad /data/today --excluded-hosts host2,host3 --excluded-locality ROCK2,ROCK3
-```
-
 ### du
 
 The `du` command outputs the total size and amount stored in Alluxio of files and folders.

docs/en/overview/JobService.md

@@ -21,11 +21,6 @@ The Job Workers queues tasks from the Job Master and manages a configurable fixe
 
 ## Different Types of Jobs
 
-### Load
-
-Load Job is used as part of the `fs distributedLoad` CLI command and loads files to Alluxio 
-with specified amount of replication
-
 ### Migrate
 
 Migrate Job is used as part of the `fs distributedCp` CLI command and is responsible

stress/common/src/main/java/alluxio/stress/jobservice/JobServiceBenchOperation.java

@@ -16,7 +16,6 @@
  */
 public enum JobServiceBenchOperation {
   CREATE_FILES("CreateFiles"),
-  DISTRIBUTED_LOAD("DistributedLoad"),
   NO_OP("NoOp");
 
   private final String mName;

stress/shell/src/main/java/alluxio/stress/cli/StressJobServiceBench.java

@@ -15,8 +15,6 @@
 import alluxio.ClientContext;
 import alluxio.Constants;
 import alluxio.annotation.SuppressFBWarnings;
-import alluxio.cli.fs.command.DistributedLoadCommand;
-import alluxio.cli.fs.command.DistributedLoadUtils;
 import alluxio.client.file.FileOutStream;
 import alluxio.client.file.FileSystem;
 import alluxio.client.file.FileSystemContext;
@@ -51,7 +49,6 @@
 import java.io.IOException;
 import java.util.ArrayList;
 import java.util.Arrays;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.concurrent.Callable;
@@ -247,14 +244,6 @@ private void runInternal() throws Exception {
     private void applyOperation(String dirPath)
         throws IOException, AlluxioException, InterruptedException, TimeoutException {
       switch (mParameters.mOperation) {
-        case DISTRIBUTED_LOAD:
-          mResult.setRecordStartMs(mContext.getStartMs());
-          long startNs = System.nanoTime();
-          // send distributed load task to job service and wait for result
-          long endNs = runDistributedLoad(dirPath);
-          // record response times
-          recordResponseTimeInfo(startNs, endNs);
-          break;
         case CREATE_FILES:
           FileSystem fileSystem = FileSystem.Factory.create(mFsContext);
           long start = CommonUtils.getCurrentMs();
@@ -278,9 +267,9 @@ private void applyOperation(String dirPath)
             long recordMs = mContext.getStartMs() + FormatUtils.parseTimeSize(mParameters.mWarmup);
             mResult.setRecordStartMs(recordMs);
             mContext.getRateLimiter().acquire();
-            startNs = System.nanoTime();
+            long startNs = System.nanoTime();
             runNoop();
-            endNs = System.nanoTime();
+            long endNs = System.nanoTime();
             long currentMs = CommonUtils.getCurrentMs();
             // Start recording after the warmup
             if (currentMs > recordMs) {
@@ -303,24 +292,6 @@ private void recordResponseTimeInfo(long startNs, long endNs) {
       }
     }
 
-    private long runDistributedLoad(String dirPath) {
-      int numReplication = 1;
-      DistributedLoadCommand cmd = new DistributedLoadCommand(mFsContext);
-      long stopTime;
-      try {
-        long jobControlId = DistributedLoadUtils.runDistLoad(cmd, new AlluxioURI(dirPath),
-                numReplication, mParameters.mBatchSize,
-                new HashSet<>(), new HashSet<>(),
-                new HashSet<>(), new HashSet<>(), false);
-        cmd.waitForCmd(jobControlId);
-        stopTime = System.nanoTime();
-        cmd.postProcessing(jobControlId);
-      } finally {
-        mResult.incrementNumSuccess(cmd.getCompletedCount());
-      }
-      return stopTime;
-    }
-
     private void createFiles(FileSystem fs, int numFiles, String dirPath, long fileSize)
         throws IOException, AlluxioException {
       CreateFilePOptions options = CreateFilePOptions.newBuilder()