Improve Auto-Analyze Mechanism for Better Concurrency

[Rustin170506]

Background

The current implementation of AnalyzeHighestPriorityTables uses a WaitGroupWrapper to execute analysis jobs concurrently. This approach lacks flexibility in managing running jobs.

Proposed Changes

1. Implement a dynamic job management system that maintains a constant number of running jobs up to the maximum concurrency limit.
2. Add functionality to track currently running analysis jobs by table ID.
3. Allow for dynamic adjustment of concurrency based on the AutoAnalyzeConcurrency parameter.
4. Replace the WaitGroupWrapper with individual goroutines for each job.

Implementation Details

1. Add a runningJobs map to the Refresher struct to track active jobs.
2. Modify AnalyzeHighestPriorityTables to:
   • Check the current number of running jobs before starting new ones.
   • Use goroutines for each job instead of a WaitGroupWrapper.
   • Dynamically start new jobs as others complete, maintaining the desired concurrency.
3. Implement proper locking mechanisms to ensure thread-safety when managing the job queue and running jobs.
4. Add a method to get the current running jobs.

Expected Benefits

• More efficient use of system resources by maintaining a constant number of running jobs.
• Improved flexibility in managing analysis jobs, allowing for better responsiveness to system changes.
• Enhanced ability to monitor and control the auto-analyze process.
• Prepare for the new priority queue.