As of June 17th, 2021, we are imposing a limit of 10,000 submitted jobs per user. Previously, there had been no limit, and this has caused issues in the past where the cluster scheduler was overloaded when there were 100s of thousands of jobs in the queue. Going forward, if you try to submit more than 10,000 jobs, you will receive the following error:
Unable to run job: job rejected: only 10000 jobs are allowed per user (current job count: 10000)
You will need to either submit your jobs in smaller batches, or, preferably, use an array job to submit your jobs. Arrays jobs can be used to submit multiple instances of the same script where different arguments or data is used for each instance. Please see https://jhpce.jhu.edu/question/how-do-i-run-array-jobs-on-the-jhpce-cluster for more details and examples.