Last updated Aug 12, 2020 11:58 AM

Advanced Cluster Computing

Presented on 8/12/2020

Advanced Cluster Computing Slides


Commands used during the webinar:

squeue - see the queue status so we can find our jobs and the node(s) they're running on squeue -u $USER

dstat - used for monitoring system stats, output easily written to a file: dstat -tcmsn -N bond0.100

htop - used for a graphical view of CPU and memory usage, as well as a list of processes: htop

Example sbatch scripts

Automatically staging data to /scratch, moving relevant outputs to /Users, removing intermediates on /scratch
        #!/bin/bash
#SBATCH --partition=short
#SBATCH --job-name=rsync_example
#SBATCH --mail-type=ALL
#SBATCH --mail-user=YOU@colorado.edu
#SBATCH --output=/scratch/Users/jode5573/eo/slurm_test_%j.out
#SBATCH --error=/scratch/Users/jode5573/eo/slurm_test_%j.err
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --time=01:00:00
#SBATCH --mem=8G


# Move our entire directory of samples to /scratch 
rsync -r --stats $HOME/samples /scratch/Users/$USER

# Change into working directory, perform work
cd /scratch/Users/$USER/samples
zcat *.fastq.gz >> concat.fastq

# Copy only our results back to /Users space to be backed up 
rsync --stats /scratch/Users/$USER/samples/concat.fastq $HOME/samples

# We don't want ANYTHING to continue to live on /scratch
rm -rf /scratch/Users/$USER/samples

Fiji Cluster
Posted in Category: Cluster Computing