AQUA FIELD GUIDE
Aqua field guide · Notifications

Get a ping when
your job is done.

Phone notifications when your Aqua jobs start, finish, fail or get held, so you stop checking the queue by hand.

Why notifications?

Let the jobs tell you.

Instead of checking the queue yourself, get a push notification on your phone when a job starts, finishes, fails or gets held.

  1. A small program runs on your computer

    Every 3 minutes it asks Aqua for your jobs’ state with qstat, over one shared SSH connection.

  2. It notices what changed

    It keeps a local history and compares each check with the last one, so you hear about each change once. A 30-task array sends one summary, not 30 messages.

  3. Your phone gets a push

    Through ntfy, a free notification service with iOS and Android apps. No account needed.

This is optional.

You can always check jobs yourself, ideally by reading status files through a mount. Notifications suit long jobs and big arrays you don’t want to keep checking.

Set up once

Install hpc-watch.

hpc-watch-ntfy is a small, dependency-free, open-source (MIT) Python tool from the same community toolkit. Its setup script explains every step as it goes.

You need key-based SSH to Aqua (no password prompt) and Python 3. The installer sets up a Linux systemd timer; on other systems you can run ./collector.py poll on your own schedule.

  1. Get it and run the setup

    Your computer · Linux 
    git clone https://github.com/margauxmedwards/hpc-watch-ntfy.git ~/hpc-watch
cd ~/hpc-watch
./setup.sh --explain   # read the walkthrough first (changes nothing)
./setup.sh             # then set it up for real

  2. Subscribe on your phone

    Install the ntfy app (App Store, Google Play or F-Droid), tap Subscribe to topic, and enter the topic name the setup script generated for you.

  3. Check it works

    Your computer · from ~/hpc-watch 
    ./collector.py test-push   # one test notification to your phone
./collector.py doctor      # checks SSH, qstat, ntfy and the timer
./collector.py status      # what it currently sees
Your topic name is a password.

On the public ntfy server, anyone who knows the topic can read your notifications (job names, node names, exit codes). Keep the long random name the setup script generates, and don’t share it.

What you’ll hear about

Only the changes that matter.

When a job… Notification
starts running Normal, with the node it landed on
finishes with exit code 0 Normal
finishes with any other exit code High priority
is held High priority
prints an error in its log (optional) High priority, even while still running

Queued jobs are recorded but don’t notify. ./collector.py report also compares the time and memory you requested with what jobs actually used, which helps you ask for less and start sooner.

Optional

Say what the job is doing.

Point hpc-watch at your log folders (HPC_WATCH_LOG_DIRS) and it can also report progress and catch errors that still exit 0. A few lines in the job script make that clear:

Inside job.pbs · after redirecting output 
source "$HOME/hpc-watch/contrib/hpcwatch.sh"
hpcwatch_init "resnet50_cifar10_seed0"
hpcwatch_stage "training resnet50 on cifar10"
python train.py || hpcwatch_fail "resnet50/cifar10"

These helpers only print lines to the job’s log. A job that uses them still runs normally when hpc-watch isn’t installed.

Good habits

One watcher is enough.

  • hpc-watch shares one SSH connection and checks every 3 minutes. Don’t poll faster, and don’t run other SSH polling loops alongside it.
  • If you use an AI agent, tell it hpc-watch is running so it doesn’t start its own watcher.
  • Your computer must be awake and able to reach Aqua (VPN if you need it). If it sleeps, you get the missed changes when it wakes.