Skip to content

Threshold

Jump to bottom
Ally Rice edited this page Apr 3, 2023 · 2 revisions

How to determine the threshold

For the Bm-Bcall-Detector

Description

After creating a groundtruth and generating a kernel, you must determine the optimal detector threshold for your dataset. Only then can you run the detector on the full dataset.

You will...

  • Run the detector on one xwav file at incremental thresholds (💡 Tip: this will be the xwav that covers the same time period as your groundtruth)
  • Compare the detector output at these thresholds to your groundtruth
  • Choose the threshold that has the highest precision and recall

Requirements

  • MATLAB 2012b (or later)
  • triton_dev_sherbert folder
  • triton1.93.20160524-B call detector folder
  • DetPickCompare.m

Determine the threshold

  1. Complete Steps 1-12 of running the detector

    • For Step 3 the input directory should contain one xwav file instead of a whole directory and the threshold value should be set to 25

    • If you get the error

      Undefined function or variable 'Detections'

      make sure MATLAB is looking in the right place for nilus.jar by modifying your javaclasspath. If you don't know what nilus.jar is, follow the set up directions for the Bm-Bcall-Detector.

  2. Once Matlab has finished running, open the xls containing the automatic detections

  3. Convert the cells containing detections (first column) to a time format

  4. Using the first and last detection times you noted from your groundtruth, delete all of the detections that come before and after your manual picks

  5. Convert the remaining detections to a number format

    excel-num-lg

  6. Save the excel sheet to replace the unedited detector output

  7. The optimal threshold should be somewhere from 25-45. Continue to run the detector on the same xwav, increasing the threshold by 2 each time you run it

  8. Once you have run through all the thresholds, edited each xls file so that it contains only detections within the timespan of your manual groundtruth, and converted all detections into a number format, open DetPickCompare.m

  9. At line 6 you can enter the location of your disk or the location of the folder containing your manual picks

    DPC

  10. Run DetPickCompare

    • When prompted to select an xls file with manual picks, select the xls file that is the groundtruth for the dataset you are working with
    • When prompted to select a directory of xls files with detections at various thresholds, select the folder where you have all of the files at different thresholds
    • Your manual picks will be compared to the detector picks. The number of missed calls, true calls, and false calls will be output in the command window
    • An ROC curve of the different thresholds will pop up

  11. Determine which threshold is optimal using the ROC curve and the precision and recall values produced by MATLAB (values will appear in the workspace window in Matlab. Double click P_prec and P_rec to see these values)

    • Ideally, precision and recall will both be 0.85 or higher (try to stay close to that – higher is better)

    • When choosing the optimal threshold, keep in mind that you want to have a balance between precision and recall

      ROC

  12. Save your ROC curve (either in its own folder or in the folder containing the different thresholds) and make a note somewhere of the exact precision and recall values of the threshold you chose

Next steps

You are now ready to run the detector on your full dataset!

Clone this wiki locally