GETTING_STARTED.md

Getting Started with FrozenSeg

This document provides a brief intro of the usage of FrozenSeg.

Please see Getting Started with Detectron2 for full usage.

Inference Demo with Pre-trained Models

We provide demo.py that is able to demo builtin configs. Run it with:

python demo.py \
  --input input1.jpg input2.jpg \
  [--other-options]
  --opts MODEL.WEIGHTS /path/to/checkpoint_file

The configs are made for training, therefore we need to specify MODEL.WEIGHTS to a model from model zoo for evaluation. This command will run the inference and show visualizations in an OpenCV window.

For details of the command line arguments, see demo.py -h or look at its source code to understand its behavior. Some common arguments are:

• To run on your webcam, replace --input files with --webcam
• To run on a video, replace --input files with --video-input video.mp4.
• To run on cpu, add MODEL.DEVICE cpu after --opts.
• To save outputs to a directory (for images) or a file (for webcam or video), use --output.

Training & Evaluation in Command Line

We provide a script train_net.py, that is made to train all the configs provided in FrozenSeg.

To train a model with "train_net.py", first setup the corresponding datasets following datasets/README.md, download SAM checkpoints and save it under pretrained_checkpoint/. then run:

python train_net.py --num-gpus 4\
  --config-file configs/coco/frozenseg/convnext_large_eval_ade20k.yaml

The configs are made for 4-GPU training. Since we use ADAMW optimizer, it is not clear how to scale learning rate with batch size. To train on 1 GPU, you need to figure out learning rate and batch size by yourself:

python train_net.py \
  --config-file configs/coco/frozenseg/convnext_large_eval_ade20k.yaml \
  --num-gpus 1 SOLVER.IMS_PER_BATCH SET_TO_SOME_REASONABLE_VALUE SOLVER.BASE_LR SET_TO_SOME_REASONABLE_VALUE

To evaluate a model's performance without OpenSeg Ensemble:

python train_net.py \
  --config-file configs/coco/frozenseg/convnext_large_eval_ade20k.yaml \
  --eval-only MODEL.WEIGHTS /path/to/checkpoint_file \
  TEST.USE_SAM_MASKS False 

For using OpenSeg Ensemble:

1. generate SAM mask predictions (default saveing under output/SAM_masks_pred):

python save_sam_masks.py --data_name pc_val --sam_model vit_h

2. run with:

python train_net.py \
  --config-file configs/coco/frozenseg/convnext_large_eval_ade20k.yaml \
  --eval-only MODEL.WEIGHTS /path/to/checkpoint_file \
  TEST.USE_SAM_MASKS True