Исследование устойчивости моделей интерактивной сегментации к состязательным кликовым подсказкам
Авторы
-
А. В. Москаленко
-
В. И. Шахуро
-
Д. С. Ватолин
Ключевые слова:
компьютерное зрение
интерактивная сегментация
сегментация с подсказками
кликовые подсказки
состязательные атаки
оценка устойчивости
Аннотация
Модели интерактивной сегментации широко используются для выделения объектов и ускорения разметки данных с помощью пользовательских подсказок. Наиболее простым, интуитивным и распространенным в большинстве систем разметки типом таких подсказок являются клики по интересующим объектам. Однако большинство существующих подходов к обучению и тестированию таких моделей опирается на нереалистичные искусственно сгенерированные кликовые подсказки, тогда как влияние реальных пользовательских вводов остается недостаточно изученным. На основе краудсорсингового исследования с участием 1800 пользователей в этой работе мы показали, что стандартные стратегии генерации кликов не всегда соответствуют реальному пользовательскому поведению, а высокое качество на традиционных тестах не гарантирует надежной работы в практических условиях. Полный перебор возможных положений пользовательских вводов мог бы решить проблему оценки устойчивости моделей, однако он требует колоссальных вычислительных ресурсов и неосуществим за разумное время. Поэтому мы переформулировали задачу как эффективную состязательную атаку белого ящика на модель интерактивной сегментации. Мы предложили метрики оценки устойчивости и с их помощью провели экспериментальное сравнение моделей на популярных наборах данных.
Раздел
Методы и алгоритмы вычислительной математики и их приложения
Библиографические ссылки
- C. Rother, V. Kolmogorov, and A. Blake, “GrabCut – Interactive Foreground Extraction Using Iterated Graph Cuts,” ACM Transactions on Graphics (TOG) 23 (3), 309–314 (2004).
doi 10.1145/1186562.1015720
- D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A Database of Human Segmented Natural Images and Its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics,” in Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, Vancouver, BC, Canada, July 07–14, 2001(IEEE Press, New York, 2001), pp. 416–423.
doi 10.1109/ICCV.2001.937655
- K. McGuinness and N. E. O’Connor, “A Comparative Evaluation of Interactive Segmentation Algorithms,” Pattern Recognition 43 (2), 434–444 (2010).
doi 10.1016/j.patcog.2009.03.008
- N. Xu, B. Price, S. Cohen, et al., “Deep Interactive Object Selection,” in Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), Las Vegas, Nevada, USA, June 26–July 1, 2016(IEEE Press, New York, 2016), pp. 373–381.
doi 10.1109/CVPR.2016.47
- M. Everingham, L. Van Gool, C. K. I. Williams, et al., The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results(2012).
- T.-Y. Lin, M. Maire, S. Belongie, et al., “Microsoft COCO: Common Objects in Context,” in Proceedings 13th European Conference. Computer Vision – ECCV 2014, Zurich, Switzerland, September 6–12, 2014.ECCV 2014. Lecture Notes in Computer Science, Vol. 8693. (Springer, Cham, 2014), pp. 740–755.
doi 10.1007/978-3-319-10602-1_48
- Z. Li, Q. Chen, and V. Koltun, “Interactive Image Segmentation with Latent Diversity,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, June 18–23, 2018(IEEE Press, 2018), pp. 577–585.
doi 10.1109/CVPR.2018.00067
- F. Perazzi, J. Pont-Tuset, B. McWilliams, et al., “A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, June 27–30, 2016(IEEE Press, 2016), pp. 724–732.
doi 10.1109/CVPR.2016.85
- B. Hariharan, P. Arbelaez, L. Bourdev, et al., “Semantic Contours from Inverse Detectors,” in 2011 International Conference on Computer Vision, Barcelona, Spain, November 06–13, 2011(IEEE Press, 2011), pp. 991–998.
doi 10.1109/ICCV.2011.6126343
- W.-D. Jang and C.-S. Kim, “Interactive Image Segmentation via Backpropagating Refinement Scheme,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, June 15–20, 2019(IEEE Press, 2019), pp. 5292-5301.
doi 10.1109/CVPR.2019.00544
- K. Sofiiuk, I. Petrov, O. Barinova, and A. Konushin, “f-BRS: Rethinking Backpropagating Refinement for Interactive Segmentation,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, June 13–19, 2020(IEEE Press, 2020), pp. 8620–8629.
doi 10.1109/CVPR42600.2020.00865
- K. Sofiiuk, I. A. Petrov, and A. Konushin, “Reviving Iterative Training with Mask Guidance for Interactive Segmentation,” in 2022 IEEE International Conference on Image Processing (ICIP), Bordeaux, France, October 16–19, 2022(IEEE Press, 2022), pp. 3141–3145.
doi 10.1109/ICIP46576.2022.9897365
- B. Cheng, R. Girshick, P. Dollár, et al., “Boundary IoU: Improving Object-Centric Image Segmentation Evaluation,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, June 20–25, 2021(IEEE Press, 2021), pp. 15329–15337.
doi 10.1109/CVPR46437.2021.01508
- H.-E. Gueziri, M. McGuffin, and C. Laporte, “Latency Management in Scribble-Based Interactive Segmentation of Medical Images,” IEEE Transactions on Biomedical Engineering 65 (5), 1140–1150 (2018).
doi 10.1109/TBME.2017.2777742
- E. Agustsson, J. R. R. Uijlings, and V. Ferrari, “Interactive Full Image Segmentation by Considering All Regions Jointly,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, June 15–20, 2019(IEEE Press, 2019), pp. 11614–11623.
doi 10.1109/CVPR.2019.01189
- D. P. Papadopoulos, J. R. R. Uijlings, F. Keller, and V. Ferrari, “Extreme Clicking for Efficient Object Annotation,” in 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, October 22–29, 2017(IEEE Press, 2017), pp. 4940–4949.
doi 10.1109/ICCV.2017.528
- H. Cheng, S. Xu, X. Jiang, and R. Wang, “Deep Image Matting with Flexible Guidance Input,” in Paper presented at 32nd British Machine Vision Conference, BMVC 2021, Virtual, Online
doi 10.5244/C.35.198
- A. Kirillov, E. Mintun, N. Ravi, et al., “Segment Anything,” in 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France, October 01–06, 2023(IEEE Press, 2023), pp. 3992–4003.
doi 10.1109/ICCV51070.2023.00371
- C. Kamann and C. Rother, “Benchmarking the Robustness of Semantic Segmentation Models,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, June 13–19, 2020(IEEE Press, 2020), pp. 8825–8835.
doi 10.1109/CVPR42600.2020.00885
- E. A. Vorob’ev and D. E. Namiot, “Adversarial Testing of Image Segmentation Models,” Programmnaya Inzheneriya 16 (4), 190–198 (2025). [in Russian].
doi 10.17587/prin.16.190-198
- N. D. Lokshin, A. V. Khvostikov, and A. S. Krylov, “Augmenting the Training Set of Histological Images with Adversarial Examples,” Programming and Computer Software 49 (3), 187–191 (2023).
doi 10.1134/S0361768823030027
- Z. Guan, M. Hu, Z. Zhou, et al., “BadSAM: Exploring Security Vulnerabilities of SAM via Backdoor Attacks (Student Abstract),” in Proceedings of the 38th AAAI Conference on Artificial Intelligence (AAAI-24), Vancouver, Canada, February 20–27, 2024Vol. 38, Iss. 21. (AAAI Press, Washington, DC, USA, 2024), pp. 23506–23507.
doi 10.1609/aaai.v38i21.30448
- C. Zhang, C. Zhang, T. Kang, et al., “Attack-SAM: Towards Attacking Segment Anything Model with Adversarial Examples,” arXiv preprint arXiv:2305.00866 (2023).
doi 10.48550/arXiv.2305.00866
- Z. Zhou, Y. Song, M. Li, et al., “DarkSAM: Fooling Segment Anything Model to Segment Nothing,” Advances in Neural Information Processing Systems 37, 49859–49880 (2024).
doi 10.52202/079017-1579
- Y. Qiao, C. S. Hong, T. Kang, et al., “Robustness of SAM: Segment Anything under Corruptions and Beyond,” IEEE Transactions on Pattern Analysis and Machine Intelligence 48 (10), 12082–12099 (2026).
doi 10.1109/TPAMI.2026.3698837
- Y. Wang, Y. Zhao, and L. Petzold, “An Empirical Study on the Robustness of the Segment Anything Model (SAM),” Pattern Recognition 155 (12), Article Number 110685 (2024).
doi 10.1016/j.patcog.2024.110685
- W. Brendel, J. Rauber, and M. Bethge, “Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models,” in 6th International Conference on Learning Representations, Vancouver, Canada, April 30–May 3, 2018
https://openreview.net/group?id=ICLR.cc/2018/Conference.
- A. N. Bhagoji, W. He, B. Li, and D. Song, “Practical Black-Box Attacks on Deep Neural Networks Using Efficient Query Mechanisms,” in Proceedings of the European Conference on Computer Vision (ECCV 2018)Lecture Notes in Computer Science, Vol. 11216. (Springer, Cham, 2018), pp. 158–174.
doi 10.1007/978-3-030-01258-8_10
- J. Su, D. V. Vargas, and K. Sakurai, “One Pixel Attack for Fooling Deep Neural Networks,” IEEE Transactions on Evolutionary Computation 23 (5), 828–841 (2019).
doi 10.1109/TEVC.2019.2890858
- X. Chen, Z. Zhao, F. Yu, et al., “Conditional Diffusion for Interactive Segmentation,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, October 10–17, 2021(IEEE Press, 2021), pp. 7325–7334.
doi 10.1109/ICCV48922.2021.00725
- Q. Liu, Z. Xu, G. Bertasius, and M. Niethammer, “SimpleClick: Interactive Image Segmentation with Simple Vision Transformers,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France, October 01–06, 2023(IEEE Press, 2023), pp. 22233–22243.
doi 10.1109/ICCV51070.2023.02037
- S. Sun, M. Xian, F. Xu, et al., “CFR-ICL: Cascade-Forward Refinement with Iterative Click Loss for Interactive Image Segmentation,” in Proceedings of the 38th AAAI Conference on Artificial Intelligence (AAAI-24), Vancouver, Canada, February 20–27, 2024Vol. 38, Iss. 5. (AAAI Press, Washington, DC, USA, 2024), pp. 5017–5024.
doi 10.1609/aaai.v38i5.28306
- M. Zhou, H. Wang, Q. Zhao, et al., “Interactive Segmentation as Gaussian Process Classification,” in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, June 17–24, 2023(IEEE Press, 2023), pp. 19488–19497.
doi 10.1109/CVPR52729.2023.01867
- L. Ke, M. Ye, M. Danelljan, et al., “Segment Anything in High Quality,” Advances in Neural Information Processing Systems 36. 29914–29934 (2023).
doi 10.52202/075280-1303
- C. Zhang, D. Han, Y. Qiao, et al., “Faster Segment Anything: Towards Lightweight SAM for Mobile Applications,” arXiv preprint arXiv:2306.14289 (2023).
doi 10.48550/arXiv.2306.14289
- X. Ma, Y. Zhou, X. Xu, et al., “Towards Layer-Wise Image Vectorization,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, June 18–24, 2022(IEEE Press, 2022), pp. 16293–16302.
doi 10.1109/CVPR52688.2022.01583
- D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” arXiv preprint arXiv:1412.6980 (2014).
doi 10.48550/arXiv.1412.6980
- L. R. Dice, “Measures of the Amount of Ecologic Association between Species,” Ecology 26 (3), 297–302 (1945).
doi 10.2307/1932409