A Safety Framework for Critical Systems Utilising Deep Neural Networks

Xingyu Zhao; Alec Banks; James Sharp; Valentin Robu; David Flynn; Michael Fisher; Xiaowei Huang

doi:10.1007/978-3-030-54549-9_16

A Safety Framework for Critical Systems Utilising Deep Neural Networks

Xingyu Zhao
, Alec Banks
, James Sharp
, Valentin Robu
, David Flynn
, Michael Fisher
, Xiaowei Huang^*

^*Corresponding author for this work

Formal Methods

Heriot-Watt University, Edinburgh
DSTL (Defence Science & Technology Laboratory)
University of Liverpool

Research output: Chapter in Book/Conference proceeding › Conference contribution › peer-review

Abstract

Increasingly sophisticated mathematical modelling processes from Machine Learning are being used to analyse complex data. However, the performance and explainability of these models within practical critical systems requires a rigorous and continuous verification of their safe utilisation. Working towards addressing this challenge, this paper presents a principled novel safety argument framework for critical systems that utilise deep neural networks. The approach allows various forms of predictions, e.g., future reliability of passing some demands, or confidence on a required reliability level. It is supported by a Bayesian analysis using operational data and the recent verification and validation techniques for deep learning. The prediction is conservative – it starts with partial prior knowledge obtained from lifecycle activities and then determines the worst-case prediction. Open challenges are also identified.

Original language	English
Title of host publication	Computer Safety, Reliability, and Security - 39th International Conference, SAFECOMP 2020, Proceedings
Editors	António Casimiro, Pedro Ferreira, Frank Ortmeier, Friedemann Bitsch
Publisher	Springer London
Pages	244-259
Number of pages	16
ISBN (Print)	9783030545482
DOIs	https://doi.org/10.1007/978-3-030-54549-9_16
Publication status	Published - 31 Jul 2020
Event	39th International Conference on Computer Safety, Reliability and Security - Lisbon, Portugal Duration: 16 Sept 2020 → 18 Sept 2020

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12234 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	39th International Conference on Computer Safety, Reliability and Security
Abbreviated title	SAFECOMP 2020
Country/Territory	Portugal
City	Lisbon
Period	16/09/20 → 18/09/20

Keywords

Assurance arguments
Bayesian inference
Deep learning verification
Quantitative claims
Reliability claims
Safe AI
Safety cases

Access to Document

10.1007/978-3-030-54549-9_16

Cite this

Zhao, X., Banks, A., Sharp, J., Robu, V., Flynn, D., Fisher, M., & Huang, X. (2020). A Safety Framework for Critical Systems Utilising Deep Neural Networks. In A. Casimiro, P. Ferreira, F. Ortmeier, & F. Bitsch (Eds.), Computer Safety, Reliability, and Security - 39th International Conference, SAFECOMP 2020, Proceedings (pp. 244-259). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12234 LNCS). Springer London. https://doi.org/10.1007/978-3-030-54549-9_16

Zhao, Xingyu ; Banks, Alec ; Sharp, James et al. / A Safety Framework for Critical Systems Utilising Deep Neural Networks. Computer Safety, Reliability, and Security - 39th International Conference, SAFECOMP 2020, Proceedings. editor / António Casimiro ; Pedro Ferreira ; Frank Ortmeier ; Friedemann Bitsch. Springer London, 2020. pp. 244-259 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{4ff61d1104844bd6a88878390cb1a534,

title = "A Safety Framework for Critical Systems Utilising Deep Neural Networks",

abstract = "Increasingly sophisticated mathematical modelling processes from Machine Learning are being used to analyse complex data. However, the performance and explainability of these models within practical critical systems requires a rigorous and continuous verification of their safe utilisation. Working towards addressing this challenge, this paper presents a principled novel safety argument framework for critical systems that utilise deep neural networks. The approach allows various forms of predictions, e.g., future reliability of passing some demands, or confidence on a required reliability level. It is supported by a Bayesian analysis using operational data and the recent verification and validation techniques for deep learning. The prediction is conservative – it starts with partial prior knowledge obtained from lifecycle activities and then determines the worst-case prediction. Open challenges are also identified.",

keywords = "Assurance arguments, Bayesian inference, Deep learning verification, Quantitative claims, Reliability claims, Safe AI, Safety cases",

author = "Xingyu Zhao and Alec Banks and James Sharp and Valentin Robu and David Flynn and Michael Fisher and Xiaowei Huang",

year = "2020",

month = jul,

day = "31",

doi = "10.1007/978-3-030-54549-9\_16",

language = "English",

isbn = "9783030545482",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer London",

pages = "244--259",

editor = "Ant{\'o}nio Casimiro and Pedro Ferreira and Frank Ortmeier and Friedemann Bitsch",

booktitle = "Computer Safety, Reliability, and Security - 39th International Conference, SAFECOMP 2020, Proceedings",

address = "United Kingdom",

note = "39th International Conference on Computer Safety, Reliability and Security, SAFECOMP 2020 ; Conference date: 16-09-2020 Through 18-09-2020",

}

Zhao, X, Banks, A, Sharp, J, Robu, V, Flynn, D, Fisher, M & Huang, X 2020, A Safety Framework for Critical Systems Utilising Deep Neural Networks. in A Casimiro, P Ferreira, F Ortmeier & F Bitsch (eds), Computer Safety, Reliability, and Security - 39th International Conference, SAFECOMP 2020, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12234 LNCS, Springer London, pp. 244-259, 39th International Conference on Computer Safety, Reliability and Security, Lisbon, Portugal, 16/09/20. https://doi.org/10.1007/978-3-030-54549-9_16

A Safety Framework for Critical Systems Utilising Deep Neural Networks. / Zhao, Xingyu; Banks, Alec; Sharp, James et al.
Computer Safety, Reliability, and Security - 39th International Conference, SAFECOMP 2020, Proceedings. ed. / António Casimiro; Pedro Ferreira; Frank Ortmeier; Friedemann Bitsch. Springer London, 2020. p. 244-259 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12234 LNCS).

Research output: Chapter in Book/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - A Safety Framework for Critical Systems Utilising Deep Neural Networks

AU - Zhao, Xingyu

AU - Banks, Alec

AU - Sharp, James

AU - Robu, Valentin

AU - Flynn, David

AU - Fisher, Michael

AU - Huang, Xiaowei

PY - 2020/7/31

Y1 - 2020/7/31

N2 - Increasingly sophisticated mathematical modelling processes from Machine Learning are being used to analyse complex data. However, the performance and explainability of these models within practical critical systems requires a rigorous and continuous verification of their safe utilisation. Working towards addressing this challenge, this paper presents a principled novel safety argument framework for critical systems that utilise deep neural networks. The approach allows various forms of predictions, e.g., future reliability of passing some demands, or confidence on a required reliability level. It is supported by a Bayesian analysis using operational data and the recent verification and validation techniques for deep learning. The prediction is conservative – it starts with partial prior knowledge obtained from lifecycle activities and then determines the worst-case prediction. Open challenges are also identified.

AB - Increasingly sophisticated mathematical modelling processes from Machine Learning are being used to analyse complex data. However, the performance and explainability of these models within practical critical systems requires a rigorous and continuous verification of their safe utilisation. Working towards addressing this challenge, this paper presents a principled novel safety argument framework for critical systems that utilise deep neural networks. The approach allows various forms of predictions, e.g., future reliability of passing some demands, or confidence on a required reliability level. It is supported by a Bayesian analysis using operational data and the recent verification and validation techniques for deep learning. The prediction is conservative – it starts with partial prior knowledge obtained from lifecycle activities and then determines the worst-case prediction. Open challenges are also identified.

KW - Assurance arguments

KW - Bayesian inference

KW - Deep learning verification

KW - Quantitative claims

KW - Reliability claims

KW - Safe AI

KW - Safety cases

UR - https://arxiv.org/abs/2003.05311

UR - https://www.scopus.com/pages/publications/85089750379

U2 - 10.1007/978-3-030-54549-9_16

DO - 10.1007/978-3-030-54549-9_16

M3 - Conference contribution

AN - SCOPUS:85089750379

SN - 9783030545482

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 244

EP - 259

BT - Computer Safety, Reliability, and Security - 39th International Conference, SAFECOMP 2020, Proceedings

A2 - Casimiro, António

A2 - Ferreira, Pedro

A2 - Ortmeier, Frank

A2 - Bitsch, Friedemann

PB - Springer London

T2 - 39th International Conference on Computer Safety, Reliability and Security

Y2 - 16 September 2020 through 18 September 2020

ER -

Zhao X, Banks A, Sharp J, Robu V, Flynn D, Fisher M et al. A Safety Framework for Critical Systems Utilising Deep Neural Networks. In Casimiro A, Ferreira P, Ortmeier F, Bitsch F, editors, Computer Safety, Reliability, and Security - 39th International Conference, SAFECOMP 2020, Proceedings. Springer London. 2020. p. 244-259. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-54549-9_16

A Safety Framework for Critical Systems Utilising Deep Neural Networks

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this