Title:
Geospatial machine learning in safety critical systems
External Stakeholder:
UK Hydrographic Office (UKHO)
The Research:
The project will fulfil the following objectives (described as research questions):
1) What is the minimum volume of labelled data needed in order to achieve an accurate, geo-generalisable model?
How many samples required for machine learning is a currently active and open research question. Recent results have examined the theoretical sample complexity but in this project it is the reliability of the labels of the training data that is in question.
2) How can User Interfaces facilitate the labelling of large datasets?
Our prior expertise is in the domain of effective user interfaces for labelling large time-series data sets. We will extend our research into this application area and integrate further tools to solve the problems below.
3) How can variation in the interpretation of imagery by labellers be measured, and what degree of confidence should we have in a labelled dataset?
Non-expert labellers could produce adequate labels with fewer resources compared to experts, but a research question is how much we could rely on those labels, and whether we could track any uncertainty through the training to the model. Current approaches examine consensus labelling or label noise from various angles, but this is still an open area of research.
4) What impact does a label corpus's characteristics have on model accuracy?
Once an initial model is trained, various metrics can be used to describe the accuracy and success of the model. We need to analyse performance in the presence of label noise and label bias. Label noise, mislabelling or bias impacts models and some mitigation factors are known. We need to ascertain what proportion of the model's performance is a result of inaccuracy, uncertainty or bias in the labelled data and how bias, accuracy and uncertainty can be formally quantified and propagated through different stages of a model's development. Model training, testing and refinement is ideally a non-linear process, instead model training is iterative as each stage of model creation informs the next (and previous steps in a loop). Techniques such as re-labelling, resampling and pseudo-labelling are available to update a labelled dataset. Based on assessments of initial model performance.