We’ve recently had a short paper accepted to Computer Human Interaction (CHI) where we describe the outcomes from a qualitative study on what ontologists do when they author an ontology. The paper is called “Design Insights for the Next Wave Ontology Authoring Tools” and the motivation is a desire to understand how people currently go about authoring ontologies (the work was carried out by Markel Vigo in collaboration with Caroline Jay and me). Ultimately we want to understand how people judge the consequences of adding axioms so that we can support the answering of “what if…?” questions (and we’ll do this by creating models of ontology authoring). So, if I add some axiom to a large system of axioms what will be the consequences. As we work up to this, we’re doing studies where we record what people are doing as they author an ontology, logging all activities in the Protégé 4 environment, as well as capturing screen recordings and eye-tracking data.
This first study was a qualitative study where we asked 15 experienced ontology builders a series of open questions:
- Can you describe the authoring tasks you perform?
- How do you use tools to support you in these tasks?
- What sort of problems do you encounter?
You can read the details of the method and analysis in the paper. We chose to do this study with experienced ontology authors as this will, in the fullness of time, inform us about how authoring takes place without any confounding factors such as not fully understanding ontologies, OWL or tools being used. Understanding issues faced by novices also needs to be done, but that’s for another time.
The 15 participants partition into three groups of five: Ontology researchers; ontology developers; and ontology curators. These, in turn, are CS types who do research on ontology and associated tools and techniques; ontology developers are CS types that work closely with domain experts to create ontologies; curators are those that have deep domain knowledge and maintain, what are often, large ontologies.
The tools participants use are (here I list those with numbers of users above one): Protégé (14 users), OWL API (6), OBO-Edit (4) and Bioportal (3). We didn’t choose participants by the tools they used; these are the tools that the people we talked to happened to use.
The analysis of the interviews revealed themes based on the major tasks undertaken by ontologists; the problems they encounter and the strategies they use to deal with these problems.
- Sense-making, exploration and searching: Knowing the state of the ontology, finding stuff, understanding how it’s all put together – “making sense” of an ontology.
- Ontology building: Efficiently adding axioms to an ontology en mass and effectively support what we called “definition orientated” ontology building.
- Reasoning: Size and complexity of ontologies hampering use of ontologies.
- Debugging: Finding faults and testing ontologies.
- Evaluation: is it a good thing?
The paper describes in more detail the strategies people use in these five themes. For instance, speeding up reasoning by restricting the ontology to a profile like OWL EL and using a fast reasoners like ELK; chopping up the ontology to make reasoning faster; relying on user feedback for evaluation; using repositories and search tools to find ontologies and re-use parts of them; using the OWL API to programmatically add axioms; and so on (the paper gives more of the strategies people use).
There will be other issues; there will be ones we may not have encountered through our participants and there will be important issues that were in our interviews, but may not have been common enough to appear in our analysis.
There may well be tools and techniques around that address many of the issues raised here (we’ve done some of them here in Manchester). However, this sample of ontology authors don’t use them. Even if tools that address these problems exist, are known about and work, they don’t work together in a way that ontology authors either can use or want to use. So, whilst we may have many useful tools and techniques, we don’t have the delivery of these techniques right. What we really need to build the new wave of ontology authoring environments are models of the authoring process. These will inform us about how the interactions between author and computational environment will work. This qualitative study is our first step on our way to elucidating such a model. The next study is looking at how experienced ontology authors undertake some basic ontology authoring tasks.