Questions for a panel at Semantic Web Applications and Tools for Life Sciences

September 1, 2015

James Malone and I are the Scientific Chairs for Semantic Web Applications and Tools for Life Sciences (SWAT4LS) 2015, Cambridge UK in December 2015. As we’re coming to the end of the paper submission period we’ll be doing reviewing and forming the programme. Rather than wall-to-wall talks, James and I want to break up the day a bit; panel sessions are a standard way of adding a little variety to a programme and can be both lively and informative – if the questions are right etc.

One can either form a panel and choose questions appropriate to a panel; choose questions and form the panel to suit; or some combination of both. For SWAT4LS 2015 we want to open up the questions or theme for the panel(s) to the SWAT4LS audience, gather a corpus of good panel questions and then form a panel around these questions, with some moderation by who we can get to be on the panel.

So, this is a “call for panel questions” or a CFPQ. Add your proposed questions along to the short CFPQ survey at https://www.surveymonkey.com/r/RLF3VPG James and I will take a look and along with the rest of the SWAT4Ls organising committee choose some panel questions. Do send along anything you wish to ask about Semantic Web technologies and the life sciences; we’ll use these questions to prime the panels sessions, which will then be opened up to the audience. Those people that have their question used for a panel will receive:

  1. An acknowledgement of their contribution (unless they choose not to do so).
  2. A £10 Amazon voucher (or equivalent in some other countrys currency).

This CFPQ will close on 1st October 2015.

Where a Pizza Ontology may help

August 13, 2015

A few people have pointed me at a recent news story on the BBC Web site about valid and invalid types of pizza and suggested that the Pizza Ontology should have helped. The story is entitled “the day I ordered pizza that doesn’t exist” and is written by Dany Mitzman in Bologna. The nub of the story as far as this blog is concerned is that she ordered a marinara pizza, which should be a simple pizza of just tamato and garlic. In Pizza Ontology terms this would be:

 

Class: marinaraPizza

    SubClassOf: NamedPizza,

    HasTopping some TomatoSauceTopping,

    HasTopping some Garlictopping,

    hasTopping only ( TomatoSaucetopping or GarlicTopping)

 

the NamedPizza class supplies the pizza base; MarinaraPizza is disjoint with all other NamedPizza and the restrictions on the MarinaraPizza class says that there is a tomato sauce topping, a garlic topping and that those are the only toppings that appear on this type of pizza.

 

What Dany, the journalist did, was ask for a marinara pizza with mozzarella; that is:

 

Individual: “Dany’s very own marinara pizza”

 

    Types:

        MarinaraPizza,

        hasTopping some MozzarellaTopping

 

the article reports that the pizza maker found this request inconsistant, as would the Pizza Ontology, if the class of pizza existed as above and an automated reasoner were used – the MarinaraPizza class describes that only tomato sauce and garlictoppings occur on this type of pizza, so adding another topping, such as the mozzarella, no matter what (as long as it cannot be inferred to be the same as either tomato sauce or garlic topping) mean that the stated constraints are broken and we all descend into a maelstrom of sin and corruption (or there is an inconsistency reported by the reasoner). Quoting the quote of the pizza maker in question “”You can’t have a marinara with mozzarella,” she says. “It doesn’t exist.”” – the marinara with mozzarella cannot exist as described in our ontology.

 

Of course, describing a class of pizza that has tomato sauce, garlic and mozzarella as its toppings would be fine, just as long as one doesn’t claim it’s a marinara pizza. Creating such a new pizza in the Pizza Ontology is possible; it’s just not a marinara according to the ontology.

 

This ability to make a bespoke class or individual pizza may also be the case with the pizza maker encountered by Dany. It may all be down to the name; asking “may I have a pizza with tomato sauce, garlic and mozzarella” may have elicited a different response, unles it is believed that only specified types of pizza exist. In this were the case, one would need a covering axiom in the ontology. Such an axiom would look like

 

NamedPizza

EquivalentTo:

        (MargheritaPizza or MarinaraPizza or NapolitanoPizza)

 

If one wanted a world in which only these three pizza existed (which I don’t and neither would any sensible person). This axiom asserts that if there is a pizza then it must be one of the three pizza covering the NamedPizza class In the Pizza Ontology itself there are many named pizza and should one wish to construct such a covering axiom, keeping it up to date with the disjoint named pizza in a tool such as protégé would be found to be tedious – Tawny-OWL allows covering axioms and so on to be generated with ease programmatically.

 

So, what have we learnt? From the original BBC article we find that, for at least one Italian pizza maker, that a pizza’s name is as good as its definition; a marinara pizza with another ingredient is no longer a marinara pizza. The Pizza Ontology plus a reasoner can convey the same kind of stance. For the pizza maker, it seems that the name of the pizza is its definition. In an OWL ontology we make such definitions explicit. Of course, even though the Pizza Tutorial adopts the use case of the “intelligent pizza finder” that uses the Pizza Ontology to allow diners to pick topping to include and exclude, form a DL query behind the scenes and select pizza that fulfil that query, the Pizza Ontology is not really going to help in such cross-cultural circmstances as described in the original article. One may perhaps imagine an ontology driven app on a mobile device where one describes one’s pizza of choice and one is told what it is and how to order it in a particular cultural setting, but the phrase “hammer to crack a nut” comes to mind.

Open data and the need for ontologies

July 24, 2015

This is an abstract for “Digital Scholarship and Open Science in Psychology and the Behavioural

Sciences”, a Dagstuhl Perspectives Seminar (15302) held in the week commencing 20 July 2015. The workshop brought together computer scientists, computational biologists and people from the behavioural sciences. The workshop explored eScience, data, data standards and ontologies in psychology and other behavioural sciences. This abstract gives my view on the advent of eScience in parts of biology and the role open data and metadata supplied by ontologies played in this change.

There is a path that can be traced with the use of open data in the biological domain and the rise in the use of ontologies for describing those data. Biology has had open repositories for its nucleic acid and protein sequence data and controlled vocabularies were used to describe those data. These sequence data are core, ground truth in biology; all else comes from nucleic acids and, these days, the environment. As whole genome sequences became available, different organism communities found that the common vocabulary used to represent sequences facilitated their comparison at that level, but a lack of a common vocabulary for what was known about those sequences blocked the comparison of the knowledge of those sequences. Thus we could tell that sequence A and sequence B were very similar, but finding that the function, processes in which they were involved and where they were to be found etc. was much more difficult, especially for computers. Thus biologists created common vocabularies, delivered by ontologies, for describing the knowledge held about sequences. This has spread too many types of data and many types of biological phenomenon, from genotype to phenotype and beyond, so that there is now a rich, common language for describing what we know about biological entities of many types.

At roughly the same time was the advent of eScience. The availability of data and tools open and available via the Web, together with sufficient network infra-structure to use them, led to systems that co-ordinated distributed resources to achieve some scientific goal, often in the form of workflows. Open tools, open data, open standards, open, common metadata all contribute to this working, but it can be done in stages; not all has to be perfect for something to happen – just availability of data will help, irrespective of its metadata. Open data will, however provoke the advent of common data and metadata standards, as people wish to do more and do it more easily.

In summary, we can use the FAIR principles (Findable, Accessible, Interoperable and re-usable) to chart this story. First we need data and tools to be accessible and this means openness. Metadata, via ontologies, also have a role to play in this accessibility – do we know what those data are etc.? Metadata has an obvious role in making tools and data findable – calling the same things by the same term and knowing what those terms mean makes things findable. The same argument works for interoperable tools and data.

OBOPedia: An Encyclopaedia made from Open Biomedical Ontologies

March 31, 2015

A little while ago I wrote a blog about using an ontology or ontologies as background knowledge about a field of interest in order to learn about that domain, rather than simply annotating data or building some kind of hierarchical search interface. The idea is that an ontology captures knowledge about a field of interest; I should be able to look at that ontology and gain some kind of understanding about that domain by examining the terms used to name the class and its definition about how to recognise objects in that class (both the natural language definition and the axioms that describe that class’ objects in terms of relationships to other objects. In that blog article I conjectured that an encyclopaedia style presentation of many ontology entries could work as a way of presenting the large amount of background knowledge captured in the ontologies the community has built. My feeling is that the standard graphical presentation of blobs and lines isn’t necessarily a good way of doing this, especially when there are several ontologies at which to look. Encyclopaedia are also meant for “looking up things” and finding out about them – but we can exploit Web technologies and the structure of an ontology to get the best of both worlds. The OBO ontologies are particularly attractive for an encyclopaedia because:

  • They cover a broad spectrum of biology – from sequence, through proteins, processes, functions, cells, cellular components to gross anatomy.
  • Each entry in an ontology has a human readable label, synonyms, a natural language definition, all of which are”standard” parts of an encyclopaedia entry.
  • The relationships between entries in the ontology can provide the “see also” links for an encyclopaedia entry.

 

One of my undergraduate project students for this year, Adam Nogradi, has built OBOPedia for me as an example of this kind of presentation for a group of ontologies. OBOPedia may be found via http://www.OBOPedia.org.uk. The current version of OBOPedia has nine ontologies, including the OBO Foundry ontologies plus a few more and has over 210,000 entries; the ontologies currently available are:

  • The Gene Ontology.
  • The Protein Ontology.
  • The Chemical Entities of Biological Interest ontology.
  • The Ontology of Biomedical Investigation.
  • The Phenotypic Quality Ontology.
  • The Zebra Fish Anatomy and Development Ontology.
  • The Xenopus Anatomy and Development Ontology.
  • The Human Disease Ontology.
  • The Human Phenotype Ontology.
  • The Plant Ontology.

     

An example of an entries page for OBOPedia can be seen in the picture below:

 

This shows Entries are arranged alphabetically. The screen here shows some entries from “E”, after some scrolling; on view are “”ether metabolic process” from GO and “ethmoid cartilage” from the Zebrafish Anatomy and Development Ontology. Each entry has the main label as the entry’s title, the various synonyms, the natural language definition and some “see also” links. The letters down the left hand side takes one to the beginning of entries starting with that letter. Entries are shown 50 at a time. One nice aspect of this style of presentation can be the serendipity of looking at entries surrounding the target entry and seeing something of interest; a typical hierarchical display automatically puts entries that are semantically related more or less in the same place – this encyclopaedia presentation doesn’t, but preserves the hierarchy via the “see also” links (though what those link to is rather hidden until arrival at the end of the link, which isn’t the case in most graphical presentations). Each entry shows the ontology whence the entry came – there are several anatomies containing the entry “lung” and knowing whence the entry comes is just a good thing. The picture also shows the (possible) exact, broader, narrower and related synonyms taken from the entries in the ontologies. At the moment OBOPedia only uses the subsumption links for “see also”s, but the aim is to expand this to other relationships in the fullness of time. I’d also like to include the ability to use DL-queries to search and filter the encyclopaedia, but time in this project has not permitted.

 

The picture below shows OBOPedia’s search in action.

The search was for “lung” and entries were retrieved from the Gene Ontology and the Human Disease Ontology; some of the entries brought back and available for viewing were “lung disease”, “lung leiomynoma”, “lung induction”, “lung lymphoma”, “hyperlucent lung” and many others…

Along with each OBOPedia entry there is also a “rate this definition” feature. This definition rating uses a simple five point scale that allows people to rate the natural language definition (capturing comments will come at some point). The idea here is that feedback can be gathered about the natural language definitions and eventually this will form an evaluation of the definitions.

 

OBOPedia is an encyclopaedia of biology drawn from a subset of the OBO ontologies (there’s no reason not to include more than the nine currently on show, except for resources), exploiting their metadata, especially their natural language definitions. OBOPedia is not a text-book and it’s not a typical blob and line presentation of an ontology. It’s an encyclopaedia that presents many ontologies at once, but without the reader necessarily knowing that he or she is using an ontology. It’s an attempt to give an alternative view on the knowledge captured by the community in a range of ontologies in a way that gives easy access to that knowledge. OBOPedia may be a good thing or a bad thing. Send comments to Robert.Stevens@manchester.ac.uk or add comments to this blog post.

Patterns of authoring activity when using Protégé for building an ontology

February 9, 2015

We’ve continued our work investigating the human-computer interaction of authoring an ontology. We had a couple of papers last year looking at some qualitative aspects of ontology authoring through interviews with experienced ontologists. We wanted to follow this up with quantitative work looking at the activities during the addition of axioms when authoring an ontology. I’m pleased to say we’ve just had a long paper accepted for CHI 2015 with the following details:

 

Markel Vigo, Caroline Jay and Robert Stevens. Constructing Conceptual Knowledge Artefacts Activity Patterns in the Ontology Authoring Process. Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems: CHI 2015; 18 Apr 2015-24 Apr 2015; Seoul, Korea.

 

I reported some early work in this quantitative study. In this latest work we’ve taken the following approach:

  • We’ve instrumented a version of Protégé 4.3 (P4) to record every keystroke, mouse click and so on in a time-stamped log file (it’s called Protégé4US – the “Us” is for “user studies”). We divided the events into interaction events (interacting with the ontology and its axioms via the class and property hierarchies and the axiom description window), authoring events (typing an axiom, class declaration, etc.) environment events (invoking the reasoner, getting an explanation, etc.).
  • We had experienced ontology authors perform a series of tasks to build an ontology of potatoes. Three tasks of increasing difficulty involving making various defined classes over descriptions of some 15 potato varieties, the creation of which was also part of the tasks.
  • Whilst this happened we recorded what was happening on the screen.
  • Finally, we recorded eye-tracking data as to where the author’s gaze fell during the ontology authoring.

 

In capturing eye-tracking data, the screen of Protégé4US is divided up into areas of interest as shown below. This picture shows the main view as an area of interest; other views involve classes, properties and individuals and these have their own areas of interest defined. These AOI are used to determine the dwell time of eye gaze during the tasks.

 

 

The patterns of ontology authoring activity we found were:

 

  1. An exploration cycle. The asserted class hierarchy is expanded after ontology loading – in over 31% of the time an expansion is followed by an expansion as users appear to familiarise themselves with the structure of the ontology. Eventually, this behaviour appears to become directed as an author chooses a class to edit. In contrast, the expansion of the inferred class hierarchy appears to be more exploratory as the authors check what has happened post reasoning, perhaps answering the question “have I found all the changes?”.
  2. An editing cycle. Here an entity is selected, followed by selection of another entity 37% of the time or selection of the description area 29% of the time. Once selected, a description will be modified 63% of the time and followed by selection of another entity 59% of the time. This looks like selecting an entity, inspecting its description and then either editing it or moving on to another entity, each decision based on the content of the description.
  3. A reasoning cycle. Just prior to the reasoner being invoked, the ontology is saved 40% of the time; a defined class is created (17%). After the reasoner is run, 41% of the time participants observe the change on the asserted class hierarchy and then look at a description where the effects of reasoning can be seen. The inferred class hierarchy is inspected post-reasoning 30% of the time, which is again followed by the expansion of the hierarchy 43% of the times.

These activity patterns are shown in the following pictures.

 

Overall, we can see the following flow of events:

  • Initial exploration of the ontology.
  • A burst of exploration coupled with editing.
  • Reasoning followed by exploration.

 

An activity pattern is a common sequence of events. The details of our analysis that led to these activity patterns are in the paper, but some of the pretty pictures and the basic analysis steps that gave us these patterns are below.

 

 

This is a simple log plot of the number of each type of event recorded across all participants. The top three events – entity selected, description selected and edit entity:start account for 54% of events. Interaction events account for 65% of events, authoring events for 30% and environment events for 5%. There’s a lot of a few events and interaction with P4 accounts for most things.

 

This picture shows the N-grams of consecutive events. We can see lots of events like expanding the class hierarchy (either asserted or inferred) occurring many times one after the other, indicating people moving down through the hierarchy – the class hierarchy seems to be a centre of interaction – looking for classes to edit and checking for the effects of reasoning.

Those are the events themselves, but what happens after each event. Below there is a plot of transitions from event to event (note the circles around to the same event and the thickness of the lines indicating the likelihood of the event occurring). A matrix of number of transitions from event to event gives a fingerprint for each user. We see that the fingerprints revealed by these transitions from state to state are the same within individuals for each task; that is, each task is operationalised in P4 in the same way.

The inter-user similarity is also high, suggesting patterns of events (though there is also evidence of some different styles here too). What is below is a 16×16 matrix showing the correlation of the fingerprints (i.e. transition matrices of all participants).

 


 

The eye-tracking data showed that the class hierarchy received by far the most fixations (43%) and their attention 45% of the time. The edit entity dialogue has 26% of the fixations and the same for attention, and the description area 17% of the fixations and 15% attention. If we look at events over time we begin to see patterns, but with gaps. Some of these gaps can be filled by looking at where the eye gaze dwell – e.g., a user is looking at the description area and not interacting via events. The picture below shows the distribution of dwell times on each area of interest on the P4 user interface – note these numbers tell the same sort of story as the P4US event logging.

 


Each cell of the following matrix conveys the number of fixations between areas of interest. In other words, it indicates where users will look in t based on t-1 (the x-axis indicate the origin while the y-axis is the destination). The darker the cell is the more transitions there are between areas. We find that given a fixation on a given area the most likely next fixation is on the same area.


We also see other transitions:

  • From class hierarchy to description area (and vice versa).
  • From the class addition pop-up to class hierarchy.
  • From the edit entity dialogue to the class hierarchy.
  • From the edit entity dialogue to the description area.

 

Again, we se the class hierarchy being central to the interactions.

 

To find the activity patterns themselves, we next merged the eye-tracking and N-gram analyses. First we collapsed consecutive events and fixations of the same type. We then took the resulting N-grams of size > 3 and extended them one at a time until it only yielded repeated and smaller N-grams of the merged data. This analysis resulted in the editing, reasoning and exploration activities outlined at the top.

 

So what do we now know?

 

It appears that the class hierarchy is the centre of activity in Protégé. Authors look at the asserted class hierarchy to find the entity he or she wishes to edit and then edits in the description window. The inferred class hierarchy is used to check that what is expected to have happened as a result of reasoning has indeed happened. While activity in each of these windows involves a certain amount of “poking about”, the activity in the asserted class hierarchy looks more directed than that in the inferred class hierarchy. Design work that can ease navigation and orientation within the ontology and checking on the results of reasoning – all results, not just those the author expects to happen – would be a good thing.

 

Adding lots of axioms by hand is hard work and presumably error prone. Bulk addition of axioms is desirable, along with the means of checking that it’s worked OK.

 

There is a lot of eye-gaze transition between the class hierarchy and the editing area. In P4 these are by default these are top-left and bottom right. Defaulting to adjacency of these areas could make authoring a little more efficient.

 

We see experienced authors tend to save the ontology before reasoning. An autosave feature would seem like a good thing – even if the reasoners/Protégé never fell over.

 

Finally, rather than letting the author hunt for changes in the class inferred hierarchy changes should be made explicit in the display; in effect showing what has happened. This would be a role for semantic diff. Knowing what has changed between the ontology before and after a round of editing and reasoning could help – authors look for changes in the inferred class hierarchy – presumably the ones they are expecting to have happened; there may be other, unforeseen consequences of changing axioms and showing these semantic differences to users could be a boon. To do this we’ll be looking to exploit the work the work at Manchester on the Ecco semantic diff tool.

 

The paper has more detailed design recommendations. However, what this work does show is that we can gain insight into how ontologists operationalise their work by extracting patterns from log data. The task we used here (everyone built the same ontology to do the same tasks) is not so ecologically valid. It does provide a base-line and allowed us to develop the analysis methods. One pleasing aspect is that the findings in this quantative work largely supported that of our qualitative work. The next thing is to use Protégé4US while people are doing their everyday authoring jobs (do contact me if you’d be willing to take part and do a couple of hours work on Protégé 4US and receive our gratitude and an Amazon voucher). I expect to see the same patterns, but perhaps with variations, even if it’s just in timings, frequency and regularity.

The International Conference on Biomedical Ontology

January 24, 2015

It’s good to see the calls for the next International Conference on biomedical Ontology coming out; this time it’s to be held in Lisbon, Portugal on 27 – 30 July 2015. I encourage biomedical ontology folk to write papers and schedule themselves to get themselves along to Lisbon in July 2015.

 

The blurb about ontologies on the ICBO site is: “Ontologies are increasingly used in the semantic description of biological and medical data from creation to publication and consumption by semantically enabled applications. To be effective, such ontologies must be of high quality and work well together. Therefore, issues like techniques for ontology use, good ontology design, ontology maintenance, and ontology coordination need to be addressed. The International Conference on Biomedical Ontology (ICBO) series, http://www.icbo-conference.org, is designed to meet this need. ICBO 2015, the sixth event in the highly successful series, will bring together representatives of all major communities involved in ontology use and development in biomedical research, health care, and related areas.”

 

This covers the usual topics in biomedical ontologies, but that’s because there’s still a lot to do. Biomedical ontology is maturing, but there is still too much that is ad hoc and craft based; there are islands of good practice, but we need to learn from those and spread that best practice and really find out what works in the development, assessment and application of biomedical ontologies.

 

Here I especially want to encourage submissions to the workshops at ICBO that Melanie Courtot and Joao Ferreira are charing. Workshops are a great opportunity to focus upon a particular aspect of biomedical ontologies and, ideally, explore that topic deeply with interaction with the participants – workshops where work is done…

The deadline for workshop submissions is 10 February 2015 via the ICBO 2015 EasyChair site. I want to attend a workshop on programmatic development of ontologies where I can experience the various ways of avoiding hand-crafting of axioms and achieve bulk upload of axioms according to patterns etc etc. So, someone organise that and I should come along (to that one), but I’ll come to ICBO 2015 anyway; ICBO is informative and fun, and Lisbon is ace.

Fundamental barriers to accessibility to digital materials

November 27, 2014

It’s nearly twenty years ago since I did my D.Phil work on the principles for auditory access to complex notations. I’ve moved away from the field of research into the HCI of accessibility since then, but it’s always been in my mind and recent conversations have prompted me to write a bit about some of my thoughts on the underlying issues of using computers (digital presentations of material) when one cannot see.

 

My D.Phil. research work was motivated by my experience of doing mathematics using algebra notation as part of my masters in Biological Computation at the University of York. I did this after giving up, as a consequence of sight loss, my previous Ph.D. work in biochemistry. There was a lot of mathematics in the master’s degree; not necessarily that hard as mathematics goes, but made much harder by not being able to see the notations and use the notations for manipulation and effective/efficient thinking.

 

At the root of my difficulty, it appeared to me, was not having a piece of paper upon which I could do my algebraic manipulations. The paper remembers the equation for me, I can write down re-arrangements of the symbols, cross things out, all the time only using my brain to work out what to do, both strategically and tactically, then remembering each bit by externalising it on the page in front of me. Without the paper, most of this had to happen in my head – I mimicked some of paper and pencil’s attributes in Braille or, even worse, in text in a text editor, but it isn’t the same at all – and this prompted me to think exactly why it isn’t the same. I discussed these problems with Alastair Edwards and eventually did my masters project with him looking at rendering and browsing algebra written in some linear code in audio and being able to render and browse that audio presentation. This led on to my D.Phil research work with Alastair where I looked at the human computer interaction of the problem of doing algebra, and other complex notations, in audio.

 

There’s no need to go into the details of my D.Phil. work here, because I want to look at basics of interacting with information when one cannot see; in particular what’s possible (if not beautiful) In terms of interaction and what the real “can’t do very well” problems are that, as far as I can tell, still remain.

 

Reading “plain” text (words written one after another in normal natural language form) is more or less OK. I use a screenreader and I can read and write straight-forward text without too much of a problem. My window upon the screen of text is rather small; it’s essentially one line. I can get my screenreader to read bigger portions of text, but the quick look, the scan is still problematic. I can move around the text to find what I want and inspect it with relative ease; the interaction is, to my mind, clunky, but it’s all doable. As soon as moves away from simple, linear strings of works and into two-dimensions, as in algebra notation, and into informationally dense material (again algebra is dense and complex or complex because it’s dense), speech based screenreaders don’t offer an effective reading solution.

 

This comes to two of the things that I worked out during my D.Phil.:

  1. A listening reader tends to be a passive reader. As a listening reader, I tend to lack agility in my control of information flow. In the worst case, e.g., with an audio book, I listen at the rate dictated by the reader, not what my eyes and brain want to do. Obviously I control information flow with keystrokes that makes my screenreader say things, but it’s all a bit clunky, slow and intrusive compared to what one does with ones eyes – they move around the screen (or paper) in a way that basically gets me to the right portion of text, either word by word, or bigger chunks, without my having to consciously do very much at all. So, speed and accuracy in the control of the flow of information turns the reader from being passive to being active.
  2. I lack an adequate external memory. The paper or the screen has the text upon it and it remembers it for me, but as it’s slow and clunky to get at it, I rely more on my brain’s memory and that’s a bit fragile. Of course there is an external memory – the information I have access to on a computer – but it only really plays the role of an external memory if there is sensible (fast and accurate) control in the access to that external memory.

     

    The external memory in conjunction with speed and accuracy in control of information flow makes eyes and paper/screen all rather effective. It was these two issues that I addressed in my D.Phil. work.

     

Despite these issues, access to straight-forward text is OK. I, along with lots of other people, read and write perfectly well with screenreader’s and word processors. In the small the interaction works well, but I find reading and comprehending larger documents much harder work; it’s a burden on my memory and flipping backwards and forwards in text is relatively hard work – not impossible, but harder work than it was when I could see.

 

Some of this difficulty I describe with the large grained view of information comes from the ability, or the lack of it, to glance at material. Typesetters have spent centuries working out styles of layout that make things easy to read, there are visual clues all over pages to aid navigation and orientation. Algebra notation is laid out to group operands in a way that reflects the order of precedence of the operators – it makes a glance at an expression written in algebra easier. Similarly, diagrams need to at least give the illusion of being able to see the whole thing (see below) – the glance at the whole diagram. Work on glancing has been done, including some by myself, and there are ways of doing it for individual information types, but I don’t know of a generic solution and certainly one that is available to me for everyday use.

 

  1. Glancing at information to assess reading strategies, help orientation and navigation, and choices in reading is difficult

     

My final chore is the looking at two things at once problem. Eyes give one the impression that two things can be looked at at once. In the small this is true, the field of accurate vision is narrow, but does see several things in detail at once. However, the speed and accuracy in control of information flow afforded by the eyes, combined with the layout of information (when done well), on an external memory means that eyes can move back and forth between items of information rather well. This is hard in speech and audio- so much layout information is lost – when reading research papers, moving back and forth from the narrative text to the references was easy with eyes; it’s hard with speech (what I do is to have two windows open and move between the windows – this is hard work).

 

My interaction with spreadsheets always seems v clunky to me. My natural view with a speech based screenreader is one cell at a time; looking at column or row headers to see what they are is naturally a matter of flicking one’s eyes up or along to remember the orientation and that’s fine. I can do this, but the means of doing so is intrusive. Similarly, dealing with any tabular information is painful. The ability to compare rows, columns, cells is central; indexing via column and row headings is vital. I have the keystrokes to do it all in my screenreader, but it’s hard work – in contrast, one flicks one’s eyes back and forth and one appears to be looking at two things at once. Tables are nothing in terms of difficulty when it comes to diagrams; even if there is access to the material (e.g., simple line graphs, histograms, and node and arc diagrams) one has to build up a picture of the whole thing piecemeal. The “looking at two things at once” ability of eyes makes this task relatively easy and the inability to do this with speed, accuracy and so on means many interactions are either very hard or impossible.

 

  1. Looking at two things at once is nigh on impossible

 

In conclusion, I think there are still two main unsolved problems in audio interaction with information:

  1. Glancing;
  2. Looking at two things at once.

Once I have general solutions to these two things, I’ll be a much more effective and efficient reader that is satisfied with my reading.

Getting emotional about ontologies

October 22, 2014

It’s taken a long time, but we’re finally publishing our paper about evaluating the Emotion Ontology (EM). Evaluating ontologies always seems to be hard. All too often there is no evaluation, or it ends up being something like “I thought about it really hard and so it’s OK”, or “I followed this method, so it’s OK”, which really amounts to not evaluating the ontology. Of course there are attempts made to evaluate ontologies with several appealing to the notion of “large usage indicates a good ontology”. Of course, high usage is to be applauded, but it can simply indicate high need and a least bad option. High-usage should imply good feedback about the ontology; so we may hope that high usage would be coupled with high input to, for instance, issue trackers (ignoring the overheads of actually issuing an issue request and the general “Oh, I can’t be bothered” kind of response) – though here I’d like to see typologies of requests issued and how responses were made.

 

Our evaluation of the Emotion Ontology (EM) fits into the “fitness for purpose” type of evaluation – if the EMdoes its job, then it is to some extent a good ontology. A thorough evaluation really needs to do more than this, but our evaluation of the EM is in the realm of fitness for purpose.

 

An ontology is all about making distinctions in a field of interest and an ontology should make the distinctions appropriate to a given field. If the ontology is doing this well, then an ontology delivers the vocabulary terms for the distinctions that need to be made in that field of interest – if we can measure how well people can use an ontology to make the distinctions they feel necessary, then the ontology is fit for purpose. In our paper we attempted to see if the EM makes the distinctions necessary (and thus the appropriate vocabulary) for a conference audience to be able to articulate their emotional response to the talks – in this case the talks at ICBO 2012. That is, the EM should provide the vocabulary distinctions that enables the audience to articulate their emotional response to talks. The nub of our null hypothesis was thus that the EM would not be able to let the audience members articulate their emotions such that we can cluster the audience by their response.

 

The paper about our evaluation of the EM is:

 

Janna Hastings, Andy Brass, Colin Caine, Caroline Jay and Robert Stevens. Evaluating the Emotion Ontology through use in the self-reporting of emotional responses at an academic conference. Journal of Biomedical Semantics, 5(1):38, 2014.

 

The title of the paper says what we did. As back-story, I was talking to Janna Hastings at ISMB in Vienna in 2011 and we were discussing the Emotion Ontology that she’d been working on for a few years and this discussion ended up at evaluation. We know that the ontology world is full of sects and that people can get quite worked up about work presented and assertions made about ontologies. Thus I thought that it would be fun to collect the emotional responses of people attending the forthcoming International Conference on biomedical Ontology 2012 (ICBO), where I was a programme co-chair. If the EM works, then we should be able to capture self-reported emotional responses to presentations at ICBO. We, of course, also had a chuckle about what those emotions may be in light of the well-known community factions. We thought we should try and do it properly as an experiment, thus the hypothesis, method and analysis of the collected data.

 

 

 

 

Colin Caine worked as a vacation student for me in Manchester developing the EmOntoTag tool (see Figure 1 above), which has the following features:

  • It presented the ICBO talk schedule and some user interface to “tag” each talk with terms from the EM. The tags made sentences like “I think I’m bored”, “I feel interested” and “I think this is familiar”.
  • We also added the means by which users could say how well the EM term articulated their emotion. This, we felt, would give us enough to support or refute our hypothesis – testing whether the EM gives the vocabulary for self-reporting emotional response to a talk and how well that term worked as an articulation of an emotion. We also added the ability for the user to say how strongly they felt the emotion – “I was a bit bored”, “I was very bored” sort of thing.
  • We also used a text-entry field to record what the audience members wanted to say, but couldn’t – as a means of expanding the EM’s vocabulary.

 

We only enabled tagging for the talks that gave us permission to do so. Also, users logged in via a meaningless number which were just added to conference packs in such a way that we couldn’t realistically find out whose responses were whose. We also undertook not to release the responses for any individual talk, though we sought permission from one speaker to put his or her talk’s emotional responses into the paper.

 

The “how well was the EM able to articulate the necessary emotion you felt?” score was significantly higher than the neutral “neither easy nor difficult” point. So, the ICBO 2012 audience that participated felt the EM offered the appropriate distinctions for articulating their emotional response to a talk. The bit of EmOntoTag that recorded terms the responders wanted, but weren’t in the EM included:

  • Curious
  • Indifferent
  • Dubious
  • Concerned
  • Confused
  • Worried
  • Schadenfreude
  • Distracted
  • Indifferent or emotionally neutral

 

There are more reported in the paper. Requesting missing terms is not novel. The only observation is that doing the request at the point of perceived need is a good thing; having to change UI mode decreases the motivation to make the request. The notion of indifference or emotionally neutral is interesting. It’s not really a term for the EM, but something I’d do at annotation time, that is, “not has emotion” sort of thing. The cherry-picked terms I’ve put above are some of those one may expect to be needed at an academic conference; I especially like the need to say “Schadenfreude”. All the requested terms, except the emotionally neutral one, are now in the EM.

 

There’s lots of data in the paper, largely recording the terms used and how often. A PCA did separate audience members and talks by their tags. Overall, the terms used were of the positive valence “interested” as opposed to “bored”. These were two of the most used terms; other frequent terms were “amused”, “happy”, “this is familiar” and “this is to be expected”.

 

The picture below

 

Shows the time line for the sample talk for which we had permission to show the emotional responses. Tags used were put into time slot bins and the size of the tags indicates the number of times that tag was used The EM appraisals are blue, the EM’s emotions are red and the EM’s feelings are green. We can see that, of the respondants, there was an over-whelming interest, with one respondant showing more negative emotions: “bored”, “bored”, “tired”, “restless” and “angry”. Of course, we’re only getting the headlines; we’re not seeing the reason or motivation for the responses. However, we can suspect that the negative responses mean that person didn’t like the presentation, but that there was a lot of interest, amusement and some pleasure derived from understanding the domain (“mastery pleasure”).

 

We think that this evaluation shows that the EM’s vocabulary works for the self-reporting of emotional response in an ontology conference setting. I’m prepared to say that I’d expect this to generalise to other settings for the EM. We have, however, only evaluated the ontology’s vocabulary; in this evaluation we’ve not evaluated its structure, its axiomatisation, or its adherence to any guidelines (such as how class labels are structured). There is not one evaluation that will do all the jobs we need of evaluation; many aspects of an ontology should be evaluated. However, fitness for purpose is, I think, a strong evaluation and when coupled with testing against competency questions, some technical evaluation against coding guidelines, and use of some standard axiom patterns, then an evaluation will look pretty good. I suspect that there will be possible contradictions in some evaluations – some axiomatisations may end up being ontologically “pure”, but militate against usability and fitness for purpose. Here one must make one’s choice. All in all, one of the main things we’ve done is to do a formal, experimental evaluation of one aspect of an ontology and that is a rare enough thing in ontology world.

 

Returning to the EM at ICBO 2012, we see what we’d expect to see. Most people that partook in the evaluation were interested and content, with some corresponding negative versions of these emotions. The ontology community has enough factions and, like most disciplines, enough less than good work, to cause dissatisfaction. I don’t think the ICBO audience will be very unusual in its responses to a conference’s talks; I suspect the emotional responses we’ve seen would be in line with what would be seen in a twitter feed for a conference. Being introduced to the notion of cognitive dissonance by my colleague Caroline jay was enlightening. People strive to reduce cognitive dissonance; if in attending a conference one decided, for example, that it was all rubbish one would realise one had made a profound mistake in attending. Plus, it’s a self-selecting audience of those who, on the whole, like ontologies, so overall the audience will be happy. It’s a pity, but absolutely necessary, that we haven’t discussed (or even analysed) the talks about which people showed particularly positive or negative responses, but that’s the ethical deal we made with participants. Finally, one has to be disturbed by the two participants that used the “sexual pleasure” tags in two of the presentations – one shudders to think.

Patterns of bioinformatics software and database usage

September 27, 2014

 

I published a blog on the rise and rise of the Gene Ontology. This described my Ph.D. student Geraint Duck’s work on bioNerDS, a named entity recogniser for bioinformatics databases and software. In a survey of Genome Biology and BMC Bioinformatics full text articles we saw that the Gene Ontology is in the top ten of mentioned resources (a fact reflected in our survey of the whole of 2013’s PMC). This interesting survey was, however, a bit of a side-show to our goal of trying to extract descriptions of bioinformatics and computational biology method from text. Geraint has just presented a paper at ECCB 2014 called:

 

Geraint Duck, Goran Nenadic, Andy Brass, David L. Robertson, and Robert Stevens. Extracting patterns of database and software usage from the bioinformatics literature. Bioinformatics, 30(17):i601-i608, 2014.

 

That has edged towards our ultimate goal of extracting bioinformatics and computational method from text. Ideally this would be in a form that people wishing to use bioinformatics tools and data to analyse their data could consult a resource of methods and see what was commonly done, how and with what it was done, what’s the latest method for data, who’s done each method and so on and so on.

 

Geraint’s paper presents some networks of interacting bioinformatics software and databases that shows patterns of commonly occurring pairs of resources appearing in 22,418 papers from the 2013 PMC corpus that had the MeSH term “Bioinformatics” as a tag. When assembled into a network, there are things that look remarkably like methods, though they are not methods that necessarily appear in any one individual paper. What Geraint did in the ECCB paper was:

 

  1. Take the results of his bioNerDS survey of the articles in PMC 2013 labelled with the MeSH term “Bioinformatics”.
  2. Removed all resources that were only mentioned once (as they probably don’t really reflect “common” method).
  3. Filter the papers down to method sections.
  4. Get all the pairs of adjacent resources.
  5. Assuming the most used ordering (“Software A takes data from Database B” or “Data from Database B is put into Software A”), we used a binomial test to find the dominant ordering and assumed that was the correct ordering (our manually sampled and tested pairs suggests this is the case).
  6. Resources were labelled as to whether they are software or a database. A network is constructed by joining the remaining pairs together.

The paper gives the details of our method for constructing patterns of usage and describes the evaluations of each part of the method’s outputs.

 

Some pictures taken from the paper of these networks created from assembling these ordered pairs of bioinformatics resources are:

 

Figure 1 A network formed from the software recovered by bioNerDS at the 95% confidence level

 

This shows the network with only bioinformatics software. In Figure 1 we can see a central set of sequence alignment tools, split into homologue search, multiple sequence alignment and pairwise sequence alignment tools), which reflects the status of these core, basic techniques in bioinformatics based analyses. Feeding into this are sequence assembly, gene locator and mass spectroscopy tools. Out of the sequence analysis tools come proteomic tools, phylogeny tools and then some manual alignment tools. Together these look like a pipeline of core bioinformatics tasks, orientated around what we may call “bioinformatics 101” – it’s the core, vital tasks that many biologists and bioinformaticians undertake to analyse their data.

 

The next picture shows a network created from both bioinformatics software and databases. Putting in both software and databases in Figure 2, we can see what the datasets are “doing” in the pipelines above: UniProt and GEO are putting things into BLAST; GenBank links into multiple sequence alignment tools; PDB links into various sequence prediction and evaluation tools.

 

Figure 2 A network formed from the bioinformatics and database recovered by bioNerDS at the 95% confidence level

 

Finally, we have the same network of bioinformatics software and databases, but with the Gene Ontology node (which we count as a database) highlighted.

 

Figure 3 The same network of bioinformatics software and databases, but with the Gene Ontology and its associates highlighted.

 

In another blog I spoke about the significance of the Gene Ontology, as recorded by bioNerDS, and this work also highlights this point. In this network we’re seeing GO as a “data sink”, it’s where data goes, not where it comes from – presumably as it is playing its role in annotation tasks. However, its role in annotation tasks, as well as a way of retrieving data, fits sensibly with what we’ve seen in this work. It may well be that we need a more detailed analysis of the language to pick up and distinguish where GO is used as a means of getting a selection of sequences one wants for an analysis – or to find out if people do report this activity. Again we see GO with a central role in bioinformatics – a sort of confirmation of its appearance in the top flight of bioinformatics resource mentions in the whole PMC corpus.

 

What are we seeing here? We are not extracting methods from the text (and certainly not individual methods from individual papers). What we’ve extracted are patterns of usage as generalised over a corpus of text. What we can see, however, are things that look remarkably like bioinformatics and computational biology method. In particular, we see what we might call “bioinformatics 101” coming through very strongly. It’s the central dogma of bioinformatics… protein or nucleic acid sequences are taken from a database and then aligned. Geraint’s paper also looks at patterns over time – and we can see change. Assuming that this corpus of papers from PMC is a proxy for biology and bioinformatics as a whole and that, despite the well-known inadequacy of method reporting, the methods are a reasonable proxy for what is actually done, BioNerDS is offering a tool for looking at resources and their patterns of usage.

Being a credible virtual witness

September 19, 2014

Tim Clark introduced me to the notion of a scientific paper acting as a virtual witness upon a scientific investigation. We, the readers, weren’t there to see the experiment being done, but the scientific article acts as a “witness statement” upon the work for us to judge that work. There’s been a deal of work over recent time about how poorly methods are described in scientific papers – method is key to being able to judge the findings in a paper and then to repeat and reproduce the work. Method is thus central to a scientific paper being a “credible virtual witness”. One of the quotes on Wikipedia’s description of credible witness is “Generally, a witness is deemed to be credible if they are recognized (or can be recognized) as a source of reliable information about someone, an event, or a phenomenon”. We need papers to be credible witnesses on the phenomena they report.

 

We’ve recently added to this body of work on reproducibility with a systematic review of method reporting for ‘omic experiments on a set of parasite host investigations. This work was done by Oscar Florez-Vargas, a Ph.D. student supervised by Andy Brass and me; the work was also done with collaborators in Manchester researching into parasite biology. The paper is:

 

Oscar Flórez-Vargas, Michael Bramhall, Harry Noyes, Sheena Cruickshank, Robert Stevens, and Andy Brass. The quality of methods reporting in parasitology experiments. PLoS ONE, 9(7):e101131, July 2014.

 

Oscar has worked for 10 years on the immunogenetics of Chagas disease, which is caused by one of the Trypanosoma parasites. Oscar wished to do some meta-analyses by collecting together various results from ‘omics experiments. He came with one issue of apparently contradictory results – Some papers say that the Th17 immune response, T regulatory cells and Nitric Oxide may be critical to infection and others say that they are not. Our first instinct is to go to the methods used in apparently similar experiments to see if differences in the methods could explain the apparent contradiction; the methods should tell us whether these results can be reasonably compared. Unfortunately the methods of the papers involved don’t give enough information for us to know what’s going on (details of the papers are in Oscar’s article). If we are to compare results from different experiments, we have to base that comparison on the methods by which the data were produced. In a broader context, method lets us judge the validity of the results presented and should enable the results in a paper to be reproduced by other scientists.

 

This need to do meta-analyses of Trypanosoma experiment data caused us to look systematically at a collection of ‘omic experiments from a series of parasite host experiments (Trypanosoma, Leishmania, Toxoplasma, Plasmodium, Trichuris and Schistosoma, as well as the non-parasitic Mycobacterium). Oscar worked with our collaborating parasitologists to develop a checklist of what essential parameters that should be reported in methods sections. This included parameters in three domains – the parasite, the host and the experimental infection. Oscar then used the appropriate PRISMA guidelines in a systematic review of 23 Trypanosoma spp. papers and 10 from each of the other organisms from the literature on these experiments (all the details are in the paper – we aimed to have our method well reported…).

 

We looked for effects on the level of reporting from organism and publication venue (various bibliometric features such as impact factor, the journal’s h-index and citations for the article).

 

Perhaps not unsurprisingly the reporting of methods was not as complete as one may wish. The mean of scores achieved by Trypanosoma articles through the checklist was 65.5% (range 32–90%). The method reporting in the other organisms was similarly poor, except in Trichuriasis experiments, which achieved the highest scores and included the only paper to score 100% in all criteria. We saw no effect of publication (some negative correlation with Google Scholar citation levels, though this is confounded by the tendency of older publications to have more citations). There has been no apparent improvement in reporting over time.

 

Some highlights of what we found were:

  • Species were described, but strains were not and it’s known that this can have a large effect on outcome;
  • Host’s sex has an influence on immunological response and it was sometimes not described;
  • The passage treatment of the parasite influences its infectivity and this treatment was often not reported;
  • Housing and treatment of hosts (food, temperature, etc.) effects infectivity and response to infection and these were frequently not reported.

     

We know method reporting tends to be poor. It is unlikely that any discipline is immune from this phenomenon. Human frailty is probably at the root – as authors, we’d like to think all the parameters we describe are taken into account in the experimental design. The fault is presumably in the reporting rather than the execution. Can we get both authors and reviewers to use checklists? The trick is, I suspect, to make such checklists help scientists do their work – not a stick, but some form of carrot. This is a similar notion that Phil Lord has used in discussing semantic publishing – the semantics have to help the author do their work, not be just another hindrance. We need checklists in a form that help scientists write their methods sections. Methods need to become a first class citizen in scientific writing, rather than a bit of a chore. Method is vital to being a credible virtual witness and we need to enable us all to be credible in our witness statements.


Follow

Get every new post delivered to your Inbox.

Join 162 other followers