Table 9 Metrics to evaluate and compare ontology and traditional data model approaches
Criteria | Metrics for ontology evaluation | References | Metrics for data model evaluation | References |
|---|---|---|---|---|
Flexibility | Easily adapted to multiple views in terms of parameters such as modularity, partitioning, context-boundedness | Gangemi et al. 2006 | Ability to deal with changes in business and/or regulatory rules/context? | Moody and Shanks 2003 |
| Â | Ability to accept input of new data from various research groups and disciplines | Maiga and Williams 2008 | Ability to add new data elements and relationships if project scope or regulatory rules (e.g. patient identification) change | Kahn et al. 2012 |
| Â | Easily re-define the extraction procedure logics and adapt it to user needs | Pannarale et al. 2012 | Flexibility of data models include "extensibility", "scalability", and "adaptability" as defined operationally below. | Kahn et al. 2012 |
| Â | Easily manage the changes of the database schema or the ontology | Pannarale et al. 2012 | Â | Â |
Reusability | Ability to integrate data so that it is useful to different users and disciplines | Maiga and Williams 2008 | Â | Â |
| Â | Ability to match user requirements across different disciplines | Pinto 2004 | Â | Â |
Scalability | Â | Â | Can data model be sized in smaller or larger data sets? | Kahn et al. 2012 |
Completeness | Â | Â | Does the data model contain all user requirements? | Moody and Shanks 2003 |
| Â | Â | Â | Can the data model store and retrieve data to meet investigator needs? | Kahn et al. 2012 |
Correctness | Â | Â | Does the data model conform to the rules of the data modelling techniques? | Moody and Shanks 2003 |
| Â | Â | Â | Does the model conform to good data modelling practices such as limited data storage redundancy? | Kahn et al. 2012 |
Extensibility | Â | Â | Can the data model expand data elements, data types and include new data domains? | Kahn et al. 2012 |
Adaptability | Â | Â | Can the data model represent a broad data domain? | Kahn et al. 2012 |
Cohesiveness | A measure of the separation of responsibilities and independence of components of ontologies | Yao et al. 2005 | Â | Â |
Precision | A measure of the amount of knowledge correctly identified in the ontology w.r.t. the whole domain knowledge available | Brewster et al. 2004 | Â | Â |
Recall | A measure of the amount of knowledge correctly identified with respect to all the knowledge that it should identify | Brewster et al. 2004 | Â | Â |
Fitness for purpose | Can the ontology define and assess if routinely collected EHR data is fit for purpose? | Wand and Wang, 1996; | Can the data model store and retrieve data to meet investigator needs correctly? (Note: Kahn defined this as completeness of the data model) | Kahn et al. 2012 |
| Â | Â | Liaw et al. 2011 | Â | Â |