APPLICABILITY OF AN ASSESSMENT MODEL FOR HEALTHCARE INFORMATION SYSTEMS IN A PUBLIC HOSPITAL

Assessment processes are essential to guarantee quality and continuous improvement of software in healthcare, as they measure software attributes in their lifecycle, verify the degree of alignment between the software and its objectives and identify unpredicted events. This article analyses the use of an assessment model based on software metrics for three healthcare information systems from a public hospital that provides secondary and tertiary care in the region of Ribeirão Preto. Compliance with the metrics was investigated using questionnaires in guided interviews of the system analysts responsible for the applications. The outcomes indicate that most of the procedures specified in the model can be adopted to assess the systems that serves the organization, particularly in the attributes of compatibility, reliability, safety, portability and usability.


INTRODUCTION
The global health observatory data repository, from the WHO, with information about healthcare investments in over 190 countries, shows a rising curve of expenses per capita in health (WHO,2013).According to Newell (2011), the costs for funding healthcare have grown globally due to factors such as increased life expectancy, advances in healthcare technology and policies for universal access to healthcare, in spite of government actions to mitigate budgetary impacts, with public budget constraints, above all after the economic crises that have occurred on a global scale, since 2008.In countries where healthcare systems are private or mixed, they also try to minimize these costs, to make insurance and health plan operators economically feasible.
In healthcare organizations, healthcare information systems (HIS) add information technology and communication to address their processes (Ammenwerth et al., 2004) and integrate people, procedures and technologies to collect, store, manipulate and recover information (Wager, Lee & Glaser, 2009).They are characterized by complex and multidisciplinary deployment, produce impacts in learning and in the adaptation to organizational routine and involve several groups of stakeholderspatients, service providers, regulating agents and professionals (Fichman, Kohli& Krishnan, 2011).
Investments in information systems can constitute part of the healthcare organization policies to reduce the tension between costs and budgets, in order to improve efficiency and quality in the processes that occur in this sector.The information systems improve healthcare organization efficiency, reduce medical prescription error rates, help professionals and managers in decision-making and in preventive medicine (Hillestad et al., 2006;Ammenwerth et al., 2003) and have great potential to reduce costs and improve healthcare outcomes (Fichman, Cohli&Krisnan, 2011;Oliveira et al., 2011).Research outcomes on the assessment of information systems for the healthcare sector also show how administrators recognize the importance of information systems as critical resources and that there is great demand to align information systems to the management process (D´Souza &Sequeira, 2011).
The development and maintenance of healthcare information systems are complex activities, due to: (a) lack of standardization and interoperability difficulties between applications (Hillestad et al.,2006),(b) the interdisciplinary characteristic of healthcare that demands added knowledge from several user professionals in the construction of information systems (Fichman, Kohli& Krishnan, 2011;Carvalho&Eduardo, 1998) and (c) the fragmented nature of the healthcare sector and the difficulties to systematize processes in applications (Abouzahr&Boerma, 2005), besides the actual change in paradigm, of a reactive model, centered on the disease, to a preventive model, that makes communication flow difficult among the three levels of attention and in continuous attention (OPAS, 2011).
Within this context of complexity, assessment is an essential activity to guarantee healthcare software quality as well as its continuous enhancement.Software assessment activities measure the attributes of a system in several phases of its lifecycle, help in the optimization of outcomes, identify unpredicted events (Ammenwerth et al.,2004) and allow us to analyze the degree to which information systems address their objectives (Yusofet al., 2008).This article specifies 42 software metrics as a technique to measure quality attributes established in a structured model guided towards healthcare information system assessment and verifies the feasibility of these metrics in applications that offer support to clinical, outpatient and administrative processes in a public hospital that serves the macro region of Ribeirão Preto.

THE QUALITY ASSESSMENT MODEL
The model proposed by Morais & Costa (2013), presented in Figure 1 was used as a theoretical reference.The model uses attributes from the product's quality dimension in Rule ISO/IEC 25010 (ISO, 2011a), which includes eight quality characteristics: functional supportability, performance efficiency, compatibility, usability, reliability, safety, maintainability and portability.Each characteristic is made up of a set of sub-characteristics, which are described in Attachment I.
Due to the degree of subjectivity found in the sub-characteristicsinherent to the model, as it can be applied to any software producteach attribute was associated to a set of indicators, obtained in a systematic research process in databases, which selected 32 indicators from seven relevant papers: Paiand Huang (2011), Viitanen et al.(2011), Hubner-BloderandAmmenwerth (2009), Ribière et al.(1999), Otieno et al.(2008), Anderson and Aydin(2005) and Lima et al.(2009).

(ISO, 2011b
) and ISO/IEC 9123-3 (ISO, 2003).The inclusion of these indicators covered all the quality dimension characteristics of the product, thus establishing the framework.
For each indicator, the model proposes assessment questions and/or software inspection procedures, applicable to different stakeholders, to objectively assess the indicator.The model includes the following as stakeholders:  Managers: users of the system at its strategic level;  Health Professionals: users of the system at tactical and operational levels;  Patients/users: access the system only for queries;  IT Professionals: compose the technical staff for the application, such as developers and/or maintainers of the information systems.
The assessment questions were elaborated and grouped into questionnaires guided towards user profiles (managers, health professionals and patients) and are specified in Morais & Costa (2013).The elaboration of questions was guided in the writing of clearlyexpressed texts without ambiguities and in language that was easy to understand.Measuring scales were also developed observing clear and appropriate reading and coverage of the universe of possible answers with uniform distribution.Ordinal and interval scales were used, according to the definition from Malhotra (2006), with five response options organized in increasing order of adequacy.
The inspection procedures proposed in the model must be directed towards IT professionals who maintain applications, stakeholders who have access to information about the requirements of the systems, track record of changes, track records of defects and system failures and other relevant information to obtain the measurements predicted.These procedures must use the documental analysis, tests/simulations of software use or performance measurements/essays as ways of obtaining measurements.
Attachment II describes the detailing and specifications of the model's assessment questions and inspection procedures.The first column of Attachment II indicates the stakeholder users of the application for which the assessment questions were guided towards.The metrics specified in the last column are directed towards the application maintainers.
When including questionnaires directed to the application users and inspection procedures guided towards maintainers, one distinguishing characteristic of this model is the fact that it "listens to both sides".Many times, conflicts are established due to lack of communication (Dallavalle, 2000), the distance between the organization and the service provider (Albertin& Moura, 1995) or due to inadequate implantation processes identified by Caldas & Wood (2000), as acquisition of software without clear-cut criteria, by imposition or low involvement of the user.

METHOD
The work done was qualitative and it was divided into a first conceptual phase, with the specification of inspection procedures for the model used and a second empirical phase, with the investigation of compliance to the metrics obtained in the first phase for three applications in a public hospital offering regional medium and high complexity services.
The first phase of this work is classified as methodological research as it refers to an instrument that takes in the reality studied: according to Vergara (2005, p. 47), the methodological research "is associated to paths, forms, ways, procedures to reach a certain end," characterized in this study.
The specifications described in this first phase used the following documents ISO/IEC 25023 (ISO, 2011b) and ISO/IEC 9126-3 (ISO, 2003) as a basis with context adaptations for healthcare information systems.The scales for the metrics specified in the inspection procedures are valued between zero and one: the closer they are to one, the greater the compliance of the system with the specified metric.
A survey was made in the second phase of the research using interviews guided towards system analysts responsible for the applications.A questionnaire was answered in the interviews, where the interviewer asked about the feasibility of each specified metric for the application maintained by the system analyst, with the response options "feasible/unfeasible/not applicable to the context".The option "not applicable to the context" refers to cases where the interviewee assesses that the metrics proposed were incongruous with the target system.Remarks from professionals were noted, regarding feasibility or nonfeasibility of the inspection procedures.
The interviews with the application maintainers were carried out in July 2013, in sessions of 90 minutes' maximum and included part of the time for suggestions given by the professionals.As a limitation, it must be observed that these studies were restricted to the analysis of procedures only for the stakeholders included in the assessment model, to verify and describe the applicability of the metrics proposed in the model.An assessment and application of instruments guided towards other stakeholders is a proposal for future research work.

The specifications of the software inspection procedures
Chart 1 describes the indicators and software metrics inspection procedures specified for the assessment model.The first column numbers its 38 indicators, classified in the model's characteristics and sub-characteristics.Inspection procedures were not specified for four indicators: "Extensive training is not needed to learn about the system", "The interface (screens, forms, data entry, reports or graphs) as well as all the terms and concepts used in the system are clear and have no ambiguities", "The system is easy to use, intuitive" and "The system presents a uniform and standardized interface", all of them have the usability characteristic.Psychometric assessment questions guided towards users were elaborated for these indicators, to answer how many system functions comply with the indicators, using an ordinal scale of five points: 0% (none), up to 25% (few), between 25 and 75% (about half), more than 75% (most) and 100% (all).
The second column of Chart 1 describes the metric specifications for the indicators assessed using inspection procedures.37 procedures are based on the use of metrics from the document ISO/IEC 25023(ISO, 2011b), four are based on rule ISO/IEC 9126-3 (ISO, 2003) and one was proposed by the author.Every metric is labeled with an abbreviation of the characteristic it refers to and numbered sequentially.Morais,M.C.,Somera,S.C.,Goes,W. M.,Costa,A. L. JISTEM,Brazil Vol. 13 NAS=verified number of simultaneous accesses by users, with performance within adequate standards NMA=maximum number of simultaneous accesses predicted in the system requirements EFDESEMP4=NAS/NMA Compatibility/Coexistence 1.The system predicts access to its data from other systems ISO/IEC 25023 -Coexistence availability metric: NADC: number of applications for which system makes coexistence available NARE: number of applications that require coexistence of system COMPAT1 = NADC/NARE Compatibility/ Interoperability 2.The system can be integrated or connected to exchange information with other systems ISO/IEC 25023 -Connectivity with external systems metric: NIOA: number of interfaces with other correctly deployed applications, to exchange data.NIR: number of interfaces required, for connectivity with other systems.COMPAT2 = NIOA/NIR Usability/Learnability 1.The system has manuals, tutorials, documentation for training and access to data and/or help online available ISO/IEC 25023 -user documentation completeness/help facilities metric NFDD: number of correctly described functions in the user documentation and/or in the online help functions for users NTF: total number of functions in the system accessible to users USAB1 = NFDD/NTF Usability/Operability 2. Browsing through the system is quick and standardized 3. The system offers adequate feedback to user for tasks performed 4. The software allows adaptations to address local/specific needs, by user himself ISO/IEC 25023 -Operational consistency metric: NFNU: number of functions with uniform/standardized browsing.NTF: total number of system functions accessible to users.USAB2 = NFNU/NTF ISO/IEC 25023 -Message clarity metric: NFMC: number of functions with messages that include easily understood terms NTF: total number of system functions accessible to users USAB3 = NFMC/NTF ISO/IEC 25023 -Customization possibility metric: NFC=number of functions that can be customized by user in operations, among those that require this resource.NTFC=number of functions that demand, in their requirements, the possibility of customization.USAB4=NFC/NTFC Usability/ Protection from user error 5. Simple, easy and safe to correct an error (reversibility) 6.The system deploys verification of valid values in data entries 7. The system avoids incorrect operations ISO/IEC 9126-3 -Ease to cancel metric: NFC: number of functions that allow cancellation of execution before the end.NTF: total number of functions in the system that predict cancellation of its requirements.USAB5 = NFC/NTF ISO/IEC 9126-3 -Ease to undo metric: NFD: number of functions that allow undoing of execution after ended NTF: total number of functions in the system that predict cancellation of execution in requirements USAB6 = NFD/NTF ISO/IEC 25023 -Entry verification and validation metric: NACV = number of entry attributes with verification for valid data NTAE = total number of entry attributes USAB7 = NACT / NTAE ISO/IEC 25023 -Prevention of incorrect operations metric: NFCO = number of functions that prevent execution of incorrect operations NTF = total number of system functions USAB8 = NFCO / NTF Usability/Esthetics of user interface 8. The arrangement of the interface fields are adjustable to the user's work ISO/IEC 25023 -Appearance customization of user interface metric: NIC: number of interfaces that can be customized in their appearance by user NTI: total number of interfaces with system user USAB9 = NIC/NTI Usability/Accessibility 9.The system includes access facilities for users with special physical needs/per age ISO/IEC 25023 -Physical accessibility metric: NFNE = number of functions accessible to people with special needs/elderly NTF = total number of functions in system USAB10 = NFNE / NTF Reliability/Maturity 1.The system presents low rates of software maintenance calls 2. The system is reliable, stable and errors do not occur when it is being used 3.The corrections, improvements or updating of version do not cause instability in the system ISO/IEC 25023 -MTBF -Media Time Between Failures metric: Observing the MTBF of the system as from its implantation (or the implantation of its latest version), in regular periods (half a year, one year), assess the evolution of metric CONF1= 0 if MTBF decreasing, 0.5 if MTBF constant and 1.0 if MTBF increasing ISO/IEC 9126-3 -Failure detection metric: CONF2=quantity of failures detected in last review/estimated failures Maintenance re-incidence metric (metric proposed by author): MR= for a given period, compile number of maintenance calls arising from prior maintenance calls, that generate side effects (and as a consequence make the system Reliability/ Availability 4. The system is always available, accessible to the user ISO/IEC 25023 -Rate of system in production metric: HSP = Hours in which system was in production, available to the user in a pre-defined period (last month, quarter, for example) NTHD = Hours in which system should be in production in a pre-defined period CONF4 = HSP / NTHD Reliability/ Tolerance to failure 5 Portability/Adaptability 1.The system operates in standard market environments (operational system, database, development tools, etc.) 2. The system presents independence and mobility to store and recover information (notebooks, tablets, PDAs, etc.) ISO/IEC 25023 -Software environment adaptability metric: NFAS = number of system functions tested successfully in other software environments (SGBDs, operational systems, development environments, etc.), besides the native environment NTF = total number of functions in system.PORT1=NFAS/NTF ISO/IEC 25023 -Hardware environment adaptability metric: NFAH = number of functions successfully tested in the system in other hardware environments besides the native environment.NTF = total number of functions in system.

PORT2=NFAH/NTF
Portability/ Capacity to be installed 3. Installation of system in user environment is easy and fast ISO/IEC 25023 -Ease of installation metric: NIS = number of successful installations, where installation occurred according to user convenience and in adequate timeframe.NTIS = total number of installations and attempts to install system PORT3 = NIS / NTIS

Feasibility study of inspection procedures used in assessment model
In this phase, a survey was carried out with information technology professionals from Hospital das Clínicas in the Medical School of Ribeirão Preto (HCFMRP-USP), to investigate the applicability of inspection procedures specified for the proposed assessment model, according to that specified in the method section.
The survey was performed at the Information and Analysis Center (CIA -Centro de Informações e Análises), the department responsible for information technology management at the hospital.The CIA provides the hospital with its own IT development and infrastructure that was gradually organized as from 1995, when the applications maintained by PRODESP were migrated.PRODESP previously offered information technology support for the hospital.
Currently, the CIA maintains 65 information systems in deployment or operation, 55 developed internally and 10 contracted from third-parties.As collaborators, it has 20 business and system analysts, 2 software quality engineers, 3 network administrators and 1 project manager, as well as a group of information technicians to provide support and service to the users.
The applications are developed in the Microsoft.Netplatform and the Oracle database manager, they use a UML (Unified Modelling Language) for analysis specifications and prototype techniques and tools for survey and specification of requirements.Some legacy Delphi systems are still maintained gradually being submitted to reengineering for technological update and review of functionalities.
The Senior Management of HCFMRP has sponsored policies to promote improvement of the software development process quality, with SBIS/CFM (SBIS, 2013) and MPS-BR (SOFTEX, 2013) certifications, to improve IT infrastructure development enhancement.
For the survey on applicability of inspection procedures proposed, three information systems were investigated.They support clinical and hospital processes at HCFMRP:  Laboratory Information System (LIS -Sistema de Informações Laboratoriais): offers support for management of scheduled examinations and performed in the different clinical analysis laboratories at the hospital.
 Clinical Service Elaboration (EAC -Elaboração de Atendimento Clínico): shows the observations and evolutions in patient care pointed out by doctors, paramedics and nurses, through electronic forms.
 Surgical Procedure Management (CIRÚRGICO-3 -Gerenciamento de Procedimentos Cirúrgicos): for surgeries performed at the hospital, this system manages waiting lists, scheduling, notes on surgical procedures, issuing of surgical records, consumption of material as well as analytical and managerial reports.
Chart 2 describes the outcomes obtained in the survey performed using questionnaires, where the system analysts responsible for the applications were interviewed, according to that described in the method section.The chart enumerates the metrics for each quality attribute.Note that the same names attributed to the metrics in Chart 1 were used.The affirmative answers are indicated with the symbol "▲", while the negative ones with "▼".The symbol "▬" indicates that the metric does not apply to the system context and "?"indicates that the question was not answered by the interviewee.The last column summarizes the explanations from the respondents about compliance with procedures.

Inspection procedures Applicati on
Observations/Comments from professionals interviewed

Quality Attributes Metrics
Functional supportability / Functional completeness SUPFUNC1 ▼ ▲ ▲ LIS and CIR3 have no requirement documentation or are not updated, different from EAC.

Functional supportability
/Functional adequacy SUPFUNC7 ▲ ▬ ▼ Depends on the information in requirement documents.
Performance efficiency/ Behavior with regard to time EFDES1 ▬ ▼ ▼ For the LIS analyst, performance is not a critical problem of the system; The analysts of EAC and CIR3 observe that there are no performance specifications in the non-functional requirements for these systems.
For USAB4, there were questions about the customization concept.
USAB5 and USAB6 depend on information found in requirement documentation, not always available.

CONF2
? ▲ ▲ CONF3 ▲ ▲ ▲ Reliability / Availability CONF4 ▲ ▲ ▲ For the metric CONF2 (MTBF), it was observed that the size of the system has to be considered, for its effectiveness.
For MAINT1 and MANUT2, it was questioned whether the metrics are in fact associable to the indicator.For MAINT4 and MAINT5, the absence of answers was due to lack of understanding/clarity in the definition of the metrics.
Metric MAINT6, that assesses testability, depends on the availability of test cases planned for the target system.There was a consensus for these attributes.PORT2 ▲ ▲ ▲ Portability/ Capac. to be installed PORT3 ▲ ▲ ▲ In absolute numbers, 69 feasibility indications were computed for the metrics proposed against 40 indications of non-feasibility (there were also 7 comments about absence of context of use and 10 questions were not answered).Graph 1 shows the distribution of answers per quality characteristic.

Graph 1. Distribution of responses in the model's quality attributes
As shown in graph 1, there was a positive consensus for all the metrics of Compatibility, Safety and Portability characteristics (fully compliant) and for most of the Usability sub-characteristics (partially compliant).It was observed that the applicability of metrics for Functional Supportability were conditioned to the existence of functional requirement documentation for the applicationsnot always available for the applications, which resulted in partial compliance with the procedures, as well as with two Usability metrics.
For the Reliability metric group, there was also a positive consensus, with the exception of the metric that assesses component redundancy, as there was no specification on the non-functional requirements of the infrastructure.For the metric MTBF (Media Time Between Failures), it was correctly observed that the procedure does not consider the size of the system and that the number of application functions must consider the measurement for calculation.
The absence of non-functional requirement specifications for response time, authentication time and simultaneous accesses made it unfeasible to use all the Performance Efficiency metrics.For the Maintainability characteristic, there was little feasibility: the outcomes show that the Testability depends on the availability of text cases for each system and indicate that some metric specifications in these groups must be reviewed, for better clarity, as this group concentrated the largest number of non-answered questions.
As suggested by this outcome, the analysts interviewed agree that many of the specified procedures can be adopted for assessment of the systems that address HCFMRP, also that they are aligned with the specifications recommended by certification SBIS/CFM (SBIS, 2013), recently implanted in the IT development and infrastructure areas.

FINAL REMARKS
For an assessment model for healthcare information systems based on the quality dimension of the product in rule ISO/IEC 25010 and in quality indicators researched in literature, this article describes the specification of a set of software inspection procedures associated to the indicators proposed and assesses the feasibility of these procedures at HCFMRP, a public hospital with regional coverage.
The architecture of the assessment model has as a distinguishing factor the application of assessment instruments to stakeholders with conflicting interests: its architecture predicts questionnaires directed towards different user profiles of the system and inspection procedures, with collection of software metrics with the system maintainers, using documental analysis, tests and software simulation use.This work includes as a study object the second group of assessment instruments for the model, directed towards application maintainers.
The feasibility study developed for systems that address HCFMRP suggests that most of the specified procedures are to the hospital context, particularly the metrics for the attributes Compatibility, Reliability, Safety, Portability and Usability.A longitudinal measurement strategy, with a view of historical assessment data can guide improvement processes and reengineering of systems.
Future work includes assessment together with other stakeholders not included in the assessment, other assessment studies of applications and the submission of the model and outcomes of assessments to specialist panels in healthcare information, organized in focal groups, for validation and adjustments of the framework developed.
When covering the main quality aspects of the healthcare information system domain, with a holistic range of indicators, questionnaires and software metrics, this work can contribute as another reference of studies that involve assessment processes of the technical quality of healthcare software and other areas of application, with adaptations.
Many managers have as a motto that everything that is managed must be measuredafter all, measuring allows for quantification and, consequently, more effective management.When identifying a demand to discuss the quality of healthcare information systems, it is also expected that the outcome of this work can add content and/or provide subsidies for projects that deal with standardization of assessment plans and monitoring of system quality and in enhancement projects of these software assets.

Sub-characteristics
Functional supportability: capacity of software product to provide for addressing explicit and implicit needs for which it was conceived.
Functional completeness: capacity of software product to provide an appropriate set of PALAVRA FALTANDO for user-specified tasks and objectives.Functional correctness: capacity of software product to provide, with the degree of precision necessary, outcomes or correct effects or as agreed upon.Functional adequacy: capacity of software product to facilitate the performance of user tasks and objectives.
Performance efficiency: capacity of software product to maintain an appropriate level of performance, when used in specified conditions.
Behavior with regard to time: capacity of software product to supply appropriate response and processing times, when the software executes itsPALAVRA FALTANDO, under established conditions.Use of resources: capacity of software product to use appropriate types and quantities of resources, when executing its PALAVRA FALTANDO under the conditions established.Capacity: Maximum limits of system parameters (items that can be stored, number of competing users, bandwidth, velocity of transactions, size of database, etc.) that address the requirements.
Compatibility: capacity of software product allowing exchange of information with other applications and/or sharing the same hardware or software environment.
Coexistence: capacity of software product to coexist with other independent software products, in a common environment, sharing common resources.Interoperability: capacity of software product to interact with one or more specified systems, through information exchange and use of information that has been exchanged.
Usability: capacity of software product, when it has effectiveness and efficiency to be understood, learned, operated and be attractive to user, when used under specified conditions.
Intelligibility: capacity of software product to make user understand if software is adequate and how it can be used for specific tasks and conditions.Depends on software documentation.Learning capacity of software product to make it possible for user to learn how to use it.Depends on software documentation.Operability: capacity of software product to make it easy for user to operate and control it.Protection against user error: capacity of software product in protecting the user from errors.Esthetics of user interface: capacity of software product to be attractive to user, when offering an interface with pleasant interaction.Accessibility: capacity of software product to be used by an ample range of people, including people with disabilities and with limitations associated to age.
Reliability: capacity of software product to execute its function in a continuous manner.
Maturity: capacity of software product to avoid failures arising from software defects, maintaining normal operation.Availability: capacity of software product to be operational and accessible when its use is required.Tolerance towards failure: capacity of software product to operate at a specified performance level in cases of software or hardware defects.Recoverability: capacity of software product to reestablish its specified level of performance and recover data directly affected in case of failure.
Safety: capacity of software product to protect information and data -non-authorized people or systems cannot read nor modify it and access to authorized people or systems is denied.
Confidentiality: capacity of software product to guarantee that the data is accessible only to people who have access to it.Integrity: capacity of software product to avoid non-authorized access for access or modification of programs or data.No questioning: capacity of software product to guarantee that occurrence of actions or events can be proved, avoiding future questioning.Accountability: capacity of software product to help the traceability of access to operations.Authentication: capacity of system to validate user identity.Maintainability: capacity of software product to be modified.The modifications can include corrections, improvements or software adaptations due to changes in the environment and in the requirements or functional specifications Modularity: capacity of system to have discreet components so that the modification of a component has a minimum impact in other components.Reusability: capacity of software components being used in other software or in the building of other components/systems.Analyzability: capacity of software product to allow for diagnosis of deficiencies or causes of failures, or the identification of parts to be modified.Modifiability: capacity of software product to allow for a specified modification to be deployed.
Testability: capacity of software product to allow software to be validated when modified.
Portability: capacity of software product to be transferred from one environment to another.Adaptability: capacity of software product to be adapted to different specified environments, without the need to apply other actions or measures beyond those supplied for this purpose by the software considered.Capacity to be installed: capacity of software product to be installed in a specified environment.Capacity to substitute: capacity of software product to be used to substitute another specified software product, with the same purpose and in the same environment.ISO/IEC 25023 -Coexistence availability metric 13) How many systems does the application have integration with through exchange of information and use of information that is exchanged, in cases where interoperability is required?ISO/IEC 9126-3 -Ease of cancellation metric ISO/IEC 9126-3 -Ease of undoing metric 24) In how many system functions can the interface be modified, with the appearance customized by the user?
ISO/IEC 25023 -user interface appearance customization metric 25) For how many functions does the system block incorrect operations?ISO/IEC 25023 Incorrect operation hindrance metric 26) For how many entries (attributes, fields) does the system not allow the entry of invalid or incorrect data?Always or almost always not available / Unavailable most of the time / There are regular periods when the system is unavailable / Eventually unavailable / Always available ISO/IEC 25023 -System in production rate metric 32) When there is data loss in the system (power outage, equipment failure, etc.), we can say that the data: is lost, cannot be recovered / is rarely recovered / Sometimes data is recovered, sometimes not/ Frequently, almost always it is recovered / Always recovered ISO/IEC 9126-3 -Restorability metric

Figure 1 .
Figure 1.Proposed assessment framework , No. 3, Set/Dez., 2016 pp.459-478 www.jistem.fea.usp.brChart 1. Indicators and software inspection procedures for the assessment model system offers support/ helps in decision-making 2. The system obeys legal information rules (CID10, DRG, data transmission, etc.) 3. The system helps to prevent medication errors 4. The clinical documentation generated by the system is correct and complete 5.The information treated by the system addresses the users' transactional operations ISO/IEC 25023 -Functional deployment coverage metric: NFDR= number of functions included in requirement documents with support for decisions NFAI = number of absent or incorrect functions among those identified in NFDR SUPFUNC1 = 1 -(NFAI / NFDR) NFDR= number of functions related to requirement documents that demand compliance with legal information rules NFAI = number of absent or incorrect functions among those identified in NFDR SUPFUNC2 = 1 -(NFAI / NFDR) NFDR= number of functions related to requirement documents that demand compliance with verification and prevention of errors in medication NFAI = number of absent or incorrect functions among those identified in NFDR SUPFUNC3 = 1 -(NFAI / NFDR) NFDR= number of functions related to requirement documents that demand functionalities that include clinical documentation NFAI = number of absent or incorrect functions among those identified in NFDR SUPFUNC4 = 1 -(NFAI / NFDR) NFDR=number of functions related to requirement documents referring to transactional operations NFAI = number of absent or incorrect functions, among those identified in NFDR SUPFUNC5 = 1 -(NFAI / NFDR) 25023 -Computational accuracy metric: NACI = number of attributes with incorrect computations in an execution verification in a certain time interval (execution simulation) NTCV = total number of attributes observed in the verification of execution SUPFUNC6 = 1 -(NACI/NTCV) 25023 -Functional adequacy metric: NFAI = number of system functions that do not deploy or partially deploy the automated support for integration between functional areas or departments, among those that demand this function NTFI = total number of functions that demand automated support for integration between functional or department areas, predicted in requirement documents SUPFUNC7 = 1 -(NFAI/NTFI) Performance efficiency/ Behavior with regard to time 1.The performance of the system is satisfactory: data is processed in an acceptable period of time; the system responds quickly to entries 2. The clinical documentation generated by the system addresses the time constraints demanded 3. Authentication time for system access is adequate ISO/IEC 25023 -Response-time and turnaround time metrics: TE: execution time of a simulation of online operation execution TME: maximum execution time of an online operation, specified in each non-functional performance requirement document of system ATR = 1 if TE<=TME; Opposite case EFDESEMP1 = Arithmetic average of ATR values, obtained in simulations on online operation executions TE: execution time of a simulation of the execution of a job TME: maximum execution time for a job, specified in each non-functional performance requirement document of system ATR = 1 if TE<=TME; Opposite case EFDESEMP2 = arithmetic average of ATR values TE: execution time for an authentication simulation TME: maximum execution time for an authentication operation, specified in nonfunctional performance requirement document of system ATR = 1 if TE<=TME; Opposite case EFDESEMP3 = arithmetic average of ATR values, obtained in simulations of authentication operation execution Performance efficiency/ Capacity 4. The system offers adequate simultaneous access, with satisfactory performance ISO/IEC 25023 -Number of simultaneous accesses metric: . The system has resources to store redundant data ISO/IEC 25023 -Component redundancy metric: NCR = number of components installed redundantly to mitigate failures NCRI = number of redundant components installed predicted in system requirements CONF5 = NCR/NCRI Reliability/Recoverability 6.The system present level of data loss and efficient restoration mechanisms ISO/IEC 9126-3 -Restorability metric: NRR = number of restorations required, in a given time interval NRRS = number of successful restorations required, in a given time interval CONF6 = NRR/NRRS Safety/ No questioning 1.The system includes a digital signature ISO/IEC 25023 -Digital signature use metric: NEAD = number of events processed using a digital signature NTAD = total number of events that demand a digital signature SEG1 = NEAD / NTAD Safety/ Confidentiality, Authentication and Integrity 2. There is no risk of unauthorized access to the system information ISO/IEC 25023 -Access controllability metric: TCAI = number of types of access controls (verification of illegal operation) correctly deployed and verified TCAP = number of types of access controls predicted in system requirements SEG2 = TCAI / TCAP ISO/IEC 25023 -Data encryption metric: NIDCE = number of data items correctly encrypted/decrypted NIEP = number of data items with encryption predicted in system requirements SEG3 = NIDCE / NIEP Safety/ Accountability 3. The system has auditing and /or tracing mechanisms ISO/IEC 25023 -Access auditability metric: NFRL = number of functions with log register, with information about access and/or modification of data made by a user NTF = total number of functions in system SEG4 = NFRL / NTF Maintainability / Modularity and Modifiability 1. Software integrates (easily) with new /systems ISO/IEC 25023 -Condensability metric: NCNA= number of components not affected by alterations in other components NTC= total number of components MANUT1 = NCNA/NTC ISO/IEC 25023 -Modification success rate metric: PRAM = number of problems/complaints before maintenance PRDM = number of problems/complaints after (same) maintenance MANUT2 = (PRAM-PRDM)/PRAM Note: The system calculates an average for the set of modifications over a given period Maintainability/ Reusability 2. The system has reusable software components ISO/IEC 25023 -Degree of reusability metric: NCRS = number of reusable software components/artefacts, that can be used in more than one system or used to build other systems NTCB = total number of components/software artefacts that can be reused in object library reused in development environment of software product MANUT3 = NCRS / NTCB Maintainability / Analyzability 3. Demands little effort to locate causes of failure in software ISO/IEC 25023 -Audit record of error causes metric: NRCE = number of records of error causes in system operations NPRCE = number of records of error causes planned sufficiently to monitor the system status during its operation MANUT4 = NRCE / NPRCE ISO/IEC 25023 -Diagnostic function sufficiency metric: NFDF = number of functions for diagnosis of failures available for the system NPFDF = number of functions for diagnosis of failures predicted in system requirements MANUT5 = NFDF / NPFDF Maintainability / Testability 4. The system can be efficiently tested after a modification ISO/IEC 25023 -Test case coverage metric: NCT = number of test cases prepared to test system functionalities NTCT = total number of test cases estimated for functional verification of system Note: Include test cases for integration with other systems/applications MANUT6 = NCT / NTCT

ISO
(learnability, operability, protection from user error, esthetics in the user interface and accessibility (QA=13, PI=9) G PS US 14) For how many functions does the system have documentation available (manuals, tutorials and training material) and/or help online for users?ISO/IEC 25023 -user documentation completeness and/or help metric 15) How many functions in the system are easy to understand, as from the system documentation?Not specified 16) In how many functions does the system have standardized access, with similar browsing and rapid access?ISO/IEC 25023 -Operational consistency metric

ISO
(maturity, availability, recoverability, tolerance to failure and non-questioning) (QA=6, PI=7) G PS 27) How frequently does the software go through maintenance to correct errors?Never or rarely/ Eventually/ Moderately/ Regularly/ Very frequently ISO/IEC 25023 -MTBF metric (media time between failures) and ISO/IEC 9126-3 -Failure detection metric 28) Do the corrections, improvements or updates of the version that occur in the system cause instability or require excessive effort or time?maintenance calls arising from previous maintenance, that generate side effects and do not allow for installation of system.NM= total number of maintenance calls CONF= MR/NM (metric proposed by author) 29) How many system functions that require digital signatures require this resource?often is the data available to the user, when required?

Source: adapted from ISO(2011a) Attachment 2. Assessment questions, scales and software metrics of assessment model Quality characteristics/sub-characteristics Stakeholders/Assessment questions (QA) Scales and Inspection Procedures (PI)
Morais, M.C., Somera, S.C., Goes, W. M., Costa, A. L.For how many functions does the system produce adequate feedback, with clear messages, that allow understanding of tasks that are executed?How many functions of the system present clear interfaces (screens/forms/data entries/reports/graphs), as well as terms and concepts used in the system, which are clear and have no ambiguities?In how many system functions can an action be reversed, in a simple, easy and safe manner?