The use of electronic patient record systems and their associated information applications has received considerable political support over the past decade. For example, in the United States, health care informatics was a primary component in the presidential campaigns of both Senators Barrack Obama and John McCain (Hoffman and Podgurski, 2008). Accordingly, after Obama was elected to the seat, the American Recovery and Reinvestment Act was passed in 2009. This law is traditionally referred to as the “Stimulus Bill”. In the following year after it was implemented, the Unite States government invested $148 billion in the health care of which about twenty billion was assigned to health care information technology. The objective of such an investment in IT was establishing the United States national system for the on-line storage and transfer of patient records. It is projected that all medical physicians and clinicians will acquire the status of “meaningful users” of such technology if they serve in Federal health care programs.
The Act created the position of National Coordinator for Health Information Technology. The people who hold this position are responsible for setting and managing the standards that regulation the functioning of electronic health records. Their roles revolve around improving the security of the healthcare information resources by setting standards that render inferior local solutions for privacy and encryption to be ineffective and insufficient (Johnson et al., 2009). The law also formed a Federal Coordinating Council for Comparative Effectiveness Research, which was tasked with making sure that future software innovations brought about obvious benefits regarding patient outcomes and delivery of service. The formation of such a council is significant since political support for such an initiative is, In part, dependent on the expectation that any financial input will raise the effectiveness of long-term health care provision. However, there have been several incidents of IT failure since the law was passed, one of the most notable being the United States’ Veterans’ Affairs server failure of 2007. This paper analyses this incident to try and develop suitable recommendations to avoid or handle such a failure in the future.
Currently, many public and private healthcare organizations are putting in significant financial investments into the development of healthcare information technology. Such applications are considered to bear several benefits to the user. For instance, software systems may facilitate communication between health care facilities regardless of the geographical distance between them. These systems support standardized procedures that may aid to improve consistency between various service providers. Electronic patient records provide for minimum standards along the line of care when patients move from one specialization to another. Health care information systems also present economic significance through efficiency savings, for instance, whereby they provide the information necessary for the identification of potential bottlenecks in providing and administering care. However, there have been several incidents of IT failure since the law was passed, one of the most notable being the United States’ Veterans’ Affairs server crash of 2007. This paper analyses this incident to try and develop suitable recommendations to avoid or handle such a failure in the future. The objective is to make sure that future attempts to extend the incorporation of electronic patient records will stem from the lessons learned from such failure.
The Veterans’ Affairs VistA Server Failure of 2007
In August 2007, there was a series of high-profile failure of health care information technology involving the Veterans’ Affair Administration, specifically in its Sacramento facility. This facility served as a data center created after a centralization process of the VA.
Background of the Incident
Every one of the VA’s a hundred and fifty medical facilities had their own budgets, staff, and IT services before centralization in 2005. After this remodeling, the VA transferred local responsibility for IT services to four data processing centers, a pair in the east and the other pair in the west. This centralization also affected certain development practices in the VA since changes were often made regionally or locally before 2005. An appropriate example would be decisions involving updates to the Veterans’ Health Information Systems and Technology Architecture (VistA) (Johnson, 2010). Such decentralized development practices before 2005 brought about scenarios whereby there were differing versions of an application from one facility to another. Local IT officials coordinated in such a way that many considered to be very responsive to local needs. Nonetheless, it undermined the standardization that is perceived to be vital for closer integration in such a large organization. When development practices are as spread out as they were previously, there are concerns over several non-functional elements such as security, disaster recovery, and infrastructure administration. Centralization made reporting and control models that changed the previously distributed framework of software development. For instance, a coding compliance tool was introduced in medical centers in Region One. Thus, all of the VA facilities in that region were supposed to run the same version of an application.
The August incident was the most disastrous of a series of more than fourteen IT breakdowns that took place since April 2007. It took more than eight hours to revive the systems in the seventeen centers that felt the impact of the failure, unlike other incidents which lasted for only several minutes (Johnson, 2010). The knock-on effects were felt in other VA hospitals and clinics as well as local pharmacies. The local pharmacies were affected because they used VistA application for orders and labeling.
Contingency Plan
At around 7:30 am on the day of the incident, VistA end-users in medical centers around Northern California found it impossible to log on to the Computerized Patient Record System (CPRS). They could not access the on-line records of the veterans under care in their institutions (Johnson, 2010). Given the patient safety concerns that resulted from this failure, staff turned to a three tier contingency plan.
The two data processing centres cater for what are referred to as Regions One and Two in Sacramento, California, and Denver (Johnson, 2010). The first of these contingency plans required that the services which were offered by Sacramento earlier to be handled by Region Two, which was in Denver. The second contingency plan followed a similar approach but considered it impossible for local sites to make updates on the central copy of the patient data in Sacramento. The changes in patient care were to be logged on a local basis, and the update on the central records would have to be made after the system got back to normal functionality. The last level of defense required the affected health care facilities to utilize the local files stored on their computers even though they only contained short summaries of patient information (Schaffhauser, 2007). These records also only had information for patients on-site and those who were booked for appointments in the following two days. Therefore, the IT failure meant that clinicians would not access information concerning any patient who needed immediate and unscheduled care.
Recovery Actions
Whereas, before 2005, local staff would seek the help of their local support officers in estimating the probable duration of a disruption, centralization meant that this personal contact would be lost almost entirely. The VA increased the duties assigned to regional data centers. When the failure occurred, the technical staff of the VA was working hand-in-hand with an external contractor in evaluating the effectiveness of a hardware platform that was running on a specified virtual memory setting (Johnson et al., 2009). Therefore, at the time of the incident, there was a significant number of people to start diagnosing the root of the matter and share the workload in the response to the failure. However, this increased the risk associated with maintaining shared data across a large group of workmates.
When all the clinical staff had begun using the paper-based approach or the “read only” access to the other servers, the staff at Region One started the process of evaluating the root of the technical error. The initial report held that the cause of the problem was a change in the network port settings for the servers that allow for the sharing of resources between the VA facilities. The change caused a mismatch between the speed of the servers of Region One and that of the telecommunication switch (Johnson et al., 2009). It was determined that this change in settings was executed without complying with documentation and approval practices that would have allowed the different support teams to have knowledge of the modification. Therefore, the change in port configuration had to be “rolled back” and the reconfiguration was carried out afresh.
The senior management of the VA decided that the seventeen VistA systems that were run by the Sacramento facility had to be shut down so that they could be revived one after the other. The order that was to be followed was dependent on the workload of each system starting with the least functional. In so doing, they hoped that, if bringing back these systems was going to expose more problems, then the effects would be minimized since the facility would not be running at full capacity (Johnson et al., 2009). During this period, the communication between the support staff and the healthcare centers was almost continual as they were assessing the impact fo the recovery plan. Even the team established that the port reconfiguration was the primary cause of the problem, the bigger task was restoring data integrity.
It was necessary to update the on-line records with the latest orders and procedures that were entered in all the seventeen centers other linked sites that were affected by the failure. It was six days after the system was successfully restored that the medication administration records were brought up to date. It took a further two months for the administrative staff to deal with the paper backlog from the tests and consultations that could not be entered directly into VistA and the other systems after the failure at Region One data center (Johnson et al., 2009). Amidst concerns over patient information safety, the Associate Chief of Staff of the VA in Northern California reported that,
“Many providers have reverted to the Computerized Patient Record System within VistA and tried to regenerate a summary of the paper notes that they wrote to mitigate the risk of missing information. This work to reinstate medical records’ integrity will continue for many months because a lot of information was recorded on paper that day. Considering that several screening exams for Post Trauma Stress Disorder, depression, use of alcohol, smoking among other records are all now on paper and are not in a form that can be accessed quickly in the electronic registry, the burden of the single mishap will persist for a long time” (Conn, 2007, para. 8).
Conclusions and Recommendations
The introduction presents the political aspect of Information Technology management in health care institutions in the United States. It seeks to show the importance of information governance in the political atmosphere in the country. However, implementing such laws has proven to be costly in some cases, as shown in the United States’ Veterans’ Affairs server failure of 2007. The failure occurred in series, the most severe one happening in August that year. Even though, the system was revived in the affected facility, plenty of time, for instance, was lost during recovery. This calls for recommendations regarding the most appropriate practices that health care institutions would adopt to avoid such a failure in the future.
Adopting a Custom System
Health care institutions, like the Veteran Affairs’, should consider adopting a custom system because it is considerably faster than a proprietary one. Custom systems would provide these institutions with more flexibility in many cases. On the other hand, proprietary software is slow to give fixes for bugs, as is the case in the discussion in the previous section.
Project Metrics and Portfolio Management
Information Technology projects require significant amounts of financial investment, thereby creating the need for evaluation – project metrics. Assessments enable stakeholders to determine what aspect of the project has worked and what has not. The lessons learned aid in fostering effective health care IT implementation and application. Portfolio management is equally important because it provides stakeholders with the collaboration and sight needed when responding to critical mandates in a prompt fashion while developing other areas of the organization.
Study the Feasibility of a Uniform Coding System
Health care institution should conduct research on the effectiveness of a single coding system that would be used in all health care services performed by all providers in all contexts. There would be concerns about the impact in the event of failure, but a study on the safest coding system would lower the probability of such an occurrence.
References
Conn, J. (2007). California System Faced Epic Vista Failures: Report. Modern Healthcare, 1st October. Available at http://www.modernhealthcare.com/article/20071001/INFO/310010001
Hoffman, S., & Podgurski, A. (2008). Finding a Cure: The Case for Regulation and Oversight of Electronic Health Record Systems.
Johnson, C. W., Fletcher, L. L., Holloway, C. M., & Shea, C. (2009). Configuration Management as a Common Factor in Space-Related Mishaps.
Johnson, C. W. (2010). Case Studies in the Failure of Healthcare Information Systems.
Schaffhauser, D. (2007). The VA's Computer Systems Meltdown: What Happened and Why. ComputerWorld, 20th November.
