Challenges for Regulated Life Sciences Companies within the IaaS Cloud
By the GAMP Cloud Computing Special Interest Group (SIG)
This article presents the key items that must be addressed to adopt an IaaS model within a regulated firm.
In the introductory article of this series, Cloud Computing in a GxP Environment: The Promise, the Reality and the Path to Clarity (Pharmaceutical Engineering: January/February 2014), an overview was provided of some of the primary challenges and/or concerns the regulated industry continues to debate as to whether cloud solutions can be adopted or not. Key issues from that discussion are highlighted as:
- The regulated company must be willing to give up levels of “hands-on” control and oversight to the cloud provider, yet remain accountable for the integrity and compliance of the solution and data.
- The regulated company should apply the same good outsourcing/supplier management practices that it operates for all other outsourcing activities.
- Cloud vendors typically approach quality assurance or the traditional Quality Management System (QMS) differently from the regulated industry
The GAMP COP Cloud SIG was formed of representatives of a cross-section of small and large life sciences companies and cloud service provider SMEs. The perspectives provided within this article represent real-world experience, research, and interaction with both the cloud provider industry, and various regulatory bodies.
This article will focus on the key items that must be addressed to adopt an IaaS model within a regulated firm. To do that, one must first understand what IaaS is, the variation of the model offered by the provider, and what compliance-enabling services the provider can supply. One of the most useful IaaS definitions is provided by the National Institute of Standards and Technology (NIST), which defines IaaS as
“The capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, and deployed applications; and possibly limited control of select networking components (e.g., host firewalls).”
Figure 1 depicts the shift in ownership of infrastructure from the traditional IT approach. (Light blue denotes the elements of greatest risk in this model. Note: this figure includes the full range of IT outsourcing models for the sake of completeness. This article focuses on the models enclosed within the dotted lines.)
To understand IaaS, one must understand that the IAAS model is not just hardware. When a customer requests an environment, a significant amount of automation has been developed behind the scenes to build and deploy an environment, potentially inclusive of networking, storage, and virtual machine.
There are some who advocate that this infrastructure layer has become a commodity, and that the abstraction enabled by virtualization, creates an environment that has extremely low potential for impacting the platforms and applications in the upper layers. By virtue of not being tied to any specific server, this infrastructure model provides a layer more fault-tolerant and highly available than dedicated hardware. Despite these benefits, the regulated customer’s data is processed by such virtual machines and is stored at the supplier on the supplier’s hardware, and therefore the current definitions of a computer system per the FDA and EU Annex 11, still apply, (as do the associated requirements for the IaaS supplier to maintain specifications and diagrams). When it comes to the size of the supplier, the question becomes the ability to “partner.” Larger suppliers may have great performance metrics, but the pure size of the customer base makes it difficult for them to agree to contract terms or perform activities and documentation unique to any one customer.
All IaaS providers are not the same and a variety of business models with varying capabilities exist. All hope to support the regulated life sciences industry. Figure 2 is one possible way to classify these providers, but is not in any way considered absolute. We expect this to continue evolving as awareness and understanding increases. Choosing a supplier with lower cost generally places a greater burden on the regulated company to establish qualification processes.
Figure 2 is based on experience from the SIG members, and illustrates the potential for a wide disparity in IaaS vendors’ awareness and their understanding of quality assurance or a QMS as it pertains to the life sciences.
Many of the IaaS providers view quality as providing high availability and data integrity, which are only elements of what the regulated company defines as quality. Is a QMS implemented? And if so, has the infrastructure undergone formal qualification in line with GAMP® 5 practices, or has it been implemented following good IT practices? The following parameters lead to the understanding of the differences between IaaS cloud providers in the market:
- Qualification Documents: the provider executes and maintains the objective evidence that the IaaS solution is qualified and sufficiently documented as expected with reference to the Infrastructure GPG. Generally, good documentation practices are followed.
- Customer Specific Change Agreement: the provider agrees to key customer specific requirements related to the change process, and incorporates these elements into their daily operations.
- Qualification/Verification Assistance: the IaaS provider is aware of the regulated life sciences industry QMS definition, but not having an established QMS, may have tools, white papers, etc., that can be leveraged to assist the regulated customer in executing qualification documents.
- Permits Onsite Audit: the provider will allow the customer to conduct quality and security audits at the provider site(s). Dependent on the provider, this may be considered standard, or will be allowed via pre-agreed limitations and incremental fees (time and material).
- Enterprise Scale: the SIG’s experience indicates that non-GxP service providers typically have business models and capacity to handle requests for large scale operational tasks.
- Cost profile: the providers with no formal QMS or quality awareness are usually the lowest cost – and could be the right solution for some low risk business activities. As additional customer specific services are provided, the cost increases
Deciding What to Move to the Cloud
The approach to IaaS should be based on the regulated company’s risk assessment factoring in product and process in combination with the intended use of the IaaS. The risk assessment should incorporate the five key-elements from GAMP 5 in Figure 3, but should focus on the two areas at the supplier end (those in the green circle). For the most part, impact/risk to the end user will be consistent, since in most cases the infrastructure layer will not have a product-specific issue. The five-stage risk assessment framework in the accordance with GAMP 5 principles would be helpful.
Key risk considerations are:
- An understanding of both the functions and risks of the supporting and business process(es) is needed before the risks associated with specific functions of the IaaS can be assessed.
- Focus on risk to patient safety, product quality, and data integrity. Specification of requirements should include mitigating critical risks. The extent and detail of the requirements specification should be based on the associated risk, complexity, and novelty of the IaaS. Once the risks are understood, refer to the the GAMP Testing GPG,1 Appendix E2 – Testing of Cloud Applications, for further guidance on the overall testing approach.
In some situations, the outcome of the risk assessment may indicate, that an IaaS solution is not acceptable; whereas in other situations it could be used with very few controls. In any case, the decision must never have a negative impact on patient safety and data integrity.
IAAS and Regulatory Controls
The required controls for the regulated company are independent of whether the infrastructure is in house or is outsourced to a supplier, e.g., in a IaaS cloud solution. Table A lists the basic regulatory requirements of the end-user and related consequences to the IT delivery/IAAS from an operational level.
In general, the activities at the provider site should comply with the principles of the regulations listed in Table A, even if the approach taken differs from the client company’s internal processes. From a technical point of view, many of the requirements are adopted from, e.g., ISO 900033 and ISO 27001, section 7.4 The IaaS implementation should be based on the supplier’s domain knowledge and QMS (or equivalent), and not verbatim from the regulated company’s point of view.
- 1ISPE GAMP® Good Practice Guide: A Risk-Based Approach to Testing of GxP Systems, International Society for Pharmaceutical Engineering (ISPE), Second Edition), December 2012, www.ispe.org.
- 3ISO 90003:2004, Software engineering — Guidelines for the Application of ISO 9001:2000 to Computer Software.
- 4ISO 27001:2005, Information Technology - Security Techniques - Information Security Management Systems – Requirements
|
Regulation |
Phrase |
Consequence |
|
|
EU, GMP Volume 4, Annex 11: Computerized Systems |
Principle |
The application should be validated; IT infrastructure should be qualified. |
In practice, basic principles described in the ISPE GAMP® Good Practice Guide: IT Infrastructure Control and Compliance Guide should be applied, both at the provider’s site, and at the regulated company. |
|
§1 |
As part of a risk management system, decisions on the extent of validation and data integrity controls should be based on a justified and documented risk assessment of the computerized system. |
It requires that data integrity and security controls at both the provider site and the regulated company are handled as described in GAMP® 5 (Appendix M3: Science Based Quality Risk Management) and the ISPE GAMP® Good Practice Guide: IT Infrastructure Control and Compliance Guide. The Challenge: the provider has the key knowledge related to their business, and should answer questions related to the IaaS. The regulated company should have controls in place to verify if the solution satisfies business needs and risk tolerance. |
|
|
§2 |
All personnel should have appropriate qualifications, level of access and defined responsibilities to carry out their assigned duties. |
The principles described in the ISPE GAMP® Good Practice Guide: IT Infrastructure Control and Compliance Guide should be applied at the provider site. In practice it means “limit the required qualifications” and focus on the essential discipline for infrastructure. This is intended to simplify and not complicate. |
|
|
§3.1 |
…. and these agreements should include clear statements of the responsibilities of the third party. |
In practice: 1) Provisioning the service. 2) Ongoing management and maintenance of the service will be for a substantially longer period. In both cases, roles and responsibilities should be clear, and both the SOW (provisioning) and SLA (ongoing) are important. |
|
|
§3.2 |
The need for an audit should be based on a risk assessment. |
The requirement doesn’t state that an audit must be conducted – but the regulated company must from a risk-based approach evaluate the provider’s ability to comply with regulations. Principles from GAMP® 5 (Appendix M2 Supplier Assessment) could be applied. . Relevant certifications may provide an opportunity for a simplified or expedited assessment of the provider. The scope for the assessment should be clarified based on the risk assessment. |
|
|
§3.4 |
Quality system and audit information relating to suppliers or developers of software and implemented systems should be made available to inspectors on request. |
It should be possible to obtain evidence of an appropriate assessment process and subsequent judgement of supplier suitability, including significant findings and outcomes should be made available to regulators on request. Some detailed aspects of assessment finding, especially those related to supplier intellectual property and technology may be covered by confidentiality agreements between the regulated company and the supplier. The Challenge: if a regulator requests supplier information, a request may be passed on to the supplier - and when necessary further confidentiality agreements discussed. In the case of service suppliers of high risk processes, contracts should notify them of the possibility for direct inspection and request timely access to their QMS if needed during regulatory inspection. |
|
|
4.5 |
The regulated user should take all reasonable steps to ensure that the system has been developed in accordance with an appropriate quality management system. The supplier should be assessed appropriately.
|
In general, GAMP® 5, Chapter 4 related to the project phase describes all validation activities aligned with general validation principles. These activities should also cover the IAAS elements of automation used to provision the infrastructure with reference to the principles in Annex 11. |
|
|
§7.1 |
Data should be secured by both physical and electronic means against damage. Stored data should be checked for accessibility, readability and accuracy. Access to data should be ensured throughout the retention period. |
This requirement should be implemented via controls both at supplier site and at the regulated company site. This involves implementation of the proper security controls, verification of backups, periodic testing of restoration, and contingency planning.
In practice the requirements for disaster recovery at supplier side may be limited to just Recovery Time Objective (RTO) in reality. Recovery Point Objective (RPO) may also be stated in the contract. |
|
|
§12.1 |
Physical and/or logical controls should be in place to restrict access to computerized systems to authorized persons. Suitable methods of preventing unauthorized entry to the system may include the use of keys, pass cards, personal codes with passwords, biometrics, restricted access to computer equipment and data storage areas. |
Consideration should be given to, e.g., establishing VPN connectivity, so that the regulated company standard identity access management practices could be leveraged. These requirements are fully aligned with good engineering practice as described in e.g. ISO 27001, section 7.2 |
|
|
21 CFR 211 Current Good Manufacturing Practice in Manufacturing, Processing, Packing, or Holding of Drugs; General and Current Good Manufacturing Practice for Finished Pharmaceuticals |
§211.25(a) |
… shall have education, training, and experience, or any combination thereof, to enable that person to perform the assigned functions.
|
The principles described in the ISPE GAMP® Good Practice Guide: IT Infrastructure Control and Compliance Guide should be applied at the provider site. In practice it means “limit the required qualifications” and focus on the essential discipline for infrastructure. This is intended to simplify and not complicate. |
|
§211.68(b) |
A backup file of data entered into the computer or related system shall be maintained …. designed to assure that backup data are exact and complete and that it is secure from alteration, inadvertent erasures, or loss shall be maintained.
|
This requirement relies on the associated risk assessment, and must be conducted according to the principles described in the described in GAMP® 5 (Appendix M3 Science Based Quality Risk Management) and the ISPE GAMP® Good Practice Guide: IT Infrastructure Control and Compliance Guide, both at the supplier site and at the regulated company. |
- 2ISO 27001:2005, Information Technology - Security Techniques - Information Security Management Systems – Requirements
To secure quality and security in the deliverables from the IaaS provider site, there should be quality assurance activities in place monitoring processes and methods used to ensure quality. The methods by which this is accomplished could include ensuring conformance to one or more standards, such as ISO 9000 and/or ISO 27001.
In practice, there should be a quality responsible person at the IaaS provider organization with the authority to reject or approve deliverables to the customer. Generally, a quality-responsible person should also approve periodic evaluation of the different GxP-related services at the IaaS provider.
Industry Expectations Based on GAMP Approach
Qualification/verification is a process that demonstrates an entity has fulfilled specified requirements. In the context of an IaaS provider, this means demonstrating the ability of components such as network, firewalls, and Storage Area Networks (SANs) to fulfill the specified requirements for the various platforms regardless of whether they are specific or of a generic nature.
Basic requirements of qualification in the context if IaaS includes:
- Involvement of IaaS quality-responsible person at the appropriate level
- Formally verified and approved design solutions that meet specified requirements
- Documentation quality, detail, and degree of testing consistent with the level of risk of the specific use case.
- Tests and verifications aimed at establishing conformance to specifications
- Provisions to ensure that the qualified status of the entity is maintained
- Traceability of actions and activities
An IaaS providers’ activities should be based on a lifecycle approach covering all phases from initial requirements until retirement including design, specification, programming, testing, installation, operation, and maintenance, even if it is infrastructure.5 In reality, the ongoing deployment of new components and the client awareness of such may be based on the size of the vendor and ability to have a change agreement in place. The goal is to leverage the tools and the processes of the provider that have been assessed to meet compliance requirements, and incorporate outputs from these tools in the regulated company qualification documents as supporting evidence. A regulated firm’s qualification strategy should be established after performing the assessment of the provider’s QMS and core processes. Scope and depth of the qualification should be determined by a risk assessment.
When a customer wants its own service, based on an IaaS cloud providers standard service, this should be stated in the contract with additional details provided within the service level agreement (discussed later in the contract). In general, a gap analysis with focus on general requirements from the regulated company should be conducted, to verify customer needs and policies. Gaps should be handled appropriately.
| Qualification of components (building block system for VLANS, switches, routers, back-up systems, monitoring etc.). Activities also reflect tools and implicit automation software for deployment of IaaS | |
|---|---|
| IaaS Providers Activities | Activities in the Regulated Company |
|
Applies to components to be included in the technical solution regardless of automated or manual deployment of the IaaS: Input to the process: Output from the process: |
Vendor assessment/on-site audit Input to the process: The output of the gap analysis is a list of activities to be done before the approval status can be stated. This could include: |
| Qualification of Automated Deployment Process | |
|---|---|
| IaaS Providers Activities | Activities in the Regulated Company |
|
Input to the process: Output from the process: |
Input to the process: Output from the process: |
| Operation and Maintenance | |
|---|---|
| IaaS Provider Activities | Activities in the Regulated Company |
|
The IT infrastructure typically changes frequently – sometimes on a daily or hourly basis depending on the size of the infrastructure. The IT infrastructure should be maintained in a documented state of control by ensuring appropriate: • Change Management It is expected, that the IaaS provider periodically evaluates the complete service (standard service and associated customer approved services) to ensure that the service are in control. This should be accomplished via management review, internal audit, or similar business responsibility and include any/all of the following: • Key Performance Indicator’s (KPI) |
The regulated company will maintain the application in a manner equivalent to how applications hosted internally would be managed. These items include:
Vendor management becomes a key task in a cloud model. Ongoing activities in this model include: • SLA Management |
Service Provider and Regulated Firm Shared Activities
The adoption of an IaaS model will normally be done in two steps:
- Qualification of the environment including:
- - Qualification of the infrastructure components to be used as shown in Table B
- - Qualification of the automated deployment and configuration of the customer specific IaaS (if applicable) as shown in Table C
- Definition of the operation and maintenance roles and responsibilities between the IaaS provider and the regulated company. The activities listed in Table D are all activities, that are related to the operational and maintenance of the IaaS from the regulated point of view.
These two bullets above should be reflected in the SOW and or contract.
The tables shown list different activities from a lifecycle approach related to IaaS provider and regulated company.
(The deliverables in the following tables reflect what may be required. However, depending on the risk associated with the desired use case and degree of automation available from the provider, a subset of these deliverables may be enough to satisfy requirements).
Table D indicates activities related to the automated deployment process that may be compared to and considered equal to the disciplines of a facility process control system. The automated deployment process is normally based on a generic tool, where appropriate business specific code is implemented to reflect the IaaS providers’ proprietary method of deploying infrastructure solutions.
The operation and maintenance activities listed in Table 3 should secure the IaaS availability and performance in operation for which it was designed. The activities are generally based on the GAMP® Good Practice Guide: IT Infrastructure Control and Compliance Guide and don’t differ from the main concept in this guidance. Regardless of the difference in cloud models, the intensions in the guidance still are applicable.
The Contract
The contractual agreement with an IaaS provider is a critical tool for effectively managing the relationship between the regulated company and the provider. It must be treated as a partnership between the two entities. Beyond the standard legal agreements and warranties, the contract should have at least the following three key sections to ensure expectations are clear. These sections may be exhibits, addendums, or independent documents but should include:
Compliance Requirements or Quality Agreement
Although the regulated company still has the responsibility for deploying and maintaining the operating system(s) and applications running in the IaaS model, the data is located at the supplier, and therefore, the supplier must provide and/or maintain certain elements over the lifecycle of the relationship.
Compliance requirements. To the regulated firm, the presence of the elements is not the end point. SOPs need to have a periodic review, and there needs to be evidence that existing procedures are followed. Compliance requirements include:
- Change Management SOPs - change records should be executed for all changes to infrastructure and maintained for the life of the combined set of hardware, software, networks, facilities, etc. that comprise the infrastructure.
- Incident Management - scope of incidents should include excursions from specified environmental parameters, unexpected outages/failures, and items impacting performance of hardware/networking equipment. Incident records should be maintained and associated with any root cause analysis or CAPA items resulting from the incident investigation.
- Individuals responsible for data center operations and infrastructure support shall be adequately trained and qualified for the role(s) being performed.
- Training procedures shall exist to ensure personnel associated with the procured services are trained for their role. Evidence of training shall be captured and maintained.
- Infrastructure and network diagrams. The diagrams should be readily available to reflect current state.
- Backup and Recovery SOPs of the supplier’s infrastructure and configuration settings should be maintained, with documented evidence of periodic testing captured.
- Adherence to SOPs for the qualification/deployment of infrastructure items with documented evidence available. If highly automated, tools should be available to provide reports verifying that the provisioned service matches requirements.
- Timely support in the event that the regulated company is subject to a regulatory authority audit.
- Notification in the event the supplier becomes subject to such an audit.
- Exit Strategy – based on the scenario, expectations of the supplier in the event that the relationship terminates, should establish what data, any other assets, and method of retrieval/media would be used to recover the information.
Statement of Work (SOW)
Specified deliverables in the SOW should be focused on supplier commitments that are required for the regulated company to go-live in production with the desired services. These should include:
- Any documents as described in Table A that are regulated customer specific should be in place prior to moving forward with the supplier.
- Remediation items identified during the audit/assessment process should be contractually agreed to with target dates for implementation. These items should be verified as being in place prior to any deployment.
Service Level Agreement (SLA)
Defined operational measures of the service on an ongoing basis. Elements should include:
- Escalation and communication processes should be established in the event of major incidents that have significant risk to the service availability, reliability, and data integrity.
- Specific requirements of the supplier in the event the regulated company is audited by a regulatory authority, such as providing process documentation and evidence within a specified time frame.
- Notification of proposed/planned changes. The regulated company requires time to evaluate the change, and determine the degree of testing required to be performed before migration to production. Changes that pose little or no risk to the regulated company platform, should be classified as such, and may be implemented without the regulated company testing.
- Key compliance metrics such as definition of incident severity, time to perform a root cause analysis report from an incident occurrence, change controls executed successfully first time, etc.
Conclusion
In an IaaS model, the regulated company is still the accountable entity for the application and data. IaaS solutions should be qualified based on the principles in the GAMP® Good Practice Guide: IT Infrastructure Control and Compliance. This article demonstrates that the predicate rules are applicable regardless of where the infrastructure resides. The continuing challenge is how the service provider’s network, supporting systems, and processes for deployment of the IaaS solution meet what is described in the GAMP® Good Practice Guide: IT Infrastructure Control and Compliance Guide.
Sufficient documentation and controls, and effective supplier management may permit an IaaS to be verified as fit for purpose according to principles from ISO90003 and ISO27001. The exact format of the controls may be based on the technology used to provide the controls and should be assessed in that light.
The intention of the future SIG work is to describe in detail how to address supplier activities related to the key concepts shown in Figure 3 in a manner that meets regulations and satisfies client needs. Future articles in this series will focus on PaaS and SaaS models, and illustrate how the risks associated with cloud exist in all models, but the risk elements shift as the provider’s processes and controls become more closely tied to the O/S and application(s).
- 5Annex 11 Definition of Lifecycle
