PURPOSE

This document is intended to provide general procedures for requesting, reviewing, approving, and tracking changes to centrexIT (cIT) and Customer Production Systems.

SCOPE

This procedure applies to changes to production systems that would require an update to any controlled system documentation, diagrams, procedures, or system availability, or cause any production systems to be modified in such a way as to present new risk, changes in cost, new potential vulnerability, or potential loss of function to any critical network, or application systems (PBA/s) with which a customer or cIT relies on for daily operations.

REQUIREMENTS and PREREQUISITES

·         Configuration Item or Items (CI) have been identified

·         Team Member and or department doing the change has been identified

·         Financial Impact has been assessed

·         Impact of the change has been assessed

·         The full scope of the change including all configuration change steps are identified

DEFINITIONS

·         Production System – any system that is in-use by a customer and not otherwise designated as a development or non-production system, this can also be known as a Primary Business Application (PBA) or potentially any of a PBA’s components or underlying network requirements. Systems include hardware, software, and process that may include people and business policy. Examples include but are not limited to: Servers, Storage, Networks, Active Directory, Backup, end user service, or any business software application.

·         Change – A modification to a system or systems that would require an update to any controlled system documentation, diagram, procedure, or system availability, or cause any production system to be modified in such a way as to present new risk, changes in cost, new potential vulnerabilities, or potential loss of function to any critical network, or application systems (PBA/s) with which a customer or cIT relies on for daily operations. Examples include but are not limited to firewall configuration changes, network configuration changes, Compute and Configuration, or policy changes, service pack updates if not normally applied as part of general pre-approved maintenance of that system, security fixes, and physical configuration changes such as the movement of critical hardware from one location to another, or the addition or removal of critical hardware.

o   Types of Changes

§  Normal Changes or Normal Request for Change (NRFC) – This document covers the scope of a normal change including those undertaken during an approved project.

§  Standard Changes – These are predefined changes and must be completed per the predefined steps and the documentation required by the pre-approved change.

§  Emergency Changes – Emergency changes generally follow the same process as a Normal Change however in the event a major system is down and a change must be completed to return that system to full operability, an immediate minimal change may be made prior to approval.  All normal change steps must be followed once the system is back online, and service is restored and the change request must still be requested following the Normal Change instructions.  No changes beyond the minimal needed to restore functionality should be made prior to authorized approval being granted.

·         cIT SME – The cIT subject matter expert (SME) is an individual or individuals who must review and approve of a change prior to implementation.  This individual is also a member of the CAB.

·         virtual IT Manager (vITM) – This is the client SME and must approve all changes for client systems alongside the cIT SME. For cIT systems, the vITM role is filled by the Senior IT Manager.

·         Change Advisory Board (CAB) – The Change Advisory Board is the review body for all Change Controls. The CAB will ensure that the change was completed successfully, that all documentation was updated, that the approved changes are followed, that all risk is accounted for and limited to the fullest extent possible under the circumstances, and that changes have completed an appropriate design review with peers.

STEPS

1.       Create a change control in the ITSM system (pzzle)

a.       Open Pzzle

b.       Go to ITSM

c.       Select Change normal from Change Management Section

d.       Click the “+” in the upper right from the Normal RFC list view screen

2.       Draft Stage

a.       Fill out the main details of the change (Draft Status)

                                                              i.      Number – The change reference number and will be pre-set by the system.  It can be used to reference this change.

                                                            ii.      Set appropriate company

                                                          iii.      Set appropriate company location

                                                           iv.      Title – Describe the change in its basic form

                                                             v.      Description – A detailed explanation of the change, why the change is needed, and the end goal of the change, reason for the change, as well as any other pertinent information to the change and the approval of the change

b.       Details Tab (found in tab list on the left side of the Normal RFC window).

                                                              i.      Category – Set the appropriate category from the drop down list

                                                            ii.      Subcategory – Select if this is an Add, change, or remove, of the above category item

                                                          iii.      Assignment Group – The group doing the change

                                                           iv.      Assignment User – The member of the assigned group doing the change and is normally the one filling out the change request.

                                                             v.      Planned start date – The date and time in which the change is tentatively scheduled to take place

                                                           vi.      Planned Duration - The estimated amount of time it will take to complete the change in hours, minutes, and seconds (HH:MM:SS).

                                                         vii.      Has Risk toggle switch – Indicates there is risk involved with this change and will add the Risk tab to the left hand side navigation.

                                                       viii.      Financial impact toggle switch – Indicates a financial impact and will add the financial impact tab to the left hand side navigation.

c.       People & CIs Tab

                                                              i.      Affects (CI) – Add all affected CIs here.  (For services, such as email, add the underlying system, IE Exchange or Microsoft 365, not the end service of email)

                                                            ii.      People – Add the people associated with the change. This would include anyone completing parts of the change or advising on the change from cIT (assisting), anyone from the client that maybe involved or impacted by the change or the client system owner (watching), any vendors involved in the change (vendor), and the primary person leading this change (Lead).

d.       Risk Tab (Visibility of this tab is set by Risk toggle on the Details Tab listed above)

                                                              i.      If this change has risk, it must be defined and discussed in this section

                                                            ii.      Risk Level – Identify the level of risk 1 = low, 9 = high

                                                          iii.      Risk Assessment – Define the risk, its implications, and its potential impact on the systems or company.

                                                           iv.      Risk Management Plan – Define and describe how risk will be limited and managed.

e.       Financial Impact Tab (Visibility of this tab is set by Financial Impact toggle on the Details Tab listed above)

                                                              i.      Financial impact description – Describe the financial impact

f.        Execution Plan Tab

                                                              i.      Build out the change plan and the individual steps.  Phases are large sections with the (+) under each phase allowing for individual tasks/steps to be defined. All phases and steps are completed in linear fashion in the execution phase of the project so build out the steps accordingly.  Each step should have associated changes items and describe the evidence that will show prior change settings and then updated settings. 

                                                            ii.      Example:        

a.       Phase 1 – Reconfigure cIT-NOC-FW-1

                                                                                                                                      i.      Step 1 – Change IP address of cIT-NOC-FW-1 from 192.168.1.1 to 10.1.10.1

2.       Evidence of this change needed in the execution phase would include screenshot of the configuration on cIT-NOC-FW-1 showing 192.168.1.1 prior to the change and an additional screenshot showing the updated configuration and new IP of 10.1.10.1 Additionally, a screenshot of the updated network diagram showing the cIT-NOC-FW-1 device with the new IP of 10.1.10.1 updated on that diagram as indicated in the change.

g.       Testing and Issues Tab

                                                              i.      Testing Plan – Describe in detail how changes will be test after the change is completed.  These steps or the evidence showing this steps were complete will also need to be included in any post change evidence.  Should any of these steps fail, the Escalation plan will come into effect.

                                                            ii.      Escalation Plan – Describe in detail the steps to be taken should the Testing or change phases fail to be completed.  If the escalation plan fails, the backout plan will come into effect.

                                                          iii.      Backout Plan – Describe, in detail, the backout or rollback plan should the testing and escalation plans fail.  Describe how the system or CIs will be put back into configuration as to return them to the pre-change state.

h.      Attachments Tab

                                                              i.      Include any attachments relevant to the change or showing evidence of the change.  Also include any pre-change and post-change network diagrams.  Label all documents clearly in the file name of the document.

i.        Save it

                                                              i.      Once all the details are entered save the Normal RFC by clicking the save and stay icon in the upper right.

3.       Submit for approval – Approval Lifecycle Phase

a.       Once a change is ready to be submitted for approval, select drop down arrow on the right of the blue “Draft” status at the top right of the window.  Change the status to “Approval.” This will initiate the approval phase.  Click save and stay to save this status.

b.       Approvals are automated and sent to the approvers associated with the company record.  First approval is the client vITM.  Second approval is the cIT SME team. Final approvals will include the cIT financial reviewer if applicable.

c.       Approval time frame – Each approver has up to one week to review and approve.  If no approval is given from either group within that time frame the request will automatically be rejected must be re-submitted.

d.       Approval rejection – If either the vITM or the cIT SME reject the change it will immediately show a status of “rejected” and must be placed back in Draft status and resubmitted with updated changes to mitigate the reason it was rejected.  Reach out to the rejecting party for further details on the rejection.

4.       Scheduled Lifecycle Phase

a.       Once the NRFC shows “Scheduled” as the status the change has been approved by both groups of approvers in the approval phase.  Nothing is needed during this phase until the execution phase is to begin.  Once execution of the change is ready to be carried out select drop down arrow on the right of the blue “Scheduled” status at the top right of the window.  Change the status to “Execution.”

b.       Should changes to the NRFC be needed, the status should be changed back to “Draft” and the details amended as needed.  Then the change should be resubmitted for approval.  No changes should be made to the execution plan, testing plan, or any other parts of the NRFC without the change being resubmitted for approval prior to those changes being made.

5.       Execution Lifecycle Phase

a.       Once an NRFC enters the execution lifecycle it can no longer be cancelled or changed.  Changes must be completed or reverted and the change control must move forward through all steps as indicated by the change, the testing plan, escalation plan, and back out plan.

b.       To complete the changes move to the Execution Tab.

c.       Changes should show up as individual line items set for execution with the assigned user highlighted for each change.  Complete each task and complete all steps and include all evidences as needed.  If changes are not to be made, the change step should be set to reverted and evidence showing as such included for review.

d.       Once all steps are completed and NRFC is saved the status of the change will automatically be converted to review.  Attach all additional evidence, updated documentation, etc. Save the NRFC.

6.       Review Lifecycle Phase

a.       Review of the NRFC will take place at the next CAB meeting.  The cab will review the following:           

                                                              i.      All changes completed to specification

                                                            ii.      All items impacted were tested and evidence of that testing is sufficient

                                                          iii.      If any exceptions or other challenges were indicated that they were sufficiently handled, documented, and the change and end goal was still achieved or in the event of a back out, that the changes were reverted.

                                                           iv.      All evidence is detailed enough, documentation is updated, and any additional items of importance are covered including but not limited too:

1.       Passwords

2.       Network diagrams

3.       CMDB configurations

4.       Client documentation

                                                             v.      Once satisfied that the change was completed, the CAB will set the lifecycle to “Review Complete.” If any items such as billing are outstanding, the CAB will get confirmation of those items prior to complete closure.  Should any items be missing, the CAB will set the lifecycle to “review rejected,” provide details of the rejection, and reassign it to the assigned engineer to complete the needed tasks.  Engineer will then be responsible for updating the NRFC as required and resubmitting it for review.

                                                           vi.      A CAB member will denote the review was completed and that no outstanding items remain prior to setting it to the final status of completed.

7.       Once the CAB is ready to close a member of the CAB will set the NRFC to “Completed.” This is the end step and denotes a completed change. Nothing further is needed.       

REFERENCES

 Any additional resources such as external links.