Enterprise Infrastructure Solutions (EIS)
Request for Proposals
Section E
Inspection and Acceptance
Issued by:
General Services Administration
Office of Integrated Technology Services
1800 F St NW
Washington, DC 20405
October 2015
Table of Contents
E.1.1 FAR
52.252-2 Clauses Incorporated by Reference (FEB 1998)
E.1.1.1 Clauses
Incorporated by Reference Table.
E.2.1 Business
Support Systems Verification Testing
E.2.2 EIS
Services Verification Testing
E.2.2.1 General
Testing Requirements
E.2.2.5 Test
Results and Acceptance
E.1 Introduction
This section describes the inspection and acceptance plan for the following:
· Contractor’s Business Support Systems (BSS)
· EIS Services
E.1.1 FAR 52.252-2 Clauses Incorporated by Reference (FEB 1998)
This contract incorporates one or more FAR clauses by reference, with the same force and effect as if they were given in full text. Upon request, the Contracting Officer (CO) will make their full text available. The full text of a clause may be accessed electronically at this address: www.acquisition.gov/far.
The clauses in Table E.1.1.1 apply at the contract and order levels, as applicable, depending upon the contract type of the order, or as specifically referenced in the applicable order.
E.1.1.1 Clauses Incorporated by Reference Table
|
Clause # |
FAR Clause # |
Title and Date |
Date |
|
52.246-2 |
Inspection of Supplies - Fixed Price |
AUG 1996 |
|
|
52.246-4 |
Inspection of Services - Fixed Price |
AUG 1996 |
|
|
52.246-6 |
Time-and-Material and Labor-Hour |
MAY 2001 |
|
|
52.246-12 |
Inspection of Construction |
AUG 1996 |
|
|
52.246-16 |
Responsibility for Supplies |
APR 1984 |
|
|
52.246-17 |
Warranty of Supplies of a Noncomplex Nature |
JUN 2003 |
|
|
52.246-19 |
Warranty of Systems and Equipment under Performance Specifications or Design Criteria |
MAY 2001 |
|
|
52.246-20 |
Warranty of Services |
MAY 2001 |
E.2 Test Methodology
The contractor shall follow the verification and acceptance test methodology described in this section in developing the verification test plan(s) for:
· Contractor’s Business Support Systems (BSS)
· EIS Services
E.2.1 Business Support Systems Verification Testing
The contractor shall provide a draft BSS Verification Test Plan (BSS Test Plan) with its proposal and a final BSS Test Plan 30 days after notice to proceed. This plan shall comply with the test methodology for BSS defined in Sections E.2.1.1 – E.2.1.5. The government reserves 21 days from the date of receipt of final plan to accept or reject the plan. If it is rejected, the contractor will be given the opportunity to update the plan based on government comments.
Based on GSA Conexus readiness and other factors, the government may, at its option, offer the opportunity for contractors to perform preliminary testing prior to award, but after submission of proposals. If this opportunity is offered, it will be offered to all contractors who submit a proposal, pending only the contractor’s demonstration of primary security features such as anti-virus protection. Any such preliminary testing shall not replace formal testing post-award. The government offers no guarantees that the GSA Conexus configuration offered as the preliminary system will be identical to the final system used in post-award formal testing. If the government offers such preliminary testing, the government will issue terms and conditions for such testing which the contractor must accept prior to accessing the test system.
The contractor shall provide updates to the BSS Test Plan within 14 days of receipt of government comments. The government reserves 14 days after receipt of the updated plan to accept or reject it. If necessary to gain approval, the contractor may repeat this process.
See section G.2.3 BSS Final Contract Acceptance for final acceptance requirement.
In support of BSS testing, GSA will provide test parameters as follows: BSS test scenarios are provided in Section E.2.1.2. BSS test cases are provided in Section E.2.1.3. Test metadata (including approximate number of TOs, SOs, and other data sets to be used in testing) will be provided not later than NTP. System reference data, as described in Section J.2.3, will be provided not later than NTP. Test data, as described in Section E.2.1.3, and test verification criteria will be provided at the time of BSS testing.
The BSS testing will be performed during normal business hours, 8:00am-5:00pm Monday-Friday, Eastern Time.
E.2.1.1 Scope
The contractor shall meet the following Inspection and Acceptance requirements:
· BSS testing shall verify that all BSS functional, regression and security requirements have been successfully met.
· BSS testing shall be performed for all management and operation functions supporting Ordering, Billing, Inventory Management, Disputes, SLA Management and Trouble Ticketing processes described in Section G and Section J.2.
· Security testing shall be based on the requirements described in Section G.5.6 BSS Security Requirements. The security requirements acceptance shall be based on:
o Assessment and Authorization (A&A)
o FedRAMP certification (if applicable)
· BSS testing shall include multiple test cases that are defined in Section E.2.1.3 Test Cases.
· BSS testing shall include test cases for quality, utility and customer access features.
· The contractor shall allow government representative(s) to observe all or any part of the verification testing.
o If the government so requests, the contractor shall perform tests to ensure continued compliance each time a new service is offered or the contractor modifies features/functionality of the BSS that affect the functional requirements described in Section G and Section J.2.
o If the government requests this retest, the contractor shall provide a BSS Verification Test Results report, including analysis, within seven (7) days after performance of the tests. The government reserves 14 days to accept or reject the test results, in part or in whole. If the government rejects the test results the contractor shall retest until such time the results are acceptable to the government.
· The contractor shall perform BSS verification testing according to the accepted BSS Test Plan at a mutually acceptable date with the government.
E.2.1.2 BSS Test Scenarios
E.2.1.2.1 Testing Prerequisites
Prior to initiating BSS testing, the contractor shall:
· Provide written notice to the government that the contractor’s BSS has passed its internal testing and is ready to begin BSS interface testing with GSA Conexus.
· Provide a finalized BSS Test Plan that is accepted by GSA.
The purpose of the verification and acceptance testing is to ensure that the contractor’s BSS meets requirements in Section G and Section J.2. The contractor shall support BSS security and functional testing as defined in Section G.5.6 BSS Security Requirements and Section G.2.3 BSS Final Contract Acceptance.
E.2.1.2.2 Test Scenarios
The following table contains a high-level list of BSS Test Scenarios for which the contractor’s BSS must pass the defined acceptance criteria. The contractor shall address the test scenarios based on the functional requirements defined in the relevant portions of Section G and/or Section J.2 (see the “RFP References” column for references).
The scenarios shall address relevant data exchange mechanisms and validation of data exchanged. Each Test Scenario is associated with one or more Test Cases defined in Section E.2.1.3.
|
Test Scenario # |
RFP References |
Description |
Acceptance Criteria |
|
BSS-TS01 |
· G.5.3.2 · J.2.9 |
Exchange structured data using the defined direct data exchange methods: · XML via secure web services · Pipe, “|”, delimited table via SFTP |
The contractor shall demonstrate bidirectional exchange of defined data structure that meets the interface specifications as defined in Section G.5.3.2 and Section J.2.9. |
|
BSS-TS02 |
· G.3 · J.2.3 |
The contractor’s BSS manages the following as specified in Section J.2.3: · Accept System Reference Data · Provide Direct Billed Agency Setup |
The contractor shall demonstrate successful TO Data Management initial setup and updates. |
|
BSS-TS03 |
· J.2.3 |
The contractor’s BSS manages Role based access control to all BSS functions (e.g. ordering, billing, inventory management, trouble management, SLA management) |
The contractor shall demonstrate that its BSS provides the ability to define role based users with privileged access to the BSS to meet the requirements as defined in Section J.2.3. |
|
BSS-TS04 |
· G.3 · J.2.4 |
The contractor’s BSS manages the processing of orders and generation of required acknowledgments and notifications. Order types include: · New service for each of the services specified in Section C.2, Technical Requirements, that are included in the awardee’s contract · Service Moves · Service Disconnects · Service Feature Changes · Telecommunications Service Priority (TSP) · Auto-sold CLINs · Bulk Orders |
The contractor shall demonstrate that an authorized government user can place an order using the methods specified in Section J.2.4 and the order populates the fields in the contractor’s BSS in a way that meets the requirements in Sections G.3, G.5 and J.2. Using the direct data exchange method defined in Section J.2.4, the contractor shall demonstrate that its BSS can provide all required CDRLs including: 1) Service Order Acknowledgement 2) Service Order Rejection Notice 3) Service Order Confirmation 4) Firm Order Commitment Notice 5) Service Order Completion Notice |
|
BSS-TS05 |
· G.3 · J.2.4 |
The contractor’s BSS handles order supplements/updates that impact other, in-progress orders: · Cancel orders · Service Feature Changes · Location changes · Changes to Customer Want Date · Changes to administrative data
|
The contractor shall demonstrate that an authorized government user can place a change or cancel order using the methods specified in Section J.2.4 and the order populates the fields in the contractor’s BSS in a way that meets the requirements in Sections G.3, G.5 and J.2. Using the direct data exchange method defined in Section J.2.4, the contractor shall demonstrate that its BSS can provide all required CDRLs including: 1) Service Order Acknowledgement 2) Service Order Rejection Notice 3) Service Order Confirmation 4) Firm Order Commitment Notice 5) Service Order Completion Notice |
|
BSS-TS06 |
· G.3 · J.2.4 |
The contractor’s BSS handles orders for administrative changes to the records for previously provisioned services as described in G.3
|
The contractor shall demonstrate that an authorized government user can place an administrative change order using the methods specified in Section J.2.4 and the order populates the fields in the contractor’s BSS in a way that meets the requirements in Sections G.3, G.5 and J.2. Using the direct data exchange method defined in Section J.2.4, the contractor shall demonstrate that its BSS can provide all required CDRLs including: Service Order Administrative Change |
|
BSS-TS07 |
· G.3.5.6 |
The contractor’s BSS manages Self-Service Provisioning and other Rapid Provisioning orders and provides the correct notices. |
The contractor shall successfully demonstrate the completion of these orders. Non-self-service orders shall be tested using both correctly placed orders and orders with related errors. Self-service orders shall be tested with correctly placed orders and to ensure that the contractor’s BSS does not permit the placement of incorrect orders. Using the direct data exchange method defined in Section J.2.4, the contractor shall demonstrate that its BSS can provide all required CDRLs including: 1) Service Order Acknowledgement 2) Service Order Completion Notice |
|
BSS-TS08 |
· G.4 · J.2.5 · J.2.6 · J.2.7 · J.2.10 |
The contractor’s BSS properly manages inventory and billing: · Generates the inventory of services delivered by the contractor · Produces output that is consistent with order and billing details · Generates the detailed billing in accordance with the Billing Invoice (BI) CDRL · Properly handles usage-based billing · Calculates billing based on the contractor’s awarded pricing · Correctly calculates the AGF due to GSA and produces the required AGF CDRLs · Provides accurate calculation of rounding and proration related to Billing, Taxes, Fees and Surcharges |
The contractor shall demonstrate that its service inventory management system maintains a complete and accurate inventory of EIS service orders in a way that meets the requirements in G.7 and G.5 and Section J.2 CDIP and is verified and accepted by GSA. The contractor shall demonstrate that the output of its billing data elements is consistent with the orders entered into its BSS and that the billing data elements meet the requirements in Sections G.3, G.5 and J.2. Using the direct data exchange method defined in Sections J.2.5-J.2.7, the contractor shall demonstrate that its BSS can provide all required CDRLs including: 1) Billing Invoice 2) Billing Adjustment 3) Tax Detail 4) AGF Detail 5) AGF Electronic Funds Transfer Report 6) Inventory Reconciliation |
|
BSS-TS09 |
· G.4.4 · J.2.6 |
The contractor’s BSS properly manages all dispute types with appropriate handling for: · Billing disputes · Inventory disputes · SLA disputes · Dispute tracking and reporting |
The contractor shall demonstrate that its BSS can accept and issue disputes as well as tracking them to resolution. Using the direct data exchange method defined in Sections J.2.5-J.2.7, the contractor shall demonstrate that its BSS can provide all required CDRLs including: 1) Dispute 2) Dispute Report |
|
BSS-TS10 |
· G.8 · J.2.8 |
The contractor’s BSS properly manages SLA Management: · SLA Reporting · SLA Credit Request handling and response |
The contractor shall demonstrate that its BSS successfully tracks SLAs with associated KPIs as well as reporting SLA performance and providing sufficient information to response to SLA Credit Requests. Using the direct data exchange method defined in Sections J.2.5-J.2.7, the contractor shall demonstrate that its BSS can provide all required CDRLs including: 1) SLA Report 2) SLA Credit Request Response |
|
BSS-TS11 |
· J.2 |
The contractor’s BSS produces acceptable open-format reports defined in the CDIP: · Monthly Billing Information Memorandum · Trouble Management Incident Performance Report · Trouble Management Performance Summary Report |
The contractor shall demonstrate, via sample reports, that the open-format reports specified are sufficiently detailed and clear so as to meet the government’s requirements. |
|
BSS-TS12 |
· J.2 |
The contractor’s BSS testing includes regression testing of all key features including ordering, service assurance, and billing.
NOTE: Applies only to testing conducted as part of system changes, not initial BSS development. |
The contractor shall demonstrate that its BSS meets regression testing.
The BSS test plan shall include regression testing; however, actual regression testing will not be part of initial test and acceptance. |
|
BSS-TS13 |
· G.5.6 |
The contractor’s BSS has passed A&A as defined in Section G.5.6. |
The contractor shall demonstrate that the BSS meets FISMA Moderate requirements. |
E.2.1.3 BSS Test Cases
The contractor shall accept, incorporate into the BSS Test Plan, and successfully execute test cases provided for each of the test scenarios above.
· The contractor shall accept the following test conditions:
o No testing between the contractor and GSA shall occur until both the contractor’s BSS and GSA Conexus have passed unit testing.
o All testing to be performed on the actual system to be used in delivering service (i.e. special purpose “test systems” shall not be used).
o Unless otherwise specified, all data transfers are to use the mechanism specified in Section J.2 for that data set.
· The contractor shall use GSA provided test data for all BSS verification testing unless specified otherwise:
o This data shall be used for testing purposes only.
o No customer “live” data shall be used for testing.
o This data shall be a realistic simulation of actual customer data.
o The test data shall include, in some tests, intentional errors intended to test the contractor’s BSS error handling.
· BSS testing shall follow a tiered approach:
o The contractor shall accept multiple test cases for the test scenarios defined in Section E.2.1.2.
o The contractor shall accept, incorporate into the BSS Test Plan, and successfully execute each test case with one or more test data sets.
o In providing test data sets, GSA will group them into Test Subcases:
§ Each Test Subcase shall contain data sets intended to test a specific “real world” test case (e.g. a complete and accurate disconnect order)
§ Each test subcase shall include at least two complete test data sets
· BSS functional testing acceptance:
o The contractor’s BSS shall not have completed functional testing until all BSS Test Scenarios (Section E.2.1.2) are passed.
o A test scenario shall not be considered passed until the contractor’s BSS properly handles each associated test case.
o A test case shall not be considered passed until the contractor’s BSS properly handles each associated subcase twice in succession using different data sets.
o A subcase shall not be considered passed until the contractor’s BSS properly handles the data sets following the prescribed actions with no errors or warnings.
· BSS security testing acceptance is defined in Section G.5.6 and associated references.
The individual test cases are defined in the tables in the subsections below. Each test case table includes the following headings:
· Test Scenario: The associated test scenario from Section E.2.1.2.
· Test Case ID: Identification number for the test case.
· Test Case Description: Brief title of the test case.
· Requirements Reference(s): Where the functional requirements that are being tested can be found.
· Prerequisites: Actions that must be completed prior to implementing the test case (in addition to the general prerequisites for all testing).
· Government Input(s): Data the government will provide as input to the test case.
· Expected BSS Output(s): Expected data or actions from the contractor’s BSS.
· Data Set Description: Brief description of the data sets the government intends to provide as part of testing.
· Acceptance Criteria: Factors to be checked prior to acceptance of the test results. The table below defines the terms used:
|
Criteria |
Definition |
|
Successful data transfer |
The contractor demonstrates that data transmitted was received intact without error using the formats and mechanisms described in the test case. |
|
Correct technical aspects |
The contractor demonstrates that data transfers were completed using the correct mechanism, in the correct format, and with the correct structure. |
|
Evidence of failure notification |
The contractor demonstrates that expected notifications of failure were properly issued. |
|
No partial import |
The contractor demonstrates that their BSS does not import partial data in cases where the data is corrupt or otherwise cannot be imported in full. |
|
All required CDRLs |
The contractor demonstrates that the expected CDRLs (listed in the expected output section) are all delivered. |
|
Accurate data based on inputs |
The contractor demonstrates that the data provided in CDRLs is accurate and reflects the data provided by the government inputs and the prerequisites. |
|
Access granted |
The contractor demonstrates that the user gains access to resources as expected. |
|
Access denied |
The contractor demonstrates that the user is denied access to resources as expected. |
|
No errors displayed |
The contractor demonstrates that the user is not shown any unexpected errors. |
|
Appropriate errors are displayed |
The contractor demonstrates that the user is shown the expected errors. |
|
CDRLs are internally consistent |
The contractor demonstrates that the data provided in the expected CDRLs is internally consistent between the set of CDRLs. |
|
Complies with calculation rules |
The contractor demonstrates that the data provided in the CDRLs matches that which would be expected based on rounding and proration calculation requirements. |
|
Each CDRL meets requirements |
The contractor demonstrates that the provided CDRLs meet the requirements specified by the government (used for CDRLs without detailed format requirements). Standard requirements include, at minimum: · CDRL contains the required information · CDRL is clear and readily understood |
|
Contractor BSS receives ATO |
The contractor demonstrates that the BSS has received Authorization to Operate (ATO) and has been approved by GSA based on relevant security requirements as defined in Section G.5.6 and references therein. |
E.2.1.3.1 BSS-TS01: Direct Data Exchange
E.2.1.3.1.1 BSS-TS01-01: XML over Secure Web Services
|
Test Scenario |
BSS-TS01: Direct Data Exchange |
|
Test Case ID |
BSS-TS01-01 |
|
Test Case Description |
XML over Secure Web Services |
|
Requirements Reference(s) |
· G.5.3.2 · J.2.9 |
|
Prerequisites |
· N/A |
|
Government Input(s) |
· Properly formatted government data set listed as using web services as the transfer mechanism in Section J.2 |
|
Expected Output(s) |
· Properly formatted contractor data set that meets the following criteria: o Listed as using web services as the transfer mechanism in Section J.2 o Includes data derived from the government input |
|
Acceptance Criteria |
· Successful data transfer · Correct technical aspects |
|
Data Set Description |
· Each government-provided test data set will include: o One government data set listed as using web services as the transfer mechanism in Section J.2 |
E.2.1.3.1.2 BSS-TS01-02: PSV over SFTP
|
Test Scenario |
BSS-TS01: Direct Data Exchange |
|
Test Case ID |
BSS-TS01-02 |
|
Test Case Description |
PSV over SFTP |
|
Requirements Reference(s) |
· G.5.3.2 · J.2.9 |
|
Prerequisites |
· N/A |
|
Government Input(s) |
· Properly formatted government data set listed as using SFTP as the transfer mechanism in Section J.2 |
|
Expected Output(s) |
· Properly formatted contractor data set that meets the following criteria: o Listed as using SFTP as the transfer mechanism in Section J.2 o Includes data derived from the government input |
|
Acceptance Criteria |
· Successful transfer of PSV data · Correct technical aspects |
|
Data Set Description |
· Each government-provided test data set will include: o One government data set listed as using SFTP as the transfer mechanism in Section J.2 |
E.2.1.3.1.3 BSS-TS01-03: Error Handling: XML over Secure Web Services
|
Test Scenario |
BSS-TS01: Direct Data Exchange |
|
Test Case ID |
BSS-TS01-03 |
|
Test Case Description |
Error Handling: XML over Secure Web Services |
|
Requirements Reference(s) |
· G.5.3.2 · J.2.9 |
|
Prerequisites |
· N/A |
|
Government Input(s) |
· Invalid government data set listed as using web services as the transfer mechanism in Section J.2 |
|
Expected Output(s) |
· Notification to contractor of failed import |
|
Acceptance Criteria |
· Evidence of failure notification · No partial import |
|
Data Set Description |
· Each government-provided test data set will include: o One government data set listed as using web services as the transfer mechanism in Section J.2 o One or more XML formatting errors (e.g. missing delimiters) |
E.2.1.3.1.4 BSS-TS01-04: Error Handling: PSV over SFTP
|
Test Scenario |
BSS-TS01: Direct Data Exchange |
|
Test Case ID |
BSS-TS01-04 |
|
Test Case Description |
Error Handling: PSV over SFTP |
|
Requirements Reference(s) |
· G.5.3.2 · J.2.9 |
|
Prerequisites |
· N/A |
|
Government Input(s) |
· Invalid government data set listed as using SFTP as the transfer mechanism in Section J.2 |
|
Expected Output(s) |
· Notification to contractor of failed import |
|
Acceptance Criteria |
· Evidence of failure notification · No partial import |
|
Data Set Description |
· Each government-provided test data set will include: o One government data set listed as using SFTP as the transfer mechanism in Section J.2 o One or more formatting errors (e.g. missing delimiters) |
E.2.1.3.2 BSS-TS02: Task Order Data Management
E.2.1.3.2.1 BSS-TS02-01: Direct Billing Account Setup
|
Test Scenario |
BSS-TS02: Task Order and Account Management Setup |
|
Test Case ID |
BSS-TS02-01 |
|
Test Case Description |
Direct Billing Account Setup |
|
Requirements Reference(s) |
· G.3 · J.2.2 · J.2.3 |
|
Prerequisites |
· N/A |
|
Government Input(s) |
· Task order controlled data as defined in Section J.2.3 · Task order associated data as defined in Section J.2.3 · System Reference data as defined in Section J.2.3 |
|
Expected Output(s) |
· Direct Billed Agency Setup (DBAS) CDRL |
|
Acceptance Criteria |
· All required CDRLs as defined in Section J.2.3 · Accurate data based on inputs · Correct technical aspects |
|
Data Set Description |
· Each government-provided test data set will include: o Sample TO controlled data transferred in the form of a sample TO o Sample TO associated data transferred in free format (not previously defined) unless the contractor specifies in their proposal that customer registration is to be submitted via their web interface and that interface collects all of the required data o Sample System Reference data transferred using the mechanism specified in Section J.2 |
E.2.1.3.2.2 Reserved BSS-TS02-02: Central Billing Accounts
E.2.1.3.3 BSS-TS03: Role Based Access Control
E.2.1.3.3.1 BSS-TS03-01: Authorized User Access Verification
|
Test Scenario |
BSS-TS03: Role Based Access Control |
|
Test Case ID |
BSS-TS03-01 |
|
Test Case Description |
Authorized User Access Verification |
|
Requirements Reference(s) |
· G.5 · J.2.3 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS |
|
Government Input(s) |
· User attempts to: o Sign into BSS o Access BSS areas to which they are an authorized user o Exercise the full range of functionality (read/write) permitted for their role |
|
Expected Output(s) |
· User is permitted to: o Access to BSS o Access BSS areas as authorized o Exercise assigned functionality |
|
Acceptance Criteria |
· Access is granted · No security errors displayed |
|
Data Set Description |
· Each government-provided test data set will include: o Role to be tested o Functionality to be tested |
E.2.1.3.3.2 BSS-TS03-02: Unauthorized User Access Denial Verification
|
Test Scenario |
BSS-TS03: Role Based Access Control |
|
Test Case ID |
BSS-TS03-02 |
|
Test Case Description |
Unauthorized User Access Denial Verification |
|
Requirements Reference(s) |
· G.5 · J.2.3. |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS |
|
Government Input(s) |
· User attempts to: o Sign into BSS o Access BSS areas to which they are not an authorized user o Exercise functionality (read/write) not permitted for their role |
|
Expected Output(s) |
· User is denied access at the point appropriate the role, area and functionality specified in the test data set: o Access to BSS o Access to specific BSS areas o Access to specific functionality · User is shown security error message |
|
Acceptance Criteria |
· Access is denied · Appropriate errors are displayed |
|
Data Set Description |
· Each government-provided test data set will include: o Role to be tested (may be specified as none if the set is intended to show denial of unauthorized users) o Functionality to be tested |
E.2.1.3.4 BSS-TS04: Service Ordering
E.2.1.3.4.1 BSS-TS04-01: New Order via Web Interface
|
Test Scenario |
BSS-TS04: Service Ordering |
|
Test Case ID |
BSS-TS04-01 |
|
Test Case Description |
New Order via Web Interface |
|
Requirements Reference(s) |
· G.3 · G.5.3.1 · J.2.4 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS |
|
Government Input(s) |
· Service Order (SO) with all required data elements as described in Section J.2.10.2.1.15 |
|
Expected Output(s) |
· Service Order notification CDRLs as defined in Section J.2.4: o Service Order Acknowledgment (SOA) o Service Order Confirmation (SOC) o Firm Order Commitment Notice (FOCN) o Service Order Completion Notification (SOCN) |
|
Acceptance Criteria |
· All required CDRLs as defined in Section J.2.4 · Accurate data based on inputs · Correct technical aspects |
|
Data Set Description |
· Each government-provided test data set will include: o A complete SO for one or more services listed in Section C.2 of the contract o SO to be entered into the contactor’s BSS via the contractor’s web interface as described in Section G.5.3.1 |
E.2.1.3.4.2 BSS-TS04-02: New Order via Email
|
Test Scenario |
BSS-TS04: Service Ordering |
|
Test Case ID |
BSS-TS04-02 |
|
Test Case Description |
New Order via Email |
|
Requirements Reference(s) |
· G.3 · G.5.3.1 · J.2.4 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS |
|
Government Input(s) |
· Service Order (SO) with all required data elements as described in Section J.2.10.2.1.15 |
|
Expected Output(s) |
· Service Order notification CDRLs as defined in Section J.2.4: o Service Order Acknowledgment (SOA) o Service Order Confirmation (SOC) o Firm Order Commitment Notice (FOCN) o Service Order Completion Notification (SOCN) |
|
Acceptance Criteria |
· All required CDRLs as defined in Section J.2.4 · Accurate data based on inputs · Correct technical aspects |
|
Data Set Description |
· Each government-provided test data set will include: o A complete SO for one or more services listed in Section C.2 of the contract o SO submitted via a means listed in J.2.4 other than the contractor’s web interface |
E.2.1.3.4.3 BSS-TS04-03: Disconnect Order
|
Test Scenario |
BSS-TS04: Service Ordering |
|
Test Case ID |
BSS-TS04-03 |
|
Test Case Description |
Disconnect Order |
|
Requirements Reference(s) |
· G.3 · J.2.4 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS · Previously provisioned circuit or service element entered into the contractor BSS |
|
Government Input(s) |
· Service Order (SO) with all required data elements as described in Section J.2.4 for the disconnect of o A circuit or service element (CLIN) o A feature of a circuit or service element |
|
Expected Output(s) |
· Service Order notification CDRLs as defined in Section J.2.4: o Service Order Acknowledgment (SOA) o Service Order Confirmation (SOC) o Firm Order Commitment Notice (FOCN) if required based on the requirements described in Section J.2.4 o Service Order Completion Notification (SOCN) |
|
Acceptance Criteria |
· All required CDRLs · Accurate data based on inputs · Correct technical aspects |
|
Data Set Description |
· Each government-provided test data set will include: o A complete SO for the disconnect of a circuit or service element or a feature of a circuit or service element as described in Section G.3 and Section J.2.10.1.1.4.2 |
E.2.1.3.4.4 BSS-TS04-04: Feature Addition Order
|
Test Scenario |
BSS-TS04: Service Ordering |
|
Test Case ID |
BSS-TS04-04 |
|
Test Case Description |
Feature Addition Order |
|
Requirements Reference(s) |
· G.3 · J.2.4 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS · Previously provisioned circuit or service element entered into the contractor BSS |
|
Government Input(s) |
· Service Order (SO) with all required data elements as described in Section J.2.4 for the addition of a feature to a circuit or service element |
|
Expected Output(s) |
· Service Order notification CDRLs as defined in Section J.2.4: o Service Order Acknowledgment (SOA) o Service Order Confirmation (SOC) o Firm Order Commitment Notice (FOCN) o Service Order Completion Notification (SOCN) |
|
Acceptance Criteria |
· All required CDRLs · Accurate data based on inputs · Correct technical aspects |
|
Data Set Description |
· Each government-provided test data set will include: o A complete SO for the addition of a feature to a circuit or service element as described in Section G.3 and Section J.2.10.1.1.4.2 |
E.2.1.3.4.5 BSS-TS04-05: Move Order
|
Test Scenario |
BSS-TS04: Service Ordering |
|
Test Case ID |
BSS-TS04-05 |
|
Test Case Description |
Move Order |
|
Requirements Reference(s) |
· G.3 · J.2.4 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS · Previously provisioned circuit or service element entered into the contractor BSS |
|
Government Input(s) |
· Two Service Orders (SOs) that combine to specify the move of a circuit or service element with all required data elements as described in Section J.2.4 o One SO for the disconnect of the circuit or service element at the old location o Second SO for the installation of the identical circuit or service element at the new location |
|
Expected Output(s) |
· Service Order notification CDRLs as defined in Section J.2.4: o Service Order Acknowledgment (SOA) o Service Order Confirmation (SOC) o Firm Order Commitment Notice (FOCN) o Service Order Completion Notification (SOCN) |
|
Acceptance Criteria |
· All required CDRLs · Accurate data based on inputs · Correct technical aspects |
|
Data Set Description |
· Each government-provided test data set will include: o A pair of complete SOs for the move of a circuit or service element from one valid location to another as described in Section G.3 and Section J.2.10.1.1.4.2 § SO for the disconnect from the old location § SO for the installation of the identical service at the new location |
E.2.1.3.4.6 BSS-TS04-06: TSP Order
|
Test Scenario |
BSS-TS04: Service Ordering |
|
Test Case ID |
BSS-TS04-06 |
|
Test Case Description |
TSP Order |
|
Requirements Reference(s) |
· G.3 · J.2.4 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS |
|
Government Input(s) |
· Service Order (SO) requesting TSP with all required data elements as described in Section J.2.4 |
|
Expected Output(s) |
· Service Order notification CDRLs as defined in Section J.2.4: o Service Order Acknowledgment (SOA) o Service Order Confirmation (SOC) o Firm Order Commitment Notice (FOCN) o Service Order Completion Notification (SOCN) |
|
Acceptance Criteria |
· All required CDRLs as defined in Section J.2.4 · Accurate data based on inputs · Correct technical aspects |
|
Data Set Description |
· Each government-provided test data set will include: o A complete SO with a TSP code for one or more services listed in Section C.2 of the contract |
E.2.1.3.4.7 BSS-TS04-07: Auto-Sold CLINs
|
Test Scenario |
BSS-TS04: Service Ordering |
|
Test Case ID |
BSS-TS04-07 |
|
Test Case Description |
Auto-Sold CLINs |
|
Requirements Reference(s) |
· G.3 · J.2.4 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS |
|
Government Input(s) |
· Service Order (SO) that includes CLINS with associated Auto-Sold CLINs and contains all required data elements as described in Section J.2.4 |
|
Expected Output(s) |
· Service Order notification CDRLs as defined in Section J.2.4: o Service Order Acknowledgment (SOA) o Service Order Confirmation (SOC) o Firm Order Commitment Notice (FOCN) o Service Order Completion Notification (SOCN) |
|
Acceptance Criteria |
· All required CDRLs as defined in Section J.2.4 · Accurate data based on inputs · Correct technical aspects |
|
Data Set Description |
· Each government-provided test data set will include: o A complete SO that includes Auto-Sold CLINs for one or more services listed in Section C.2 of the contract |
E.2.1.3.4.8 BSS-TS04-08: Task Order Unique CLINs (TUCs)
|
Test Scenario |
BSS-TS04: Service Ordering |
|
Test Case ID |
BSS-TS04-08 |
|
Test Case Description |
Task Order Unique CLINs (TUCs) |
|
Requirements Reference(s) |
· G.3 · J.2.4 |
|
Prerequisites |
· TO setup data loaded into contractor BSS o TO Data defines one or more TUCs · Account Management data loaded into contractor BSS |
|
Government Input(s) |
· Service Order (SO) containing Task Order Unique CLINs (TUCs) and all required data elements as described in Section J.2.4 |
|
Expected Output(s) |
· Service Order notification CDRLs as defined in Section J.2.4: o Service Order Acknowledgment (SOA) o Service Order Confirmation (SOC) o Firm Order Commitment Notice (FOCN) o Service Order Completion Notification (SOCN) |
|
Acceptance Criteria |
· All required CDRLs as defined in Section J.2.4 · Accurate data based on inputs · Correct technical aspects |
|
Data Set Description |
· Each government-provided test data set will include: o A complete SO containing Task Order Unique CLINs (TUCs) for one or more services listed in Section C.2 of the contract |
E.2.1.3.4.9 Reserved
E.2.1.3.4.10 BSS-TS04-10: Bulk Orders
|
Test Scenario |
BSS-TS04: Service Ordering |
|
Test Case ID |
BSS-TS04-10 |
|
Test Case Description |
Bulk Orders |
|
Requirements Reference(s) |
· G.3 · J.2.4 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS |
|
Government Input(s) |
· Bulk Service Order (SO) with all required data elements as described in Section J.2.4 |
|
Expected Output(s) |
· Service Order notification CDRLs as defined in Section J.2.4: o Service Order Acknowledgment (SOA) o Service Order Confirmation (SOC) o Firm Order Commitment Notice (FOCN) o Service Order Completion Notification (SOCN) |
|
Acceptance Criteria |
· All required CDRLs as defined in Section J.2.4 · Accurate data based on inputs · Correct technical aspects |
|
Data Set Description |
· Each government-provided test data set will include: o A SO including at least 20 line items for services listed in Section C.2 of the contract o SO data provided via a delimited text file or MS Excel file |
E.2.1.3.4.11 BSS-TS04-11: Error Checking: Missing Info
|
Test Scenario |
BSS-TS04: Service Ordering |
|
Test Case ID |
BSS-TS04-11 |
|
Test Case Description |
Error Checking: Missing Information |
|
Requirements Reference(s) |
· G.3 · J.2.4 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS |
|
Government Input(s) |
· Service Order (SO) missing one or more required data elements as described in Section J.2.4 |
|
Expected Output(s) |
· Service Order notification CDRLs as defined in Section J.2.4: o Service Order Acknowledgment (SOA) o Service Order Rejection Notice (SORN) |
|
Acceptance Criteria |
· All required CDRLs · Accurate data based on inputs · Correct technical aspects |
|
Data Set Description |
· Each government-provided test data set will include: o SO missing one or more required data elements for one or more services listed in Section C.2 of the contract |
E.2.1.3.4.12 BSS-TS04-12: Error Checking: Invalid Info
|
Test Scenario |
BSS-TS04: Service Ordering |
|
Test Case ID |
BSS-TS04-12 |
|
Test Case Description |
Error Checking: Invalid Info |
|
Requirements Reference(s) |
· G.3 · J.2.4 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS |
|
Government Input(s) |
· Service Order (SO) with one or more invalid data elements as described in Section J.2.4 |
|
Expected Output(s) |
· Service Order notification CDRLs as defined in Section J.2.4: o Service Order Acknowledgment (SOA) o Service Order Rejection Notice (SORN) |
|
Acceptance Criteria |
· All required CDRLs · Accurate data based on inputs · Correct technical aspects |
|
Data Set Description |
· Each government-provided test data set will include: o SO orders with one or more invalid data information for one or more services listed in Section C.2 of the contract o Invalid data can include improperly formatted data or data that is inconsistent with the TO or Account Management data |
E.2.1.3.5 BSS-TS05: Supplements to In-Progress Orders
E.2.1.3.5.1 BSS-TS05-01: Cancel Orders
|
Test Scenario |
BSS-TS05: Supplements to In-Progress Orders |
|
Test Case ID |
BSS-TS05-01 |
|
Test Case Description |
Cancel Orders |
|
Requirements Reference(s) |
· G.3 · J.2.4 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS |
|
Government Input(s) |
· Service Order (SO) with all required data elements as described in Section J.2.4 · Service Order (SO) for a cancellation of the previous order with all required data elements as described in Section J.2.4 issued prior to completion of the previous order |
|
Expected Output(s) |
· Service Order notification CDRLs as defined in Section J.2.4: o Service Order Acknowledgment (SOA) o Updates to Service Order Confirmation (SOC) if required o Updates to Firm Order Commitment Notice (FOCN) if required o Service Order Completion Notification (SOCN) |
|
Acceptance Criteria |
· All required CDRLs · Accurate data based on inputs · Correct technical aspects |
|
Data Set Description |
· Each government-provided test data set will include: o A complete SO for one or more services listed in Section C.2 of the contract o A second SO canceling the first Service Order as defined in Section J.2.10.1.1.4.3 o The Cancel Order may be issued before or after the deadline described in Section G.3 |
E.2.1.3.5.2 BSS-TS05-02: Service Feature Change
|
Test Scenario |
BSS-TS05: Supplements to In-Progress Orders |
|
Test Case ID |
BSS-TS05-02 |
|
Test Case Description |
Service Feature Change |
|
Requirements Reference(s) |
· G.3 · J.2.4 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS |
|
Government Input(s) |
· Service Order (SO) with all required data elements as described in Section J.2.4 · Service Order (SO) for a service feature change to the previous order with all required data elements as described in Section J.2.4 issued prior to completion to previous order |
|
Expected Output(s) |
· Service Order notification CDRLs as defined in Section J.2.4: o Service Order Acknowledgment (SOA) o Updates to Service Order Confirmation (SOC) if required o Updates to Firm Order Commitment Notice (FOCN) if required o Service Order Completion Notification (SOCN) |
|
Acceptance Criteria |
· All required CDRLs · Accurate data based on inputs · Correct technical aspects |
|
Data Set Description |
· Each government-provided test data set will include: o A complete SO for one or more services listed in Section C.2 of the contract o A second SO describing a service feature change to the first Service Order as defined in Section J.2.10.1.1.4.3 |
E.2.1.3.5.3 BSS-TS05-03: Location Change
|
Test Scenario |
BSS-TS05: Supplements to In-Progress Orders |
|
Test Case ID |
BSS-TS05-03 |
|
Test Case Description |
Location Change |
|
Requirements Reference(s) |
· G.3 · J.2.4 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS |
|
Government Input(s) |
· Service Order (SO) with all required data elements as described in Section J.2.4 · Service Order (SO) for a location change to the previous order with all required data elements as described in Section J.2.4 issued prior to completion to previous order |
|
Expected Output(s) |
· Service Order notification CDRLs as defined in Section J.2.4: o Service Order Acknowledgment (SOA) o Updates to Service Order Confirmation (SOC) if required o Updates to Firm Order Commitment Notice (FOCN) if required o Service Order Completion Notification (SOCN) |
|
Acceptance Criteria |
· All required CDRLs · Accurate data based on inputs · Correct technical aspects |
|
Data Set Description |
· Each government-provided test data set will include: o A complete SO for one or more services listed in Section C.2 of the contract o A second SO describing a location change to the first Service Order as defined in Section J.2.10.1.1.4.3 |
E.2.1.3.5.4 BSS-TS05-04: Change to Customer Want Date
|
Test Scenario |
BSS-TS05: Supplements to In-Progress Orders |
|
Test Case ID |
BSS-TS05-04 |
|
Test Case Description |
Change to Customer Want Date |
|
Requirements Reference(s) |
· G.3 · J.2.4 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS |
|
Government Input(s) |
· Service Order (SO) with all required data elements as described in Section J.2.4 · Service Order (SO) for a change to the Customer Want Date for the previous order with all required data elements as described in Section J.2.4 issued prior to completion of the previous order |
|
Expected Output(s) |
· Service Order notification CDRLs as defined in Section J.2.4: o Service Order Acknowledgment (SOA) o Updates to Service Order Confirmation (SOC) if required o Updates to Firm Order Commitment Notice (FOCN) if required o Service Order Completion Notification (SOCN) |
|
Acceptance Criteria |
· All required CDRLs · Accurate data based on inputs · Correct technical aspects |
|
Data Set Description |
· Each government-provided test data set will include: o A complete SO for one or more services listed in Section C.2 of the contract o A second SO describing a change to the Customer Want Date for the first Service Order as defined in Section J.2.10.1.1.4.3 o The Customer Want Date Change Order may be issued before or after the deadline described in Section G.3 |
E.2.1.3.5.5 BSS-TS05-05: Change to Administrative Data
|
Test Scenario |
BSS-TS05: Supplements to In-Progress Orders |
|
Test Case ID |
BSS-TS05-05 |
|
Test Case Description |
Change to Administrative Data |
|
Requirements Reference(s) |
· G.3 · J.2.4 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS |
|
Government Input(s) |
· Service Order (SO) with all required data elements as described in Section J.2.4 · Service Order (SO) for a change to the administrative data for the previous order with all required data elements as described in Section J.2.4 issued prior to completion of the previous order |
|
Expected Output(s) |
· Service Order notification CDRLs as defined in Section J.2.4: o Service Order Acknowledgment (SOA) o Updates to Service Order Confirmation (SOC) if required o Updates to Firm Order Commitment Notice (FOCN) if required o Service Order Completion Notification (SOCN) |
|
Acceptance Criteria |
· All required CDRLs · Accurate data based on inputs · Correct technical aspects |
|
Data Set Description |
· Each government-provided test data set will include: o A complete SO for one or more services listed in Section C.2 of the contract o A second SO describing a change to the administrative data for the first Service Order as defined in Section J.2.10.1.1.4.3 |
E.2.1.3.6 BSS-TS06: Administrative Change Orders
E.2.1.3.6.1 BSS-TS06-01: Administrative Change Order
|
Test Scenario |
BSS-TS06: Administrative Change Order |
|
Test Case ID |
BSS-TS06-01 |
|
Test Case Description |
Administrative Change Order |
|
Requirements Reference(s) |
· G.3 · J.2.4 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS · One or more previously provisioned orders |
|
Government Input(s) |
· Administrative Change Order that specifies a change to the administrative data associated with a previously provisioned service as described in Section G.3 |
|
Expected Output(s) |
· Service Order notification CDRLs as defined in Section J.2.4: o Service Order Administrative Change (SOAC) |
|
Acceptance Criteria |
· All required CDRLs · Accurate data based on inputs · Correct technical aspects |
|
Data Set Description |
· Each government-provided test data set will include: o A complete administrative change order for a change to the administrative data for a previously provisioned service as described in Section G.3 |
E.2.1.3.7 BSS-TS07: Rapid Provisioning & Self-Provisioning Orders
E.2.1.3.7.1 BSS-TS07-01: Rapid Provisioning Orders
|
Test Scenario |
BSS-TS07: Rapid Provisioning & Self-Provisioning Orders |
|
Test Case ID |
BSS-TS07-01 |
|
Test Case Description |
Rapid Provisioning Orders |
|
Requirements Reference(s) |
· G.3 · G.5.3.1 · J.2.4 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS |
|
Government Input(s) |
· Service Order (SO) for one or more services subject to rapid provisioning as defined in Section G.3 with all required data elements as described in Section J.2.4 |
|
Expected Output(s) |
· Service Order notification CDRLs as defined in Section J.2.4: o Service Order Acknowledgment (SOA) if provisioning requires more than 24 hours o Service Order Completion Notification (SOCN) |
|
Acceptance Criteria |
· All required CDRLs as defined in Section J.2.4 · Accurate data based on inputs · Correct technical aspects |
|
Data Set Description |
· Each government-provided test data set will include: o A complete SO for one or more services subject to rapid provisioning as defined in Section G.3 |
E.2.1.3.7.2 BSS-TS07-02: Self-Provisioning Orders
|
Test Scenario |
BSS-TS07: Rapid Provisioning & Self-Provisioning Orders |
|
Test Case ID |
BSS-TS07-02 |
|
Test Case Description |
Self-Provisioning Orders |
|
Requirements Reference(s) |
· G.3 · G.5.3.2 · J.2.4 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS |
|
Government Input(s) |
· Service Order (SO) with all required data elements as described in Section J.2.4 for one or more services that are: o Subject to rapid provisioning as defined in Section G.3 o Available for self-provisioning as defined in Section G.3 and Section C.2 |
|
Expected Output(s) |
· Service Order notification CDRLs as defined in Section J.2.4: o Service Order Acknowledgment (SOA) if provisioning requires more than 24 hours o Service Order Completion Notification (SOCN) |
|
Acceptance Criteria |
· All required CDRLs as defined in Section J.2.4 · Accurate data based on inputs · Correct technical aspects |
|
Data Set Description |
· Each government-provided test data set will include: o Service Order (SO) with all required data elements as described in Section J.2.4 for one or more services that are subject to rapid provisioning as defined in Section G.3 and that are available for self-provisioning as defined in Section G.3 and Section C.2 |
E.2.1.3.7.3 BSS-TS07-03: Self-Provisioning Orders: Error Checking
|
Test Scenario |
BSS-TS07: Rapid Provisioning & Self-Provisioning Orders |
|
Test Case ID |
BSS-TS07-03 |
|
Test Case Description |
Self-Provisioning Orders: Error Checking |
|
Requirements Reference(s) |
· G.3 · G.5.3.2 · J.2.4 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS |
|
Government Input(s) |
· Service Order (SO) for one or more services subject to rapid provisioning as defined in Section G.3 and available for self-provisioning · Populated via Contractors Portal with one or more missing or invalid data elements as described in Section J.2.4 |
|
Expected Output(s) |
· Service Order notification CDRLs as defined in Section J.2.4 o Service Order Rejection Notice (SORN) o User is shown error message indicating failure |
|
Acceptance Criteria |
· All required CDRLs as defined in Section J.2.4 · Accurate data based on inputs · Correct technical aspects · Appropriate errors are displayed |
|
Data Set Description |
· Each government-provided test data set will include: o Service Order (SO) with all required data elements as described in Section J.2.4 for one or more services that are subject to rapid provisioning as defined in Section G.3 and that are available for self-provisioning as defined in Section G.3 and Section C.2 o SO to be provided via Contractors Portal |
E.2.1.3.8 BSS-TS08: Inventory and Billing
E.2.1.3.8.1 BSS-TS08-01: Inventory Reconciliation
|
Test Scenario |
BSS-TS08: Inventory and Billing |
|
Test Case ID |
BSS-TS08-01 |
|
Test Case Description |
Inventory Reconciliation |
|
Requirements Reference(s) |
· G.7 · J.2.7 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS · One or more previously provisioned orders |
|
Government Input(s) |
· N/A |
|
Expected Output(s) |
· Inventory Reconciliation (IR) CDRLs as described in Section J.2.7 |
|
Acceptance Criteria |
· All required CDRLs · Accurate data based on inputs · Correct technical aspects |
|
Data Set Description |
· N/A, see Prerequisites |
E.2.1.3.8.2 BSS-TS08-02: Billing
|
Test Scenario |
BSS-TS08: Inventory and Billing |
|
Test Case ID |
BSS-TS08-02 |
|
Test Case Description |
Billing |
|
Requirements Reference(s) |
· J.2.5 · J.2.10 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS · One or more previously provisioned orders |
|
Government Input(s) |
· N/A |
|
Expected Output(s) |
· Billing CDRLs as defined in Section J.2.4: o Billing Invoice (BI) o Tax Detail Report (TAX) o Associated Government Fee Detailed (AGFD) o AGF EFT Report (ATR) |
|
Acceptance Criteria |
· All required CDRLs · Accurate data based on inputs · Correct technical aspects · CDRLs are internally consistent · Complies with calculation rules |
|
Data Set Description |
· N/A, see Prerequisites |
E.2.1.3.8.3 BSS-TS08-03: Usage Based Billing
|
Test Scenario |
BSS-TS08: Inventory and Billing |
|
Test Case ID |
BSS-TS08-03 |
|
Test Case Description |
Usage Based Billing |
|
Requirements Reference(s) |
· G.3 · J.2.4 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS · One or more previously usage based provisioned orders |
|
Government Input(s) |
· Sample usage data for one or more UBI based on Usage Based CLIN(s) |
|
Expected Output(s) |
· Billing CDRLs as defined in Section J.2.4: o Billing Invoice (BI) o Tax Detail Report (TAX) o Associated Government Fee Detailed (AGFD) o AGF EFT Report (ATR) |
|
Acceptance Criteria |
· All required CDRLs · Accurate data based on inputs · Correct technical aspects · CDRLs are internally consistent · Complies with calculation rules |
|
Data Set Description |
· Each government-provided test data set will include: o Sample usage data for one or more UBI based on Usage Based CLIN(s) o See also Prerequisites |
E.2.1.3.8.4 BSS-TS08-04: Billing Adjustments
|
Test Scenario |
BSS-TS08: Inventory and Billing |
|
Test Case ID |
BSS-TS08-04 |
|
Test Case Description |
Billing Adjustments |
|
Requirements Reference(s) |
· G.4 · J.2.5 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS · One or more previously provisioned orders · At least one previously submitted Billing Invoice (BI) |
|
Government Input(s) |
· Sample adjustment request to change or modify a billing line item |
|
Expected Output(s) |
· Billing Adjustment (BA) as defined in Section J.2.5: o Reflects requested adjustment |
|
Acceptance Criteria |
· All required CDRLs · Accurate data based on inputs · Correct technical aspects |
|
Data Set Description |
· Each government-provided test data set will include: o Sample adjustment request to change or modify a billing line item |
E.2.1.3.9 BSS-TS09: Dispute Handling
E.2.1.3.9.1 BSS-TS09-01: Government Initiated Dispute
|
Test Scenario |
BSS-TS09: Dispute Handling |
|
Test Case ID |
BSS-TS09-01 |
|
Test Case Description |
Government Initiated Dispute |
|
Requirements Reference(s) |
· J.2.3 · J.2.6.3 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS · One or more previously provisioned orders · At least one previously submitted Billing Invoice (BI) |
|
Government Input(s) |
· Government issues at least 2 Disputes (D) as defined in Section J.2.6 · Notification to close a dispute after first Dispute Report is issued (see expected outputs) |
|
Expected Output(s) |
· Dispute Report (DR) o Reflects open Disputes · A second Dispute Report (DR) o Reflects open and closed Disputes |
|
Acceptance Criteria |
· All required CDRLs · Accurate data based on inputs · Correct technical aspects |
|
Data Set Description |
· Each government-provided test data set will include: o At least two Disputes (D) o Notification to close one or more disputes |
E.2.1.3.10 BSS-TS10: SLA Management
E.2.1.3.10.1 BSS-TS10-01: SLA Reporting
|
Test Scenario |
BSS-TS10: SLA Management |
|
Test Case ID |
BSS-TS10-01 |
|
Test Case Description |
SLA Reporting |
|
Requirements Reference(s) |
· G.3 · J.2.4 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS · One or more previously provisioned orders |
|
Government Input(s) |
· Services to show as SLAs met or missed |
|
Expected Output(s) |
· SLA Report (SLAR) |
|
Acceptance Criteria |
· All required CDRLs · Accurate data based on inputs · Correct technical aspects |
|
Data Set Description |
· Services by UBI to show as SLAs met or missed |
E.2.1.3.10.2 BSS-TS10-02: SLA Credit Request
|
Test Scenario |
BSS-TS10: SLA Management |
|
Test Case ID |
BSS-TS10-02 |
|
Test Case Description |
SLA Credit Request |
|
Requirements Reference(s) |
· G.3 · J.2.4 |
|
Prerequisites |
· TO controlled, TO associated, and System Reference data loaded into contractor BSS o One or more previously provisioned orders o At least one previously submitted Billing Invoice (BI) o SLA Report with at least one SLA missed |
|
Government Input(s) |
· SLA Credit Request (SLACR) |
|
Expected Output(s) |
· SLA Credit Request Response |
|
Acceptance Criteria |
· All required CDRLs · Each CDRL meets requirements |
|
Data Set Description |
· Each government-provided test data set will include: o SLA Credit Request |
E.2.1.3.11 BSS-TS11: Open-Format Reporting
E.2.1.3.11.1 BSS-TS11-01: Open-Format Reporting: Samples
|
Test Scenario |
BSS-TS11-01: Open-Format Reporting |
|
Test Case ID |
BSS-TS11-01 |
|
Test Case Description |
Open-Format Reporting: Samples |
|
Requirements Reference(s) |
· G.4 · G.5 |
|
Prerequisites |
· N/A |
|
Government Input(s) |
· N/A |
|
Expected Output(s) |
· Sample copies of the contractor’s standard reports for: o Monthly Billing Information Memorandum o Trouble Management Incident Performance Report § Describes service outage or degradation that are user initiated and/or automated monitoring created reports o Trouble Management Performance Summary Report |
|
Acceptance Criteria |
· Each CDRL meets requirements |
|
Data Set Description |
· N/A |
E.2.1.3.12 BSS-TS12: Regression Testing
E.2.1.3.12.1 BSS-TS12-01: Regression Testing
|
Test Scenario |
BSS-TS12: Regression Testing |
|
Test Case ID |
BSS-TS12-01 |
|
Test Case Description |
Regression Testing: Test Cases TBD |
|
Requirements Reference(s) |
· G.5.5 |
|
Prerequisites |
· Contractor BSS has completed Development testing and ATO |
|
Note |
· Inputs, outputs, acceptance criteria and datasets to be determined based on Change Management provided in G.5.5 · Additional test cases may be defined as needed |
E.2.1.3.13 BSS-TS13: Security Testing
E.2.1.3.13.1 BSS-TS13-01: Security Testing
|
Test Scenario |
BSS-TS13: Security Testing |
|
Test Case ID |
BSS-TS13-01 |
|
Test Case Description |
Security Testing |
|
Requirements Reference(s) |
· G.5.6 |
|
Prerequisites |
· Defined in Section G.5.6 and references therein |
|
Government Input(s) |
· Defined in Section G.5.6 and references therein |
|
Expected Output(s) |
· Defined in Section G.5.6 and references therein |
|
Acceptance Criteria |
· Contractor BSS receives ATO |
|
Data Set Description |
· Defined in Section G.5.6 and references therein |
E.2.1.4 Test Results
The contractor shall demonstrate that it successfully meets the BSS acceptance criteria for the various test scenarios/test cases defined in Sections E.2.1.2 and E.2.1.3.The contractor shall provide test results that will provide details of testing the following:
· Functional requirements for the Ordering, Billing, Inventory Management, Disputes, SLA Management and trouble ticketing processes as described in Section G and Section J.2.
· System to system data exchange mechanism requirements defined in Section G.5 for each CDRLs defined in Section J.2.
· Correct CDRLs are used in the data exchange.
· Mandatory data elements for each CDRL defined in Section J.2.10 Data Dictionary are populated and accurate.
· Available optional data elements for each CDRL defined in Section J.2.10 Data Dictionary are populated and accurate.
· Timely and successful system to system data exchange to meet defined performance SLAs and provisioning intervals.
The test results shall detail at a minimum the following:
Test scenario # / Test case # / Test Data Set # / Test #; Date of Test performed, Acceptance Criteria, Test Result (Pass/Fail), etc.
E.2.1.5 Deliverables
E.2.1.5.1 Verification Test Plan for Contractor’s BSS
The contractor shall submit a BSS Verification Test Plan (BSS Test Plan) based on the following timeline:
· Draft: with proposal
· Final: 30 days after NTP
· Revisions: 14 days after receipt of government comments
The BSS Test Plan shall:
· Reflect the test methodology defined in Section E.2.1.
· Include the contractor’s approach to testing each test scenario and test case
· Include the contractor’s timeline and test sequencing
E.2.1.5.2 Verification Test Results Report for Contractor’s BSS
The contractor shall provide a BSS Verification Test Results Report that includes analysis of the current testing and a summary table of all previously submitted test results, within seven (7) days after performance of the tests. The government reserves fourteen (14) days to accept or reject the test results, in part or in whole.
· Contractor shall perform re-test of test cases with test data sets that failed until they are accepted by the government.
· The contractor shall rerun tests, in part or in whole, as deemed necessary by the government, to verify that the government’s comments on the test results are satisfactorily addressed.
E.2.2 EIS Services Verification Testing
The contractor shall provide an EIS Services Verification Test Plan (EIS Test Plan) based on the test methodology defined in this section(test scenarios, test cases, test data sets, acceptance criteria) in response to the RFP for each of the proposed EIS services.
E.2.2.1 General Testing Requirements
The contractor shall meet the following EIS Services testing requirements:
· Provide a verification and acceptance testing approach for all awarded EIS services defined in Section C.2.
· Develop an EIS Test Plan that includes, but is not limited to:
o The test methodology for each EIS Service with test cases that will define the parameters to be measured, the measurement procedure, and the acceptance (pass/fail) criteria.
o Fallback approach to describe the fallback process and procedures in case of testing failure.
o An EIS Test Plan shall be required for all new services during the life of the contract.
o Identification of any services from Section C.2 to which testing is not applicable.
The following conditions also apply:
· An agency may define additional testing in the TO.
· The contractor shall allow government representative(s) to observe all or any part of the EIS services verification testing.
· The contractor shall provide all necessary test equipment: data terminals, load boxes, test cables, and any other hardware and software required for testing.
E.2.2.2 Test Scenarios
The EIS Test Plan shall include, but not be limited to, the following test scenarios:
E.2.2.2.1 EIS Services Verification Test Scenarios
|
Test Scenario # |
RFP Sec # |
Description |
Acceptance Criteria |
|
Service TS-01 |
Demonstrate that the contractor’s Cloud Services is compliant with Federal Risk and Authorization Management Program (FedRAMP) requirements as defined. · http://cloud.cio.gov http://cloud.cio.gov/fedramp/csp · NIST.gov publications http://csrc.nist.gov/groups/SMA/ispab/documents/minutes/2013-06/ispab_june2013_goodrich.pdf |
The contractor shall provide FedRAMP certification and ensure that it is verified and accepted by GSA. |
|
|
Service TS-02 |
Demonstrate that awarded services are delivered based on the KPIs and SLAs defined. |
The contractor shall demonstrate that the service works properly according to KPIs defined in Section C.2. |
|
|
Service TS-03 |
Verification Testing of Dark Fiber Services |
See Section C.2.1.6.4 for acceptance criteria |
E.2.2.3 Test Cases
The contractor shall provide test cases for each of the test scenarios defined in Section E.2.2.2. The test cases will be defined in the EIS Test Plan.
E.2.2.4 Test Data Sets
The contractor shall successfully test all of the test cases defined in the EIS Test Plan using one or more test data sets proposed by the contractor. The contractor shall test all services and service features proposed at the TO. The contractor shall use test data sets that reflect real-world service conditions and locations and shall address all relevant test cases.
E.2.2.5 Test Results and Acceptance
The contractor shall provide an EIS Services Verification Testing Report (EIS Testing Report) that shows successful completion of testing defined in the EIS Test Plan. The contractor shall complete verification and acceptance testing based on the acceptance criteria defined in the government accepted EIS Test Plan. Acceptance shall include government compliance requirements such as FedRAMP for cloud services, and the ATO for FISMA related security requirements for EIS services. The contractor shall provide the following in order for the government to approve the test results:
· FedRAMP certification if cloud services are included in the TO.
· EIS Testing Report showing that each service provisioned works properly according to the KPIs defined in Section C.2 and the acceptance criteria defined in the EIS Test Plan.
Once verification testing is completed successfully the government may complete acceptance testing based on the acceptance criteria defined in the EIS Test Plan. The acceptance test will verify satisfactory end-to-end service performance and proper operation of all ordered features and functions. Performance will be considered satisfactory when services, equipment, systems and their associated features and functions perform as specified in the contract and TO. The contractor may not assign an effective billing date to an EIS Service until the agency accepts it in accordance with the agreed-upon acceptance testing procedures described in the EIS Test Plan.
The government reserves the right to perform additional tests to confirm proper operation of a delivered EIS service as defined by the TO. If the government does not report a problem to the contractor during this test period, the effective billing date will be the completion date on the SOCN. The contractor shall not begin billing for services if the government rejects the services within three (3) days of receipt of the SOCN. A longer period for test and acceptance may be specified in the TO. The contractor shall issue a new SOCN for services after correcting the reasons for rejection.
The service will be considered accepted if the government does not reject the service within the acceptance period defined above. If the government rejects the service, it may at its option:
1. Direct the contractor to repeat the procedure outlined above;
2. Withdraw the service from acceptance testing;
3. Direct the contractor to facilitate the return of the services to their original provider (for services transitioned or migrated from another contractor’s network);
4. Request a replacement of the service (in whole or in part); or
5. Cancel the service order without penalty.
If the government exercises any of these options as a consequence of unacceptable acceptance testing results, all expenses incurred by the government shall be borne by the contractor.
If the government elects option 1 above, the contractor shall immediately initiate corrective actions to remedy the problem reported on the trouble ticket and shall keep the government informed of progress. In such cases, the government reserves the right to exercise option 2, 3, 4 or 5 at any time.
If the government elects any of the options above other than option 1, all expenses incurred by the government, including recurring charges and non-recurring charges (NRC) to return services to the previous network configuration, shall be borne by the contractor. In cases when the government cannot successfully complete acceptance testing due to circumstances beyond the control of the contractor, the contractor shall notify government of the details surrounding the deficiencies and the steps the contractor has taken to overcome the deficiencies.
These cases shall be discussed between the government and the contractor. On a case-by-case basis, the GSA CO or the OCO may choose to waive the acceptance testing or extend the testing period. Waiver of the acceptance testing may be considered in those instances when the contractor has demonstrated that the problems encountered are not the fault of the contractor and government has determined that the contractor has taken all reasonable actions to correct all problems. The waiver issued by the GSA CO or the OCO will specify the grounds for the waiver. If the waiver is not granted, the contractor shall be obligated to continue to attempt correction of the deficiencies encountered in order to successfully accomplish the acceptance testing.
E.2.2.6 Deliverables
The contractor shall provide an EIS Test Plan in its proposal that describes the testing of EIS Services based on test methodology described in Sections E.2.2.1 – E.2.2.5. Updates shall be submitted for any new services that are added to the contract with the modification proposal.
The contractor shall provide an EIS Testing Report as defined in Section E.2.2.5 within 3 days of service installation and testing.