Term
|
Definition
Test & Evaluation
The Generation, Gathering & Evaluation of Data in Support of the Decision Maker |
|
|
Term
| What are the 2 reason discussed in class for conducting T&E? |
|
Definition
1. To Make The System Work
2. To determine the system works |
|
|
Term
|
Definition
The process to Obtain, Verify or Provide Data for determining:
Progress
Performance
Vulnerability & Lethality of Systems |
|
|
Term
|
Definition
| The process for review and analysis of qualitative or quantitative data obtained from design review, hardware inspection, modeling and simulation, testing, or operational usage of equipment to aid in making decisions. |
|
|
Term
| 2 Principle Types of T&E? |
|
Definition
Developmental(DT)
Operational (OT) |
|
|
Term
|
Definition
Multi-Service
Joint
Live Fire
Nuclear, Chemical, Biological
Nuclear Hardening/Survivability |
|
|
Term
| DoD agencies with primary responsibility for the DoD T&E process? |
|
Definition
Director, Systems Engineering, Defense Systems
Director, OT&E: OT and LF
ASD (NII) (JITC) |
|
|
Term
| Which area of T&E does Director, Systems Engineering, Defense Systems have the primary responsibility? |
|
Definition
|
|
Term
| Which area of T&E does Director, OT&E: OT and LF have the primary responsibility? |
|
Definition
| Operational Testing & Live Fire |
|
|
Term
| Which area of T&E does ASD (NII) (JITC)have the primary responsibility? |
|
Definition
| IT Interoperability Testing |
|
|
Term
| How is T&E integrated in the Systems Engineering Process? |
|
Definition
| Thru the process of the Verification Loop |
|
|
Term
| What are types of Verification that integrate T&E into the Systems Engineering Process? |
|
Definition
Inspection
Demonstrations & Certifications
Simulations & Analyses
Testing |
|
|
Term
| What are the 5 steps involved in the DoD T&E process? |
|
Definition
Identify the Information Needed
Pre-Test Analysis
Test Activity
Post-Test Synthesis & Evaluation
Decision Maker Weighs info |
|
|
Term
| What are 3 purposes of OT&E? |
|
Definition
Determine Operational Effectiveness & Operational Suitability under realistic conditions
Identify Operational Risk
Determine whether minimum acceptable operational performance requirements (CDD thresholds) have been satisfied |
|
|
Term
| List at least 2 types of OT&E? |
|
Definition
Early Operational Assessment (EOA)
Early User T&E (EUTE)
Limited User Test (LUT)
Initial Operational T&E (IOT&E) or OPEVAL (Navy)
Follow-On OT&E (FOT&E)
Qualification OT&E (QOT&E) |
|
|
Term
| Operational Effectiveness |
|
Definition
| The overall degree of mission accomplishment of a system when used by representative personnel in the environment planned or expected (e.g. natural, electronic, threat etc.) for operational employment of the system considering organization, doctrine, tactics, survivability, vulnerability, and threat |
|
|
Term
|
Definition
| The degree to which a system can be placed satisfactorily in field use with consideration given to availability, compatibility, transportability, interoperability, reliability, wartime usage rates, maintainability, safety, human factors, manpower supportability, logistics supportability, natural environmental effects and impacts, documentation and training requirements. |
|
|
Term
| What is the purpose of IOT&E? |
|
Definition
| To support a Production or Procurement Decision |
|
|
Term
Three characteristics of OT&E.
(There are apparently more than 3) |
|
Definition
1. Realistic operational 'Combat' environment including Enemy Counter-Measures when possible
2. Accomplished by typical operational & support expected to use and maintain the deployed system
3. Contractor involvement is restricted/prohibited
4. Designed to Obtain Operational Effectiveness & Operational Suitability data
5. Systems tested are "Production Representative:, usually early units from LRIP
6. Must have sufficient numbers of systems to gain OT operational data - "a unit set"
7. Test design driven by operational doctrine
8. Managed by Operational Test Agency (OTA) which is independent of Contractor and Developing Agency |
|
|
Term
| What is the purpose of LVFT&E? |
|
Definition
Live Fire Test & Evaluation
Determines Lethality & Survivability |
|
|
Term
ACAT I & II systems cannot pass LRIP without completing LVFT&E.
(T/F) |
|
Definition
|
|
Term
LVFT&E applies realistic combat conditions to systems and pits them against actual threats.
(T/F) |
|
Definition
|
|
Term
| What are the benefits of LFT&E? |
|
Definition
Design enhancements & Survivability analysis
Battle Damage Assessment and Repair (BDAR)
Tactics Development |
|
|
Term
|
Definition
| When development testing and operational testing take place at the same time as two parallel but seperate distinct activities |
|
|
Term
|
Definition
| Refers to a single test program conducted to support development test (DT) and operational test (OT) objectives |
|
|
Term
| What 2 OT&E reports must be submitted to Congress prior to FRPDR? |
|
Definition
|
|
Term
| What is the purpose of the Test & Evaluation Strategy? |
|
Definition
1. The strategy balances Program Technical & Operational Risk with T&E cost and schedule.
2. T&E Strategy identifies the verification method for system technical and operational performance
3. Certification
4. Demonstration
5. Modeling & Simulation
6. Contractor & Government Testing |
|
|
Term
T&E is not a significant cost and schedule driver
(T/F) |
|
Definition
False
It is a significant driver |
|
|
Term
Rank the following T&E strategy costs from lowest to highest
Certification
M&S
Demonstration
Contractor & Govt. |
|
Definition
1. Certification (no/low cost)
2. Demonstration (low cost)
3. M&S (variable, but less than test)
4. Contractor & Government Testing (highest cost) |
|
|
Term
|
Definition
1. Reduces cost and duration of T&E programs
2. Iterative costs are extremely low, so excellent statistical confidence
3. Ability to influence system design early
4. Ability to see system impacts in large-scale operations |
|
|
Term
|
Definition
1. Initially expensive and time consuming to create
2. Must be validated and verified for data acceptance
3. Limited situational & technical variables
4. Decision Maker Skepticism |
|
|
Term
| Who is responsible for preparing the TEMP? |
|
Definition
|
|
Term
|
Definition
Documents the overall structure and objectives of the T&E program
Provides for the generation detailed T&E plans and documents associated schedule and resource implications
Identifies the necessary DT&E, OT&E, & LFT&E |
|
|
Term
|
Definition
Can be functionally decomposed
Traceable to System Requirements
Have accountability by task
Progress can be monitored
Are bound by operating principles & constraints |
|
|
Term
| How is SW different from HW? |
|
Definition
Lacks physical nature
Greater logic complexity
Tends to propagate change effects
Is data & logic
Limited standardized design methods, components or structure |
|
|
Term
| Why does DoD have difficulty in managing SW development? |
|
Definition
SW development us dynamic & rapidly evolving
PMs & decision makers are not SW knowledgeable
DoD cannot compete with industry for SW engineers
Inability to accurately convey requirements
Requirements Creep
SW/HW development is unccordinated
Poor or non-existent SW development metrics
Inadequate SW testing
Typical SW development project underestimates the development schedule by half |
|
|
Term
| What are the two broad categories of DoD SW? |
|
Definition
Weapon Systems SW
Management Information Systems |
|
|
Term
| Two types of Weapon System SW? |
|
Definition
|
|
Term
|
Definition
Information System Resource (ISR)
Automated Information System (AIS)
Information Resource Management (IRM)
All other Non-Weapon Systems SW |
|
|
Term
|
Definition
| Designed into or dedicated to as an integral part of the system - usually not directly accessible |
|
|
Term
| What is the growth of embedded SW LOC? |
|
Definition
|
|
Term
|
Definition
Command, Control & Communications
Communicates, assimilates, coordinates, analyzes and interprets data providing decision support to commanders |
|
|
Term
|
Definition
Often combined with C3 to make C3I
Facilitates surveillance, tracking and warning decision support |
|
|
Term
|
Definition
SW Maintenance & SW Redevelopment
Those software support activities that occur after the deployment of the system |
|
|
Term
60%-80% of a programs SW costs fall in PDSS.
(T/F) |
|
Definition
|
|
Term
| What is critical to a successful PDSS? |
|
Definition
Determine your life cycle support strategy early
Remember that software support is actually software redevelopment
Select a good contractor
Specify required documentation and verification methods in the appropriate CDRLs
Establish a CRIPT
Identify necessary software development and support tools in CRISD
Documentation, source code, and a description of the software design and test procedures. |
|
|
Term
| What management device is used by the PM to develop, field, & support SW? |
|
Definition
|
|
Term
|
Definition
| Computer Resource Lifecycle Management Plan. |
|
|
Term
| What entity accomplishes the CRLCMP |
|
Definition
| Computer Resources IPT (CRIPT) |
|
|
Term
|
Definition
Application Specific Integrated Circuit
Custom Chip with embedded firmware not electronically reprogrammable |
|
|
Term
|
Definition
| When Weight, Speed, & Space are critical |
|
|
Term
| What position did the Clinger Cohen Act Establish? |
|
Definition
|
|
Term
| What are the imperatives of the Clinger Cohen Act? |
|
Definition
Orient IT investments to a strategic business, mission oriented focus
Manage resources based on performance & results
Mandate performance measures for all IT systems
Use business process reengineering prior to information system acquisition |
|
|
Term
| Why did the Clinger Cohen Act come about? |
|
Definition
Address long-standing, government-wide weaknesses in information resource management.
Improve the way the Government acquires and manages IT. |
|
|
Term
| What does the Clinger Cohen Act Address? |
|
Definition
Strategic information resource plans, measurements.
Implementation of best business practices
Requirement for a Chief Information Officer (CIO) (ASD(NII) at DoD)
Information Technology Systems: Any equipment handling data or information.
National Security Systems (NSS’s): Any telecommunications or IT
Intelligence activities
Cryptologic activities related to national security
Command and control of military forces
Integral part of a weapons system,
Critical to the direct fulfillment of military or intelligence missions. |
|
|
Term
| What is critical to a successful PDSS? |
|
Definition
Determine your life cycle support strategy early
Remember that software support is actually software redevelopment
Select a good contractor
Specify required documentation and verification methods in the appropriate CDRLs
Establish a CRIPT
Identify necessary software development and support tools in CRISD
Documentation, source code, and a description of the software design and test procedures. |
|
|
Term
|
Definition
Program Acquisition Baselines – Inflexible Schedules
Requirements: Unclear, Lack Definition, Creep
Technical Reviews not driven by software development
Poor software metrics and software documentation
Poor management of software effort; Managers don’t understand software development
Software development contractor personnel turnover |
|
|
Term
Evolutionary strategies are the preferred approach for the development and lifecycle management of SW.
(T/F) |
|
Definition
|
|
Term
| Spiral Development is the preferred process for the development and lifecycle management of SW. |
|
Definition
|
|
Term
|
Definition
Series of incremental releases with each increment increasing system functionality
Risk is spread over several increments
Requirements are stabilized for each increment with users deferring changes to later increments
Requirements refined with each increment
Low to medium risk programs |
|
|
Term
|
Definition
Risk reducing approach
Iterative and recursive
Can produce IOC rapidly
Final cycle produces FOC
Not conducive to Schedule Driven program
|
|
|
Term
|
Definition
First formal software development framework
Not suited to modern development techniques
Each step is a prerequisite for following steps, increasing risk
Integration problems surface too late in the model
A complete product is not available until the end
Not recommended for major, software intensive systems development |
|
|
Term
| What are the four iterative & recursive steps of the Spiral Process? |
|
Definition
Determine objectives, alternatives and constraints
Evaluate alternatives, identify and resolve risks
Develop next level product
Plan next phase |
|
|
Term
| What is the output of the SW in the SEP? |
|
Definition
| SW System/Subsystem Design Description (SSDD) |
|
|
Term
| What are the goals of SW metrics? |
|
Definition
Monitor Requirements
Predict Development Resources
Track development progress
Understand maintenance and supportability costs |
|
|
Term
| Define Management Metrics |
|
Definition
Progress vs Plans
deals with questions regarding scheduling, personnel, requirements, volatility, cost performance and individual work unit progress |
|
|
Term
|
Definition
Quality of effort
Concerned with SW product attributes that can impact performance, user satisfaction, supportability, and ease of change |
|
|
Term
|
Definition
| Measuring the contractor's SW development capabilities |
|
|
Term
| Ground rules for the establishment of metrics |
|
Definition
Understandable
Economical
Field Tested
Highly Leveraged
Timely
Incentivize Process Improvement |
|
|
Term
|
Definition
Must be used as indicators, not absolutes
Only as good as the data
Must be understood
Metrics cannot id, explain, or predict every problem
Direct Comparisons should be avoided
Avoid using a 'single' metric |
|
|
Term
| Name the SW process model most commonly used in DoD Acquisitions. |
|
Definition
| SEI's SW Capability Maturity Model (CMM) |
|
|
Term
| How many levels does the CMM have and what are they? |
|
Definition
5
Level 1: Initial - Unpredictable & Poorly Controlled
Level 2: Repeatable - Mastered Tasks are Repeatable
Level 3: Defined - process understood & characterized
Level 4: Managed - Process Measured & Controlled
Level 5: Optimizing - Focus on Process Improvement |
|
|
Term
|
Definition
| Software Engineering Institute Carnegie-Mellon |
|
|
Term
|
Definition
| A tool for effectively integrating cost, schedule & technical performance management |
|
|
Term
| One primary benefit of EV? |
|
Definition
| provides "Early Warning" of impeding cost, schedule & performance issues |
|
|
Term
| How does WBS relate to EV? |
|
Definition
Provides "What" must be accomplished
Lowest level decomposition describes what "Work Packages" must produce
The basis that EV is built upon |
|
|
Term
| What is the FUNDAMENTAL WBS element developed by teh KTR, which let's both the Govt. and the KR both budget, and monitor cost and schedule performance? |
|
Definition
|
|
Term
|
Definition
Performance Measurement Baseline
The PMB is a roll-up of all Work Packages, creating a Time-Phased Budget Plan |
|
|
Term
| 2 Types of variances the EVM measures? |
|
Definition
Schedule Variance (SV)
Cost Variance (CV) |
|
|
Term
|
Definition
| Actual Cost of Work Performed |
|
|
Term
|
Definition
Budgeted Cost of Work Performed
Work Package Progress is aggregated identifying work performed to date. |
|
|
Term
|
Definition
Budgeted Cost of Work Schedule
Planned Progress of Work Packages |
|
|
Term
|
Definition
|
|
Term
|
Definition
| Estimate At Completion (Last Revised Estimate) |
|
|
Term
|
Definition
|
|
Term
|
Definition
Cost Variance
BCWP - ACWP
(-result) = overcost
(+result) = under cost |
|
|
Term
|
Definition
Schedule Variance
BCWP - BCWS
(-result) = behind schedule
(+result) = ahead of schedule |
|
|
Term
| Methods for measuring progress in EVM? |
|
Definition
% Complete
50/50
Level Loaded
Variant Loaded
Apportioned |
|
|
Term
|
Definition
| Credit = % of Work Package Complete |
|
|
Term
|
Definition
| 50% credited for Opening & 50% for closing a work package |
|
|
Term
|
Definition
| Equal Amount Credited at Milestones |
|
|
Term
|
Definition
| Different amount credited at Milestones |
|
|
Term
|
Definition
| Credit based on Work Package Factor |
|
|
Term
| EAC impact of a "One Time" event |
|
Definition
| Variance likely to remain constant |
|
|
Term
| EAC impact of a Product or Process "Factor" |
|
Definition
| Variance likely to increase |
|
|
Term
| When should EVM be employed by the PMO? |
|
Definition
EVMS
Non-FFP contracts / agreement for R&D > $73M; & Proc or O&M > $315M (CY)
Cost/Schedule Status Report (C/SSR)
Non-FFP contracts / agreements > $6.3M(CY) & > !2 months duration |
|
|
Term
|
Definition
| The relative ease of manufacturing an item or system |
|
|
Term
| How is the producibility determined (in general). |
|
Definition
| By the characteristics and features of a design that enables economical fabrication, assembly, inspection, and testing using available manufacturing techniques |
|
|
Term
Producibility of the system design shall be a development priority.
(T/F) |
|
Definition
|
|
Term
Producibility is the responsibility of the Contracting Officer.
(T/F) |
|
Definition
|
|
Term
| Relationship between producibility and the SEP? |
|
Definition
System Producibility is addressed through the SEP
Producibility challenges are identified early and resolved continuously through the iterative SEP problem solving process
Identified producibility risks are included in the Risk Management Plan |
|
|
Term
| Identify the producibility development activity that occurs in Concept Refinement and Tech Development. |
|
Definition
| Exploration of alternatives and associated Producibility methods & challenges |
|
|
Term
| Identify the producibility development activity that occurs in System Development and Demonstration. |
|
Definition
| Manufacturing process are demonstrated & design materials qualified |
|
|
Term
| Reviews that impact producibility |
|
Definition
|
|
Term
| SDR - how it impacts producibility |
|
Definition
System Design Review
Establishes System Functional Baseline
Evaluates System Design |
|
|
Term
| PDR - how it impacts producibility |
|
Definition
Preliminary Design Review
Establishes System Allocated Baseline
Evaluates preliminary H/W & S/W component design |
|
|
Term
| FCA - how it impacts producibility |
|
Definition
Functional Config Audit
Evaluates functional design
Evaluates critical interfaces |
|
|
Term
| CDR - how it impacts producibility |
|
Definition
Critical Design Review
Evaluates detailed design
Determines readiness for H/W fabrication & S/W final build |
|
|
Term
| PCA - how it impacts producibility |
|
Definition
Physical Config Audit
Confirms that design meets Product Baseline for all Configuration Items |
|
|
Term
| PRR - how it impacts producibility |
|
Definition
Production Readiness Review
Assesses Risk for production |
|
|
Term
| Potential Production Risk Drivers |
|
Definition
Design Instability
Unstable production rates and quantities
Over-emphasis on Schedule
Inadequate Configuration Management
Inattention to Environmental Impacts
Insufficient process verification
Materials not well-defined or available
Changes in processes, materials, vendors…
Unavailability of Special Tools
Special test equipment |
|
|
Term
| Metrics to track producibility risk drivers |
|
Definition
HW
Engineering Changes
Scrap Rework Levels
Yield Rates for Testing
SW
Rate of error discovery
Rate of Requirements change
Rate of revision changes |
|
|
Term
| Current DoD emphasis on Production Quality? |
|
Definition
| Reduce Variability in the final product |
|
|
Term
| Most prevalent Industry Standard Quality System endorsed by DoD? |
|
Definition
|
|
Term
| What is the concept of JIT? |
|
Definition
Just In Time
A “pull” system, driven by actual demand. The goal is to produce or provide one part JIT for the next operation. Reduces stock inventories, but leaves no room for schedule error. As much a managerial philosophy as it is an inventory system. |
|
|
Term
| What does JIT strive to eliminate? |
|
Definition
| Waste of Equipment, Material, Parts, Space, Workers Time. |
|
|
Term
| What is Lean Manufacturing? |
|
Definition
Includes the complete process of manufacturing a product and can include things like:
Customer Relations
Supplier Networks
Distribution |
|
|
Term
| What are the Principles of Lean Manufacturing? |
|
Definition
Low waiting time
Reduced process times
Low Inventory
Scheduling based on customer pull instead of push system
Batch to Flow |
|
|
Term
| What does Lean Manufacturing Focus on? |
|
Definition
| Eliminating all waste in the manufacturing process |
|
|
Term
| What is the most critical Acquisition Logistics decision that a PM must make? |
|
Definition
| Maintenance & Support concept for the system lifetime |
|
|
Term
| Logistic Support Elements |
|
Definition
Design Interface
Maintenance Planning
Manpower and Personnel
Supply Support
Training and Training Devices
Technical Data
Support Equipment
Computer Resources
Facilities
Packaging, Handling Storage, & Transportation |
|
|
Term
| Metrics that can be given to designers to ensure proper logistical supportability |
|
Definition
Reliability (MTBF, MTTF)
Availability (MTBM)
Maintainability (MTTR) |
|
|
Term
| PM's financial responsibilities for repair parts during the system's life cycle? |
|
Definition
| Initial Provisioning - 1 year |
|
|
Term
| Which support element has the longest lead time? |
|
Definition
|
|
Term
| Which support element is the most critical? |
|
Definition
|
|
Term
| Performance Based Logistics |
|
Definition
Preferred (required for new) sustainment strategy
Purchase of support as an integrated, affordable performance-based package
PBL meets output performance goals for a weapon system through
Clear lines of responsibility
Incentives for meeting/exceeding logistics performance goals
Buying availability and reliability, NOT parts
Managing Suppliers, not supplies |
|
|
Term
|
Definition
| Activities Conducted to ensure supportability implications are considered early and throughout the acquisition process |
|
|
Term
| Acquisition Logistics in 4 bullets |
|
Definition
Influence design for supportability
Design the support system
Concurrently deploy the support system
Maintain readiness and improve affordability |
|
|
Term
| Why is acquisition logistics so important |
|
Definition
| Because 72% of a programs TOC resides in O&S costs. Acquisition Logistics anticipates and attempts to reduce these costs. |
|
|
Term
|
Definition
Reliability
Availability
Maintainability |
|
|
Term
Logistics planning needs to begin early in the SEP.
(T/F) |
|
Definition
|
|
Term
|
Definition
|
|
Term
|
Definition
| Army Test & Evaluation Command |
|
|
Term
|
Definition
|
|
Term
| What is the overall status? |
|
Definition
|
|
Term
| What are the Program Trends? How efficient is the Contractor? |
|
Definition
CPI < 1 inefficient cost
SPI < 1 inefficient schedule |
|
|
Term
| If you are 20% into the program and your CPI is .91 or lower, what will likely happen? |
|
Definition
| Your program will most likely not recover |
|
|
