Full report

Original page 1

NASA/TM2016-219273 A Tool for Measuring NASA’s Aeronautics Research Progress toward Planned Strategic Community Outcomes Farhad Tahmasebi and Robert Pearce NASA Headquarters, Washington, DC May 2016

Original page 2

NASA STI Program . . . in Profile Since its founding, NASA has been dedicated to the advancement of aeronautics and space science. The NASA scientific and technical information (STI) program plays a key part in helping NASA maintain this important role. The NASA STI program operates under the auspices of the Agency Chief Information Officer. It collects, organizes, provides for archiving, and disseminates NASA’s STI. The NASA STI program provides access to the NTRS Registered and its public interface, the NASA Technical Reports Server, thus providing one of the largest collections of aeronautical and space science STI in the world. Results are published in both non-NASA channels and by NASA in the NASA STI Report Series, which includes the following report types:  TECHNICAL PUBLICATION. Reports of completed research or a major significant phase of research that present the results of NASA Programs and include extensive data or theoretical analysis. Includes compilations of significant scientific and technical data and information deemed to be of continuing reference value. NASA counter-part of peerreviewed formal professional papers but has less stringent limitations on manuscript length and extent of graphic presentations.  TECHNICAL MEMORANDUM. Scientific and technical findings that are preliminary or of specialized interest, e.g., quick release reports, working papers, and bibliographies that contain minimal annotation. Does not contain extensive analysis.  CONTRACTOR REPORT. Scientific and technical findings by NASA-sponsored contractors and grantees.  CONFERENCE PUBLICATION. Collected papers from scientific and technical conferences, symposia, seminars, or other meetings sponsored or co-sponsored by NASA.  SPECIAL PUBLICATION. Scientific, technical, or historical information from NASA programs, projects, and missions, often concerned with subjects having substantial public interest.  TECHNICAL TRANSLATION. English-language translations of foreign scientific and technical material pertinent to NASA’s mission. Specialized services also include organizing and publishing research results, distributing specialized research announcements and feeds, providing information desk and personal search support, and enabling data exchange services. For more information about the NASA STI program, see the following:  Access the NASA STI program home page at http://www.sti.nasa.gov  E-mail your question to help@sti.nasa.gov  Phone the NASA STI Information Desk at 757-864-9658  Write to: NASA STI Information Desk Mail Stop 148 NASA Langley Research Center Hampton, VA 23681-2199

Original page 3

NASA/TM2016-219273 A Tool for Measuring NASA’s Aeronautics Research Progress toward Planned Strategic Community Outcomes Farhad Tahmasebi and Robert Pearce NASA Headquarters, Washington, DC Available from: NASA STI Program / Mail Stop 148 NASA Langley Research Center Hampton, VA 23681-2199 Fax: 757-864-6500

Original page 4

The use of trademarks or names of manufacturers in the report is for accurate reporting and does not constitute an official endorsement, either expressed or implied, of such products or manufacturers by the National Aeronautics and Space Administration. Available from: NASA Center for AeroSpace Information 7115 Standard Drive Hanover, MD 21076-1320 443-757-5802

Original page 5

Table of Contents Abstract ........................................................................................................................................... 1 1.0 Introduction.......................................................................................................................... 1 1.1 ARMD’s Strategic Planning Process ................................................................................... 1 1.2 Implementation of ARMD’s Research Strategy .................................................................. 2 2.0 Evaluation of Progress toward Planned Strategic Outcomes............................................... 4 3.0 Description of the Portfolio Analysis Tool for Measuring Progress toward Outcomes ...... 6 4.0 References.......................................................................................................................... 12 List of Figures Figure 1. Hierarchy for ARMD’s aeronautics research planning. ...............................................3 Figure 2. Sample TC progress indicator (based on TRL, confidence assessments, and technology transfer)......................................................................................................4 Figure 3. The ColorValueTrendSymbol function........................................................................7 Figure 4. Sample chart displaying progress toward an Outcome over a desired time period......9 Figure 5. Sample chart displaying progress toward Outcomes for a specific year....................10 Figure 6. Sample chart displaying progress toward Outcomes for a specific year using clustered data. .............................................................................................................10 Figure 7. “Progress toward Outcomes” plot for 2014 (no available trend data)........................11 List of Tables Table 1. Current Aeronautics research strategic community Outcomes (being revised). ..........5 Table 2. Colors used to evaluate progress toward strategic Outcomes. .....................................6 Table 3. Most common inputs and outputs of the ColorValueTrendSymbol function. .............8 Table 4. Notional examples of “Progress toward Outcomes” criteria ratings............................9 Table 5. “Progress toward Outcomes” ratings for 2014 (no available trend data). ..................11 iii

Original page 6

Acronyms ARMD Aeronautics Research Mission Directorate NACA National Advisory Committee for Aeronautics NAS National Airspace System NASA National Aeronautics and Space Administration PAMO Portfolio Analysis and Management Office R&D Research and Development TC Technical Challenge TRL Technology Readiness Level UAS Unmanned Aircraft Systems iv

Original page 7

Abstract Description of a tool for portfolio analysis of NASA’s Aeronautics research progress toward planned community strategic Outcomes is presented. For efficiency and speed, the tool takes advantage of a function developed in Excel’s Visual Basic for Applications. The strategic planning process for determining the community Outcomes is also briefly discussed. Stakeholder buy-in, partnership performance, progress of supporting Technical Challenges, and enablement forecast are used as the criteria for evaluating progress toward Outcomes. A few illustrative examples of using the tool are also presented. 1.0 Introduction The year 2015 marks the 100th anniversary of the founding of NASA’s predecessor, the National Advisory Committee for Aeronautics (NACA). Since that seminal event, aeronautics research has expanded from the fundamentals of flight to hypersonic air vehicles, from static performance of airfoils to behavior of complex human-machine systems, and from wood-and-canvas structures to adaptive shape-changing materials. NASA has a history of undertaking research and development (R&D) efforts that are outside the scale, risk, and payback criteria that govern commercial investments, with the purpose of proactively transitioning the research findings to the aviation community. NASA’s Aeronautics Research Mission Directorate (ARMD) has delivered results producing substantial benefits for air transportation in the customary focus areas of fundamental aeronautics, air traffic management, and aviation safety. These results have, transformed aviation to the benefit of the national economy, travelers and shippers, as well as the global environment. 1.1 ARMD’s Strategic Planning Process In defining NASA’s response to future aviation needs, the 2014 NASA Strategic Plan (NASA 2014) sets forth a bold objective for aeronautics research in Strategic Objective 2.1: “Enable a revolutionary transformation for safe and sustainable U.S. and global aviation by advancing aeronautics research.” ARMD is responding with an equally bold vision embodied in its Strategic Implementation Plan (ARMD 2015) for its research activities. Based on analysis of global trends, ARMD has identified the following three overarching drivers, referred to as Mega-Drivers, which will in large part shape the needs of aeronautical research in the coming years:  Mega-Driver 1, Global Growth in Demand for High Speed Mobility: Rapid growth in traditional measures of global demand for mobility — measures such as economic development and urbanization  Mega-Driver 2, Global Climate Change, Sustainability, and Energy Use: Energy and climate issues that are likely to create severe challenges in maintaining affordability and sustainability 1

Original page 8

 Mega-Driver 3, Technology Convergence: Emerging revolutions in automation, information, and communication technologies, which will eventually combine to produce transformative aeronautical capabilities. The following six Strategic Thrusts represent ARMD’s response to the Mega-Drivers as they affect aviation:  Thrust 1: Safe, Efficient Growth in Global Operations  Thrust 2: Innovation in Commercial Supersonic Aircraft  Thrust 3: Ultra-Efficient Commercial Vehicles  Thrust 4: Transition to Low-Carbon Propulsion  Thrust 5: Real-Time System-Wide Safety Assurance  Thrust 6: Assured Autonomy for Aviation Transformation Taken together, these Strategic Thrusts constitute a vision for the future of aviation. ARMD’s strategic planning addresses research needs associated with these Strategic Thrusts through a hierarchy of Outcomes, Research Themes, and Technical Challenges (TCs). Outcomes defined in terms of three time¬frames — near-term (2015 to 2025), mid-term (2025 to 2035), and far-term (>2035) — signify the advances required to address each Strategic Thrust. Research Themes, which support the Outcomes, represent major areas of research necessary to enable the Outcomes consistent with ARMD’s roles and capabilities. Each Research Theme includes one or more TCs, which are funded activi¬ties with specific objectives. These TCs serve as the basis for planning research activi¬ties and measuring performance. Figure 1 depicts ARMD’s research planning hierarchy. In addition to portfolio analyses and inputs from subject matter experts and senior stakeholders, ARMD’s planning incorporates mechanisms for dialogue with the aviation community. To help identify important research areas and challenges of the future, ARMD has frequently engaged the aviation community to understand what its stakeholders believe are priority research areas. Regular discussions have engaged domestic and international partners and experts from industry, academia, and government. Interactions have included regular reviews of ongoing research by federal advisory committees and dialogue sessions with the National Research Council’s Aeronautics Research and Technology Roundtable. 1.2 Implementation of ARMD’s Research Strategy Research Themes comprise major areas of research aligned to specific Outcomes. Unlike Outcomes and Strategic Thrusts, which represent aviation community goals that will be achieved through the community’s joint efforts, the Research Themes are more focused. They define the roles that ARMD takes in conducting research that ultimately supports the Strategic Thrusts and Outcomes. The Research Themes are pursued through programs and project organizations within the programs, and progress is reviewed on an annual basis. The research program offices define the TCs within each Research Theme and delegate them to the project organizations for execution. The project offices continually monitor their portfolios and develop plans that document the relevant TCs and how they will be addressed, as well as measures of progress and other programmatic information. 2

Original page 9

Figure 1. Hierarchy for ARMD’s aeronautics research planning. A generic example of a TC progress indicator; which is based on Technology Readiness Level (TRL), confidence level of reaching goals, and technology transfer; is shown in Figure 2. This example is generated based on work of the ARMD’s “High Speed” and “Unmanned Aircraft Systems (UAS) in the National Airspace System (NAS)” projects. TRLs specified in Figure 2 are defined in a white paper from NASA headquarters (Mankins 1995). Levels 1, 2, and 3 in Figure 2 refer to program, project, and sub-project tiers. Detailed description of these levels/tiers can be found in the NASA Systems Engineering Handbook (NASA 2007, 8-11). 3

Original page 10

Figure 2. Sample TC progress indicator (based on TRL, confidence assessments, and technology transfer). Current Outcomes for ARMD’s six Strategic Thrusts are listed in Table 1. Some of these Outcomes are being revised and will not be used in describing the “Progress toward Outcomes” tool. Due to space constraints, enabling Research Themes and TCs are not listed in this paper. ARMD programs and their corresponding projects list their Research Themes and the TCs for which they are responsible at www.aeronautics.nasa.gov/programs.htm. 2.0 Evaluation of Progress toward Planned Strategic Outcomes Starting with the December 2015 annual program reviews, and using traffic light colors (i.e., green, yellow, and red); ARMD’s four programs (Airspace Operations and Safety Program, Advanced Air Vehicles Program, Integrated Aviation Systems Program, and Transformative Aeronautics Concepts Program) report on the status of the following four criteria for the Outcomes that they are enabling. 1. Stakeholder buy-in - Level of alignment with stakeholders (adopting/satisfied/same vision) and relevant policies. 2. Partnership performance - Progress level of technical partners (emphasizing the success level of partnership plans), availability of flight assets and test beds. 3. Progress of supporting TCs - Progress level of internal TCs that contribute to ARMD Strategic Outcomes. 4

Original page 11

1. Enablement Forecast – Overall rating based on detailed risk/opportunity analysis, progress of relevant/influencing internal TCs from other Strategic Thrusts (not the ones in item 3 above), and progress of relevant/influencing external technology developments that ARMD is watching or following. Table 1. Current Aeronautics research strategic community Outcomes (being revised). 2015 2025 Outcomes Thrust Near Term 2035 Mid Term Far Term Improved NextGen Full NextGen Integrated Beyond NextGen Operational Performance in Terminal, En Route, Dynamic Fully Individual Domains, with Surface, and Arrivals/ Autonomous 1 Some Integration Between Departures Operations to Trajectory Services Domains Realize Trajectory-based Operations Supersonic Overland Introduction of Affordable, (Outcomes beyond Certification Standard Low-boom, Low-noise, and 2035 will depend on 2 Based on Acceptable Low-emission Supersonic market needs and Sonic Boom Noise Transports technology solutions) Achievement of Community Goals for Improved Vehicle Efficiency and Environmental Performance 3 Achieve Community Goals for Improved Vertical Lift Vehicle Efficiency & Environmental Performance in 2035 (2025 and beyond) Introduction of Low-carbon Initial Introduction of Introduction of Fuels for Conventional Alternative Propulsion Alternative 4 Engines and Exploration of Systems Propulsion Systems Alternative Propulsion Systems to Aircraft of All Sizes Introduction of Advanced An Integrated Safety Automated Safety Safety Assurance Tools Assurance System Enabling Assurance Integrated Continuous System-wide with Real-time 5 Safety Monitoring Operations Enabling a Self-protecting Aviation System Initial Autonomy Human-machine Teaming Ability to Fully Applications in Key Applications Certify and Trust 6 Autonomous Systems for NAS Operations Justification and evidence will be provided by the programs for their traffic light ratings of the above four criteria. TC progress indicator shown in Figure 2 is an example of such a justification 5

Original page 12

and evidence. When appropriate, the programs also report on positive, negative, or neutral trends (projecting forward based on recent past data) for the above-mentioned self-assessments. ARMD’s Portfolio Analysis and Management Office (PAMO) reviews the programs’ selfassessments, asks relevant questions, and if necessary adjusts evaluation colors for the four progress criteria (described above). PAMO analysts may also use Orange (Red/Yellow) and Chartreuse (Yellow/Green) colors in their adjustments of self-assessments by the programs. The colors used in this portfolio analysis process as well as their definitions are described in Table 2. Table 2. Colors used to evaluate progress toward strategic Outcomes. Color Red Definition Lowest quintile possible (bad) nd Orange - Red/Yellow 2 lowest quintile possible (poor) rd Yellow 3 highest quintile possible (average) nd Chartreuse - Yellow/Green 2 highest quintile possible (good) Green Highest quintile possible (very good) 3.0 Description of the Portfolio Analysis Tool for Measuring Progress toward Outcomes To analyze the evaluation results (described in Section 2), the following steps need to be taken: 1. Bring the green, chartreuse, yellow, orange, and red evaluation colors and corresponding ® trends into Excel (Microsoft 2013). a. Positive, negative, and neutral trends are added to the same cells containing the color ratings. 2. Use the ColorValueTrendSymbol function developed in Visual Basic for Applications to perform the following automated operations. a. Excel’s Color Index properties are used to assign an appropriate value between one and three to the evaluation colors. b. Assign ↗, ↘, and → symbols to the positive, negative, and neutral trend ratings, respectively. If no input trend rating is provided (due to lack of appropriate data or any other reason), the function does not generate an output trend symbol. c. Combine assigned color values (step a above) and trend symbols (step b above) to generate the function outputs. Function ColorValueTrendSymbol is displayed in Figure 3. Most common inputs and outputs of this function are shown in Table 3. It should be noted that descriptive comments in Figure 3 are shown in green font. 1. Store the color ratings and trend symbols for the progress criteria in Excel. For each Outcome, criteria ratings for more than one year can be included. Table 4 shows notional examples of ratings for three Outcomes over the 2015 to 2017 time period. 2. Filter the stored results to graphically analyze progress toward a given Outcome over a desired time period. For example, based on the entries in Table 4, Figure 4 shows progress toward Outcome 1 over the 2015 to 2017 time period. 6

Original page 13

1. Filter the stored ratings to plot progress toward Outcomes for a particular year. For example, based on the entries in Table 4, Figure 5 displays progress toward Outcomes for 2015. Figure 3. The ColorValueTrendSymbol function. The stored data can be filtered in any other desired way to plot results for gaining more insight into progress toward the ARMD planned Outcomes. Figures 4 and 5 stack the progress criteria for each Outcome. Another way to gain insight into the data is to cluster the criteria together, as shown in Figure 6. If desired, for each Outcome, weighted sum (or average) of the progress criteria can easily be calculated in Excel. Stacked charts in Figures 4 and 5 provide similar information. However, one has to be careful not to oversimplify issues by just measuring “Progress toward Outcomes” using sum of the criteria values. 7

Original page 14

Table 3. Most common inputs and outputs of the ColorValueTrendSymbol function. Inputs Outputs Color and Trend Color Value-Trend Symbol Neutral Negative Positive Neutral Negative Positive Neutral Negative Positive Neutral Negative Positive Neutral 3→ 3↘ 2.5↗ 2.5→ 2.5↘ 2↗ 2→ 2↘ 1.5↗ 1.5→ 1.5↘ 1↗ 1→ As mentioned earlier, rating for the “Progress toward Outcomes” criteria by the ARMD programs starts with the December 2015 annual reviews. For 2014, the PAMO analysts rated the progress criteria for the Outcomes that are not being revised. The results are shown in Table 5. Note that it was not possible to determine any trends for the 2014 ratings because no prior data was available. Therefore, the outputs of the ColorValueTrendSymbol function that are shown in Table 5 contain no trend symbols. Progress toward the Outcomes listed in the leftmost column of Table 5 is displayed in Figure 7. The ratings in Table 5 are based on the 2014 annual review presentations. Due to space constraints, justifications for all of Table 5 ratings are not presented in this paper. As an example, here we just present justifications for one of the Outcomes (Introduction of Affordable, Low-boom, Low-noise and Low-emission Supersonic Transports - 2025-2035). Stakeholder buy-In is rated yellow/2 because there is only mixed funding support for ARMD’s research in enabling overland supersonic transports. Some of the stakeholders believe that only a small portion of the population with very high income will benefit from introduction of supersonic transports. Partnership performance is rated chartreuse/2.5 because industry focus is mainly on smaller supersonic business jets. Progress of TCs is rated green/3 since there are no significant issues in generating ARMD’s outputs for this Outcome. Finally, Enablement forecast is rated orange/1.5 due to the relatively high risk of delays in introduction of a certification standard for overland supersonic flights. It should be noted that the PAMO ratings shown in Table 5 and plotted in Figure 7 have not been formally reviewed and validated by all members of the ARMD leadership team. These ratings are presented here for illustration purposes only. “Progress toward Outcomes” results obtained using the tool described in this article will be used in the following ways: 1) as a mean for facilitating positive dialogue between ARMD leadership, program managers, and PAMO analysts; 2) as an indicator of where further management attention 8

Original page 15

needs to be focused to improve future strategic performance; and 3) as a utility that can be used in adjusting ARMD’s research portfolio. Starting with the December 2015 program reviews, and going forward; the tool will be used as the depository for all of the ARMD “Progress to Outcomes” ratings. Additional features will be added to the tool, as necessary. Table 4. Notional examples of “Progress toward Outcomes” criteria ratings. Figure 4. Sample chart displaying progress toward an Outcome over a desired time period. 9

Original page 16

Figure 5. Sample chart displaying progress toward Outcomes for a specific year. Figure 6. Sample chart displaying progress toward Outcomes for a specific year using clustered data. 10

Original page 17

Table 5. “Progress toward Outcomes” ratings for 2014 (no available trend data). Figure 7. “Progress toward Outcomes” plot for 2014 (no available trend data). 11

Original page 18

4.0 References National Aeronautics and Space Administration (NASA) 2014. “NASA Strategic Plan.” http://www.nasa.gov/sites/default/files/files/FY2014_NASA_SP_508c.pdf Aeronautics Research Mission Directorate (ARMD) 2015. “Strategic Implementation Plan.” http://www.aeronautics.nasa.gov/pdf/armd-strategic-implementation-plan.pdf Mankins, J. C. 1995. “Technology Readiness Levels.” White paper, NASA Office of Space Access and Technology. http:/www.hq.nasa.gov/office/codeq/trl/trl.pdf National Aeronautics and Space Administration (NASA) 2007. “NASA Systems Engineering Handbook.” NASA/SP-2007-6105 Rev1. NASA Headquarters. http://www.acq.osd.mil/se/docs/NASA-SP-2007-6105-Rev-1-Final-31Dec2007.pdf ® Microsoft . 2013. “Excel 2013 Quick Start Guide.” https://support.office.com/enus/Article/Excel-2013-Quick-Start-Guide-4337fe93-0690-47cb-89a1-8424d5fdd886 12