Full report

Original page 1

NASA/TM-2009-215745 NESC-RP-06-01/05-171-E External Tank Liquid Hydrogen (LH2) Prepress Regression Analysis Independent Review Technical Consultation Report Vickie S. Parsons/NESC Langley Research Center, Hampton, Virginia May 2009

Original page 2

NASA STI Program . . . in Profile Since its founding, NASA has been dedicated to the advancement of aeronautics and space science. The NASA scientific and technical information (STI) program plays a key part in helping NASA maintain this important role. The NASA STI program operates under the auspices of the Agency Chief Information Officer. It collects, organizes, provides for archiving, and disseminates NASA’s STI. The NASA STI program provides access to the NASA Aeronautics and Space Database and its public interface, the NASA Technical Report Server, thus providing one of the largest collections of aeronautical and space science STI in the world. Results are published in both non-NASA channels and by NASA in the NASA STI Report Series, which includes the following report types: • TECHNICAL PUBLICATION. Reports of completed research or a major significant phase of research that present the results of NASA programs and include extensive data or theoretical analysis. Includes compilations of significant scientific and technical data and information deemed to be of continuing reference value. NASA counterpart of peerreviewed formal professional papers, but having less stringent limitations on manuscript length and extent of graphic presentations. • TECHNICAL MEMORANDUM. Scientific and technical findings that are preliminary or of specialized interest, e.g., quick release reports, working papers, and bibliographies that contain minimal annotation. Does not contain extensive analysis. • CONTRACTOR REPORT. Scientific and technical findings by NASA-sponsored contractors and grantees. • CONFERENCE PUBLICATION. Collected papers from scientific and technical conferences, symposia, seminars, or other meetings sponsored or co-sponsored by NASA. • SPECIAL PUBLICATION. Scientific, technical, or historical information from NASA programs, projects, and missions, often concerned with subjects having substantial public interest. • TECHNICAL TRANSLATION. Englishlanguage translations of foreign scientific and technical material pertinent to NASA’s mission. Specialized services also include creating custom thesauri, building customized databases, and organizing and publishing research results. For more information about the NASA STI program, see the following: • Access the NASA STI program home page at http://www.sti.nasa.gov • E-mail your question via the Internet to help@sti.nasa.gov • Fax your question to the NASA STI Help Desk at 443-757-5803 • Phone the NASA STI Help Desk at 443-757-5802 • Write to: NASA STI Help Desk NASA Center for AeroSpace Information 7115 Standard Drive Hanover, MD 21076-1320

Original page 3

NASA/TM-2009-215745 NESC-RP-06-01/05-171-E External Tank Liquid Hydrogen (LH2) Prepress Regression Analysis Independent Review Technical Consultation Report Vickie S. Parsons/NESC Langley Research Center, Hampton, Virginia National Aeronautics and Space Administration Langley Research Center Hampton, Virginia 23681-2199 May 2009

Original page 4

The use of trademarks or names of manufacturers in the report is for accurate reporting and does not constitute an official endorsement, either expressed or implied, of such products or manufacturers by the National Aeronautics and Space Administration. Available from: NASA Center for AeroSpace Information 7115 Standard Drive Hanover, MD 21076-1320 443-757-5802

Original page 5

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Page #: Title: External Tank Liquid Hydrogen (LH2) Prepress 1 of 34 Regression Analysis Independent Review Technical Consultation Report External Tank Liquid Hydrogen (LH2) Prepress Regression Analysis Independent Review Technical Consultation Report Prepared by Vickie S. Parsons, PhD NASA Engineering and Safety Center (NESC) Systems Engineering Office January 12, 2006 NESC Request No.: 05-171-E

Original page 6

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Page #: Title: External Tank Liquid Hydrogen (LH2) Prepress 2 of 34 Regression Analysis Independent Review Technical Consultation Report TABLE OF CONTENTS VOLUME I: Technical Consultation Report 1.0 Authorization and Notification..................................................................................................... 4 2.0 Signature Page................................................................................................................................ 5 3.0 Team Members, Ex Officio Members, and Consultants ............................................................ 6 4.0 Executive Summary ....................................................................................................................... 7 5.0 Consultation Plan........................................................................................................................... 7 6.0 Description of the Problem, Proposed Solutions, and Risk Assessment ................................... 8 7.0 Data Analysis.................................................................................................................................. 8 8.0 Findings, Root Causes, Observations and Recommendations................................................. 22 8.1 Findings ............................................................................................................................ 22 8.2 Recommendations............................................................................................................. 22 9.0 Lessons Learned........................................................................................................................... 22 10.0 Definition of Terms...................................................................................................................... 22 11.0 Minority Report (Dissenting Opinions) ..................................................................................... 23 Volume II: Appendices Appendix A. NESC ITA/I Request Form (NESC-PR-003-FM-01) .......................................................... 24 Appendix B. Key for Interpreting Box Plots and Scatterplots................................................................... 30 Appendix C. List of Acronyms .................................................................................................................. 33 List of Figures 7.0-1 Comparison of Elevated Cycle Counts Resulting from Incorrect Diffuser Material on Three STS-114 Tanking Tests with Cycle Counts for the Remaining Data Records.................... 11 7.0-2 Comparison of Elevated Cycle Counts Resulting from Undersized Orifice on MLP-2 with Cycle Counts for the Remaining Data Records ............................................................................. 12 7.0-3 Scatterplot of the Fit Versus Actual Cycle Count for the Regression on all Variables ................. 16 7.0-4 Scatterplot of the Fit Versus Scaled Actual Cycle Count for the Regression on all Variables...... 19 7.0-5 Comparison of Scaled Output and Predictor Variables for the Clark and Kreen Comparison ......20 List of Tables 7.0-1 Data Set for the Review ................................................................................................................. 10 7.0-2 Correlation Coefficients for the Potential Independent Predictor Variables.................................. 13 NESC Request No.: 05-171-E

Original page 7

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Page #: Title: External Tank Liquid Hydrogen (LH2) Prepress 3 of 34 Regression Analysis Independent Review Technical Consultation Report 7.0-3 Stepwise Regression on all Data using P-Value to Determine Significance of Entering Variables ........................................................................................................................................ 14 7.0- 4 Stepwise Regression on all Data using F-ratio Value to Determine Significance of Entering Variables ........................................................................................................................................ 15 7.0-5 Stepwise Regression Corresponding to Clark and Krenn.............................................................. 17 7.0-6 Stepwise Regression Corresponding to Clark and Krenn using Scaled Values............................. 18 7.0-7 Stepwise Regression Based on Subset of Data that is Relevant to Current and Future Conditions......................................................................................................................................21 NESC Request No.: 05-171-E

Original page 8

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Page #: Title: External Tank Liquid Hydrogen (LH2) Prepress 4 of 34 Regression Analysis Independent Review Technical Consultation Report Volume I: Technical Consultation Report 1.0 Authorization and Notification The request to conduct an independent review of regression models, developed for determining the expected Launch Commit Criteria (LCC) External Tank (ET)-04 cycle count for the Space Shuttle ET tanking process, was submitted to the NESC on September 20, 2005. NESC acceptance of this task was approved in an out-of-board action on October 7, 2005. NESC Request No.: 05-171-E

Original page 9

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Page #: Title: External Tank Liquid Hydrogen (LH2) Prepress 5 of 34 Regression Analysis Independent Review Technical Consultation Report 2.0 Signature Page Consultation Team Members Vickie S. Parsons, NESC Lead K. Preston White, University of Virginia NESC Request No.: 05-171-E

Original page 10

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Title: Page #: External Tank Liquid Hydrogen (LH2) Prepress 6 of 34 Regression Analysis Independent Review Technical Consultation Report 3.0 Team Members, Ex Officio Members, and Consultants Vickie Parsons NESC Systems Engineer K. Preston White Statistical Consultant Bernie Mylnczak NESC Program Analyst Erin Moran Technical Writer NESC Request No.: 05-171-E NASA LaRC University of Virginia MTSO, LaRC Swales Aerospace, LaRC

Original page 11

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Page #: Title: External Tank Liquid Hydrogen (LH2) Prepress 7 of 34 Regression Analysis Independent Review Technical Consultation Report 4.0 Executive Summary The NESC team performed an independent review of regression models documented in Prepress Regression Analysis, Tom Clark and Angela Krenn, 10/27/05. These regressions were developed for determining the expected cycle count described in LCC ET-04. This independent review was limited to regression models developed based on variable measurements that are available prior to launch and corresponding to the development of “regression #1”, pp 4-7, in Prepress Regression Analysis. 5.0 Consultation Plan This consultation consisted of a peer review by statistical experts of the proposed regression models provided in the Prepress Regression Analysis. Both primary members of the NESC team reviewed the following documents: • Prepress Regression Analysis, Tom Clark & Angela Krenn (10/27/05) • STS-114 S0007#2 LCC ET-04 LH2 Prepress Cycle Count, Diane Stees (7/29/05) • STS-114 S0037#2 LH2 Prepress Test Cycle Count, Diane Stees (5/26/05) • Define Mechanization of GH2 Prepress Cycle Count LCC ET-04, Diane Stees (2/12/03) • Launch Commit Criteria (LCC) ET-04 • Launch Commit Criteria (LCC) ET-05 • LH2 Tank Prepress Overview (LCCs ET-04 & ET-05), B. Piekarski (10/27/05) • Flight Pressurization System ET-04 Assessment LH2 Vent Vale Hazard Review, Kathryn Kynard & Jonathan Looser (10/24/05) • Transient Analysis of LH2 Tank Prepress Helium Mass Flow, Adam Baran (10/27/05) • Lockheed Martin Pressurization Program, B. Piekarski (10/27/05) • Flight Pressurization System ET-04 Assessment ER21 Pressurization Model Sensitivity ER21, Tim Olive (10/24/05) A telephone conference was also conducted between the NESC team and Space Shuttle Program (SSP) members knowledgeable of the ET tanking process, LCC ET-04, LCC ET-05, and the cycle count process. NESC Request No.: 05-171-E

Original page 12

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Page #: Title: External Tank Liquid Hydrogen (LH2) Prepress 8 of 34 Regression Analysis Independent Review Technical Consultation Report 6.0 Description of the Problem, Proposed Solutions, and Risk Assessment The SSP uses LCC ET-04 as an indirect method to monitor ET LH2 vent valve leakage. LCC ET-04 monitors the LH2 tank pre-press system by counting GHe press valve command cycles. A larger than predicted command-cycle count is an indicator of potentially dangerous leakage and can lead to aborting the launch. The most recent valve replacements appear to have faster cycle times than those for the valves previously employed. Because the shorter pulse time leads to a lower helium make-up input per command cycle, a higher cycle count is required to maintain the ET pressure. The STS-114 ET tank loading observed 11 cycles which constituted 2 greater than predicted for the prior valves. This highlighted the need for revised modeling to better predict expected cycles with the current valves. The SSP’s intent is to use modeling that is currently under development to determine appropriate modifications to the applicable LCCs. The intended application of these regressions is to provide a simple tool to validate the NASA-accredited, analytical (first principles) model currently under development, against empirical data available since the transition to half-second valve pulses. Additional analyses have been performed to verify that ground valve cycle timing does approximate the average cycle count. 7.0 Data Analysis The data set for the review is provided in Table 7.0-1. The set comprises a total of twenty-seven data records, including records for seventeen flights (STS-88, -92, -93, -95, and -114), as well as one tanking test for STS-91 and three tanking tests for STS-114. The dependent (output) variable in each data record is: ActLCC: Actual (integer) number of cycles experienced plus the amount of time (as a fraction) between last and subsequent cycle at which point the LCC expires (T-43s). This combination variable was chosen by the SSP team rather than the integer number of cycles because a continuous variable is better suited to regression analyses. NESC Request No.: 05-171-E

Original page 13

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Page #: Title: External Tank Liquid Hydrogen (LH2) Prepress 9 of 34 Regression Analysis Independent Review Technical Consultation Report The independent (predictor) variables are: Mission: Mission identifying number. Orbiter: Orbiter number. MLP: Mobile launch pad number. Diffuser: Indicator variable for the diffuser material (0 indicating Single Dutch Twill and 1 indicating Double Dutch Twill). UllagePres: The trigger point software to control the tank at flight pressure (pulse fired when 2 of 3 ullage pressure transducers fall below the trigger point). SupplyPres: The GHe pressure coming into the panel and remains fairly constant throughout launch operations. SFOutPres: Peak (steady state) panel outlet pressure during initial pressurization in Slow Fill (to 5 percent full). TCOutPres: Peak panel outlet pressure during initial pressurization in Terminal Count. CycleTime: Average prepress s/o valve cycle time in Terminal Count (LCC prepress cycles only). Similar data can be obtained prior to loading to predict an LCC cycle count. PeakPress: Average peak panel outlet pressure during Terminal Count (LCC prepress cycles only). Temp: Ambient outside temperature during Slow Fill (to 5 percent full). Prepres: Peak Orbiter prepress line pressure during initial pressurization in Slow Fill (to 5 percent full). Other variables that were provided in the data set were discounted because they were not known early enough in the tanking process. For analysis, the three category variables Orbiter, MLP, and Diffuser were translated to 0-1 indicator (dummy) variables. NESC Request No.: 05-171-E

Original page 14

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Page #: Title: External Tank Liquid Hydrogen (LH2) Prepress 10 of 34 Regression Analysis Independent Review Technical Consultation Report Table 7.0-1. Data Set for the Review MissionActLCCOrbiterMLP DiffuserUllagePresSupplyPresSFOutPresTCOutPresCycleTimePeakPresTemp PrePres 88 7.79 5 3 0 41.8 1960 1064 1072 0.515 808.0 74.0 208 92 9.13 3 3 0 41.8 1980 1080 1088 0.518 828.0 73.6 232 93 8.54 2 1 0 41.8 2000 1048 1032 0.499 816.0 86.0 200 95 8.55 3 2 0 41.8 2040 1024 1016 0.518 824.0 61.2 220 96 10.11 3 2 0 41.8 2040 1000 1000 0.496 760.0 75.6 216 97 7.70 5 1 0 41.8 2020 1088 1080 0.505 802.3 67.6 216 98 11.12 4 2 0 41.8 2060 1016 1016 0.469 720.7 60.4 196 99 7.05 5 3 0 41.8 1980 1080 1088 0.512 827.4 48.8 212 100 9.09 5 1 0 41.8 2000 1056 1048 0.504 794.7 51.6 208 101 8.16 4 1 0 41.8 1980 1080 1064 0.514 813.0 75.2 212 102 8.10 3 3 0 41.8 1940 1048 1048 0.503 791.0 48.4 228 103 10.44 3 2 0 41.8 2080 1024 1016 0.466 731.2 66.4 220 104 10.07 4 2 0 41.8 2020 1016 1016 0.489 747.2 77.6 192 105 8.50 3 3 0 41.8 1980 1104 1104 0.524 858.0 78.0 236 106 10.51 4 2 0 41.8 2060 1016 1016 0.502 762.4 79.6 196 107 10.26 2 1 0 47.0 1980 1048 1040 0.511 761.6 47.2 200 108 8.02 5 1 0 41.8 2060 1080 1072 0.515 802.0 72.8 212 109 9.11 2 2 0 41.8 2040 1016 1016 0.468 722.7 52.8 188 110 9.05 4 3 0 47.0 1960 1112 1112 0.522 853.3 69.6 216 111 9.44 5 1 0 47.0 2020 1064 1056 0.518 814.2 81.2 212 112 9.19 4 3 0 47.0 1920 1104 1104 0.526 858.7 72.8 216 113 11.08 5 2 0 47.0 2060 1056 1056 0.487 800.0 55.2 204 114 11.81 3 3 0 46.5 1960 1080 1080 0.449 718.6 78.0 232 114T1 13.52 3 1 1 46.5 2080 1080 1080 0.505 799.4 59.2 236 114T2 13.65 3 1 1 46.5 2040 1064 1056 0.509 792.6 68.8 232 114T2 14.15 3 1 1 46.5 2040 1056 0.531 809.7 91T 9.25 3 1 0 46.7 2040 1048 1040 0.492 808.0 71.6 232 All three tanking tests for STS-114 yielded large cycle counts resulting from a known anomaly caused by use of an out-of-specification material (Double Dutch Twill) for diffusers. This anomaly is illustrated in the box plot in Figure 7.0-1. Appendix B provides a key for interpreting box plots. NESC Request No.: 05-171-E

Original page 15

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Title: Page #: External Tank Liquid Hydrogen (LH2) Prepress 11 of 34 Regression Analysis Independent Review Technical Consultation Report Box Plot Comparison—STS-114 TT Cycle Counts Other / Tanking Tests STS-114 TT / Tanking Tests 0 2 4 6 8 10 12 14 16 Figure 7.0-1. Comparison of Elevated Cycle Counts Resulting from Incorrect Diffuser Material on Three STS-114 Tanking Tests with Cycle Counts for the Remaining Data Records Additionally, the set of eight launches using MLP-2 (STS-95, -96, -98, -103, -104, -106, -109, and -113) have larger cycle counts, resulting from a known anomaly caused by a restrictive upstream orifice. This anomaly is illustrated in the box plot in Figure 7.0-2 (where the anomalous STS-114 tanking tests have been removed from the data set). NESC Request No.: 05-171-E

Original page 16

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Title: Page #: External Tank Liquid Hydrogen (LH2) Prepress 12 of 34 Regression Analysis Independent Review Technical Consultation Report Box Plot Comparison--Cycle Counts by MLP 0 2 4 6 MLP3 / MLP Data MLP2 / MLP Data MLP1 / MLP Data 8 10 12 14 Figure 7.0-2. Comparison of Elevated Cycle Counts Resulting from Undersized Orifice on MLP-2 with Cycle Counts for the Remaining Data Records The causes of these anomalous cases have been corrected and are unlikely to be repeated. Meaningful regressions require that the independent variables be unrelated. To assess potential co-linearities in the predictor variables, cross-correlation coefficients were computed for each pair of predictor variables, as shown in Table 7.0-2. When both variables were interval level measurements, Pearson’s correlation coefficient was used; when either variable was categorical, Spearman’s correlation coefficient was used. Those values that were significant at .05 or better are shown in bold, indicating that those pairs of variables would not be good candidates within the same regression. NESC Request No.: 05-171-E

Original page 17

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Page #: Title: External Tank Liquid Hydrogen (LH2) Prepress 13 of 34 Regression Analysis Independent Review Technical Consultation Report Table 7.0-2. Correlation Coefficients for the Potential Independent Predictor Variables Mission Diffuser UllagePresMLP SupplyPresSFOutPresTCOutPresCycleTimePeakPressTemp. Mission Diffuser 0.545 UllagePres 0.606 0.290 MLP -0.118 -0.387 -0.139 SupplyPres 0.142 0.324 -0.063 -0.500 SFOutPres 0.271 0.146 0.368 0.178 -0.539 TCOutPres 0.248 0.122 0.337 0.301 -0.560 0.980 -0.313 0.516 0.478 CycleTime 0.097 0.204 0.110 0.060 PeakPres -0.146 -0.030 0.172 0.147 -0.418 0.704 0.679 0.852 Temp -0.071 -0.154 -0.029 0.035 -0.033 0.116 0.077 0.098 0.159 PrePres 0.085 0.407 0.273 0.163 -0.157 0.208 0.107 0.501 0.497 0.387 A stepwise linear regression was performed using all of the data records except STS-114 tanking test 2 (where data values were missing for three predictors). The results using p-values to determine the significance levels of the variable entering are shown in Table 7.0-3. The results using F-ratio values to determine the significance levels of the variable entering is shown in Table 7.0-4. The results are clearly very similar, with Diffuser, UllagePres, MLP=2 all significant predictors in both cases. In the first regression, CycleTime also is significant; in the second regression, PeakPres replaces CycleTime (in the final step) and Temp is also significant. • Diffuser variable dominates the regression. In essence, diffuser is an indicator for the known anomalies on the STS-114 tanking tests included in the data. • Similarly, MLP=2 is an indicator for the known anomalies caused by an undersized upstream orifice. In other words, the regression flags the anomalies, as it should. • PeakPres and CycleTime are known to be highly correlated and one or the other (but not both) will be included, if the associated data indicate significance, as these do. • Temp is included in the second regression and not the first, but is an order of magnitude less significant than any of the other included predictors. These results confirm that Clark and Krenn have included the appropriate, and only the most appropriate, predictors in their regression. NESC Request No.: 05-171-E

Original page 18

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Title: Page #: External Tank Liquid Hydrogen (LH2) Prepress 14 of 34 Regression Analysis Independent Review Technical Consultation Report Table 7.0-3. Stepwise Regression on all Data using P-Value to Determine Significance of Entering Variables Multiple R-Square Summary R 0.9285 0.8620 Degrees of Sum of ANOVA Table Freedom Squares Adjusted StErr of Durbin R-Square Estimate Watson 0.8358 0.671216686 2.2440 Mean of F-Ratio p-Value Squares Explained 4 59.12244675 14.78061169 32.8070 < 0.0001 Unexplained 21 9.461168632 0.45053184 Standard Coefficient Regression Table Error Constant 10.73676745 5.096139221 Diffuser 3.999890394 0.532004437 CycleTime -26.5983152 7.861564618 UllagePres 0.265240602 0.061726836 MLP = 2 1.077573737 0.347955906 Multiple R-Square Step Information R Diffuser 0.7108 0.5052 CycleTime 0.8458 0.7154 UllagePres 0.8939 0.7990 MLP = 2 0.9285 0.8620 NESC Request No.: 05-171-E Lower Upper t-Value p-Value Limit Limit 2.1068 0.0473 0.138765773 21.33476913 7.5185 < 0.0001 2.8935266 5.106254187 -3.3833 0.0028 -42.9473338 -10.2492966 4.2970 0.0003 0.136872618 0.393608586 3.0969 0.0055 0.353959817 1.801187657 Adjusted StErr of Enter or R-Square Estimate Exit 0.4846 1.189133891 Enter 0.6906 0.921254182 Enter 0.7716 0.791489885 Enter 0.8358 0.671216686 Enter

Original page 19

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Title: Page #: External Tank Liquid Hydrogen (LH2) Prepress 15 of 34 Regression Analysis Independent Review Technical Consultation Report Table 7.0- 4. Stepwise Regression on all Data using F-ratio Values to Determine Significance of Entering Variables Multiple Adjusted StErr of Durbin R-Square Summary R R-Square Estimate Watson 0.9367 0.8774 Degrees of Sum of ANOVA Table Freedom Squares 0.8467 0.648478308 2.2195 Mean of F-Ratio p-Value Squares Explained 5 61.44086087 12.28817217 34.4074 < 0.0001 Unexplained 20 7.142754517 0.357137726 Standard Coefficient Regression Table Error Constant 6.847746674 3.728120462 Diffuser 3.817194075 0.477827428 UllagePres 0.298677801 0.05454566 MLP = 2 0.93811152 0.322786866 Temp 0.021727083 0.010771308 PeakPres -0.01552637 0.003583316 Lower Upper t-Value p-Value Limit Limit 1.8368 0.0812 -0.928976336 14.62446968 7.9886 < 0.0001 2.820463526 4.813924625 5.4757 < 0.0001 0.184897548 0.412458053 2.9063 0.0087 0.264789916 1.611433125 2.0171 0.0573 -0.00074147 0.044195638 -4.3330 0.0003 -0.02300103 -0.00805170 Multiple Adjusted StErr of Enter or R-Square Step Information R R-Square Estimate Exit Diffuser 0.7108 0.5052 CycleTime 0.8458 0.7154 UllagePres 0.8939 0.7990 MLP = 2 0.9285 0.8620 Temp 0.9367 0.8774 PeakPres 0.9473 0.8974 CycleTime 0.9465 0.8959 0.4846 1.189133891 Enter 0.6906 0.921254182 Enter 0.7716 0.791489885 Enter 0.8358 0.671216686 Enter 0.8467 0.648478308 Enter 0.8650 0.608639466 Enter 0.8698 0.597610012 Exit The scatterplot of the fit versus the actual cycle count output in Figure 7.0-3 confirms that the residuals appear to be random and that the linear model correctly captures the relationships in the data. NESC Request No.: 05-171-E

Original page 20

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Title: Page #: External Tank Liquid Hydrogen (LH2) Prepress 16 of 34 Regression Analysis Independent Review Technical Consultation Report Scatterplot of Fit vs ActLCC 16.0 14.0 12.0 10.0 Fit 8.0 6.0 4.0 2.0 0.0 0.0 2.0 4.0 6.0 8.0 10.0 12.0 14.0 16.0 ActLCC Figure 7.0-3. Scatterplot of the fit versus actual cycle count for the regression on all variables Next, CycleTime, MLP-2, and Orbiter variables were eliminated from the data set to be consistent with the data employed by Clark and Krenn in Prepress Regression Analysis, which also restored the record for STS-114 tanking data omitted from the previous regression. Stepwise regression was applied to the resulting data set. The results agree exactly with the twovariable “regression #1” reported by Clark and Krenn, as shown in Table 7.0-5. NESC Request No.: 05-171-E

Original page 21

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Title: Page #: External Tank Liquid Hydrogen (LH2) Prepress 17 of 34 Regression Analysis Independent Review Technical Consultation Report Table 7.0-5. Stepwise Regression Corresponding to Clark and Krenn Multiple R-Square Summary R 0.9297 0.8643 Degrees of Sum of ANOVA Table Freedom Squares Adjusted StErr of Durbin R-Square Estimate Watson 0.8466 0.723270396 2.1341 Mean of F-Ratio p-Value Squares Explained 3 76.60911257 25.53637086 48.8155 < 0.0001 Unexplained 23 12.0317615 0.523120065 Standard Coefficient Regression Table Error Constant 13.78872186 3.642964275 Diffuser 3.831753215 0.488181847 PeakPres -0.02012713 0.003559934 UllagePres 0.263784397 0.064649407 Multiple R-Square Step Information R Diffuser 0.7841 0.6147 PeakPres 0.8752 0.7660 Lower Upper t-Value p-Value Limit Limit 3.7850 0.0010 6.252676093 21.32476764 7.8490 < 0.0001 2.821872122 4.841634309 -5.6538 < 0.0001 -0.02749142 -0.01276285 4.0802 0.0005 0.130046909 0.397521885 Adjusted StErr of Enter or R-Square Estimate Exit 0.5993 1.168755963 Enter 0.7465 0.929623706 Enter The regression was repeated with the same data scaled from 0 to 1. The scaled value of any variable xi is zi=(xi-ximin)/(ximax-ximin). The regression statistics shown in Table 7.0-6 are the same (as these must be) as before, but the scaling makes the regression coefficients easier to interpret ― the coefficients are now proportional to the significance of the corresponding variables in the regression. NESC Request No.: 05-171-E

Original page 22

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Title: Page #: External Tank Liquid Hydrogen (LH2) Prepress 18 of 34 Regression Analysis Independent Review Technical Consultation Report Table 7.0-6. Stepwise Regression Corresponding to Clark and Krenn using Scaled Values Multiple Adjusted StErr of Durbin R-Square Summary R R-Square Estimate Watson 0.9297 0.8643 0.8466 0.10186907 2.1341 Degrees of Sum of Mean of F-Ratio p-Value ANOVA Table Freedom Squares Squares Explained 3 1.519720543 0.506573514 48.8155 < 0.0001 Unexplained 23 0.23867807 0.010377307 Standard Coefficient Regression Table Error Lower Upper t-Value p-Value Limit Limit Constant 0.465006748 0.042789934 10.8672 < 0.0001 0.376489024 0.553524471 Scaled Diffuser 0.539683551 0.068758007 7.8490 < 0.0001 0.397446778 0.681920325 Scaled PeakPress -0.39715663 0.07024602 -5.6538 < 0.0001 -0.542471598 -0.25184167 Scaled Ullage 0.193194206 0.047348862 4.0802 0.0005 0.095245623 0.291142789 Multiple Adjusted StErr of Enter or R-Square Step Information R R-Square Estimate Exit Scaled Diffuser 0.7841 0.6147 0.5993 0.164613516 Enter Scaled PeakPress 0.8752 0.7660 0.7465 0.130932916 Enter 2 The three-variable regression yields an adjusted R of .8466. However, the regression again is dominated by the Diffuser variable, which alone explains about 61.47 percent of the variation. The scatterplot of the fit versus the actual cycle count output shown in Figure 7.0-4 confirms that the residuals appear to be random and that the linear model correctly captures the relationships in the data. NESC Request No.: 05-171-E

Original page 23

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Title: Page #: External Tank Liquid Hydrogen (LH2) Prepress 19 of 34 Regression Analysis Independent Review Technical Consultation Report Scatterplot of Fit vs Scaled ActLCC 1.2 1.0 0.8 Fit0.6 0.4 0.2 0.0 0.0 0.2 0.4 Scaled ActLCC 0.6 0.8 1.0 1.2 Figure 7.0-4. Scatterplot of the Fit Versus Scaled Actual Cycle Count for the Regression on all Variables Scaling the data also permits box plots for all of the variables and variable interactions in the regression on the same scale, as shown in Figure 7.0-5. The data anomalies previously described are clearly reflected in the outliers in these plots. NESC Request No.: 05-171-E

Original page 24

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Title: Page #: External Tank Liquid Hydrogen (LH2) Prepress 20 of 34 Regression Analysis Independent Review Technical Consultation Report Box Plot Comparison Scaled ActLCC / Scaled data Scaled Diffuser / Scaled data Scaled PreakPress / Scaled data Scaled Ullage / Scaled data 0 0.2 0.4 0.6 0.8 1 1.2 Figure 7.0-5. Comparison of Scaled Output and Predictor Variables for the Clark and Kreen Comparison Scatterplots of the output versus each of the three predictor variables included are shown in Appendix B. It should be noted that examining the residuals (the differences between the observed and predicted output at each data point) for randomness is important in any regression analysis. A pattern in the residuals indicates that the underling relationship is nonlinear and that superior regression can be achieved by a suitable transformation of the data. In all of the regressions developed in this review, examination of the residuals confirmed the apparent linearity of the relationship modeled. Regression #1, developed by Clark and Krenn, is based on three independent (predictor) variables which can be measured well prior to launch. However, the data set used to develop this regression includes data that is no longer representative of the tanking operations that should occur in the future. Primarily, the diffuser variable dominates the regression and the out-ofspecification material problem with the diffusers in the STS-114 tanking tests has been resolved. All diffuser material will be single twill in the future. In addition, the MLP-2 restriction problem upstream of the panel has been fixed and MLP-2 cycles should be within the realm of the other MLPs in the future. Since the data on which this regression was based are not homogenous, NESC Request No.: 05-171-E

Original page 25

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Page #: Title: External Tank Liquid Hydrogen (LH2) Prepress 21 of 34 Regression Analysis Independent Review Technical Consultation Report replacing variables (Diffuser & UllagePress) with values that have now become standard is not statistically acceptable. Therefore, an additional stepwise linear regression was performed excluding the data points where the Double Dutch Twill Diffuser was used (STS 114 tanking tests) and where MLP was equal to 2. The results of this regression are shown in Table 7.0-7. Table 7.0-7. Stepwise Regression Based on Subset of Data that is Relevant to Current and Future Conditions Multiple R-Square Adjusted StErr of Summary R R-Square Estimate 0.8419 0.7088 0.6640 0.655232234 Degrees of Sum of Mean of F-Ratio p-Value ANOVA Table Freedom Squares Squares Explained 2 13.58341936 6.791709681 15.8193 0.0003 Unexplained 13 5.581280638 0.42932928 Coefficient Standard t-Value p-Value Confidence Interval 95% Regression Table Error Lower Upper Constant 11.76671334 6.039583426 1.9483 0.0733 -1.281013396 24.81444007 UllagePres 0.275004813 0.067714984 4.0612 0.0013 0.128715484 0.421294141 CycleTime -29.46607337 9.404504049 -3.1332 0.0079 -49.78326915 -9.148877596 Multiple R-Square Adjusted StErr of Enter or Step Information R R-Square Estimate Exit UllagePres 0.6992 0.4889 0.4523 0.836487017 Enter CycleTime 0.8419 0.7088 0.6640 0.655232234 Enter 2 While this regression equates to an R of 0.664, the somewhat lower significance than the proposed three-variable regression is balanced by the logic of using data points that are representative of the future configuration. NESC Request No.: 05-171-E

Original page 26

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Page #: Title: External Tank Liquid Hydrogen (LH2) Prepress 22 of 34 Regression Analysis Independent Review Technical Consultation Report 8.0 Findings, Root Causes, Observations and Recommendations 8.1 Findings F-1. Using the same assumptions, regression results obtained during this review confirmed that the SSP regressions have included appropriate, and only the most appropriate, predictors in their regression. F-2. A linear model is the correct choice for this regression and correctly captures the relationships in the data. F-3. Use of the full data set provided is not justified since several of the independent variables reflected out-of-specification materials and components, which will not be repeated in the future. The reduced data set of 16 data points is adequate for regression with less than four independent variables. 8.2 Recommendations R-1. Linear regression is an appropriate tool to validate the NASA-accredited analytical (first principles) model, currently under development, against empirical data available since the transition to half-second valve pulses. (F-1 and F-2) R-2. Regression should be based on the reduced data set, excluding STS-114 tanking tests where the incorrect diffuser material was used and MLP-2 restricted flow was created from out-of-specification conditions, since these situations have been corrected. (F-3) 9.0 Lessons Learned There were no lessons learned during this consultation. 10.0 Definition of Terms Corrective Actions Changes to design processes, work instructions, workmanship practices, training, inspections, tests, procedures, specifications, drawings, tools, equipment, facilities, resources, or material that result in preventing, minimizing, or limiting the potential for recurrence of a problem. NESC Request No.: 05-171-E

Original page 27

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Page #: Title: External Tank Liquid Hydrogen (LH2) Prepress 23 of 34 Regression Analysis Independent Review Technical Consultation Report Finding A conclusion based on facts established during the assessment/inspection by the investigating authority. Lessons Learned Knowledge or understanding gained by experience. The experience may be positive, as in a successful test or mission, or negative, as in a mishap or failure. A lesson must be significant in that it has real or assumed impact on operations; valid in that it is factually and technically correct; and applicable in that it identifies a specific design, process, or decision that reduces or limits the potential for failures and mishaps, or reinforces a positive result. Observation A factor, event, or circumstance identified during the assessment/inspection that did not contribute to the problem, but if left uncorrected has the potential to cause a mishap, injury, or increase the severity should a mishap occur. Problem The subject of the technical assessment/inspection. 11.0 Minority Report (Dissenting Opinions) There were no dissenting opinions during this consultation. Volume II: Appendices A NESC ITA/I Request Form (NESC-PR-003-FM-01) B Key for Interpreting Box Plots and Scatterplots C List of Acronyms NESC Request No.: 05-171-E

Original page 28

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Page #: Title: External Tank Liquid Hydrogen (LH2) Prepress 24 of 34 Regression Analysis Independent Review Technical Consultation Report Appendix A. NESC ITA/I Request Form (NESC-PR-003-FM-01) NESC Request No.: 05-171-E

Original page 29

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Page #: Title: External Tank Liquid Hydrogen (LH2) Prepress 25 of 34 Regression Analysis Independent Review Technical Consultation Report NESC Request No.: 05-171-E

Original page 30

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Page #: Title: External Tank Liquid Hydrogen (LH2) Prepress 26 of 34 Regression Analysis Independent Review Technical Consultation Report NESC Request No.: 05-171-E

Original page 31

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Page #: Title: External Tank Liquid Hydrogen (LH2) Prepress 27 of 34 Regression Analysis Independent Review Technical Consultation Report NESC Request No.: 05-171-E

Original page 32

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Page #: Title: External Tank Liquid Hydrogen (LH2) Prepress 28 of 34 Regression Analysis Independent Review Technical Consultation Report NESC Request No.: 05-171-E

Original page 33

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Page #: Title: External Tank Liquid Hydrogen (LH2) Prepress 29 of 34 Regression Analysis Independent Review Technical Consultation Report NESC Request No.: 05-171-E

Original page 34

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Page #: Title: External Tank Liquid Hydrogen (LH2) Prepress 30 of 34 Regression Analysis Independent Review Technical Consultation Report Appendix B. Key for Interpreting Box Plots and Scatterplots Figure B-1. Key for Interpreting Box Plots NESC Request No.: 05-171-E

Original page 35

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Title: Page #: External Tank Liquid Hydrogen (LH2) Prepress 31 of 34 Regression Analysis Independent Review Technical Consultation Report Scatterplot of Scaled ActLCC vs Scaled Diffuser of Scaled data 1.2 1 0.8 0.6 0.4 Scaled ActLCC / Scaled data0.2 0 0 0.2 0.4 0.6 0.8 1 1.2 Scaled Diffuser / Scaled data B-2. Scatterplot for the Scaled Output and Diffuser Predictor Scatterplot of Scaled ActLCC vs Scaled Ullage of Scaled data 1.2 1 0.8 0.6 0.4 Scaled ActLCC / Scaled data0.2 0 0 0.2 0.4 0.6 0.8 1 1.2 Scaled Ullage / Scaled data B-3. Scatterplot for the Scaled Output and UllagePres Predictor NESC Request No.: 05-171-E

Original page 36

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Title: Page #: External Tank Liquid Hydrogen (LH2) Prepress 32 of 34 Regression Analysis Independent Review Technical Consultation Report Scatterplot of Scaled ActLCC vs Scaled PreakPress of Scaled data 1.2 1 0.8 0.6 0.4 Scaled ActLCC / Scaled data0.2 0 0 0.2 0.4 0.6 0.8 1 1.2 Scaled PreakPress / Scaled data B-4. Scatterplot for the Scaled Output and PeakPres Predictor NESC Request No.: 05-171-E

Original page 37

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Page #: Title: External Tank Liquid Hydrogen (LH2) Prepress 33 of 34 Regression Analysis Independent Review Technical Consultation Report Appendix C. List of Acronyms ET External Tank GHe Gaseous Helium GN2 Gaseous Nitrogen LaRC Langley Research Center LCC Launch Commit Criteria LH2 Liquid Hydrogen MLP Mobile Launch Pad MTSO Management Technical and Support Office NASA National Aeronautics and Space Administration NESC NASA Engineering and Safety Center NRB NESC Review Board SEO Systems Engineering Office SSP Space Shuttle Program STS Space Transportation System NESC Request No.: 05-171-E

Original page 38

NASA Engineering and Safety Center Document #: Version: Technical Assessment Report RP-06-01 1.0 Title: Page #: External Tank Liquid Hydrogen (LH2) Prepress 34 of 34 Regression Analysis Independent Review Technical Consultation Report Approval and Document Revision History Approved: Original signed on file 1/24/06 NESC Director Date Version Description of Revision 1.0 Initial Release NESC Request No.: 05-171-E Office of Author Effective Date NESC System Engineer’s 1/24/06 Office

Original page 39

Form Approved REPORT DOCUMENTATION PAGE OMB No. 0704-0188 The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS. 1. REPORT DATE (DD-MM-YYYY) 2. REPORT TYPE 01- 05 - 2009 Technical Memorandim 4. TITLE AND SUBTITLE 3. DATES COVERED (From - To) September 2005-January 2006 5a. CONTRACT NUMBER External Tank Liquid Hydrogen (LH2) Prepress Regression Analysis Independent Review Technical Consultation Report 6. AUTHOR(S) Parsons, Vickie S. 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) NASA Langley Research Center Hampton, VA 23681-2199 5b. GRANT NUMBER 5c. PROGRAM ELEMENT NUMBER 5d. PROJECT NUMBER 5e. TASK NUMBER 5f. WORK UNIT NUMBER 869021.06.07.04.99 8. PERFORMING ORGANIZATION REPORT NUMBER L-19680 NESC-RP-06-01/05-171-E 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR'S ACRONYM(S) National Aeronautics and Space Administration Washington, DC 20546-0001 12. DISTRIBUTION/AVAILABILITY STATEMENT Unclassified - Unlimited Subject Category 15-Launch Vehicles and Launch Operations Availability: NASA CASI (443) 757-5802 13. SUPPLEMENTARY NOTES 14. ABSTRACT NASA 11. SPONSOR/MONITOR'S REPORT NUMBER(S) NASA/TM-2009-215745 The request to conduct an independent review of regression models, developed for determining the expected Launch Commit Criteria (LCC) External Tank (ET)-04 cycle count for the Space Shuttle ET tanking process, was submitted to the NASA Engineering and Safety Center NESC on September 20, 2005. The NESC team performed an independent review of regression models documented in Prepress Regression Analysis, Tom Clark and Angela Krenn, 10/27/05. This consultation consisted of a peer review by statistical experts of the proposed regression models provided in the Prepress Regression Analysis. This document is the consultation's final report. 15. SUBJECT TERMS ET; LCC, NESC, SSP; STS 16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF ABSTRACT a. REPORT b. ABSTRACT c. THIS PAGE U U U UU 18. NUMBER 19a. NAME OF RESPONSIBLE PERSON OF PAGES STI Help Desk (email: help@sti.nasa.gov) 19b. TELEPHONE NUMBER (Include area code) 39 (443) 757-5802 Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std. Z39.18