Full report

Original page 1

NASA/TM-2005-213936 Results of a Television Station Managers’ Telephone Survey of NASA’s Destination Tomorrow™ Scott Endo University of California at San Diego, San Diego, California Thomas E. Pinelli Langley Research Center, Hampton, Virginia Randall H. Caton Christopher Newport University, Newport News, Virginia October 2005

Original page 2

The NASA STI Program Office . . . in Profile Since its founding, NASA has been dedicated to the advancement of aeronautics and space science. The NASA Scientific and Technical Information (STI) Program Office plays a key part in helping NASA maintain this important role. The NASA STI Program Office is operated by Langley Research Center, the lead center for NASA’s scientific and technical information. The NASA STI Program Office provides access to the NASA STI Database, the largest collection of aeronautical and space science STI in the world. The Program Office is also NASA’s institutional mechanism for disseminating the results of its research and development activities. These results are published by NASA in the NASA STI Report Series, which includes the following report types: • TECHNICAL PUBLICATION. Reports of completed research or a major significant phase of research that present the results of NASA programs and include extensive data or theoretical analysis. Includes compilations of significant scientific and technical data and information deemed to be of continuing reference value. NASA counterpart of peerreviewed formal professional papers, but having less stringent limitations on manuscript length and extent of graphic presentations. • TECHNICAL MEMORANDUM. Scientific and technical findings that are preliminary or of specialized interest, e.g., quick release reports, working papers, and bibliographies that contain minimal annotation. Does not contain extensive analysis. • CONTRACTOR REPORT. Scientific and technical findings by NASA-sponsored contractors and grantees. • CONFERENCE PUBLICATION. Collected papers from scientific and technical conferences, symposia, seminars, or other meetings sponsored or co-sponsored by NASA. • SPECIAL PUBLICATION. Scientific, technical, or historical information from NASA programs, projects, and missions, often concerned with subjects having substantial public interest. • TECHNICAL TRANSLATION. Englishlanguage translations of foreign scientific and technical material pertinent to NASA’s mission. Specialized services that complement the STI Program Office’s diverse offerings include creating custom thesauri, building customized databases, organizing and publishing research results ... even providing videos. For more information about the NASA STI Program Office, see the following: • Access the NASA STI Program Home Page at http://www.sti.nasa.gov • E-mail your question via the Internet to help@sti.nasa.gov • Fax your question to the NASA STI Help Desk at (301) 621-0134 • Phone the NASA STI Help Desk at (301) 621-0390 • Write to: NASA STI Help Desk NASA Center for AeroSpace Information 7121 Standard Drive Hanover, MD 21076-1320

Original page 3

NASA/TM-2005-213936 Results of a Television Station Managers’ Telephone Survey of NASA’s Destination Tomorrow™ Scott Endo University of California at San Diego, San Diego, California Thomas E. Pinelli Langley research Center, Hampton, Virginia Randall H. Caton Christopher Newport University, Newport News, Virginia National Aeronautics and Space Administration Langley Research Center Hampton, Virginia 23681-2199 October 2005

Original page 4

Available from: NASA Center for AeroSpace Information (CASI) 7121 Standard Drive Hanover, MD 21076-1320 (301) 621-0390 National Technical Information Service (NTIS) 5285 Port Royal Road Springfield, VA 22161-2171 (703) 605-6000

Original page 5

Abstract NASA’s Destination Tomorrow™ http://destination.larc.nasa.gov is an Emmy® awardwinning series of educational programs. These programs (1) are produced by the NASA Center for Distance Learning http://dlcenter.larc.nasa.gov and (2) are designed primarily for adult (lifelong) learners. Each program in this annual series (1) is 30 minutes long, (2) is produced in a magazine style format, and (3) is broken into segments ranging from 3 – 5 minutes to 6 – 8 minutes in length. Each NASA’s Destination Tomorrow™ program is based on (1) educational theory, (2) principles, and (3) research as they pertain to how adults learn and apply knowledge. More than 800 television stations in the United States are registered users of (meaning that they air) NASA’s Destination Tomorrow™, and we randomly selected 405 stations to participate in a telephone survey that was conducted between June 20 and July 29, 2005. The survey consisted of 37 questions and was pre-tested prior to full-scale implementation. Four hundred and three (403) responses were collected by the designated cut off date. The survey goal was to solicit feedback from television stations on a range of topics, including the perceived quality of NASA’s Destination Tomorrow™ and how NASA’s Destination Tomorrow™ compares vis-àvis to other adult (educational) programming aired by the television stations in the sample pool. About 98 percent of those surveyed stated that they were familiar with the series, and a nearly similar percentage (98 percent) indicated they had watched all or a portion of a program. On a 10-point scale, survey participants rated the overall technical quality of NASA’s Destination Tomorrow™ highly ( x = 9.48), and the educational value of the series slightly more highly ( x = 9.56). Ninety one (91) percent of the participants reported that the technical quality of NASA’s Destination Tomorrow™ was higher compared to other educational programming that airs on their station. Most stations (81 percent) indicated that NASA’s Destination Tomorrow™ was well received by their audiences, and 97 percent indicated that they had recommended or would recommend the series to a colleague. Lastly, using a 10-point scale, survey participants indicated that (1) the series successfully educates people about what NASA does ( x = 9.23), (2) the information contained in NASA’s Destination Tomorrow™ is credible ( x = 9.53), and (3) is successful in educating the public about what NASA does ( x = 9.23). Introduction The NASA Center for Distance Learning (CDL) http://dlcenter.larc.nasa.gov is recognized for (1) its leadership in the application of traditional and emerging instructional technology; (2) the development of six exciting, innovative, and inspirational instructional and educational programs that are an integral part of NASA’s Distance Learning Network; (3) its use of NASA programs, projects, facilities, and personnel to motivate and inspire teaching and learning; and (4) its ability to identify customer needs and to translate those needs into customer-focused programs. NASA’s CDL was established in 1996 as a collaboration with the Christopher Newport University. Presently, NASA’s CDL operates as a partnership with the National Institute of Aerospace (NIA), Virginia Polytechnic Institute and State University (VA TECH), and Crewestone Technologies. The six programs offered by NASA’s CDL “span the educational horizon” from K – 12, through college (grades 13 – 18), to adult (lifelong) learners. The six high-energy, Emmy®-award-winning programs produced by the NASA CDL ─ • Combine cutting-edge multimedia with technology to meet needs identified by the education community. • Support the NASA pipeline and the Agency’s workforce development initiatives.

Original page 6

2 • Use NASA projects, facilities, research, and people to motivate and inspire students, young adults, formal and informal educators, parents, and adults. • Are research-, inquiry-, and standards-based and are developed with educator input for use by the formal and informal education communities. • Use 12 partnerships, collaborations, and alliances to strengthen and enrich our programs, extend their distribution, and provide nationwide classroom mentor programs. • Epitomize continuous quality improvement through the use of theory-based practice, sound instructional models, formal and informal evaluation, and longitudinal assessment. • Use multichannel, global access, and digital distribution to maximize audience participation and minimize distribution costs. • Draw from a multifaceted, national marketing program based on customer needs and service, a reliable management information system, market analysis, and ecommerce. • Have a user base of over 400,000 registered educators (representing 10.1 million students and young adults) that is made possible through extensive marketing, registration, and “world class” customer service. • Have won countless awards and are recognized throughout the world for their educational, technical, and artistic excellence. • Are produced by the Agency’s leader in educational programming and are the programs most requested through the NASA Educator Resource Centers, NASA CORE, and Voice of America (VOA). • Collectively constitute the Agency’s single largest contribution of educational programming for NASA TV. • Are dubbed in Spanish and are used by more than 3,000 registered educators for students with limited English proficiency and in language immersion programs. • Five NASA CONNECT™ programs are being translated into Japanese and air on Japan’s science channel. • NASA’s CDL programs air in 94 countries and have been translated into many languages, including Spanish, Japanese, and Malay. • Are closed- and (audio) descriptive-captioned and 508 compliant.

Original page 7

3 Collectively, these programs (1) promote creativity, critical thinking, and problem solving skills; (2) are designed to integrate easily, in whole or in part, into an existing curriculum; (3) are used to introduce or reinforce a curriculum topic, objective, or skill and can be used by formal and informal educators; (4) increase interest, engagement, and understanding of science, technology, engineering, and mathematics (STEM); (5) motivate and inspire students to pursue careers in STEM areas; (6) establish a connection between STEM concepts taught in the classroom and those used every day by NASA researchers; (7) are readily accessible to homebound and home schooled children; (8) increase (adult) scientific and technological literacy; (9) use technology to enhance and enrich the teaching and learning process; (10) advance the theory and practice of teaching mathematics, science, and technology; (11) support the NASA education plan; (12) contribute to the nation’s science and engineering goals; (13) support the Agency’s workforce development initiatives; and (14) communicate the results of NASA discovery, exploration, innovation, and research. NASA’s Destination Tomorrow™ is an Emmy®-award-winning series of 30-minute educational programs. An associated web site contains story summaries and links to related program material. As a series, NASA’s Destination Tomorrow™ is designed to (1) create and heighten adult interest in mathematics, science, technology, and NASA; (2) increase the scientific and technological literacy of adults; (3) improve the scientific and technological literacy of adults who do not use English as their primary language; and (4) serve as a mechanism for parents and caregivers to become involved in the education of children and young adults. Program collaborators – California State Polytechnic University Pomona, South Carolina Educational Television (SC ETV), the Apple Learning Interchange (ALI), Ibiblio (University of North Carolina, Chapel Hill), MAG Rack, and the Alliance for Community Media (ACM) – provide for the widest possible (domestic) program distribution and access. Our collaboration with the VOA allows for the widest possible (international) program access. NASA’s Destination Tomorrow™ was created to increase the scientific literacy of adults. Scientific literacy for children, young adults, and adults alike has emerged as a central goal of education. A scientifically literate person (1) is aware that science, mathematics, and technology are interdependent human enterprises with (defined) strengths and limitations; (2) understands key concepts and principles of science; (3) is familiar with the natural world and recognizes both its diversity and unity; and (4) uses scientific knowledge and scientific ways of thinking. The goal of NASA’s Destination Tomorrow™ is not to make “non-scientists into scientists.” On the contrary, the goal is for adults to become familiar enough with how science works to be able to understand claims and, if necessary, to respond to claims made in the name of science, mathematics, and technology (Science for all Americans: Project 2061). NASA CDL uses formative and summative evaluation (1) to approximate the cost/benefit of our programs; (2) as an accountability tool; (3) to help make sound decisions relating to program design, personnel, and budget; and (4) to determine if stated program objectives are met. At NASA’s CDL, assessment is an ongoing process designed to provide accurate and reliable data that is used expressly for program improvement. The first step toward ensuring that the program objectives established for the NASA’s Destination Tomorrow™ are met begins by making certain that the broadcast scripts and videography support the objectives. Thereafter, a variety of tools, including focus groups and (self-reported) mail, Internet, and telephone surveys are used to obtain data from our client base of registered television stations.

Original page 8

4 Both qualitative and quantitative data are collected and used to determine the extent to which the program objectives established for NASA’s Destination Tomorrow™ are met. Registered stations constitute the “population” and a sample taken from that population is surveyed every third year. Data collected from the first and each subsequent evaluation constitutes the baseline. We will have successfully met NASA’s Destination Tomorrow™ program objectives if (1) we obtain an overall average survey score of 4.30 (using a 5-point Likert scale), (2) registered stations continue to use this product, (3) the number of registered stations continues to grow; (4) registered stations recommend the NASA’s Destination Tomorrow™ to other stations, and (5) NASA’s Destination Tomorrow™ fares well in the highly competitive, multimedia education awards competition. Methodology More than 800 television stations throughout the United States are registered with NASA’s CDL to air NASA’s Destination Tomorrow™. From the database of registered stations, we randomly selected 405 television stations to serve as the survey sample. Each station point of contact (POC) was contacted by telephone and asked to participate in a (telephone) survey that took about 15 minutes to complete. The survey instrument (appendix A) was composed of 37 questions, divided into four sections, and was pretested prior to full-scale implementation. These sections included five questions about the station, seven questions about how the stations select programming, sixteen questions pertaining specifically to NASA’s Destination Tomorrow™, and eight demographic questions. (Upon completion, survey participants were given the opportunity to offer comments about the series.) The telephone survey was conducted between June 20 and July 29, 2005. In all, 403 surveys were completed by the cut-off date. We used SPSS (Statistical package for the Social Sciences) to analyze the data. Organization of the Report The report opens with an abstract and is followed by an introduction; overview; methodology; demographics; presentation of the qualitative and quantitative data; comparison and analysis of the 2002 and 2005 qualitative and quantitative data, recommendations, conclusions, and references. There are two appendices: appendix A, the survey instrument and appendix B, the general comments. Demographics We asked participants a series of demographic questions, the answers to which enabled us to establish the following respondent profile for NASA’s Destination Tomorrow™ 2005. • About 39 percent of the respondents were female; about 61 percent were male. • About 51 percent of the television stations were located in the Eastern Time Zone, 26 percent were located in the Central Time Zone, 8 percent were located in the Mountain Time Zone, and 15 percent were located in the Pacific Time Zone. • About 57 percent of those surveyed held the title of Program Manager, 11 percent held the title of Video Technician, 14 percent held the title of General Manager, and 18 percent held the title of Executive Producer. • The mean and median number of years worked at the station is 8.0 and 6.0, respectively.

Original page 9

5 • The mean and median number of years respondents worked in television is 13.7 and 12.0, respectively Presentation of Quantitative Data The data in this survey are presented primarily in graphical form. The survey also contains questions that use a 10-point Likert (response) scale. These questions are reported in terms of the mean (average). The responses of 403 participants were used in the calculation of the statistical data reported for the 10-point Likert scale questions. Responses to the multiple choice questions are represented by bar graphs. Survey Questions Section 1 – Questions about the television stations. The first five survey questions were asked to attain more information about the station. The first question inquired about the type of station (by license) that was participating in the survey. (See fig. 1.) Most of the stations surveyed were Community Access - Educational (156, 38.7%). The next most popular type of station was Community Access – Government (84, 20.8%). As a whole, Community Access made up about 60 percent of the total survey. The least frequent station type was Commercial (15, 3.4%). 1. Which of the following best describes your television station ? 200 160 120 156 80 15 84 31 Response, n40 0 ITV Access -Education Access - Commercial Community CommunityGovernment Station Type 48 37 32 PBS Other College/University Figure 1. Television stations by type. The second question inquired about the station’s audience size. The mean (average) for this question was 89,965, the median was 29,000, and the mode was 10,000. Question Mean Median Mode 2. What is the size of your audience? 89,965 29,000 10,000 The third question pertained to the television station’s ability to downlink satellite programming. (See fig. 2.) The majority of the stations that participated in this survey do not have satellite capability (223, 55.3%).

Original page 10

6 3. Does your station have satellite (downlink) capability? No Yes 180 0 50 100 150 Response, n 223 200 250 Figure 2. Television station satellite (downlink) capability. Question four included only the participants who had satellite capability. Of the 180 stations that have satellite capability, the mean percentage of programming downlinked was 31.78%. The most frequent response (mode) was 10%, which leads to the conclusion that few stations are using their satellites exclusively to attain programming. Question Mean Median Mode 4. What percentage of your programming is down linked via satellite? 31.78 20 10 The last station question determined participants preferred media format. (See fig. 3.) The first four bars going from left to right are digital format and the remaining bars are analog formats. 5. What type of media do you prefer? 200 179 150 100 Response, n50 23 25 14 0 DVD DVCamDVC Pro Mini DV 68 50 44 0 VHS 3/4" SVHS Beta-SP Media Type Figure 3. Television station media preferences.

Original page 11

7 Section 2 – Questions about how television stations select programming. Most prefer digital format (241, 59.8%). The type of media that was overwhelmingly preferred was DVD (179, 44.4%). The second most popular media type was S-VHS (68, 16.9%). The least popular media type was ¾ in. (0, 0%) and the second least popular type of media was Mini DV (14, 3.5%). The next seven survey questions pertain to how the television station selects programming. Responses to questions 6 – 11 were measured using a 10-point Likert scale. The importance of viewer recommendations registered the lowest mean at x = 7.54. The highest mean of the six questions was x = 9.09. This result came from the question about the importance of program cost, which is consistent with the fact that most stations that participated were Community Access. Because this type of station operates on a small budget, it is a logical conclusion that cost would have a strong influence on the programs aired. Question 6. How important is program cost? 7. How important is the program’s technical quality? 8. How important is easy program acquisition? Standard Mean Median Deviation Minimum Maximum Response, n 9.09 10 2.06 1 10 403 8.36 9 1.72 1 10 403 8.33 9 1.62 1 10 403 9. How important is the program’s educational value? 8.91 10 1.49 1 10 403 10. How important is having few restrictions on program use? 11. How important are viewer recommendations? 8.12 8 1.90 1 10 403 7.54 8 2.26 1 10 403 Educational value was rated the overall single most important factor when selecting a program to air (172, 43%). (See fig. 4.) The second most important factor was cost (140, 35%). The least important factor was having few restrictions on program use (15, 4%). 12. Which of the following considerationsisthe single most important attribute that affectsyour decision to air a program? 200 180 160 140 120 100 140 80 60 26 25 Response, 40 20 0 Cost Technical QualityAcquisition Ease of Attribute 172 15 25 Educational ValueRestrictionsFew ProgramRecommendations Viewer Figure 4. Attributes and program selection by television stations.

Original page 12

8 Section 3 – Questions about NASA’s Destination Tomorrow™ The following sixteen questions pertain to NASA’s Destination Tomorrow™. As shown in figure 5, about 98 percent of those surveyed were familiar with NASA’s Destination Tomorrow™. As shown in figure 6, about 97 percent of those familiar with the series had watched at least a part of a NASA’s Destination Tomorrow™ program. Survey participants were asked to identify the “time slot” in which they air NASA’s Destination Tomorrow™ (see fig. 7). About 75 percent of the stations surveyed air NASA’s Destination Tomorrow™ a combination of times. 13. Are you familiar with NASA's Destination Tomorrow™? 500 400 300 395 200 Response, 100 6 0 Yes No 2 Not sure Figure 5. Participants’ familiarity with NASA’s Destination Tomorrow™. 14. Have you watched at least part of a NASA's Destination Tomorrow™program? 500 400 300 383 200 19 Response, n100 0 Yes No 1 Not sure Figure 6. Participants having watched NASA’s Destination Tomorrow™. 15. On your station, is NASA's Destination Tomorrow™ generally aired in the morning, afternoon, evening, night, or some combination of times? 400 300 200 25 47 esponse,100 R 23 0 300 8 Morning Afternoon Evening Night Combination Time of Day Figure 7. Time slots occupied by NASA’s Destination Tomorrow™.

Original page 13

9 Respondents were questioned about the technical quality and educational value of the series. Many respondents thought very highly of the series technical quality ( x = 9.48). Respondents rated the educational value of the series even higher ( x = 9.56). As shown in figure 8, the most appealing attribute of NASA’s Destination Tomorrow™ is its educational value (251, 62.3%). The least appealing attribute of the series is its lack of restrictions concerning program use (12, 3%). Compared to other (similar) educational programming the stations air, 91 percent (368) of survey participants reported that the technical quality of NASA’s Destination Tomorrow™ is better than average (see fig. 9). About 9 percent (35) indicated that the quality was equal to other programming, while 0 percent indicated that the technical quality was worse. Question 16. How would you rate the overall technical quality of NASA’s Destination Tomorrow™ on a scale of 1 to 10, with 10 being the highest? 17. How would you rate the overall educational value of ™ NASA’s Destination Tomorrow ? Min. denotes minimum. Max. denotes maximum. Standard Mean Median Deviation Min. Max. Response, n 9.48 10 0.86 5 10 403 9.56 10 0.78 5 10 403 18. Froma programming standpoint, which of the following attributes of NASA's Destination Tomorrow™appeal most to you? 300 250 200 150 100 Response, 24 28 50 73 0 251 12 15 Cost Technical Quality Ease of Acquisition Educational Value Few Program Viewer Attribute Restrictions Recommendations Figure 8. Rating of NASA’s Destination Tomorrow™ by selected attributes. 19. Compared to other educational programming your station airs, is the technical quality of NASA's Destination Tomorrow™series better whan average, about average, or worse than average? 400 350 300 368 250 200 Response, n150 100 35 50 0 0 Better than Average About Average Worse than Average Figure 9. Comparison of NASA’s Destination Tomorrow™ to other programs.

Original page 14

10 About 81 percent (325) of the respondents reported that NASA’s Destination Tomorrow™ is very well received by their audience (see fig. 10). About 19 percent (78) reported that the series is somewhat well received, while no one reported that the series was not well received. As shown in figure 11, about 97 percent (403) of the survey participants stated that they have or would recommend NASA’s Destination Tomorrow™ to a colleague. 20. Froman audience perspective, wouldyou saythat NASA's DestinationTomorrow™has beenverywell received, somewhat well received, or not well received? 350 325 300 250 200 Response,150 78 100 50 0 0 Very well received Somewhat well Not well received received Figure 10. Audience perception of NASA’s Destination Tomorrow™. 21. Would or have you recommended NASA's Destination Tomorrow™to a colleague? No 11 Yes 0 100 200 Response, n . 392 300 400 500 Figure 11. Recommendation of NASA’s Destination Tomorrow™. Question 22. One goal of NASA’s Destination ™ Tomorrow is to educate people about what NASA does. How successful has NASA’s Destination Tomorrow™ been in that regard? 23. In your opinion, how credible is the information contained in NASA’s Destination ™? Tomorrow Standard Mean Median Deviation Min. Max. Response, n 9.23 10 1.19 5 10 403 9.53 10 0.93 3 10 403

Original page 15

11 Respondents were asked if they thought that NASA’s Destination Tomorrow™ was successful in educating the public about what NASA does. The mean value for this question is ( x = 9.23). Respondents were then asked if the information NASA’s Destination Tomorrow™ contained was credible. The mean value for this question is ( x = 9.53). Finally, survey participants were asked an “open-ended” question: “Q. 24: Can you recommend anything that would make NASA’s Destination Tomorrow™ more useful to you and to your audience?” Responses appear as part of appendix B. The response most often recorded was “make more programs.” Section 4 – Questions about NASA’s Destination Tomorrow™ Web Site The following four questions pertain to NASA’s Destination Tomorrow web site. More than 80 percent of respondents were aware of the web site. (See fig. 12.) Almost 75 percent of those who were aware of the web site had actually visited it (see fig. 13), and 97 percent of the individuals who visited the web site said it was okay as is. (See figure 14.) 25. Are you aware that NASA's Destination Tomorrow™has a web site? No 67 Yes 336 0 100 200 300 400 Response, n Figure 12. Awareness of NASA’s Destination Tomorrow™ web site. 26. If yes, have you visited the web site? No 84 Yes 252 0 50 100 150 200 250 300 Response, n Figure 13. Respondents having visited the NASA’s Destination Tomorrow™ web site.

Original page 16

12 27. If yes, would you say the web site is okayas it is, or does it need improvement? Needs improvement 7 Okay as it is 0 50 100 Response, n 245 150 200 250 300 Figure 14. Respondents’ assessment of NASA’s Destination Tomorrow™ web site. Finally, survey participants were asked an “open-ended” question: “Q. 28: Can you recommend anything that would improve NASA’s Destination Tomorrow™ web site?” Responses appear as part of appendix B. Section 5 – Demographic questions The last eight questions pertain to the demographics of the individuals who participated in the survey. As shown in figure 15, almost 95 percent of individuals surveyed personally decided what to include in their station’s programming lineup. The mean value for years worked at the station was ( x = 8.0) and the mean value for years worked in television was ( x = 13.7). 29. Doyoupersonallydecide whichprograms to include inyour station's programminglineup? No 29 Yes 0 100 Response, n 374 200 300 400 Figure 15. Respondents selection of station programming. Question Mean Median Mode 30. How many years have you worked at this TV station? 8.0 6.0 5.0 31. How many years have you worked in television? 13.7 12.0 10.0

Original page 17

13 The most common job title held by the survey participants was that of Program Manager, which roughly 58 percent of participants held. (See Figure 16.) The least common job title was Video Technician, which about 11 percent of participants held. 32. What is your jobtitle? 233 250 200 150 100 Response, n 44 59 50 0 67 Program Video General Executive Manager Technician Manager Producer Title Figure 16. Respondents’ job titles. Respondents were asked how they would rate the quality of service the NASA Center for Distance Learning provides. The mean rating for this question was 9.46, which indicates a high rate of satisfaction among those who participated in this survey. Question Mean Median Mode 33. How would you rate the quality of service the NASA Center for Distance Learning provides? 9.46 10 10 About 2/3 (61 percent) of the survey participants were male. (See fig. 17.) Slightly more than 1/3 (39 percent) of the participants were female. A map of the United States identifies the location of the 403 television stations, by state, that are represented in this study (fig. 18). 34. What is your gender? Male Gende Female 155 248 0 50 100 150 200 250 300 Response, n Figure 17. Gender.

Original page 18

14 35. State in which your station is located? Figure 18. Geographic Distribution of Survey Participants. About 51 percent of the television stations were located in the Eastern Time Zone, while 26 percent were located in the Central Time Zone. (See fig. 19.) Almost 8 percent were located in the Mountain Time Zone, and 15 percent were located in the Pacific Time Zone. 36. In what time zone is your station located? 250 200 206 150 onse, esp100 R 104 50 0 Eastern Central Time Zone 61 32 Mountain Pacific Figure 19. Distribution of Survey Participants by Time Zone.

Original page 19

Presentation of the Quantitative Data 15 When asked to recommend anything that would make NASA’s Destination Tomorrow™ more “useful” to their audience, most respondents (294) offered none, stating instead that the series is “ok” the way it is currently produced. On the other hand, 76 respondents did request that more programs, beyond the five that are produced annually, be produced. Otherwise, the remaining comments were very specific and sometimes contradictory. For example: One respondent stated that the series is too technical while another indicated that the series needed to be more technical. Four respondents offered comments regarding the educational value of NASA’s Destination Tomorrow™. Viewed as a whole, these respondents recommended that educational (curriculum) materials be added to the series. A total of 28 respondents offered comments about the web site. A simple majority (15) indicated that the web site is great. Six indicated that the web site was visually appealing and easy to navigate. Five respondents requested “better” (we assume more) links from the NASA’s Destination Tomorrow™ web site to NASA web sites that would provide specific (and more) information about a particular segment. One respondent wanted the ability to download entire programs from the web site. Comparing Selective (Quantitative) Data from the 2002 and 2005 Surveys In 2002, a telephone survey of 400 (out of 537) television stations was undertaken by Continental Research Associates, Inc., a Norfolk-based marketing research firm (Pinelli and Perry, 2002). Telephone surveys were conducted between January 16 and February 21, 2002. The protocol and survey used in the 2005 study are derivatives of the protocol and survey used in the 2002 survey. Responses from the 2002 and 2005 surveys are presented for selected questions. • The most important factor affecting the decision to air a program on your station 2002 2005 o Educational value Educational value o Audience interest Cost o Technical quality Technical quality • The most appealing factor about NASA’s o Production quality Educational value Destination Tomorrow™ from a programming standpoint • Compared to other educational programming you air, the technical quality of NASA’s Destination Tomorrow™ is better than average • Would you say that NASA’s Destination Tomorrow™ is well received by your audience? • Is NASA’s Destination Tomorrow™ successful in educating people about what NASA does? o Educational value Cost o Interesting content Technical quality o 74 percent 91 percent o 65 percent 81 percent o x = 9.1 x = 9.2 • How credible is the information contained o x = 8.0 x = 9.5 in NASA’s Destination Tomorrow™?

Original page 20

• Have you visited NASA’s Destination Tomorrow™ web site? • Would you say the web site is ok as is or does it need improvement? • Gender? Female/Male • Station type Public access PBS 16 2003 2005 o 38 percent 75 percent o 90 percent 97 percent o 36/64 percent 39/61 percent o 95 percent 60 percent o 5 percent 9 percent *In 2005 the station mix had grown to include commercial, college, ITV, and other station types. • Job title – management • Years employed at that station? o 36 percent 57 percent o x = 6.9 x = 8.0 Comparing Selective (Qualitative) Data from the 2002 and 2005 Surveys When asked to recommend anything that would make NASA’s Destination Tomorrow™ more “useful” to their audience, 47 percent (187) of respondents offered none, stating instead that the series is “ok” the way it is currently produced. On the other hand, 22 percent (86) respondents did request that more programs be produced annually. While 83 percent of those surveyed reported that NASA’s Destination Tomorrow™ was very educational, 5 percent (19) respondents recommended that educational (curriculum) materials be added to the series. Otherwise, the remaining comments were specific with the number of respondents offering comments on a topic ranging between 1, 3, and 5. Examples of specific topics include (1) media format selection, (2) language preference (e.g., Spanish), (3) a desire for supporting educational materials, and (4) customer service issues/problems such as receipt of tapes through the United States Postal Service. A total of 151 respondents offered comments about the web site. About 90 percent (136) indicated that the web site is “fine as is.” Otherwise, the remaining comments were specific with the number of respondents offering comments on a topic ranging between 1, 3, and 5. Examples of specific topics include (1) the ability to download the programs, (2) the need for greater connectivity (e.g., ability to obtain more information about a specific topic/segment), (3) the desire to obtain a transcript of the programs, and (4) a request to add supporting (educational) materials to the series. A comparison of the 2002 and 2005 qualitative survey produces several noteworthy considerations. First, the stations airing NASA’s Destination Tomorrow™ want the number of programs produced annually increased. Most television stations require a guarantee of 13 programs (per season) before a series will be afforded a “scheduled” date and time in the programming lineup. (From a production standpoint, a regularly scheduled date and broadcast time slot is the “gold standard” in television programming.) Currently, five NASA’s Destination Tomorrow™ programs are produced annually. Combining four programs from the previous season with five newly produced programs constitutes a broadcast season for NASA’s Destination Tomorrow™. While the number of television stations airing the series has nearly

Original page 21

17 doubled since 2002, the number of programs produced is problematic. Although community access, ITV, and college television stations are less bothered with this current production arrangement, PBS and commercial television stations prefer to abide by the “industry standard” of 13 programs (per season) before a series will be accepted for broadcast and afforded a “scheduled” date and time in the programming lineup. Second, most of the “customer service” problems have been eliminated by (1) adding (hiring) a customer service representative and (2) obtaining an external source to handle tape duplication and mailing. In fact, on a 1 – 10 point scale, with 10 being exceptionally good service, 2005 survey participants rated the overall quality of service offered by the NASA Center for Distance Learning as exceptional ( x = 9.5). Third, issues associated with “streaming” and downloading programs in the NASA’s Destination Tomorrow™ series have diminished as the programs are now available from servers at the following sites: the California State Polytechnic University Pomona, South Carolina Educational Television (SC ETV), the Apple Learning Interchange (ALI), and Ibiblio (University of North Carolina, Chapel Hill). Fourth, as part of an overall effort to involve all Americans in NASA discovery, exploration, innovation, and research, programs in the series are (1) now 508 compliant, and (2) are being translated into Spanish. Fifth and last, is the issue of adding educational (curriculum) resources as a program component to NASA’s Destination Tomorrow™. While the addition of educational resources would increase the utility of the series and possibly make it more likely to be used by those involved in the education of adults and possibly high school and community college, the addition brings with it significant cost, and while that cost could be significant, the addition might provide a more direct measure of determining if NASA’s Destination Tomorrow™ is accomplishing its goal – “Increased scientific literacy among adults.” Recommendations 1. Should more programs be produced? Assuming that content is available and the desire on the part of the Agency to involve all Americans in NASA discovery, exploration, innovation, and research, programs remains an important goal, a cost/benefit analysis should be conducted to determine if the cost associated with increasing the number of programs each season would be offset by the increased number of commercial and PBS television stations that would now air the programs and the corresponding increase in the potential audience. 2. Should an instructional component be added? Would the utility of the series be increased by adding an instructional component? At this point a survey of the (potential) user community should take the place of anecdotal evidence. User input aside, its unlikely that the addition of an instructional component would diminish the appeal of the series as a televised product. That said, NASA would need to determine (1) if the addition of an instructional component strengthens the Agency’s overall educational mission and objectives, and (2) if a cost/benefit analysis should be undertaken (conducted) to determine if the cost associated with adding an instructional component would be offset by the potential increased use of the series by educators.

Original page 22

18 3. Should the programs be translated into Spanish? A strong case can be made for linguistically and culturally appropriate Spanish-language educational programming. Latinos constitute the largest and fastest growing minority in the United States. Despite their increasing numbers, Latinos constitute the smallest minority group in the nation’s (overall) scientific and technical workforce. As long as the Agency continues to seek ways to involve all Americans in NASA discovery, exploration, innovation, and research, linguistically and culturally appropriate translations make good political sense. Following the axiom that “the parent is the child’s first teacher,” translating the series into Spanish might help to provide a venue for Latino parents to become more involved in the “mathematics, science, and technology” portions of their children’s education. Theorists and practitioners, alike, advocate that for young non-English speakers, math and science instruction be undertaken in their native language. Lastly, as the number of “language emersion” programs increases in America’s schools, so, too, does the need for quality language programming. Conclusions Earlier we stated that for NASA’s Destination Tomorrow™ to be considered a success, the following criteria would have to be met: 1. The series would have to obtain an overall average survey score of 4.30 (using a 5-point Likert scale). On a 10-point scale, survey participants rated the overall technical quality of NASA’s Destination Tomorrow™ high ( x = 9.48) and the educational value of the series slightly higher ( x = 9.56). Ninety one (91) percent of the participants reported that the technical quality of NASA’s Destination Tomorrow™ was higher compared to other educational programming airing on their station. Most stations (81 percent) indicated that NASA’s Destination Tomorrow™ was well received by their audiences, and 97 percent indicated that they had recommended or would recommend the series to a colleague. Lastly, using a 10-point scale, survey participants indicated that (1) the series successfully educates people about what NASA does ( x = 9.23), (2) the information contained in NASA’s Destination Tomorrow™ is credible ( x = 9.53), and (3) the series is successful in educating the public about what NASA does ( x = 9.23). 2. Registered stations would continue to use this product. Since we started registering television stations to air the series, not one single (registered) station has stopped airing the program. To the contrary, stations continue to ask that the number of programs produced annually be increased. 3. The number of registered television stations would continue to grow. The number of domestic television stations airing the series has nearly doubled since 2002. 4. Registered stations would recommend NASA’s Destination Tomorrow™ to other stations. About 97 percent (403) of the survey participants in the 2005 survey stated that they have or would recommend NASA’s Destination Tomorrow™ to a colleague.

Original page 23

19 5. NASA’s Destination Tomorrow™ would fare well in the highly competitive, multimedia education awards competition. NASA’s Destination Tomorrow™ is an Emmy®-award winning program. Additionally, the series has won the coveted New York Film Festival award and numerous others. For a complete list of awards, please visit http://destination.larc.nasa.gov/awards.cfm. By these and any other reasonable criteria, NASA’s Destination Tomorrow™ is a success. Evaluation is both ongoing and dynamic. In future evaluations, an assortment of other tools should be considered as part of an ongoing attempt to determine the extent to which NASA’s Destination Tomorrow™ does (1) create and heighten adult interest in mathematics, science, technology, and NASA; (2) increase the scientific and technological literacy of adults; (3) improve the scientific and technological literacy of adults who do not use English as their primary language; and (4) serve as a mechanism for parents and caregivers to become involved in the education of children and young adults. References American Association for the Advancement of Science (AAAS). Project 2061: Science for All Americans, A Project 2061 Report on Literacy Goals in Science, Mathematics, and Technology. Washington, DC: AAAS. 1989 (AAAS Publication 80-01S). ISBN: 0-87168-341-5. Pinelli, Thomas E. and Jeannine Perry: 2002. Evaluating the Effectiveness of NASA’s Destination Tomorrow™ 2000–2001 Program. Washington, DC: National Aeronautics and Space Administration. (Available at http://ntrs@nasa.gov)

Original page 24

20 Appendix A Appendix A contains the telephone protocol that was used for the 2005 NASA’s Destination Tomorrow™ survey. Hello. Is _______ available? Hello ________, my name is Scott Endo. I’m calling from the NASA Center for Distance ™ Learning to conduct a survey regarding NASA’s Destination Tomorrow . The survey should take about 20 minutes. Do you have time now to participate? ______(Yes) Great. ______(No) What is the best date and time to call you back? _______________________________ First, I would like to ask you “5” questions about your station. 1. What kind of programming does your station use? A. Commercial B. Community Access – Education C. Community Access – Government D. ITV E. PBS F. College/University G. Other (Please specify.) ____________________________ 2. According to our records, your audience size is ___________________________. (If not correct, change the records.) 3. What type of media do you prefer? A. Digital B. Tape i. DVD i. Beta-SP ii. DVCAM ii. SVHS iii. DVC PRO iii. VHS iv. Mini DV iv. ¾ in. 4. Does your station have satellite (downlink) capability? ______ No ______ Yes OMB Approval Number: 2700-0012

Original page 25

(If no, skip to Question 6.) 21 5. What percentage of your programming is downlinked via satellite? _______% Next, I would like to ask you “7” questions about how your station selects programs. Would you please rate each answer on a scale of 1 to 10, with 1 being not at all important and 10 being extremely important? 6. How important is program cost? Not at all important 1 2 3 4 5 6 7. How important is the program’s technical quality? Not at all important 1 2 3 4 5 6 8. How important is easy program acquisition? Not at all important 1 2 3 4 5 6 7 8 9 10 Extremely important 7 8 9 10 Extremely important 7 8 9 10 Extremely important 9. How important is the program’s educational value? Not at all important 1 2 3 4 5 6 7 8 9 10 Extremely important 10. How important is having few restrictions on program use? Not at all important 1 2 3 4 5 6 11. How important are viewer recommendations? Not at all important 1 2 3 4 5 6 7 8 9 10 Extremely important 7 8 9 10 Extremely important 12. Which of the following considerations is the single most important attribute that affects your decision to select a program to air on your station? ______ Cost ______ Technical Quality ______ Ease of Acquisition ______ Educational Value ______ Few Program Restrictions ______ Viewer Recommendations

Original page 26

22 ™ Your station is registered to receive NASA’s Destination Tomorrow . Next, I would like to ™ ask you “16” questions about NASA’s Destination Tomorrow . ™ 13. Are you familiar with NASA’s Destination Tomorrow ? ______ No ______ Yes ______ Not really sure ™ 14. Have you watched at least part of a NASA’s Destination Tomorrow program? ______ No ______ Yes ______ Don’t know ™ 15. On your station, is NASA’s Destination Tomorrow generally aired in the morning, afternoon, evening, at night, or at some combination of times? ______ morning ______ afternoon ______ evening ______ at night ______ combination of times ™ 16. How would you rate the overall technical quality of NASA’s Destination Tomorrow on a scale of 1 to 10, with 10 being the highest? Very poor quality 1 2 3 4 5 6 7 8 9 10 Extremely good quality ™ 17. How would you rate the overall educational value of NASA’s Destination Tomorrow ? No educational value 1 2 3 4 5 6 7 8 9 10 Extremely high educational value

Original page 27

23 18. From a programming standpoint, which of the following attributes of NASA’s Destination ™ Tomorrow appeal most to you? ______ Cost ______ Technical quality ______ Ease of acquisition ______ Educational value ______ Few program restrictions ______ Viewer recommendations 19. Compared to other educational programming your station airs, is the technical quality of ™ NASA’s Destination Tomorrow series ______ Better than average ______ About average ______ Worse than average ™ 20. From an audience perspective, would you say that NASA’s Destination Tomorrow has been ______ Very well received ______ Somewhat well received ______ Not well received ™ 21. Would or have you recommended NASA’s Destination Tomorrow to a colleague? ______ No ______ Yes ™ 22. One goal of NASA’s Destination Tomorrow is to educate people about what NASA does. How successful has NASA’s Destination Tomorrow™ been in that regard? Not at all successful 1 2 3 4 5 6 7 8 9 10 Extremely successful

Original page 28

24 ™ 23. In your opinion, how credible is the information NASA’s Destination Tomorrow contains? Not at all credible 1 2 3 4 5 6 7 8 9 10 Extremely credible ™ 24. Can you recommend anything that would make NASA’s Destination Tomorrow more useful to you and to your audience? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ ™ 25. Are you aware that NASA’s Destination Tomorrow has a web site? ______ No ______Yes 26. If yes, have you visited the web site? ______ No ______ Yes 27. If yes, would you say the web site is ______ Okay as it is ______ Needs improvement 28. If yes, how could we improve the web site? Lastly, I’d like to ask “7” questions about you. 29. Do you personally decide which programs to include in your station’s programming lineup? ______ No ______ Yes 30. How many years have you worked at this TV station? _________________________ 31. How many years have you worked in television? _________________________ 32. What is your job title? _________________________

Original page 29

25 33. How would you rate the quality of service the NASA Center for Distance Learning provides? Very poor quality 1 2 3 4 5 6 7 8 9 10 Extremely good quality 34. Your gender: ______ Female ______ Male 35. Your state: 36. Your time zone: Is there anything else you’d like to tell me? (If not) Thank you very much for your time.

Original page 30

Appendix B 26 Appendix B contains a compilation of responses that were received for questions 24 and 28. The majority of respondents who did submit a comment complained about the small number of shows produced per season. Q 24. Can you recommend anything that would make NASA’s Destination Tomorrow™ more useful to you and to your audience? Comment No comment. Produce more shows. Make trailers for future shows. Produce shows 1 hour in length. Number of responses 294 76 7 5 The satellite feed is hard to get/feed time online is different than actual feed time. 5 Make show so it fits educational standards. Make a college level show. Add a Chicago element to the program. Cast is too young for our audience. Format a time slot for commercials. I prefer only one topic per show. 3 2 1 1 1 1 Make the show more appealing to a younger audience. 1 More blank space at the beginning and end. NASA should offer grant money for a satellite. Provide educational resources with the show. Provide NASA news updates online. The show is too technical. The show needs more scientific detail. Q. 28. If yes, how could we improve the web site? Comment The web site is great. 1 1 1 1 1 1 Number of responses 15 I would like better links from the NASA site to the Destination Tomorrow™ site. 5 The web site is visually appealing. The web site is easy to navigate. 4 2 I would like to be able to download shows from the web site. 2

Original page 31

Form Approved REPORT DOCUMENTATION PAGE OMB No. 0704-0188 The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS. 1. REPORT DATE (DD-MM-YYYY) 2. REPORT TYPE 01- 10 - 2005 Technical Memorandum 4. TITLE AND SUBTITLE 3. DATES COVERED (From - To) 5a. CONTRACT NUMBER Results of a Television Station Managers' Telephone Survey of NASA's Destination Tomorrow™ 6. AUTHOR(S) Endo, Scott; Pinelli, Thomas E.; and Caton, Randall H. 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) NASA Langley Research Center Hampton, VA 23681-2199 5b. GRANT NUMBER 5c. PROGRAM ELEMENT NUMBER 5d. PROJECT NUMBER 5e. TASK NUMBER 5f. WORK UNIT NUMBER 23-079-99-OE 8. PERFORMING ORGANIZATION REPORT NUMBER L-19187 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR'S ACRONYM(S) National Aeronautics and Space Administration Washington, DC 20546-0001 12. DISTRIBUTION/AVAILABILITY STATEMENT Unclassified - Unlimited Subject Category 80 Availability: NASA CASI (301) 621-0390 13. SUPPLEMENTARY NOTES NASA 11. SPONSOR/MONITOR'S REPORT NUMBER(S) NASA/TM-2005-213936 Endo, University of California at San Diego; Pinelli, NASA Langley Research Center; Caton, Christopher Newport University An electronic version can be found at http://ntrs.nasa.gov 14. ABSTRACT We conducted a television station managers' telephone survey concerning NASA's Destination Tomorrow™. On a 10-point scale, survey participants rated the overall technical quality of NASA's Destination Tomorrow™ highly (mean = 9.48), and the educational value of the series slightly more highly (mean = 9.56). Ninety one percent of the participants reported that the technical quality of NASA's Destination Tomorrow™ was higher compared to other educational programming that airs on their station. Most stations (81 percent) indicated that NASA's Destination Tomorrow™ was well received by their audiences, and 97 percent indicated that they had recommended or would recommend the series to a colleague. Lastly, using a 10-point scale, survey participants indicated that (1) the series successfully educates people about what NASA does (mean = 9.23), (2) the information contained in NASA's Destination Tomorrow™ is credible (mean = 9.53), and (3) the series is successful in educating the public about what NASA does (mean = 9.23). 15. SUBJECT TERMS Distance learning; Program assessment; NASA educational programs; Survey; NASA's Destination Tomorrow™ 16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF ABSTRACT a. REPORT b. ABSTRACT c. THIS PAGE U U U UU 18. NUMBER 19a. NAME OF RESPONSIBLE PERSON OF PAGES STI Help Desk (email: help@sti.nasa.gov) 19b. TELEPHONE NUMBER (Include area code) 31 (301) 621-0390 Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std. Z39.18