Overview of Formative, Process, and Outcome Evaluation Methods Used in the VERB™ Campaign

Methods to Develop the Brand 

Using the literature review as a starting point, 48 interviews were conducted with triads of tweens, six focus groups with parents of tweens, two focus groups with adults who work with tweens (i.e., influencers), and eight in-depth interviews with industry professionals to gather additional information to help formulate the campaign's overarching message strategy. A local recruiting firm in each city was hired to place recruiting calls, screen tween participants using a pre-qualification questionnaire, and secure parental approval for their participation. Similar screening occurred for parents and adult influencers of tweens. Confirmation cards were sent or follow-up calls were placed to remind individuals of their commitment to participate. Participants were paid an honorarium for their time to secure their participation, ranging from $50 to $75, depending on the market location. All interviews were audio-taped. Data were analyzed by hand using observer notes and transcripts to identify themes that emerged across interviews. Unless otherwise noted, similar methodology was used for all qualitative research described in this paper.

During these interviews, tweens were asked what they did for fun in their free time. They were then shown several concepts for the campaign itself and asked what they liked and disliked about the concept and whether their reaction to the concept was positive or negative. Parents and influencers were asked about strategies they had used successfully to motivate their tweens to engage in healthy behaviors. Industry professionals were asked to describe what they have found to work in motivating tweens. Based on these interviews, data suggested that the campaign should focus on helping tweens enhance their self-esteem and self-confidence by participating in fun activities where they can achieve success. Messages for tweens should focus on helping tweens discover their passion and promote the idea that everyone is good at something as an important motivational message. Messages for parents should focus on how they, as parents, could help their tweens discover their passion and should remind parents that their involvement is important in helping their kids to discover what they enjoy and are good at and that this will contribute to the development of the child's self-esteem. A full report can be found at www.cdc.gov/youthcampaign/research/PDF/MessageStrategy.pdf.

Subsequently, advertisements and brands targeted to tweens were analyzed, and a semiotic analysis9 of advertisements was conducted to learn the types of images that are associated with tweens changing their behavior. The semiotic analysis involved an examination of visual and verbal conventions within advertisements to learn what they were and what they presupposed about the experience of being a tween. Analysis was based on ads for Nike, Sony, Nintendo, Tommy Hilfiger, Mountain Dew, Cheetohs, Nickelodeon, MTV, Teen People, and Sports Illustrated for Kids. The analysis revealed advertising targeting boys differed from that targeting girls, that a main message across tween-targeted advertising was how this audience was unique and distinct from adults, and the blurring of the boundaries between virtual reality and the physical environment.

Similar to the semiotic analysis, a hedonic analysis10 was conducted to learn which elements of brands resonated with tweens and intensified their enjoyment. The hedonic analysis examined brands to understand which elements, positive and negative, inspire tween affinity from a pleasure-seeking perspective. The analysis focused on the brands that played a critical role in youth culture (e.g., Shrek, Harry Potter, Nike, Teen People, Nickelodeon, Mountain Dew), led to sustained tween affinity, and with which tweens identified themselves. The analysis also sorted images into those appropriate for boys and girls of different age groups. Key motivators included narcissism, power and dominance, control and empowerment, adventure, camaraderie, and the creation of games in one's own mind.

Drawing on this information, the campaign's advertising agencies conducted formative evaluations from December 2001 through March 2002 to inform the development of the campaign's branding and program activities. The overall goal was to gain insight into youth culture and to develop a brand that resonated with tweens. Specifically, the purposes were to understand: (1) the cultural, ethnic, and economic dynamics that unify and differentiate tweens; (2) the motivators for and barriers to tweens' participation in physical activities; and (3) the motivators for and barriers to parental support of tweens' participation in physical activities.

Data were collected through qualitative methods. Twenty-five in-depth interviews with parents and tweens were conducted in their homes, lasted approximately 3 hours (2 hours with the tween, 1 hour with the parent), and were videotaped. Participants included Caucasians, African Americans, Asians, Hispanics, and Native Americans. During the interviews, tweens were observed for their after-school experience and motivations; values and attitudes were discussed, and free-time activities explored. The parent interviews explored their impressions of their tweens' activities, goals, and concerns and their feelings about their child's after-school routine. An additional 25 tween–parent dyads were asked to keep a diary depicting the tween's after-school experiences expressing their ideas, feelings, hopes, and fears through words and pictures. When necessary, follow-up phone interviews were conducted to expand on what was in the diaries.

To ensure that the campaign explicitly reached ethnic audiences that were characterized by demographics, such as race and ethnicity, length of time in the U.S., language spoken at home, and income level, an additional 12 focus groups were conducted with African-American tweens, four focus groups with Asian tweens, nine in-depth interviews with Hispanic tweens, and eight in-depth interviews with Native American tweens exploring the same issues. The same questionnaire was used to identify motivators and barriers.

Across these methods, data suggested that tweens were likely to engage in physical activity if they believed that they would have fun with friends, were exploring something new, and would not be judged. Barriers to being physically active included time constraints, lack of access to facilities, dislike of competition, appeal of other competing activities, and fear of embarrassment. Detailed reports on the brand development evaluation are found at: www.cdc.gov/youthcampaign/research/.

Once motivators and barriers were identified, an ideation session was held in early February 2002 to elicit brand and programming ideas from tweens. Twenty-four tweens participated in three separate focus groups to brainstorm ideas on how to get tweens their age more physically active. VERB's creative agencies used these and the findings from 16 additional focus groups and 19 in-depth interviews with tweens, four focus groups and 17 in-depth interviews with parents, and eight in-depth interviews with adult influencers to brainstorm ideas for the brand and develop brand concept ideas (called positionings). Twelve different positioning statements were generated that articulated how tweens should think and feel toward the brand.

Ten focus groups took place in March 2002, in which tweens expressed their views of positioning statements. Through this process, creative agency staff learned what worked and what did not work (e.g., staff learned whether a selected positioning statement would motivate kids to get active) and gathered the information necessary to refine their ideas and develop concepts for the brand.

The creative agencies' first step in concept development was to use the results of the positioning focus groups to generate a one-word description of the core message of the campaign, the brand essence. The positioning that proved most motivational was lift, suggestive of the act of rising or elevating and the exhilaration of flying or expanding ones' views. Statements of brand character were subsequently developed, progressing later to full brand concepts. Several concepts were developed, including VERB. It's what you do! Concept development was completed in April 2002.

Message testing was the last step, completed during the week beginning April 22, 2002. Feedback on two brand concept possibilities to assess brand clarity, relevance, acceptability, and credibility was obtained through 16 in-person dyadic interviews with tweens and eight telephone focus groups with parents. On the basis of findings from message testing, prior concept testing, and recommendations from the creative agencies, CDC selected as the brand name and concept VERB.™ It's what you do! 7

Methods to Develop VERB Advertising 

Once work to develop the brand was complete, the next step was to develop advertisements to create an awareness of VERB. Evaluation involving participants from the target audience was a critical component of the VERB campaign's development; such testing drove the content, tone, and execution of VERB advertisements and promotions. Developing effective messages requires a deep understanding of the audience. This understanding is attained through a process of qualitative inquiry consisting of individual and small-group interviews with representatives of the target audience. VERB used a three-step process, drawing on qualitative methods.11, 12, 13, 14 For each step, data were collected through focus groups (held in person or by telephone), individual interviews, or dyadic interviews. Participants in the focus groups and interviews were divided into three categories: (1) children aged 9–13, (2) parents of children aged 9–13, and (3) adult influencers of children aged 9–13. Each focus group consisted only of people from the same category, and each dyadic interview was with two people from the same category. The process and methods for advertising and promotion development paralleled those used for brand development.

The first step was exploratory research in which the developers of advertisements learned what motivates tweens for the developers to be able to create a general communication strategy and advertisements that would appeal to tweens. During the campaign, ten focus groups, ten in-depth interviews, and 16 dyadic interviews were conducted with tweens, and six focus groups and four in-depth interviews were conducted with parents as part of the exploratory research component of the formative evaluation. Reports from the exploratory research are at www.cdc.gov/youthcampaign/research/. On the basis of exploratory research, the creative agencies developed concepts for advertising messages to promote physical activity to the target audience. Consistently, exploratory evaluation revealed that motivators for tweens to be physically active include being with friends, engaging in self-discovery, and participating in events with an element of competition. Barriers to being physically active included fear of failure, lack of time, family responsibilities, and competing interests.

The second step, concept testing, identified the message that best promoted physical activity to tweens. Concepts were tested to validate the communication strategy and to aid in selecting concepts that resonated strongly with the target audience and that were acceptable, understandable, culturally appropriate, and motivating. During the campaign, approximately 100 focus groups, dyadic interviews, and triadic interviews were used for concept testing. Participants were presented with up to three concepts for an advertising campaign and asked to respond to a set of questions about the advertisements themselves. Rough versions of the advertisements consisted of print and video storyboards, sketch drawings, and video animations. To reduce order-effect biases, the order in which the concepts were presented for advertisements and the examples for each concept were systematically varied: the order was changed from one focus group to another and from one interview to another. Interviews focused on participants' initial reactions, their likes and dislikes, their perception of its relevance to VERB's main message, and their perception of the advertisement's ability to motivate tweens to be physically active. Campaign planners used the information gathered during concept testing to assist them in selecting which advertising concept to produce for each phase of the campaign.

Next, a rough version of the advertisement was produced and evaluated through message testing. Over the course of the campaign, approximately 100 focus groups, dyadic interviews, and triadic interviews were conducted using procedures similar to those described for concept testing. During message testing, an assessment was made whether the audience interpreted the main message about physical activity as intended and whether anything was confusing or controversial. Participants were shown almost completed versions of the final advertisements and were asked for their initial reactions, their likes and dislikes, their perception of the advertisement's relevance to VERB's main message, and their perception of the advertisement's ability to motivate tweens to be physically active. Results of the message testing guided developers in improving the final version of the advertisement.

To document the findings, VERB staff reviewed the reports from all concept and message testing and generated a summary of the various themes that emerged across all stages of testing. Because the advertising agencies were able to build on the knowledge gained from each phase of testing during the campaign, a true content analysis was not possible. For example, a phase of testing might reveal that certain images or phrases elicited negative reactions from the audience segment; these images would not be tested in subsequent phases. Thus, two VERB staff members who had not been involved in the concept or message testing were charged with reading and analyzing independently each concept and message testing report by audience segment. For each report, each reviewer generated a list of key ideas. Once each segment was reviewed, the reviewers met to discuss and compare the lists, and a third reviewer was brought in to settle any disputes.

Results suggested that messages should promote the behaviors sought by a physical activity campaign for tweens by using images and audio cues that convey happiness, excitement, and fun about being physically active. Mothers responded favorably to messages about physical activity's contribution to a child's success in other areas of life. Details of the thematic analysis are found at www.ajpm-online.net.

During all stages of formative evaluation, standard methods in qualitative research were followed. As with the interviews and focus groups related to branding, participants were divided into three categories: children aged 9–13, parents of children aged 9–13, and adult influencers. Participants from only one category were in each focus group or interview. In addition, group discussions were segmented by demographic characteristics (e.g., gender, age) to facilitate conversation and to enable planners to develop messages that were effective and efficient for all audience segments.11, 15, 16, 17 For each formative investigation, campaign planners considered which segments of the target audience were most appropriate to provide feedback. Materials developed for specific racial, ethnic, or non-English-speaking groups were tested with people in those groups. For most interviews and focus groups, moderators were of the same racial or ethnic background as participants. Moderators were given the flexibility to adapt questions to match the participants' vernacular, to respond appropriately to the group dynamics, and to add questions to augment the depth of information gathered through the process. Sample recruitment guides and moderator guides are found at: www.cdc.gov/youthcampaign/research/PDF/FormativeResearchProcess.pdf.

Analysis and Interpretation of Qualitative Data 

Data analysis and the interpretation of the qualitative data for the formative evaluation followed processes that are described in the literature for exploratory investigations13 and have also been described for testing of concepts and messages.18

When the qualitative work was exploratory, such as when investigating the motivators and barriers to physical activity, the moderator analyzed the audiotapes for recurrent themes and patterns in the responses. The focus was on understanding the participants' personal experiences. For example, tweens consistently reported that they enjoyed being physically active; being active with friends was especially appealing. The moderator was careful not to quantify the responses, but rather looked for consistency of opinions and commonality across multiple groups. Opinions that were based on personal experience carried more weight than general and impersonal statements. The participants' comments were synthesized into a written report that summarized the salient issues and themes which were then supported in the report by verbatim comments.

All of the qualitative investigations were observed by multiple members of the staff working on the VERB campaign. Observers included the CDC VERB creative team and creative agency staff, members of the CDC evaluation team, and usually staff from the evaluation contractor. Observers were in a separate room behind a one-way mirror so that verbal and nonverbal responses of the participants could be assessed and later compared to the findings of the moderator. The moderator's findings of salient themes were presented in a draft report to all the observing team members. The next part of the data analysis was critical—to what extent was there concurrence among the moderator's findings and the assessments made by observers. Discussions were often needed to reach consensus on the interpretation of participants' verbal and nonverbal responses. Discussions were iterative with the goal of reaching consensus on the points that would drive the development of the messages and the subsequent advertisements.

Unlike the exploratory evaluation, the primary purpose of concept and message testing was to elicit opinions from the participants about their understanding of the message of the ad—Did they get it?—and whether participants thought the ad was engaging and would attract their attention. The moderator needed to elicit enough information from participants to enable the creative team to judge which concept or ad best met the communication goals for the phase of the campaign. Data analyses focused less on salient themes of experience, but rather on qualitative comparisons among concepts: Which concept was the strongest? Was there a clear winner? As described previously, multiple team members were involved in the process of interpretation of the participants' remarks. Data analyses and interpretation focused on how the advertisement could be fine-tuned to improve its clarity or meaning (e.g., words could be changed and certain images could be deleted as needed).