Setting up a product test

Since in the last article we discussed the importance of action standards in product tests, let’s remain with product tests. How does one set up a test design?

It’s quite easy, if one is dealing with just two or three products. In this case, you just test one product after the other; bearing in mind, of course, that you need to rotate the order of products from one respondent to the next in order to prevent positional bias. This procedure of exposing a respondent to a product and having him/her evaluate it before proceeding to the next product is known as a sequential monadic test design.

In practice, a simple rotated test design for three products would look like this:

So, when dealing with few test products, every respondent can test and evaluate every product. However, this becomes impractical when dealing with a large number of test products. In a central location test, you could not expect any respondent to taste and evaluate more than two or three different products. And yet, your client insists on having a large number of products evaluated…

The way out of this bind is a round-robin test design with matched cells, which will allow you to include a large number of products in your test.

Here is an example of a round-robin design:

Although you have only 20 respondents per cell; i.e. combination of tested products, in      the end  each product will have been tasted and evaluated by 80 respondents. And every possible combination of test products would have been accomplished. Your total sample would be 200 respondents.

Round-robin test designs go hand-in-hand with a matched cell approach.  You must ensure that the cells are as similar as possible in terms of respondent demographics; e.g. if you have 10 males and 10 females in the first cell, you should have that precise gender distribution in every cell… Other parameters could include socio-economic class or main brand usage, etc.

What is an “Action Standard”? And why should you have one?

What is an “Action Standard”?  And why should you have one?

Action standards are especially important in product tests. They determine the criteria, which will decide whether a product will be adopted or rejected; i.e. action standards serve as benchmarks for product performance. As such, they come into play whenever a product is re-formulated or in line-extensions.

Product development is always motivated by business objectives. If one didn’t have a clear objective or goal in mind, why bother developing a new product? As the objective is known only to the client, it is the client who needs to set the action standard; i.e. identify the criteria that will lead to adoption or rejection of a prototype.

Superiority & Parity

Two concepts are important in action standards:

  • Superiority: the product must perform significantly better on a number of parameters than either the current version or competitors’ products
  • Parity: the product must perform as well as the current of competitors’ products

In both instances, the parameters of product performance need to be well defined; e.g. taste, fragrance, overall acceptability, etc. Again, it is the client who needs to identify those parameters.

Level of significance

Statistical significance tests are used to determine whether a product is rated superior, at par or inferior by consumers. Three levels are commonly reported on by research agencies:

  • 90% Confidence level: 10 times out of 100 tests the results may not be significant
  • 95% Confidence level: 5 times out of 100 tests the results may not be significant
  • 99% Confidence level: only 1 time in 100 tests will results not be significant.

While we report all three levels in our reports to the client, only one level is used to serve as benchmark. The standard benchmark is the 95% level of confidence. Why would a lower level of confidence be adopted if the 99% level seems so much more accurate?

There are two risks in setting benchmarks:

  • The benchmark is too low; the product appears like a winner – but goes on to flop in the market place
  • The benchmark was set too high; the product is rejected – although in the market place, it might have performed very well; this is a case of missed opportunities…

The 95% level of confidence is neither ridiculously low nor impossibly high; hence it is the most common benchmark

Deciding on Superiority or Parity

As mentioned above, business goals will determine whether superiority or parity are aimed for. A few examples will demonstrate this.

  • Assume a scenario, in which a product is being re-formulated in order to lower production costs, thereby increasing margins. The re-formulated product does not need to be better; it just needs to perform as well as the current formulation in order to retain its franchise. Hence, the benchmark will be parity.
  • If, on the other hand, a product is re-launched with claims of “bigger and better” or “new and improved”, the development version clearly must perform better than the current formulation; hence, superiority will be postulated.
  • A product has been reformulated with the aim of cutting into competitor’s franchise. In this case, we would aim for parity among current franchise and superiority among the competitor’s franchise.


Action standards are a crucial element determining the success of product tests. They must be in line with business goals. No research agency can – or should – second-guess a client’s business objectives. Hence, it is important for the research team on the client’s side to liaise with the marketing and R&D teams in order to formulate meaningful action standards that not only conform with the client’s business strategy, but can then be passed on to the research agency to inform the research process.

ESOMAR Global Prices Study 2010 – Africa Highlights

Every two years, ESOMAR conducts a global survey on research prices. Participating countries and/or agencies submit dummy quotes for a number of hypothetical research projects. For example:

…the standard specifications for a U&A, using face to face interviewing, are as follows:

  • Topic: Chocolate Confectionaries
  • National sample (n=500)
  • Respondents: regular users of chocolate
  • Interview length: 25 minutes
  • Deliverables: tables

…while the specs for FGDs:

  • Topic: Retail Banking Services
  • Number of Groups: 4 groups in 2 cities (2 male / 2 female)
  • Length: 2 hours
  • Deliverables: personal presentation and audio recordings

Included in the survey are other research products and approaches such as: online, CLT, face to face, telephonic, etc.   As most of these data collection methods are not (yet) relevant in our African markets, this evaluation will focus on the traditional U&A and focus groups.

Global Perspective:
The global average (median) cost for an in-home U&A is $17,580; the average cost for FGDs, $10,412.

So, where does Africa stand?

U&A studies in West & East Africa are 22% more expensive when compared to this global average. However, compared to Southern Africa, these regions are practically a bargain.  A U&A in Southern Africa is 73% more expensive than the global average and almost approaches Western European levels.

In contrast, in East and West Africa our FGDs are 12% less expensive than the global average.  In Southern Africa, though the difference is less extreme, focus groups still come in 34% above the global average.

The only sub-region in Africa in which research costs fall below the global averages is North Africa, with U&As being 13% and FGDs 35% cheaper.

Thus, project costs across West and East Africa are at an uneasy point of compromise: higher than our clients would like them to be; and lower than we need them to be if we want to continue building up local capacities and resources and developing a truly world class research industry in Africa.


Trend: Global Prices Study 2007 -  2010
Comparing prices for a U&A study across the four African countries represented in both waves of the study, South Africa would seem to have the least increase in research cost, Kenya the highest (with Nigeria not lagging far behind). However, in order to know why we’re not laughing all the way to the bank, one would need to figure in the massive devaluation of the US Dollar that occurred within that period and further the erratic fluctuations in the exchange rates of Egyptian Pound, Kenya Shilling, Nigerian Naira and South African Rand…

Factors Influencing Cost of Research in West and East Africa:

So why is research in Africa so expensive?  The key contributors (as identified by members of East Africa’s local research body but relevant across Africa) include:


  • Political & social instability
  • Project specific research permits required by many governments
  • Complex / ambiguous tax laws
  • Government corruption


  • Lack of utilities / basic infrastructure
  • Low internet penetration
  • Poor telecommunications
  • Crime / insecurity
  • Inaccessible rural areas


  • Booming / fluctuating demand
  • Currency fluctuations
  • Inflation
  • High staff costs / turnover / training
  • High travel costs
  • High communication costs
  • Corruption


  • Multiple local  languages
  • Lack of education / literacy
  • Some markets view research suspiciously


  • Lack of experienced / qualified staff
  • Small pool of researchers with high turnover
  • Briefcase researchers giving research a bad name
  • Poor secondary data resources
  • No syndicated studies / panels
  • High price / availability of technology


  • Hard to reach groups
  • Top-end – too much security so hard to access
  • Bottom end – no security for interviewer
  • Rural – poor road access
  • Some markets difficult to access (visas, direct flights, work permits)

After evaluating all of these it is easy to see where the money goes and why the African research agency struggles to grow.  Having little control over issues such as infrastructure or government policy, researchers need to direct their attention to what they can change.  For example:

  • Focus on bringing more young people into the industry and building research as a credible career
  • Work with universities to build research skills and create awareness of research as a career option
  • Build accreditation schemes for research staff at different levels to encourage professionalism and quality control
  • Build stronger relationships with clients and be willing to educate the clients on the realities of Africa

Highlights prepared by Margit Cleveland and Jane Delorie

The Potential for Social Media Marketing in Nigeria

Although Internet penetration in Africa as a whole is still at a low 8.7% (85 million users), growth rates are phenomenal; between 2000 and 2009, internet penetration grew by 1,810%. In Nigeria, in the same period of time, internet penetration grew by almost 12,000%. Internet penetration stood at 16.1% by the end of 2009; there are now almost 24 million Nigerians surfing the web. In fact, 27% of all African internet users are Nigerian. With fibre-optic cables coming online, this growth is expected to continue. But already the internet community in Nigeria exceeds the population size of Ghana; thus, marketers should seize the opportunities opened up by this new medium.

Connectivity remains an issue, however. Wireless connections (71%), mobile devices (30%) and GSM modems (27%) are the most commonly used modes of access. Cable, such as ISDN or DSL, or Vsat remain the exception. Dial up connections are still in use; as are cyber cafes and business centres, which also often make use of dial up connections.

Even without the frequent power cuts, connections tend to be erratic and unstable. Users complain that they spend more time establishing a connection than they do actually surfing the web. Another complaint is the high cost of accessing the internet, especially on mobile devices.

Despite the obstacles, Nigerians are enthusiastic about the internet: 82% connect at least once a day; 25% are connected around the clock. A minimum of two hours is spent surfing the web per day.

Being online has profound effects on users. For one, the internet is perceived as a valuable resource: knowledge, entertainment, opportunity to make friends and expand one’s network, and, last but not least, opening up new career opportunities.

But perhaps more importantly, the internet is perceived as liberation from social norms and strictures; i.e. the internet opens up an opportunity for self-actualisation without fear of censure. Curiously, users are content replacing their offline friends with an ever-growing circle of online friends and connections. In the long run, this will undoubtedly lead to a questioning of social norms that guide and confine social interaction in traditional society – in short, modernisation and cultural change from within.

Slow connections had a definite effect on browser shares. Microsoft’s Internet Explorer is used by 51% of the global internet community; in Nigeria, it comes in at just 28%. The leading browser is Mozilla’s Firefox with a share of 45%. Google Chrome attains 20% in Nigeria vis-à-vis just 8% global share. Not only are these alternative browsers faster than Internet Explorer; they can be customised to block out advertising and heavy Flash animations. This, of course, has implications for online marketing, as traditional advertising with banner ads and other adaptations of traditional print advertisements will have limited reach. Online marketing in Nigeria, however, could be rendered effective by embracing the concepts of social media marketing; i.e. initiating a dialogue with members of the online community on popular platforms like Facebook.

Participating in social networking is the single most important activity among Nigerian internet users: 95% are members of one or more social networks. Among these, Facebook is the most important with 93%. Twitter and Linked In achieve 22% each. What is the potential of a social networking strategy for engaging consumers in a dialogue with marketers? To this end, we conducted a Facebook monitoring exercise, recording activities on our private pages for a period of one week.

At first glance, the results are disappointing: the bulk of activities revolves around status updates and announcements. Facebook is used to keep tabs on one’s circle of friends and to enlarge it. Communication then occurs via instant messaging. We found no evidence of Nigerian consumers engaging with brands or corporate entities. A dialogue has not begun. Does that imply that Nigerians do not engage at all?

When Goodluck Jonathan launched his profile on Facebook, the response was overwhelming: more than 100,000 Nigerians have become his fans, posted tens of thousands of comments in response to a mere handful of presidential posts and updates. Collateral Facebook groups have already been established. Other examples of Nigerians engaging can readily be found. As such, Facebook already has become a valuable resource for monitoring the political mood in the country, as we literally hear “the voice of the people.” So, why do we not see a similar development for consumer engagement? The answer seems to be that marketers have not even begun to explore the opportunities of social media marketing.

Giants in the FMCG or telecoms industries may well maintain global corporate profiles and/or brand pages on Facebook; yet, the point would be for companies operating in Nigeria to seek the dialogue with Nigerian consumers. We checked the Facebook presence of Cadbury’s Bournvita, Nestlé’s Milo, Cowbell, Peak Milk, Globacom and MTN. Although MTN maintains a local business page on Facebook, this page suffers from blatant neglect. Globacom’s local business page is more successful in providing content; it, therefore, attracts more fans than MTN’s page. And yet, Globacom has only taken first steps toward social media marketing: official website and social media pages are poorly integrated, as if Globacom management still needed convincing of the potential of social media marketing. The very fact that Nigerian internet users have taken the initiative to establish an unofficial Globacom fan group, however, demonstrates consumers’ readiness to engage.

At this point, it is well worth reiterating that the size of the Nigerian online community is 24 million; thus exceeding the entire population of Ghana or Cameroon; i.e. markets that would not be ignored in traditional marketing. There are opportunities for social media marketing.

The success of using social media would not be measured in direct sales; rather, in the quality of engagement and user experience, ultimately resulting in enhanced loyalty. Unlike traditional marketing with unidirectional advertising, social media platforms enable dialogue. Therefore, brands are no longer positioned by marketers; brand positioning evolves interactively through a process of co-creation. Social media marketing at its best involves consumers in the process of rejuvenating brands and maintaining brand health. This, of course, requires re-orientation on the side of marketers.

In the final analysis, the question is not whether Nigerian consumers are ready to engage, but whether Nigerian marketers are ready and willing to enter a meaningful dialogue with consumers.