A Better Mousetrap? New Option for Fundraising Direct-Mail Testing

September 15, 2011 By Margaret Battistelli Gardner

5

Here's an interesting development from the world of direct-mail testing for acquisition. The folks at DonorTrends have come up with a process called DonorVoice, which allows organizations to test various elements of their direct-mail packages — without having to actually send out the packages.

The process requires three relatively simple steps on the part of the organization:

Choose which elements of the package to test: envelope, letter, front- or back-end premium, or any combination thereof. According to press materials on DonorVoice, it's best to test a single element "if you need to nail every detail of it because of its importance or because of all the options being considered. Or choose multiple elements if you need a more global read on which element matters more to overall response."
Identify which "ingredients" to test: images, message, teaser, size, colors, offers, etc.
Fill in a grid outlining the options to be tested for each ingredient.

From there, DonorVoice creates package variations to test subsets of all possible combinations, then sends an invitation to an external panel of donors who look like the acquisition target for the campaign, providing those donors with a link where they can view and rate the samples. The donor panel is made up of "double-opt-in survey respondents drawn from a pool of direct-mail-responsive, recent donors with an affinity to your cause."

When the process is complete, the organization gets a straightforward evaluation of results that assigns numerical ratings to each of the variations tested, scoring them against the control. It also has the ability to drill down into the results to see how each variation scored against the control.

The idea behind the process is that by harvesting test results without having to actually mail out test packages, organizations are able to roll out winning packages sooner and more inexpensively than with traditional testing, dedicate more resources to packages that are statistically more likely to perform better, and have more incentive to be innovative since testing is less costly and cumbersome.

You can get more information here.

So, what do you think? Does this sound like a good idea, and do you think it's a viable alternative to traditional direct-mail testing? Let us know in the comments.

Page 1 | 2

5

MORE ON
DATABASE / DONOR RELATIONSHIP
DIRECT MAIL

MORE ON DATABASE / DONOR RELATIONSHIP >>

FROM THE BOOKSTORE

Hitting the Email Inbox

Hitting the Email Inbox covers all the bases of email deliverability, including everything from the very basics, to reputation management, as well as coding and design, and the connections between email deliverability and ROI.

As any good email marketer today knows, there are a number of obstacles standing between you

...

ORDER NOW

Email Marketing that Works (2nd Edition)

Available as a PDF.

A guide to prospecting, lead generation, building an Opt-in database, tracking, social media integration, deliverability, mining content and balanced creative. While email marketing has reached maturity, there’s still plenty of life in this channel — if used wisely.

That’s the focus of this new guide to email

...

ORDER NOW

MORE ON DIRECT MAIL >>

FROM THE BOOKSTORE

<P><SPAN style="FONT-SIZE: 10pt">“Hey, stuff happens”—the cliché is a cliché for a reason: because it’s true. It always has been and always will be, and that is why insurance, and subsequently, insurance marketing, will never find itself out of a job. And, believe it or not, the oft-neglected direct mail channel has proven itself to be the most successful in lead generation when it comes to insurance marketing. </SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt">Everything you need to take advantage of this growing market is right here in Insurance Marketing: The Competitive Intelligence Report. It’s full of direct mail trends, best practices and analysis, including formats, copy and design, premium usage, seasonality trends, branding and integrative marketing.<B><BR> <BR></B>You’ll get dozens of insurance direct mail trends based on research from DMIQ’s Who’s Mailing What archive, plus in-depth analysis of grand controls and campaigns from some of the biggest insurance giants in the business, complete with: </SPAN></P>
<UL>
<LI><SPAN style="FONT-SIZE: 10pt">Volume comparisons</SPAN>
<LI><SPAN style="FONT-SIZE: 10pt">Seasonality trends</SPAN>
<LI><SPAN style="FONT-SIZE: 10pt">Trends in package format, 4-color printing, personalization and variable data</SPAN>
<LI><SPAN style="FONT-SIZE: 10pt">Trends in offers and premiums</SPAN>
<LI><SPAN style="FONT-SIZE: 10pt">Review of the mail that uses copy drivers</SPAN>
<LI><SPAN style="FONT-SIZE: 10pt">Analysis on the integrative future of insurance mail</SPAN>
<LI><SPAN style="FONT-SIZE: 10pt">And more!</SPAN></LI></UL>
<P><SPAN style="FONT-SIZE: 10pt">Plus, this book contains full company profiles and statistics of the top 20 insurance companies in the industry, including GEICO, AAA, State Farm, New York Life, Allstate, and many more. </SPAN></P>

Insurance Marketing: The Competitive Intelligence Report

“Hey, stuff happens”—the cliché is a cliché for a reason: because it’s true. It always has been and always will be, and that is why insurance, and subsequently, insurance marketing, will never find itself out of a job. And, believe it or not, the oft-neglected direct mail channel

...

ORDER NOW

Available as a PDF<BR><BR>A guide to the science of direct response testing today, including best practices, power tests, small vs. large test, analyzing results, testing mistakes, new tricks and more.<BR> <BR>Direct marketing — regardless of the channel — is that unique combination of art and science. Direct marketing strategists are constantly striving for better response rates, open rates, conversions, cost-per-order, and life-time value. <BR><BR>And what is the tool that these strategists turn to again and again? That’s right! It’s testing. <BR><BR>In direct marketing, there are plenty of elements to test — from subject lines, premiums, envelope sizes, list selects, pricing, the placement of the shopping cart on the web page ... the choices seem endless. <BR><I><BR>Secrets of Direct Marketing Testing </I>is your personal BRAIN TRUST of testing strategies that you can start to put to use today. <BR><BR>In this guide, you’ll find everything you need to learn — the why, what, when and how-to of testing. From testing structure to basic principles, and from test ideas to mistakes to avoid, you’ll get a crash course in coding, tracking, reading and applying test results. <BR><BR>You’ll learn about: <BR>• Simple copy tests that drive response <BR>• Fine-tuning your offer <BR>• Web and Email testing <BR>• Offer tests <BR>• How to make sure you’re getting reliable results <BR>• Retesting and rolling out your findings <BR>• Plus the Rules that you should test now — or ignore at your own peril! <BR><BR>Are you are searching for ways to raise response, save on your promotion costs, drive down your cost-per-order and extend the lifetime value of your customers? The DirectMarketingIQ and Target Marketing editorial teams have been researching, writing and collecting expert advice from industry leaders about the how-tos of testing for years. <BR><BR>We’ve compiled this information and made it easy for you to find all in one place with our easy-to-read report – <EM>Secrets of Direct Marketing Testing</EM>.

Secrets of Direct Marketing Testing

Available as a PDF

A guide to the science of direct response testing today, including best practices, power tests, small vs. large test, analyzing results, testing mistakes, new tricks and more.

Direct marketing — regardless of the channel — is that unique combination of art and science. Direct marketing strategists are constantly striving

...

ORDER NOW

COMMENTS

Click here to leave a comment...

Comment *

Name *
E-mail *
URL
Why are we asking this?
Remember Me

Most Recent Comments:

Andy - Posted on September 22, 2011
It seems inaccurate to say that this process identifies "packages that are statistically more likely to perform better." It identifies packages that are statistically more likely to be RATED better. Kevin below says there are case studies establishing the connection between those two things, but without more data (beyond the images available at the link), it's hard to accept that that connection is "statistical" rather than--at best--empirical in limited circumstances. That is, it seems like you'd need to have enough case studies to produce a general but precise measurement of the statistical correspondence between rating and performance before you make claims involving the statistical likelihood of better performance. My personal gut feeling is that some elements that rate well will perform well but not all will. It would be interesting to see if that is the case. "The process works" seems like too blunt a knife as a general statement...

Kevin Schulman - Posted on September 15, 2011
Hi, Good points and questions. Full disclosure right up front, I'm the CEO of DonorVoice, a sister company to DonorTrends, and we developed the pretest tool. Hopefully my weighing in is useful and furthers the conversation. The methodology we use yields reliable results because we focus on asking survey respondents/panelists questions they are good at answering while avoiding those we, as humans, are not very good at addressing. In a nutshell, we are very good at discriminating between extremes - e.g. what we consider good and bad, like/dislike. The eye test is a good analogy - is A or B better? We're very good at answering those questions. We are far less good (often lousy in fact) at ranking or rating a long list of items. As or more importantly, we never ask people WHY they picked A or B (which in this case is a complete direct mail package that is evaluated holistically from the survey taker's perspective), which can also yield very unreliable answers for a whole host of reasons. Instead, we derive the importance of every single test item (with a specific, empirical score) on the back end with statistical analysis. As both comments suggest however, the proof is in the pudding...or in this case, the mail. We have two case studies (and many others from other sectors) from the non-profit direct mail world where our survey results matched perfectly and nearly so with actual mail results. Here is link to the case studies (bottom of page) and the tool on our site, http://www.thedonorvoice.com/product/pretest-tool/ We are thankful for the additional exposure of this tool from Fundraising Success and especially grateful for the thoughtful comments and hopefully, continued discourse.

Mike Conners - Posted on September 15, 2011
It's a neat initiative if it works. I recommend testing the 'test' as part of the investigative research by comparing results from a mix of different packages to the external panel responses. Can't help but like the idea should it prove to be successful.

Pamela Barden - Posted on September 15, 2011
Very interesting idea. Would want to see some "proof" before relying on this, given that focus groups and online survey panels aren't always reliable. But it is an idea "whose time has come." Margaret, love to see a follow-up article on nonprofits' results after using this option.

SP - Posted on September 15, 2011
Interesting, but not sure how effective something like that would be. People (consumers, donors, whatever) are notoriously bad at separating what marketing they like and what marketing is effective. It's completely possible that someone evaluating a package would behave differently than they say they would in a more real-world situation. I guess it's possible that a good evaluator could compare the responses in a way that would filter out the bias, but my initial response is skepticism. You would need to compare in-market tests to the evaluations to see how accurate this could be...

« Prev / Next »

Privacy Policy

Click here to view archived comments...

Archived Comments:

Andy - Posted on September 22, 2011
It seems inaccurate to say that this process identifies "packages that are statistically more likely to perform better." It identifies packages that are statistically more likely to be RATED better. Kevin below says there are case studies establishing the connection between those two things, but without more data (beyond the images available at the link), it's hard to accept that that connection is "statistical" rather than--at best--empirical in limited circumstances. That is, it seems like you'd need to have enough case studies to produce a general but precise measurement of the statistical correspondence between rating and performance before you make claims involving the statistical likelihood of better performance. My personal gut feeling is that some elements that rate well will perform well but not all will. It would be interesting to see if that is the case. "The process works" seems like too blunt a knife as a general statement...

Kevin Schulman - Posted on September 15, 2011
Hi, Good points and questions. Full disclosure right up front, I'm the CEO of DonorVoice, a sister company to DonorTrends, and we developed the pretest tool. Hopefully my weighing in is useful and furthers the conversation. The methodology we use yields reliable results because we focus on asking survey respondents/panelists questions they are good at answering while avoiding those we, as humans, are not very good at addressing. In a nutshell, we are very good at discriminating between extremes - e.g. what we consider good and bad, like/dislike. The eye test is a good analogy - is A or B better? We're very good at answering those questions. We are far less good (often lousy in fact) at ranking or rating a long list of items. As or more importantly, we never ask people WHY they picked A or B (which in this case is a complete direct mail package that is evaluated holistically from the survey taker's perspective), which can also yield very unreliable answers for a whole host of reasons. Instead, we derive the importance of every single test item (with a specific, empirical score) on the back end with statistical analysis. As both comments suggest however, the proof is in the pudding...or in this case, the mail. We have two case studies (and many others from other sectors) from the non-profit direct mail world where our survey results matched perfectly and nearly so with actual mail results. Here is link to the case studies (bottom of page) and the tool on our site, http://www.thedonorvoice.com/product/pretest-tool/ We are thankful for the additional exposure of this tool from Fundraising Success and especially grateful for the thoughtful comments and hopefully, continued discourse.

Mike Conners - Posted on September 15, 2011
It's a neat initiative if it works. I recommend testing the 'test' as part of the investigative research by comparing results from a mix of different packages to the external panel responses. Can't help but like the idea should it prove to be successful.

Pamela Barden - Posted on September 15, 2011
Very interesting idea. Would want to see some "proof" before relying on this, given that focus groups and online survey panels aren't always reliable. But it is an idea "whose time has come." Margaret, love to see a follow-up article on nonprofits' results after using this option.

SP - Posted on September 15, 2011
Interesting, but not sure how effective something like that would be. People (consumers, donors, whatever) are notoriously bad at separating what marketing they like and what marketing is effective. It's completely possible that someone evaluating a package would behave differently than they say they would in a more real-world situation. I guess it's possible that a good evaluator could compare the responses in a way that would filter out the bias, but my initial response is skepticism. You would need to compare in-market tests to the evaluations to see how accurate this could be...