Easily create factorial survey experiments in Qualtrics

Do your Qualtrics surveys end up with an unwieldy number of blocks that contain a lot of duplicated material? Ensuring consistency while generating and maintaining a large number of blocks is not only tedious and time consuming, but also more error prone. If you create one block for each of your treatment conditions, Qualtrics has some handy features that can help avoid needless repetition. In this guide, I’ll outline the best approach that I have found to automatically generate blocks that differ in systematic ways. Using this approach, most factorial survey experiment (also called experimental vignettes) designs can be implemented using just a single block.

Qualtrics survey flow for a factorial survey experiment with one block per treatment condition
Too many blocks! We can accomplish the same thing with just one block.

The problem

Online survey experiments, such as factorial surveys (aka experimental vignettes) often consist of multiple conditions with small but systematic differences. For example, a researcher may want to manipulate just a single word or quantity in a larger paragraph. One way to implement such a design in Qualtrics is to simply create one block (or question) for each of the different conditions:

Factorial survey experiment in Qualtrics showing vignette text
Example of a survey with one block per treatment condition. Each block manipulates a word describing the size, shape, and color of an object.

Because online experiments are so fast and cheap to run, factorial designs with 2^4=16, 3^3=27, or even more conditions are becoming increasingly common. Personally, I find a survey with that many blocks to be incredibly ugly. There is something very satisfying and beautiful about clear, concise code that I also like to see in my survey designs. More importantly though, generating and updating them is more error prone and time consuming than if we only had to maintain a single block. As the software engineering maxim goes: don’t repeat yourself.

The solution

In most cases, we can achieve the desired manipulation using just a single question or block, regardless of the number of conditions. To do this, we will take advantage of Qualtrics’ embedded data and text piping features. By taking a more programmatic approach, we can abstract the manipulations from the ‘static’ portion of the text and thus avoid unnecessary duplication.

For the purposes of this guide, we will use an example based on a 2x2x2 design. The factors and corresponding levels are color={blue, red}, shape={square, circle}, and size={small, big}. We want participants to read some version of the following sentence:

Here’s a story about some shapes that are size and color.

Rather than creating 8 different blocks for each of the possible unique combinations, we’ll create just a single block that accomplishes the same thing. This is done by creating tags (piped text) in the survey text that will tell Qualtrics where we want to insert certain words or phrases. The final result will look like this in the survey editor:

Here’s a story about some ${e://Field/shape}s that are ${e://Field/size} and ${e://Field/color}.

Step 1: Create a block containing some version of your text

Simply create any version of the text that you wish to manipulate. This will serve as a sort of template that we will modify by inserting tags that tell Qualtrics what words we want to manipulate. In our case, we have two different words that describe an object’s shape, size, and color. We want to generate all possible combinations of these descriptions and have Qualtrics output them.

Here's a story about some squares that are small and blue.
Step 1: Create some version of the text that you want to manipulate.

Step 2: Create your embedded data fields

We need to tell Qualtrics what words we want to insert into the survey. To do this, we will store these words in embedded data fields. We can think of these fields as variables that can be set to a particular word or phrase.

To create an embedded data field, click on Survey Flow, click on Add element, and select Embedded data. Make sure you insert the element before the block containing your text. Set the Embedded Data Field Name to the name of the factor that you want to manipulate, e.g. shape, size, or color. Then, click on Set a Value Now and enter the text that you want Qualtrics to insert, e.g. red, square, or blue. Repeat this step for each of your levels.

Set embedded data field name to factor name
Step 2.2:  Qualtrics will automatically insert the value (red) into the designated spot in the survey.

 

Qualtrics survey flow with all embedded data elements
Step 2.3: Repeat for each of your levels.

Step 3: Set up randomization

Now that we have all the different pieces of text that we want Qualtrics to be able to insert, we have to get it to actually choose one of these. Most researchers doing experimental vignettes or factorial surveys are likely interested in randomly assigning participants to one of each of the unique combinations of the different words. To do this, we’ll use a simple randomization scheme.

Simply create a Randomizer element for each of your factors and assign the corresponding embedded data fields to them. Qualtrics will then choose one word from each of the 3 levels and set the value of the embedded data variable to that word.

Qualtrics survey flow with randomized embedded data
Step 3: Assign each level to its own Randomizer element.

Step 4: Insert piped text

Now that Qualtrics has randomly chosen one word for each of the 3 embedded data variables, we need to tell Qualtrics where we want to insert these values (i.e., the words).  To do this, we need to use the Piped Text feature.

Simply edit the text or question and place the cursor where you want the text to appear. Click on Piped Text > Embedded Data Field. Enter the name of the embedded data field in the box. Note, this is not the value of the embedded data field, but the name of the field.  In our case, this would be the name of one the 3 factors: color, shape, or size. This will insert a tag or special code where Qualtrics will automatically insert the value of the embedded data field. Repeat for the 2 remaining factors.

Inserting piped text from an embedded data field in Qualtrics
Step 4.1: Edit text and place your cursor where you want to insert the embedded data value.

 

Inserting piped text from an embedded data field in Qualtrics
Step 4.2: Click Piped Text > Embedded Data Field. Enter the name of the embedded data field that you want to insert.

 

Inserting piped text from an embedded data field in Qualtrics
Step 4.3: This will insert a tag where we want Qualtrics to insert the value of the embedded data field.

 

Inserting piped text from an embedded data field in Qualtrics
Step 4.4: Repeat.

Final result

When we preview or run the survey, Qualtrics automatically inserts the values from the embedded data fields in place of the tags:

Piping text from an embedded data field in Qualtrics
Final result: the text appears wherever the we insert the piped text tags.

Notes

This method is quite flexible. The values in the embedded data fields are not limited to single words. They can also contain full sentences or HTML (e.g. links to images, color, etc). The main advantage is that it lets you quickly generate and update surveys with many conditions by avoiding repetition. As such, it is likely to be useful to anyone who does factorial surveys or experimental vignettes with multiple factors.

So far, this is the best method that I have found for the types of factorial survey experiments that I tend to run. If you know of a better way, please share in the comments.

1 Comment

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.