Qualtrics Integration

In this section, we describe how to set up a Qualtrics survey that chooses optimal questions using your new API.

The list of API calls and how to find <your-URL> are in List of API calls.

While we set up a survey with Qualtrics in this walkthrough, the general principles apply to any survey framework that supports API calls.

Set up a Qualtrics Survey

This section assumes some familiarity with working with Qualtrics. You should be able to create survey questions and understand how to work with the Survey Flow. We provide links to some helpful Qualtrics resources.

The QSF file for the demo Qualtrics survey created from the BACE package template is contained in /integration/Qualtrics/BACE_template.qsf. As a starting point for your own Qualtrics integration and to follow along this walkthrough, you can create a new survey in Qualtrics by importing this QSF file into Qualtrics. Instructions for importing surveys are available on the Qualtrics website.

Collect respondent information

First, use a survey question to collect information about your respondent that will be used to create the respondent's profile in the database. In particular, we collect a survey_id from the respondent with a survey question. We then store this information as embedded data. More detailed instructions for this process are described on the Qualtrics site: Embedded Data.

Start by adding a question that asks for the respondent's survey ID. This ID will help identify the respondent.

Open up the Survey Flow element. Underneath the Survey ID block, click 'Add Below' → 'Embedded Data'. We can then access and store the answer to this question as the embedded data variable ${e://Field/survey_id}. To access question information, click the dropdown menu when adding embedded data → 'Insert Piped Text' → 'Survey Question' → 'survey_id' → 'What is your survey id?`.

Note, understanding how to access data from previous questions is important as we will need this in order to send info from the survey tool to your application and pipe responses into the survey. Make sure to review Qualtric's guide for setting embedded data if you have any questions.

BACE questions

Create Profile and Store Design Using Web Service

In this section, we will create a profile for the respondent by connecting to our application using a web service. Note: you can use custom JavaScript within Qualtrics to handle web requests; however, the Web Service feature is recommended for people less familiar with APIs. We recommend reviewing information on the Qualtrics web service feature if questions arise while implementing the remaining sections.

Return to the Survey Flow. Add a Web Service element below the embedded data block that you just used to set the survey_id variable. The Web Service feature allows us to send and receive information from a specific URL. To create the respondent's profile, we will send a POST request to your application. Add the survey_id as a body parameter of your POST request using the 'application/json' type. When triggered, this web service element will make a call to the URL <your-URL>/create_profile.

Qualtrics allows you to test API calls using the 'test' button. This sends your request to the specified URL and receives feedback that can be stored as embedded data. Note that all embedded data variables will not be rendered by Qualtrics until respondents have input their data. For example, once a respondent answers the survey_id question, then ${e://Field/survey_id} will be replaced by the respondent's answer. The test call will trigger a random design if the survey_id is considered ${e://Field/survey_id} and not rendered. If a test fails (particularly when debugging user-defined functions), click 'View Raw Data' to see what is being returned by your call.

Next, we will need to store output variables as embedded data. Select 'All' and click 'Add Embedded Data'. When a respondent arrives to this portion of the survey, the web service element will be triggered creating a new profile for that respondent. We store the respondent's profile_id and other profile-specific variables as embedded data.

We can now review what's going on in the Web Service Element that you just created.

• We are going to make a POST call to <your-URL>/create_profile.
• The survey_id for the respondent will be included as a body parameter in this request.
• In the background, the web application queries the database that you just created and creates a profile specific to the respondent.
• Qualtrics receives back the respondent's profile_id and information for the first design.
• We save these variables as embedded data so that we can use them to display the optimal design and they will be included in our final data file. Make sure that any variable that you want to store is saved. You can see what variables will be stored after previewing the survey by going to the 'Results' tab in Qualtrics.

Display First Design

Now that you have created a profile and have information for the first design, we want to create a survey question that displays our question information. Return to the survey builder and create a new block. Create a multiple-choice question with a horizontal alignment and two questions. Add 'Which pen would you prefer?' as the question text. In the Survey Flow, make sure the new block comes after your web calls to create a profile and choose the first design.

You can insert piped text into this question which will be dynamically populated when the respondent takes the survey. Note also that the 'Rich Content Editor...' is helpful for formatting text.

For the base (treat) option, pipe the embedded text for the choice in using ${e://Field/message_0_1} (${e://Field/message_1_1). This is an HTML formatted table that we set up to present information in a user-friendly format in /app/bace/user_convert.py.

After creating your question, make sure to set the recode value of the treated Option B to equal the relevant value in answers in /app/bace/user_config.py. For example, Option A will have a recode value of 0, and Option B will have a recode value of 1. Recode values can be set using the 'Recode values' tab in the side toolbar. This step is important because it ensures that answers are correctly sent back to the web application.

You can test the survey using the 'Preview' button. Enter a survey id, and proceed to the first question. The embedded data fields will be populated and displayed in the survey.

Update Posterior and Choose New Design

You can update the posterior based on the previous answer and receive the next optimal question using a POST call to <your-app>/update_profile.

Return to the Survey Flow. Below the first question block, add a new Web Service element.

• In our survey, the parameter question_number will change across web service elements. This parameter indicates the question number for the next question that will be displayed to the respondent. This field is optional. The template code sets question_number to the length of the design history so far.
• profile_id should be set as the embedded data variable ${e://Field/profile_id}
• answer should be the Selected Choices Recode value for the previous question in the survey. This can be set by following the dropdown menu.

The full block should look like the image below.

Return to the Survey editor. Create a new block, Question 2, to show the new question to the respondent. This block will look identical to the first block except the answer choices should be ${e://Field/message_0_2} and ${e://Field/message_1_2}. Remember to make sure that any question that uses embedded data from a web service comes after that Web Service element in the Survey Flow. Follow the same procedure to set up recode choice values to be consistent with the answers defined in your /app/bace/user_config.py file.

Repeat this process for as many questions as you would like. Always, make sure to set new embedded data elements for each question that accurately reflect the question number of the following question. Additionally, make sure that recode values are properly set and that answer references the most recently asked question. Copying and pasting from another survey block before editing can save time.

Access Posterior Estimates

After the final BACE choice question in your survey, you can calculate and store the preliminary posterior estimates using a POST request to <your-URL>/estimates.

Include profile_id and the answer to the final question in the body of the request. Follow the same procedure for storing the test output as embedded data. The Web Service element will resemble the image below:

In the sample survey file included, we show how to display these estimates in the final question of the survey. When performing your final analyses, we recommend estimating the posteriors using additional methods with the final observed choice data. See Drake, Payró, Thakral, and Tô (2024) and Drake, Thakral, and Tô (2023) for examples.

Common Mistakes

• question_number is optional if we do not need to test query an API call within the Survey Flow. If it is set, however, then:
  • question_number needs to be changed for each question. It should match the embedded data variables that will appear in the next question that you will ask.
  • Furthermore, make sure that the numbers on the embedded data variables from each call match the value of question_number in the body parameter for that call.
• Confirm that the Survey Flow is correct. Web Service elements are not displayed in the survey builder, so you need to confirm that the order of everything is correct. Make sure that questions that use information from an API call appear after that call in the survey.
• Make sure that all embedded data variables from API calls are marked to be saved. If you do not save responses, the questions in your survey will be blank to respondents, and the values will not be saved in your data.
• If an API call is not running correctly, make sure that there are no typos or mislabeled variables in your call. Also, make sure that you are using the proper type of request mentioned in the text. You can directly test API calls using the Jupyter notebook provided /tools/API_notebooks/API_Queries.ipynb (or locally using /tools/API_notebooks/API_Queries_local.ipynb).