The Effects of Order and Text Box Size of Open-ended Questions on Withdrawal Rate and the Length of Response

Web surveys can collect many responses quickly and easily, but they have a problem with a high withdrawal rate. We found that more respondents tend to leave when open-ended questions are placed first compared to last [2]. Here, we hypothesized that participants would respond with longer sentences in larger text boxes compared to smaller text boxes, and participants would most likely leave when the larger text box was presented first. Then, we conducted a survey in which the order of the open-ended questions and the size of the text box was changed and investigated the withdrawal rate and the length of the responses. The results showed that the withdrawal rate tended to be the highest when the open-ended questions were answered first and when the text boxes were large. In addition, the larger text box obtained a greater length of response. This result indicates a trade-off relationship between the withdrawal rate and the length of response. Furthermore, the withdrawal rate for smartphones tended to be high, regardless of the text box size.


INTRODUCTION
Web surveys are often used to collect responses on social research and services and preliminary data for research purposes.Compared to paper-based surveys, web surveys can gather many answers more easily.The number of people registering for crowdsourcing services is rapidly increasing because it is easy to take part and earn income.Researchers often use open-ended survey questions because they provide answers from various perspectives [1].
We conducted an experiment in which we divided the openended questions into two groups, one in which the open-ended questions were answered at the beginning of the survey and the other when they were answered at the end [2].As a result, we found that more respondents left the survey when the open-ended question was placed first compared with last.Maloshonok et al. [3] found that the quality of responses was significantly higher for respondents with larger text boxes than for respondents with smaller ones, and they were more likely to write longer responses.In addition, Mittereder et al. [4] analyzed data from web surveys.They found that while there were no differences in attrition rates between respondents using mobile devices and those using PCs at the beginning of the survey, the attrition rate for respondents using mobile devices increased as the survey progressed.We thought that the text box size would be seen differently depending on the devices used and would significantly impact the withdrawal rate.Furthermore, by combining the position of the open-ended question and the text box size, we expected to control the withdrawal rate and the length of the response.
In this study, we hypothesize that participants would answer with longer sentences in larger text boxes compared to smaller text boxes, and participants would most likely leave when the larger text box was presented first.Then, we investigate a survey in which the order of open-ended questions and the size of their text boxes differ and examine the effects of the order and size of the questions on the withdrawal rate.In addition, we clarify the effects of differences between smartphones and PCs on the withdrawal rate and responses.
The contributions of this paper are as follows.
• We conducted an experimental test on the order of openended questions and text box size, and we clarified that the withdrawal rate was higher when the open-ended questions were answered first, and the text boxes were large.• We found that the length of the sentence was longer when the text boxes were large, indicating a trade-off relationship between the withdrawal rate and the length of the sentence.
• We clarified that the withdrawal rate for smartphones tended to be high, regardless of the text box size.

RELATED WORK
Various studies have been conducted on responses and the response time in the last half of a survey.Schmidt et al. [5] found that the later the open-ended questions were asked, the lower the number of interpretable responses became.Yan et al. [6] analyzed four web surveys and found that the closer the questions were to the end of the survey, the shorter the time to answer them became.Galesic et al. [7] found that the respondents responded quickly and with a shorter text if a question was asked later.Thus, it is known that the survey presentation timing affects the quality of responses and the response time.
There have been many studies on text box size.Zuell et al. [8] investigated how the size of response columns affects item nonresponse in a web survey for college students.The results showed that large text boxes had higher item non-response rates than small text boxes.On the other hand, Stern et al. [9] found that the number of words increased for those who answered in the large text boxes compared to those who answered in the small response boxes.These studies indicate that text boxes affect the quality of responses to open-ended questions.
Many studies have focused on the factors leading to survey withdrawal.Mittereder et al. [4] analyzed data from a web survey.They found that respondents with short or variable response times were more likely to leave the survey than respondents with longer and more stable response times.Peytchev [10] found that dropouts in web surveys do not appear to be inattentive and that it is necessary to devise ways to help these dropouts maintain their survey responses.Chen et al. [11] show that the load at the start of the survey and the load accumulated after the survey starts can cause survey withdrawal.They also state that a respondent's decision to continue or stop the survey depends on the questions being presented.This paper aims to clarify the effects of open-ended questions and text box size on the withdrawal from the survey.
Many studies have been conducted on how the quality of responses varies depending on the device used to answer the questions.Tourangeau et al. [12] reported a trend of longer response times when using a smartphone than when using a PC.Antoun et al. [13] also found that smartphone users gave longer answers to open-ended questions than PC users.On the other hand, Mavletova [14] conducted a PC and mobile survey using a Russian volunteer online access panel and found that mobile users had lower completion rates and shorter answers to open-ended questions.This study investigates the quality of responses on smartphones and PCs, including the withdrawal rate and the length of response.

EXPERIMENT 3.1 Outline of the experiment
To clarify whether the order of open-ended questions and the size of their text boxes affect the withdrawal rate and the length of response, we created a system that enables the acquisition of response times and devices and conducted an experiment.
In the experiment, the order in which respondents answered the open-ended questions and the size of the text boxes were randomly assigned.The order in which the open-ended questions were responded to was varied in two groups: the first group was asked to answer the open-ended questions first (hereafter referred to as Group-FIRST), and the last group was asked to answer the openended questions last (hereafter referred to as Group-LAST).The text box size was two lines for the small group and 20 lines for the large group (see Figure 1 and Figure 2).
The groupings in the experiment are shown below, and the order of the questions is shown in Figure 3.
• Group-FIRST (Small): Open-ended questions first with a small text box size.• Group-FIRST (Large): Open-ended questions first with a large text box size.• Group-LAST (Small): Open-ended questions last with a small text box size.• Group-LAST (Large): Open-ended questions last with a large text box size.
The survey topic in this experiment concerned people who use smartphones daily.The contents and order of the four questions in the open-ended questions phase were as follows.
• Q-a: What do you use your smartphone for?The basic information in the figure is the respondent's personal information (gender and age), which is given in the form of a choicetype response.Our experiment system displayed one question per page to control the order of answers.When a participant answered a question, the system showed the next question.

Experimental procedure
We recruited 2,000 participants (1,000 males and 1,000 females) for this experiment by posting tasks on Yahoo!Crowdsourcing [15], from which the participants accessed our survey system and completed the survey.The task description screen also warned that "people who do not use smartphones daily are not eligible for this survey" and that there were 14 questions.The requests were divided into those for males only and those for females only.
The system randomly divided the participants into four groups.The order of the survey phases was as indicated in Figure 3.The system displayed the number of questions.After completing the survey, the system displayed a common code and ID.The participant was rewarded with money after returning to the crowdsourcing page, correctly selecting the code, and entering their ID.The reward was 10 PayPay-points (a popular point program in Japan), which was equivalent to 10 Japanese yen (0.08 US dollar).This reward was higher compared with the average of tasks that require a similar task completion time in Yahoo!Crowdsourcing.

RESULT
The number of respondents who accessed the survey page to answer the survey was 3,131.The number of respondents who completed the study and correctly entered their IDs (hereinafter referred to as completers) was 1,784.The time from the start to the end of recruiting was about two and a half hours.

Withdrawal rate
Table 1 shows the number of accesses, completions, and withdrawal rates for each group.The results show that the withdrawal rate for Group-FIRST (Large) was the highest.In the following analyses, only those who completed the survey are included in the analysis, except for the withdrawal rate.
Next, the number of respondents who did not answer each question was calculated for each question.Figure 4 shows the withdrawal rate for each question, calculated from the number of respondents who accessed the survey page and the number of those who left the page.The vertical axis in Figure 4 indicates the dropout rate, and the horizontal axis indicates the question number.The figure shows that 45.8% of Group-FIRST (Large) respondents accessed the survey page but still needed to answer the questions.In Group-FIRST (Small), 30.3% of the respondents answered only one question.On the other hand, about 20% of the respondents in Group-LAST (Small) and Group-LAST (Large) did not answer a single question, indicating that the withdrawal rate increased by several percent at the 12th question, which was when the respondents in Group-LAST started answering the open-ended questions.

The length of characters and answering time of open-ended questions
Table 2 shows the average number of letters in the open-ended questions.The table shows that Group-FIRST (Large) in Q-a and Qb and Group-LAST (Large) in Q-c have more letters.A two-factor analysis of variance was conducted for each question, in which the order of the open-ended questions and the text box size were considered factors, and a significant difference was found in the text box size for all open-ended questions (p < 0.01).However, there was no significant difference between the order of the open-ended questions and the two factors.Table 3 shows the average response time for the open-ended questions.The table shows that the response time was longer for the group with a large text box.In particular, the total difference in response time for Group-FIRST (Large) was about 47 seconds compared to Group-LAST (Small).A two-factor analysis of variance was conducted for each question, with the order of the open-ended questions and the text box size as factors, and significant differences were found in the order of the open-ended questions in Q-a (p < 0.01) and the text box size for all open-ended questions (p < 0.01).

Comparing the smartphone and PC
Table 4 shows the order of the open-ended questions, the text box size, and the number of respondents per device.Fifteen respondents answered on tablets, but since the number of respondents was too small to analyze, they were excluded from the analysis in this section.
Figures 5 and 6 show the withdrawal rates for each question by device.Here, the vertical axis in the figure indicates the withdrawal rate, and the horizontal axis indicates the question number.The figures show that the withdrawal rate was higher for smartphones than for PCs.Of particular note, about 60% of the respondents in Group-FIRST (Large) who responded by smartphone left the survey at the end of the survey.
Tables 5 and 6 show the average number of letters in the openended questions for each device.The tables show that the number of characters was larger for PCs than for smartphones in almost all groups and questions.A three-factor analysis of variance with the order of open-ended questions, text box size, and device as factors was conducted, and the results showed that for all open-ended questions, there were significant differences in text box size (p < 0.01), device for Q-b (p < 0.01), and two factors for Q-a, namely the order of open-ended questions and text box size (p < 0.05).However, the three factors were similar.
Tables 7 and 8 show the average response times for the openended questions for each device.The tables show longer response times for Group-FIRST (Large) and Group-LAST (Large) smartphones.On the other hand, the first (small) and last (small) groups answered PCs longer.We conducted a three-factor analysis of variance with the order of open-ended questions, text box size, and device as factors.We found significant differences in the order of open-ended questions in Q-a (p < 0.01) and text box size (p < 0.01) for all open-ended questions.Still, no significant differences in the device and no significant differences in the three factors were found.

Effects of various factors on withdrawal rates
The results in Section 4.1 showed that the withdrawal rate was highest in Group-FIRST (Large).This suggests that the number of respondents who left the survey increased when the open-ended question was presented in the first with a large text box.Group-FIRST (Small) had the next highest withdrawal rate after Group-FIRST (Large).In other words, the order of the open-ended questions may have affected the withdrawal from the survey.
In Section 4.4, we compared the responses between smartphones and PCs.As a result, we found that the withdrawal rate for smartphones was higher than that for PCs.The reasons for the higher withdrawal rate on smartphones may include the influence of external factors such as travel and waiting time and the fact that the first question in this situation was an open-ended question.This task carried a large load.In Group-FIRST (Large), more than 50% of the participants withdrew from the task at the first question.The large text box size in this study was 20 lines, and the button for moving on to the next question was not visible.This may have increased the burden on the user, resulting in a higher dropout rate.
Here, we analysed whether some respondents left the survey after entering their answers in the text boxes.Table 9 shows, for each group and question, the number of respondents who entered their answers in the text boxes but withdrew from the survey.The table shows that in Group-FIRST (Small) and Group-FIRST (Large), nine respondents entered but dropped out of Q-a.This suggests that placing the open-ended question at the beginning of the survey made the respondents try to answer it, but they needed to become more familiar with the contents of the survey, and it took them a long time to answer the questions, so they left the survey.
To minimize the number of dropouts, the open-ended questions should not be placed at the beginning of the survey, and the text box size should be small.This is especially important when conducting a survey targeting smartphone users.

Effects of factors on open-ended responses
Although the large text box size may have encouraged the respondents to leave the survey, when we focused on the number of characters entered in the open-ended questions, the results in Section 4.3 show that the number of characters was the largest in Group-FIRST (Large) for Q-a and Q-b, and in Group-LAST (Large) for Q-c.From this result, it can be inferred that respondents try to input many characters when the text box size is large.Table 4 shows that Group-FIRST (Large) took the longest to answer the open-ended question.This suggests that respondents spend more time and try to input more text when the text box is large than when the text box is small.This tendency is also true for the device-specific analyses in Tables 5-10, indicating that the size of the text box affects the response time and the amount of response, regardless of the device.In addition, the result that the number of input characters in the open-ended question was larger for PCs than for smartphones supports the result of Mavletova [14].In other words, respondents who answer the survey using a PC can obtain more answers than those who answer the survey using a smartphone.
The above results suggest that a larger text box encourages more earnest responses to the open-ended questions and is also expected to increase the volume of the responses.Therefore, in designing the survey, it is necessary to balance the withdrawal rate and the quality of the responses to the open-ended questions.In addition, it is necessary to consider carefully what type of device the users are using.

Analysis focusing on smartphone screen size
As shown in Figure 1, the large textboxes are sometimes placed in such a way that they extend beyond the screen, and it is thought that the screen size may affect the withdrawal rate.Therefore, to equalize the number of respondents as much as possible, we divided the screen size into two groups: one with a height of less than 667px (hereinafter referred to as the "small screen size") and the other with a height of 667px or more (hereinafter referred to as the "large screen size") and calculated the dropout rate.
Table 10 shows the dropout rates at the end of the survey.The results show that the withdrawal rate at the end of the survey was higher for the small screen size group in all groups.This may be because, although the survey can be answered easily, the text box size looks larger than in Figure 1 because of the small screen size and the respondents felt the burden of the survey was greater than expected and left the survey.Therefore, paying attention to the text box and radio button sizes for smartphone survey respondents is necessary.

CONCLUSION
This study investigated the effects of the text box size of openended questions and the order of responses in the survey on the withdrawal rate, also focusing on the user's devices.Specifically, comparisons were made among four groups that differed in the position of the open-ended questions and the text box size.
The experimental results showed that 30.3∼45.8% of Group-FIRST (Large) and Group-FIRST (Small) respondents left the survey without answering even the first question.In addition, when the open-ended question was responded to first in a large text box, the withdrawal rate was the highest, but the response time for the open-ended question was also the longest.Furthermore, when we compared the responses between smartphones and PCs, we found that the withdrawal rate was higher for smartphones than for PCs, that PCs input more text than smartphones, and that PCs had longer response times for the open-ended questions in all three groups except for Group-FIRST (Large).
In the future, we plan to conduct a detailed study of the factors affecting the quality of responses.We also plan to conduct multiple surveys and suggest points to be considered when designing the survey.

Figure 1 :
Figure 1: Text boxes in open-ended questions presented on a smartphone.

Figure 2 :
Figure 2: Text boxes in open-ended questions presented on a PC.

Figure 3 :
Figure3: Order of responses in the four groups (From the top Group-FIRST(Small), Group-FIRST(Large), Group-LAST(Small), Group-LAST(Large)).

Figure 4 :
Figure 4: Withdrawal rate per question in each question and group.

Figure 5 :
Figure 5: Withdrawal rate per question in each question and group for smartphones.

Figure 6 :
Figure 6: Withdrawal rate per question in each question and group for PCs.

Table 1 :
Number of people who accessed the survey, completers, and withdrawal rate (%) in each group.

Table 2 :
The average length of answers in each question and group (characters).

Table 3 :
Open-ended questions response time in each question and group (seconds).

Table 4 :
Number of people who accessed this survey and completers in each group.

Table 5 :
The average length of answers in each question and group for smartphones (characters).

Table 6 :
The average length of answers in each question and group for PCs (characters).

Table 7 :
Open-ended questions response time in each question and group for smartphones (seconds).

Table 8 :
Open-ended questions response time in each question and group for PCs (seconds).

Table 9 :
Number of people who entered the text box but left the survey.

Table 10 :
Withdrawal rate by screen size for smartphones (%).