Several of the online survey companies have decent guidance about reducing bias in survey questions.

• SurveyGizmo
• SurveyMonkey
• surveytown

I don’t necessarily endorse these vendors, but they have a decent coverage of biases that sneak into survey and a vested interest in helping their customers reduce them.

I’ll outline just a few of the issues.

Sampling Bias

The polls/surveys only exist on Elixir Forum. Only folks who regularly engage with this platform even have a chance of seeing the proposal or the poll.

Nonresponse Bias

The proposal and discussion are long. Folks might not have the mental bandwidth to additionally form an opinion and respond right away and they also might not come back to fill out the poll.

Response Bias

The existing polls don’t have consistent scales or answer sets.

Platform Bias

We might have a couple other systemic flaws related to trying to conduct a survey on Elixir Forum–not intended as a slight; it’s not designed for that.

• We can’t hide the results from respondents (encouraging follow-on effects of “oh, dear, my preferred option is losing. Hey, X, you should vote for this.”)
• Requires an Elixir Forum account.
• We can’t randomize the order of the options.
• We can’t randomly reverse the order of scales.
• We can’t randomize the order of questions.

Recommendations

Listen, I’m gonna make some mistakes here too. Accurate polling isn’t my job. These are better, but not necessarily good.

• What is your opinion of this feature proposal in its original form? [Danger: acquiescence bias]

  • Definitely Should Not Be Feature
  • Probably Should Not Be Feature
  • Probably Should Be Feature
  • Definitely Should Be Feature
  • Prefer not to respond

• What is your opinion on this feature proposal with the structs-only variation?

  • Definitely Should Not Be Feature
  • Probably Should Not Be Feature
  • Probably Should Be Feature
  • Definitely Should Be Feature
  • Prefer not to respond

• What is your opinion on mixing field puns with key value pairs?

  • Field puns strictly first
  • Key value pairs strictly first
  • Field puns and key value pairs intermixed
  • Indifferent
  • None of the above

If we must include something less scientific, we can include Net Promoter Score.

• If this were a feature of Elixir, on a scale of 1 to 10, how likely are you to recommend its use in codebases?

If we’re serious about getting community feedback, then we’d want the survey linked from the mailing list, IRC, Slack, Twitter, etc. If we do so, we might not want to like to a survey embedded in this discussion for the above platform bias reasons I mentioned.
I’m hesitant to endorse any specific vendor for this, but even free/gratis Google Forms would be a (slight) improvement.

It is very likely that I would make a biased poll too, because I have zero knowledge on how to conduct surveys. So I would love to see how you would phrase the options here or a similar example, because I believe many of us would definitely learn a lot.

Good write-up, but please, anything but NPS

Adding my 2c, strongly recommend making the survey options:

The easier it is for the respondent, the better, IMHO.