How to reduce bias in surveys and ideas on garnering community feedback

There’s lots of material out there about how to reduce bias and conduct sound surveys, but at a bare minimum, don’t start excluding options because people are telling you something you don’t want to hear.

You completely reversed the meaning to suit your position. I meant what I said: These variants are also wholly unsuitable.

2 Likes

Maybe that’s the way it came off, but it’s not what I intended. The new poll automatically has 38 against it from the first poll. I can’t carry over votes to a new poll, so this was the compromise I came up with. In my mind, that’s biasing against this being added, because everyone who has said they’re for it would have to revote, whereas, everyone who said no doesn’t.

If you have a better set of options you would like to see, could you list them? I’m working with the mods to change the survey which I’m not able to do right now.

It is very likely that I would make a biased poll too, because I have zero knowledge on how to conduct surveys. So I would love to see how you would phrase the options here or a similar example, because I believe many of us would definitely learn a lot.

6 Likes

Several of the online survey companies have decent guidance about reducing bias in survey questions.

I don’t necessarily endorse these vendors, but they have a decent coverage of biases that sneak into survey and a vested interest in helping their customers reduce them.

I’ll outline just a few of the issues.

Sampling Bias

The polls/surveys only exist on Elixir Forum. Only folks who regularly engage with this platform even have a chance of seeing the proposal or the poll.

Nonresponse Bias

The proposal and discussion are long. Folks might not have the mental bandwidth to additionally form an opinion and respond right away and they also might not come back to fill out the poll.

Response Bias

The existing polls don’t have consistent scales or answer sets.

Platform Bias

We might have a couple other systemic flaws related to trying to conduct a survey on Elixir Forum–not intended as a slight; it’s not designed for that.

  • We can’t hide the results from respondents (encouraging follow-on effects of “oh, dear, my preferred option is losing. Hey, X, you should vote for this.”)
  • Requires an Elixir Forum account.
  • We can’t randomize the order of the options.
  • We can’t randomly reverse the order of scales.
  • We can’t randomize the order of questions.

Recommendations

Listen, I’m gonna make some mistakes here too. Accurate polling isn’t my job. These are better, but not necessarily good.

  • What is your opinion of this feature proposal in its original form? [Danger: acquiescence bias]

    • Definitely Should Not Be Feature
    • Probably Should Not Be Feature
    • Probably Should Be Feature
    • Definitely Should Be Feature
    • Prefer not to respond
  • What is your opinion on this feature proposal with the structs-only variation?

    • Definitely Should Not Be Feature
    • Probably Should Not Be Feature
    • Probably Should Be Feature
    • Definitely Should Be Feature
    • Prefer not to respond
  • What is your opinion on mixing field puns with key value pairs?

    • Field puns strictly first
    • Key value pairs strictly first
    • Field puns and key value pairs intermixed
    • Indifferent
    • None of the above

If we must include something less scientific, we can include Net Promoter Score. :stuck_out_tongue_winking_eye:

  • If this were a feature of Elixir, on a scale of 1 to 10, how likely are you to recommend its use in codebases?

If we’re serious about getting community feedback, then we’d want the survey linked from the mailing list, IRC, Slack, Twitter, etc. If we do so, we might not want to like to a survey embedded in this discussion for the above platform bias reasons I mentioned.
I’m hesitant to endorse any specific vendor for this, but even free/gratis Google Forms would be a (slight) improvement.

8 Likes

Good write-up, but please, anything but NPS :stuck_out_tongue:

Adding my 2c, strongly recommend making the survey options: :+1: :woman_shrugging: :-1:

The easier it is for the respondent, the better, IMHO.

3 Likes

There are some good tips there :023:

I like some of your recommendations! I often structure poll questions like this:

I’m sure you’ll agree the format isn’t always a good fit for every situation. Also as you said, despite knowing all this about conducting surveys even you yourself may not get it quite right, or be able to please everyone. It’s not an easy job and so we may need to be a little forgiving if people don’t always get it right. (On a forum like this it’s rarely a problem as things can be fixed in subsequent iterations.)

You raise some valid points Jeff, however I think it’s important to bear in mind that there is no perfect solution. Sometimes we just have to work with what we have or what we (by we I mean individual users) think may be the best approach for their ideas. There are of course downsides to relying on survey sites.

For instance it could be argued that going to the time and trouble of creating a survey on a survey platform may end up resulting in considerably fewer votes and/or less overall engagement (partly because people may be less likely to fill out surveys and partly because other platforms, like forums, may be more likely to get responses because people are already active on them and because getting involved is usually as easy as making a post).

It may also mean a less dynamic conversation taking place because people are unlikely to post multiple surveys with updated proposals and even fewer likely to continually take part in them. It may even be near impossible to have a structured discussion as can be had on a forum - by survey alone.

Having said that, we have been internally discussing how best to tackle proposals.

One idea put forward is that suggestions and proposals go through several phases, but perhaps importantly, where they all end up being posted to the proper channel for proposals once the author is confident they have enough information to provide to the respective teams.

So for example, a user who wants to post a language proposal can choose a platform of their choice (whether that be this forum, slack, IRC) as an initial place to talk about their suggestion/idea (they’re of course also free to cross-post as they see fit). Through this they garner feedback, and when they’re confident it is something that is worth putting forward for consideration, they (or a number of participants if they have gathered enough support) create a comprehensive proposal. That proposal is then discussed (and if need be can itself go through several iterations) and when the author is happy, they take that proposal and post it in the proper channel (so that would be Elixir Lang Core mailing list for language proposals, issue trackers for Phoenix, Ecto etc). However, not only that, their submission would include a brief description, links to the relevant threads and polls (or documents and surveys if the discussion is not taking place on a forum) as well as a summary of the comments and general feeling of users of the final proposal. That way the Core Teams can easily factor in opinions (either by looking at the summary or by looking through the supplied links/documents) and then have a (final?) discussion as they normally would before they make a decision.

This means it doesn’t matter which platform you choose to discuss your ideas as the final discussion (and decision) will take place in the usual channels where they always have.

It won’t be perfect but it will let users decide how (and where) they want to go about forming their proposal and then once it is submitted to the proper channel everyone can get involved like they always could - since those are the places where such proposals take place usually.

Anyway, it’s just one of the ideas at present - as you might imagine, we’ve discussed quite a few.

Separate to user-submitted proposals, it’s worth mentioning that sometimes core teams or individual library authors may feel that in certain cases a particular platform may provide advantages or enough of a sample to be adequate for their needs. So in some cases quick feedback on Twitter or IRC might suffice, in others a forum discussion or mailing list thread may be preferred. These channels will depend on what each individual project deems fit and will usually be detailed on their site somewhere, or cross posted to Twitter (or whichever platform where they feel will be seen the most) if no official policy has been made (eg: “Please fill out this survey” Or, “Please let us know your thoughts about this proposal” which may be a link to a thread or some other document).


Our posts are not strictly on-topic btw so we’ll have to move them into a separate thread shortly.

3 Likes