Pol.is Case Study: Temperature Check
A common use case for pol.is is temperature checking a community. Let’s take a look at a successful case study.
Scoop (http://www.scoop.co.nz/), an independent newspaper in New Zealand, wanted to check in with its readership base regarding the country’s obesity epidemic. You can find the original story here. Starting from the top, we can see how they framed this question:
Proper framing is essential for pol.is conversations. Framing helps participants understand the intended scope of the conversation and helps decide which statements are ‘on topic’.
Framing the conversation is the responsibility of whoever creates the conversation. It’s more art than science — but don’t worry! Participants will add more general and more specific comments, which can lead to follow up pol.is conversations. Using pol.is iteratively to zero in on specifics is very powerful.
If you’re in doubt, ask a more general question and let participants guide the direction — it will feel natural to get more specific.
Scoop ‘seeds’ its conversation
Next, Scoop further frames the pol.is conversation beyond just asking a question by ‘seeding it’ with diverse perspectives from various stakeholders they know already exist out in the community. This is simply adding comments for people to vote on as the account owner / administrator.
The text input to seed statements can be found in the ‘configure’ tab once you’ve started your conversation, seen at left.
It’s usually a good idea to seed around 10–15 diverse comments. This has a powerful effect on early participation. We’ve found that about 1 in 10 people leave a comment (whereas 9 in 10 only agree disagree and pass on statements submitted by others). Given this ratio, if there are no statements at the outset, it can take a little while for enough statements to build up to make the conversation meaningful. This is an impediment to data collection. So seed away! Great work, Scoop :)
Scoop then embedded the conversation on their site, using the pol.is script tag / embed code. If you want to embed a pol.is conversation on your site, you can grab the embed code in the ‘share and embed’ tab above.
What participants do
Before we move on to describing the kinds of insights produced by a pol.is conversation, we should take a second to understand what the participants do during the conversation.
The primary action is voting (agreeing, disagreeing or passing) on statements that have been submitted by other participants. These statements appear in random order. Participants may optionally submit statements of their own.
That’s it!
Moderation
When statements are submitted by participants, they show up to administrators as ‘unmoderated’ comments. By default, statements will enter the conversation and be seen by other participants as soon as they are submitted, and can then be moderated ‘out’ by whoever started the conversation.
If you’d like to prevent statements from being seen until they’ve been moderated ‘in’, head back to the ‘configure’ tab and toggle on ‘strict moderation’.
Near future versions of pol.is — maybe even by the time you read this! — will include optional ‘crowd moderation’ functionality. Offloading moderation of statements to the crowd:
- reduces moderation burden on administrators
- provides more transparency regarding which statements are ‘in’ and why
- allows the AI to account for the fact that different groups ‘value’ statements differently across several dimensions — such as whether they are factual, whether they are important, whether they are relevant and whether they are offensive or abusive
Metadata
You may also notice in the above image that there is a checkbox next to each comment labeled ‘metadata’. Toggle it, and statements will be shown in the report you generate after the conversation is over. These comments will no longer affect the layout of the visualization. They will still be voted on by participants. Metadata comments appear as an annotation on each group in the report (see: “reporting”, below, to see metadata in action).
This is helpful because statements such as ‘I identify as male’ and ‘I live in a rural area’ are interesting, but will split the conversation into groups along those lines, rather than more subtle differences in perspective, which is what we’re looking for.
Insights pol.is produces
A core idea in pol.is is that all of the participants can gain insights about their own position relative to others in the conversation while they are participating.
This is accomplished by visualizing, in real time, groups of people who voted similarly. Each person sees their own social profile (or anonymous image, if they haven’t logged in), inside of a group of people that voted like they did.
Without clicking on anything, we can see there are three opinion groups, and one of them is significantly bigger than the others.
The buttons at the bottom allow us to explore the visualization and see which statements brought each group together, which statements differentiate each group from the other groups, and which statements everyone could get behind.
Let’s take a look at group B, the largest group, by clicking on it or clicking its button:
The first comment we’ll evaluate is #41. (Statements are numbered based on the order they were submitted.)
There is a significant disagreement between group B and group C regarding issues of personal choice, societal costs, and regulations. Almost everyone who voted from group B agreed with statement #41, most people from group C who voted disagreed.
This statement is quite rich with meaning. It could be broken up, or even serve as a prompt for another entire conversation.
Let’s turn to group C, and find out if there are comments that people in that group were uniquely likely to agree with…
Here, we can clearly see a continuation of this theme of personal responsibility and cost. In #31, Group C is concerned that raising the price of food and drink will hurt vulnerable minority communities.
This would fit the theme of group C being hesitant to go along with the idea that regulation should restrict the behavior of individuals and hold them responsible.
Let’s look at one more difference:
Comment #42 seals the pattern. We can now form a narrative regarding the main axis of disagreement in the conversation.
There is a highly concerned group of 20 people who are hesitant to deal with obesity at the individual level by raising taxes to discourage purchases, fearing new taxes on food will hurt vulnerable populations.
The larger group is more concerned with costs borne by the government than keeping it cheap to eat bad food, and seems convinced that higher taxes will work.
By looking at these differences across several comments, groups begin to accrue a ‘personality’ of a kind.
But is there anything these groups agree on?
Yes! Comment #17, among others. This statement has broad support. New Zealanders reading Scoop are very likely to believe that the media is a factor in both the problem and the solution.
Here we find several encouraging things: common ground, a challenge, a consensus, a vector for action that has broad support. Despite the fact that there are differences, there are commonalities.
What’s especially gratifying in many pol.is conversations is that the crowd knows all this already — ie., some acute observer knows how to bring everyone together. It’s just about giving people the opportunity to do so, in a new information structure that surfaces and values it.
Reporting
If you toggle ‘metadata’ on one or more comments, the pol.is report will add those comments as an annotation to each group in the report, rather than putting them in the visualization. You can then compare groups on this metric.
The report is static and reads top to bottom, unlike the interactive visualization.
In the report, the comments that differentiate each group can be read top to bottom, and it quickly becomes clear what ‘makes this group a group’.
At left are the comments for group B, the largest group that we examined in the conversation above.
The report also features a list of statements that were passed on by more than 30% of participants who saw them.
These statements can map knowledge of facts, events and possibilities. Statements with high pass rates can also serve as areas of education, where people are admitting they are uncertain, as well as openings in group dialogue.
What next?
Pol.is is meant to be a kind of ‘radar’ for human perception. What do you do with it all once you see what’s on the scope?
There’s no one answer — what comes next is as varied as the missions, and internal processes, of the organizations that use it. Maybe you’ll run another pol.is conversation to gain more insight. Maybe you’ll send the information up the chain to decision makers who learn something new. Maybe you’ll let people you’re in charge of know that you hear them.
In the case of Scoop, they produced a 30+ page report deliverable to their community and to decision makers, including their methods and ways the process could be improved next time. They are also planning on running multiple conversations during their election coverage.
Wrapping up
If you’ve followed along this far, you’ve gotten a sense of the whole pol.is cycle — from framing a conversation and seeding existing perspectives, through participation and moderation, and from gathering insights in real-time to generating a report you can deliver to others.
Head to https://pol.is to get started. You can send us a message within the app at any time if you have questions!
