Clickbait title but this was, partially at least, the debate I was having on Twitter earlier today.

It all started when Frank Graves, from Ekos, tweeted this:

As well as this along the conversation:

Then Darrel Bricker, from Ipsos, followed through by tweeting this:

And finally Jean-Marc Léger (from Léger360) weighed in as well:

Ok, so let's break down the various issues raised here.

1. Seat projections were bad in 2015 while polls were accurate

Yes and no. Yes seat projections, mine included, were definitely quite off. Most polls aggregators and forecasters had the Liberals winning a minority. I even had them winning a relatively small one. They ultimately won a comfortable majority. My model failed on many levels.

With that said, I get tired of the claim that polls were right. Yes polls did relatively well in 2015, but let's not pretend that they didn't underestimate the Liberals, in particular in Quebec. A raw poll average of the last week puts the Liberals at 37.8%. They ultimately got 39.5%. This is almost 2 full points below. Only one poll -Forum- actually overestimated the LPC. For polls to be "right", the poll average should be closer.

I know what you'll say: yes but margins of error. Fair enough. But the poll average has really low margins of error. So missing the mark by 1.7 points isn't nothing. For projections, a 2-points difference can easily turn a minority into a majority. Seat projections -actually seats in an election- are very sensitive to every percentage point. We don't get the luxury of being within 2-3% of the correct result and call it a success.

The polls were particularly bad in Quebec where they underestimated the Liberals by over 5 points while overestimating the NDP by 1. This alone can account for most of the reasons while seat projections had the Liberals with a minority. This isn't the only one. Models, like mine, probably failed to forecast the increased turnout in some key ridings.

I personally underestimated the Liberals even more because I don't allocate undecided proportionally. My method has worked well in Quebec, Alberta and BC but for 2015, it made me miss the Liberals even more. I stand by my choice of going with a non-proportional allocation as evidence clealry shows incumbents tend to be underestimated. Also, my final average was actually closer, overall, than the one from 308/CBC for instance. It's just that it made me miss the most important party that time. With that said, it made me get the Conservatives closer to their real result. I will take the blame on this one as I should have accounted for the last minute momentum towards the Liberals but I'll continue to aggregate polls with a non-proportional allocations of the undecided.

Still, my model, for all its shortcomings in 2015, would have predicted a Liberal majority with 170 seats with the correct percentages. So while I agree I need to improve the model -particularly when turnout can change greatly- I also don't think 2015 should be remembered only as the victory of the pollsters and the failure of seat projections. And polls don't have a share of the blame in the failure of the projections.

2. Polls aggregators are potentially bad for democracy

I can't see how. If anything, I find it way more misleading when the media report on only one poll instead of doing an average. In recent weeks for instance, we've had some Forum polls that were very different from the average -I talked about them in the past- and some newspapers ran with it. This is way more dangerous and misleading than a careful aggregation. One single poll is subject to much higher margins of error than an average (well except for 2015 where, somehow, the latest Nanos or Forum polls would have done better alone even at the regional level despite ridiculously small sample size...). I don't buy for one second the argument that it'd be less shameful or dangerous for democracy if newspapers or the medias were paying for polls and analyzing them separately. I don't even see the connection.

As for the risk of influencing the election. Meh. We hear the same arguments against polls themselves. There as well, polls aggregators at least provide a better analysis. I see seat projections as providing information and information is always welcome.

The bigger debate here was also how the medias aren't paying for polls anymore and instead "steal" the numbers from other polls they didn't pay for. I understand why it could be frustrating to see the giant CBC not ordering polls and getting a ton of traffic by simply averaging the other polls. At the same time, if pollsters aren't happy with that, they could just stop providing public polls. My understanding is that political polls have long been a way for pollsters to get exposure, not a money maker. And if that's the objective, then they should be happy sites like mine mention them. But there is maybe a good case for medias to pay for polls.

As for asking for permission to use their numbers... Sorry but this doesn't hold in my view. The numbers are publicly published. Why should I, or anyone else, actually ask for permission to use them? Bricker said he doesn't think his data should be used for riding-level projections and doesn't endorse it. He has the right to think this but that's pretty much it. I don't even understand the logic. The numbers themselves aren't intellectual property or anything, they are pure information.

Just to be clear, I always had a good relationship with all the pollsters mentioned here. But on this issue, we won't agree.

I'm not a lawyer but I'd like to hear the opinion of one.

3. The seat projection models aren't transparent enough.

I'm not sure I agree here. Eric Grenier et 308/CBC always put pretty much the entire methodology of his simple model on his site. Read the blog and you can recreate the model in 10 minutes. I don't personally put all the coefficients and details, but I provide enough of the methodology. Also, my model is publicly available for everyone to use. Some do tend to think they are the sh*t and have a secret, magical recipe. The Signal from Vox Populi was a perfect example of that.

Look, I'm not totally against publishing the source code and everything on github for instance but I'm not sure pollsters are the ones who should tell me to be more transparent.

Pollsters publish very few details of their methodology. Ipsos publishes more detailed tables but that's all and they are almost the only ones. None publishes the exact sampling method or weighting process. Let's remember that those are essential. In the US, the same raw data was giving to four scientists and they all got different estimates out of the same raw data. There was one poll last US election that gave access to its full data (ironically, that turned out to be one of the least accurate poll... well at least the estimates based on the original weighting).

If the pollsters don't have to publish their "secret sauce", I'm not sure why polls aggregators should.
Here are my first Ontario projections since the election of Doug Ford as the leader of the Progressive Conservative party. The least we can say is that changing leader so close to the election isn't currently hurting the PC, far from it. If anything, Ford's popularity in Toronto and the 905 means the Tories could win a landslide majority.

Before going into the deeper analysis, here are some bullet points for you, busy readers

- The PC's chances of winning are higher today under Ford than they were under Patrick Brown despite Ford being quite unpopular/polarizing among some demographics.

- The Liberals are in a dire situation because they are now polling consistently below 30%. It means that even their safe region of the GTA is now in jeopardy. A nightmare scenario for the OLP is now absolutely possible.

- My model was adjusted to account for the higher than predicted popularity of the PC and Ford in the 416 (Toronto) and the 905 (the GTA). These adjustments might change as new polls come in.

- The NDP could potentially benefit from Ford's election. Its leader, Andrea Horwath is often polled as the least unpopular of the three leaders. But so far the polling numbers haven't move in her directions. The desire for change -very strong- is currently going with the Tories.

Based on the polls conducted since Ford's election (One Forum, Ipsos, Leger, Campaign Research and Mainstreet research today), we get the following projections.

Voting intentions; Seat projections with confidence intervals; Chances of winning.

The chances for the PC are just shy of 100%. If the election was tomorrow, there would be a tiny chance for the Liberals. It'd however require one of the biggest polling mistakes in Canadian history as well as a very efficient vote. Make no mistake, that would be a much bigger surprise than Trump's win for instance.

Please remember that these projections and probabilities are to be interpreted as if the election was tomorrow. But it isn't. With almost 3 months until then, things can still change. But as of right now, there is very little uncertainty as to which party is the most likely to win. Even a majority seems more than likely at this point.

My model used to be quite harsh on the Tories because of the historical trends in the GTA -where the PC didn't do well at all over the last 10 years. However, we have enough polls with regional numbers to see that the current situation is very different. I have therefore made the necessary adjustments. I'll keep working on it obviously but I can't continue to ignore the fact the Conservatives are polling higher than expected. I also need to make sure my regions match with the ones from the various pollsters.

Notice also that the current projections have the PC at the high end of its confidence interval while it's the opposite for the Liberals. This means there are a good 10 seats that are currently projected to go PC by a very small margin. I'll try to be more formal about vote efficiency as we approach the election. It might also be a question of tweaking the regional adjustments. For now though, if you want to be more conservative in your predictions, put the OLP above 20 and the PC closer to 80.

Doug Ford can be a polarizing figure. Polls were showing that Christine Elliott might have been a safer choice. Multiple polls have shown that some soft-PC voters could be turn off by Ford as leader. While I don't deny it, one must admit that voting intentions since he became leader have been very good for his party (and we can't really observe any real trend before/after so far. Also, Ford's popularity in Toronto (and some of the 905) is exactly what his party needed in order to secure a majority. We'll obviously have to wait and see, especially during the campaign. Still, the PC members definitely took some risk by going with such a polarizing figure. The fact the Liberal government of Kathleen Wynne is so unpopular and the strong desire for change are currently compensating for this.

The Liberal vote can become quite inefficient below 30%, especially if the PC is gaining in the GTA. This explains why the NDP could potentially win more seats with fewer votes. And as mentioned above, with Ford being at times unpopular and a strong desire for change, the NDP might actually create a surprise in June. I'd absolutely not be shocked by a NDP victory at the end. It's all going to be about representing change for Ontarians. Right now they seem to go with the Conservatives but it could change.

Here below are the riding by riding projections. I have also updated the simulator.

Le Journal de Montréal a un nouveau sondage fédéral par Léger ce matin plaçant les Libéraux de Trudeau à égalité à 38% avec les Conservateurs de Scheer. Au-delà des intentions des votes brutes, ce sondage vient confirmer une importante et récente baisse du PLC. Et la raison semble être de plus en claire: le voyage en Inde durant lequel la couverture médiatique s'est surtout concentrée sur les "vêtements" que Trudeau portait. Un voyage qui est perçu par beaucoup d'électeurs comme un échec semble-t-il.

Ce n'est pas la première fois que le PLC connaît une baisse dans les sondages, mais la récente chute semble un peu plus importante. Il restera à voir si elle sera plus permanente que les précédentes.

Commençons par comparer les intentions pré- et post-voyage. Nous avons maintenant suffisamment de sondages faits par différentes firmes pour comparer le avant/après. Le tableau ci-dessous indique la variation du PLC dans ces sondages. Je compare firme par firme afin d'éliminer les effets de chacune.

Le PLC chute dans tous les sondages mais les comparaisons ne sont pas parfaites. Léger par exemple n'avait pas fait de sondage depuis fin novembre alors que Nanos fait une moyenne sur 4 semaines -donc inclut des observations avant et après le voyage dans le dernier sondage et la chute de 2 points pourrait sous-estimer l'effet. Abacus, Forum et, dans une moindre mesure Ipsos, nous fournissent les meilleurs estimations de l'effet. La moyenne pondérée donne moins de poids à Léger et Nanos. Si je n'utilisais que les trois sondages mentionnés, la baisse serait de de 4 points.

Comparons cela à l'évolution des Conservateurs.

Au total, on parle d'un "swing" de près de 8 points entre les deux partis. C'est une variation très importante avec possiblement des conséquences majeures en termes de sièges (voir ci-dessous).

Comparons cela à ce qui est arrivé depuis 2015. Le graphique ci-dessous, gracieuseté de Wikipédia, nous montrent que le PLC a essentiellement connu 3 chutes majeures et rapides depuis le début de 2017. La 1ere, entre mai et juin 2017 correspond essentiellement à l'arrivée d'Andrew Scheer à la tête des Conservateurs. La 2e, vers septembre-octobre de la même année avait été causé par la proposition de réforme fiscale des travailleurs autonomes -le gouvernement Trudeau avait en partie fait marche arrière après avoir observé le mécontentement.

La différence entre ces deux baisses et l'actuelle? Le PLC restait devant le PCC en moyenne -à condition de faire une moyenne sur plusieurs semaines. Or, tel que vous pouvez le voir sur le graphique, ce n'est pas le cas actuellement. Il faut cependant reconnaître que c'est aussi en partie due à une tendance à plus long terme qui voit le PLC en baisse et les Conservateurs en hausse.

Maintenant que nous avons établi le fait que le PLC et Trudeau sont effectivement à la baisse récemment, essayons de répondre à quelques interrogations:

1. La baisse est-elle vraiment causée uniquement par le voyage en Inde de la famille Trudeau?

Il semble bel et bien que ce soit, à tout le moins, une des raisons principales. La popularité du Premier Ministre a subit un coup. Le sondage Léger indique que 66% considèrent le voyage comme un échec par exemple.

Bien sûr, la période avant/après le voyage coincide aussi avec le budget. Ce dernier ne semble pas plaire aux électeurs avec seulement 29% de satisfaits selon Trudeau, toujours selon Léger. Ipsos nous montre des résultats similaire sur le fond (budget pas populaire, voyage en Inde manqué).

Certains pourront se demander s'il est logique que les électeurs répondent si fortement à ce qui était, sommes toutes, un faux-pas vestimentaire. On ne parle pas ici d'une réforme majeure ou de l'instauration d'une politique qui fera des milliers de perdants au pays. Non, le voyage en Inde n'aura logiquement pas vraiment de conséquences à long terme -ou même à court terme- sur la vie des Canadiens. Ce voyage a terni l'image de Justin Trudeau et certains accuseront les électeurs de baser leur vote sur des raisons "stupides" ou sans conséquence. Je ne suis pas ici pour débattre des mérite de cette position, mais remarquez que la popularité et l'image sont une lame à double tranchant. Trudeau est devenu Premier Ministre essentiellement grâce à sa popularité. Le PLC n'a pas gagné l'élection de 2015 en proposant des politiques que tout le monde aimait par exemple. Andrew Coyne abonde dans la même direction que moi.

Au final, l'effet observé est probablement un mélange du voyage en Inde et du budget.

2. L'effet va-t-il persister?

N'est-ce pas là la vraie question? La réponse est: je ne sais pas. Lors des deux précédentes baisses en 2017, le PLC avait rapidement rebondi pour reprendre une comfortable avance. Cela étant dit, il semble y avoir une légère tendance à long terme qui est négative pour le PLC. Aussi, il semblerait que le Premier Minister soit maintenant bien plus vulnérable qu'auparavant. Son image et sa popularité ont baissé et ont possiblement ouvert la porte à Andrew Scheer pour au moins nous offrir une élection compétitive en 2019. Cela dépendra cependant en grande partie du Québec où le Premier Ministre et son parti restent largement devant.


Quoiqu'il en en advienne, on me basant sur les sondages post-voyage, voici ce à quoi ressembleraient les projections actuelles.

À venir, il me faut les données par province du dernier Léger.


Les deux derniers sondages de Léger et Mainstreet continuent de montrer la CAQ devant -mais par des marges très, très différentes- et le PLQ en baisse. Par rapport à l'année passée, la CAQ a augmenté en moyenne de 10 points -encore une fois, cela dépend de si vous regardez les sondages Léger ou Mainstreet- et le PQ en a perdu environ 9.

En termes de projections, voici ce que cela donnerait si l'élection était demain. Ces projections sont basées sur les sondages Léger, Mainstreet et Ipsos entre février et mars 2018.

Tout un changement par rapport à 2017 où le PLQ avait des chances de gagner au-dessus de 90%.

Les chances d'une majorité CAQ sont de 57%.

Et oui je sais il me faudra mettre à jour le logo de QS.

Si la baisse du PQ et la hausse de la CAQ peuvent s'expliquer par les électeurs désireux de changer de gouvernment -ils sont nombreux!- se réunissant auprès d'un parti, la baisse des Libéraux mérite possiblement davantage d'attention.

Pendant longtemps la situation au Québec était relativement stable. Le gouvernement était impopulaire mais restait au-dessus des 30%. Avec le PQ et la CAQ se divisant l'opposition, Philippe Couillard était en fait favori combien même le taux de satisfaction était très bas. En d'autres mots, une situation un peu bizarre.

Au cours de la dernière année, nous avons vu Québec Solidaire en forte hausse après l'arrivée de Gabriel Nadeau-Dubois. Une poussée qui s'est résorbée, du moins en partie. Nous avons aussi assisté au déclin du PQ et à la lente mais certaine progression de la CAQ (+15.5 points selon Léger, +4.8% selon Mainstreet qui avait ce parti plus élevé en 2017 et plus bas en 2018 par rapport à Léger; Une différence majeure quand même).

La plus récente tendance cependant est la baisse du PLQ. Un parti qui se retrouve maintenant sous les 30%, une barre souvent considérée comme symbolique. Aussi, en raison du soutient des non-francophones, les Libéraux ne peuvent chuter sous cette barre que si les francophones l'abandonnent massivement. Sauf bien sûr si les non-franco délaissent finalement le PLQ...

Le tableau ci-dessous vous montre la moyenne pour les quatre principaux partis entre janvier et mars 2017 et 2018 en me basant sur les sondages Léger et Mainstreet. J'ai fait la moyenne sur plusieurs mois afin de vraiment mesurer les changements à long termes.

Une perte de 5 points pour le PLQ n'apparaît peut-être pas comme aussi significative que la chute du PQ, mais dans les faits, elle est plus difficile à expliquer. Tel que mentionné ci-dessus, la chute du PQ peut s'expliquer par le fait que les électeurs anti-PLQ ont finalement fait un choix, du moins temporairement. L'effet est particulièrement visible depuis la partielle dans Louis-Hébert.

Alors, comment le PLQ se retrouve-t-il sous les 30%? A-t-il perdu chez les francophones? Le tableau ci-dessous répond à la question.

Tel que vous pouvez le voir, la chute est bien plus importante chez les non-francophones. Nous assistons possiblement ici à un phénomène nouveau. Le PLQ reste naturellement largement en avance dans ce groupe mais la CAQ semble y faire des gains. Ces gains ne seraient pas suffisants pour gagner dans l'ouest de l'île, mais les Libéraux devraient commencer à se faire du souci.

D'un autre côté cela signifie possiblement que les projections sous-estiment les chances Libérales. Si la chute observée est surtout chez les non-franco, cela ne devrait pas coûter trop cher en termes de sièges. Du moins tant que la chute reste contenue. Le PLQ a de la marge dans ces comtés. En plus, il faudra voir si les non-francophones délaissent vraiment le PLQ durant une élection.

Après la guerre des sondeurs Léger-Crop, la bataille Léger-Mainstreet?

Si vous vous souvenez bien, vers la fin de 2016, nous avions une "guerre des sondeurs" entre Léger et Crop. Ces deux firmes nous montrant des chiffres et tendances fort différents pendant plusieurs mois.

Il semble que nous ayons maintenant une nouvelle bataille mais entre Léger et Mainstreet. La principale différence entre janvier et mars 2018 concerne la CAQ. Alors que Léger a ce parti vers les 37-39% après répartition des indécis, Mainstreet ne place cette formation que vers les 31-32%. Il s'agît d'une différence massive! L'avance de la CAQ sur le PLQ est de 11 points dans le dernier Léger mais seulement 4 dans le dernier Mainstreet. Pour vous donner un ordre d'idée des conséquences, les chances de gagner le plus de sièges de la CAQ sont de 86.5% (et de 32% pour une majorité) en me basant uniquement sur les sondages Mainstreet et 99.8% (83% pour une majorité) en me basant sur ceux de Léger.

Il y a bien sûr les marges d'erreur à tenir compte mais la différence entre les deux firmes est stable depuis 2-3 mois. Qui qui dit vrai? Quasi impossible à dire. Remarquez que le sondage Ipsos de février avait la CAQ à 34%... soit à peu près entre les deux firmes! Il semble ainsi que faire une moyenne reste la meilleure solution. Mais gardons un oeil sur la différence entre les deux firmes.
The most recent Ipsos poll has generated a fair share of media coverage -good for them! That's what political polls are good for actually. The reason being that it was showing the Conservatives of Andrew Scheer well ahead of the Liberals of Prime Minister Justin Trudeau (38% vs 33% to be exact). Our electoral system being what it is, a 5-points lead would indeed very likely result in a Tory victory.

My facebook as well as my Google Now feed were full of these kind of headlines, in both official languages. I was slightly annoyed at first because I always strongly believe that one poll doesn't create a trend. And the fact is that even the most recent trend was still placing Trudeau in a very favourable position. In other words, I started writing this blog post thinking the Ipsos poll was a likely outlier and the medias should maybe avoid using the clickbait headlines. But after looking at the numbers, my assertion is mostly: maybe. Yes, the Conservatives would maybe win the election. I still don't think this is the most likely scenario but it's also not that far a stretch. Here are the details of how I arrive at this conclusion.

I did what a "poll analyst" would do. I collected the most recent polls from various pollsters -you can find them here on Wikipedia. The caveat being that Nanos is quite literally spamming this list because they release partially updated polls every week. Nanos is a great firm (that has had an incredible, if not partially lucky, track record for federal elections), but I'm against using all their polls in such an average. Doing so -and other sites do that- means your average/projections are now based way too much on one single firm. So I used the most recent polls from Ipsos, Forum, Nanos, Abacus, Campaign and Mainstreet. That's a pretty good variety with the only down side that these polls were conducted over a period of 2 months. Hey, whatever, we don't get 7 polls a week like during a campaign.

You have these polls in the table below. I also added two older polls from Ipsos and Forum. You'll understand why very soon.

As you can see, not only is the CPC ahead in the latest Ipsos, it's also the case in the latest Forum. Now, I know what you'll tell me. Forum can't be trusted, right? Moreover, this firm has had the Conservatives ahead for a while (and has been alone in showing such a thing). I have written about Forum before. Their polls do tend to occasionally be weird and more volatile but as far as their track record is concerned, I have no reason to exclude them.

The general consensus in the articles about the Ipsos polls is that Trudeau is paying the cost of his less-than-optimal India trip. Or, at the very least, that this trip and the reactions to it are symptomatic of a change of mood and attitude towards Trudeau. I don't want to debate whether this trip was as disastrous as some said it was, but I think it's fair to say that it didn't turn out as expected for Trudeau. At the very least the media coverage has been fairly negative and you can only find people defending it as a success on some sub-forums on the internet.

Please notice that the Ipsos and Forum polls were conducted after the recent federal budget. Whether this helps Trudeau and the Liberals or not is, there as well, debatable. And there as well, I don't want to enter this debate. But it could be that the observed effect referred here as the India-trip one is actually a budget-related one.

So let's just assume that if the numbers for the Liberals did indeed drop recently, this is due to the India trip. I always tell my students that the real life is complicated and messy and that you can rarely pin down the reasons for a phenomenon onto one single cause. I strongly believe this is the case here as well but being able to refer to the effect as the "India trip" one makes it easier as far as writing this post.

So far we have a Forum poll that many wouldn't trust and one Ipsos poll that could simply be an outlier. Small sample to convincingly declare the Liberals would lose and that the January-February polls are now invalid. Fair enough. But we can still try to measure this "India trip effect". To do so, let's compare the Ipsos poll to the previous Ipsos' numbers (and let's do the same with Forum). This way, even if there is a so-called "house effect" -and there is definitely one for Forum, no matter if Forum is right or wrong- it'll be eliminated.

Doing so shows the Liberals down by 5-points according to Ipsos and 4-points for Forum. The two Forum polls are quite recent while we need to go back to December to find the previous Ipsos one. That's unfortunate obviously.

The Tories are up between 7 (Ipsos) and 3 points (Forum). Ipsos might be measuring a longer trend where the Tories have been slightly increasing since December, even before the India trip. The NDP is up between 1 and 2-points while the Bloc is down 1.5-points in average -the last point, along with the recent meltdown among the Bloc caucus deserve their own post that should come soon.

To me it's really interesting that we find such consistent differences between the two pollsters. Again, small sample size but consistent results, at least qualitatively.

So, who would win an election if it was held tomorrow? The answer really depends on whether you believe this India effect is real. If you don't and simply average the most recent polls -even giving the two most recent ones from Ipsos and Forum bigger weights- you find the Liberals and Conservatives essentially tied. If you give more weight to Nanos, you get the Grists slightly ahead. It doesn't really matter, please think big picture and agree that it'd be close with an advantage for the Liberals thanks to Quebec.

On the other hand if you believe that there is indeed a change of trend and that Ipsos/Forum are simply the first to observe it, then we need to adjust past polls for this effect. To do so, I did an average of the India effect estimates, giving the Forum differences a weight of 60% as the two polls were more directly comparable -this is a choice that actually reduces the effect, just so that we are clear or you believe I'm trying really hard to put the Tories ahead.

By doing this, I got the following projections.

Note for those who love details. My simulations obviously provincial numbers but those are too volatile to calculate province-specific India-trip effects. I thus used the national numbers to estimate the impact and uniformly applied it to each province. Some might object that it's a lot of assumptions to get to those projections and those people would be right. So you go ahead and write an internet comment about this.

What is striking is that even after adjusting the numbers and therefore giving the Tories a 6-points lead nationally, the Liberals would still have a chance. This is essentially because of Quebec. Even the recent Ipsos poll still puts Trudeau and his party as being incredibly popular in the French-speaking province. A lead that could translate into a ton of seats. Added to the Atlantic, that could allow the Liberals to win despite receiving way fewer votes nationwide.

The obvious million dollars question is still whether the "India trip" effect is real or not. And I can't answer that. Nobody can. What I'm saying is that we have some evidence pointing to such an effect existing and Andrew Scheer being more likely to become PM than he was a month ago. Would I go as far as saying he'd be favourite if the election was tomorrow? I'm leaning towards a no. But I also think that simply calling the trends observed by Forum and Ipsos as outliers would be an over simplification.