Comparing the Quality of Pro- and Anti-vaccination Online Information

Results

Descriptive Analysis

General tone of the webpage. Of the final sample of 1093 webpages, 514 were pro-vaccination (47 %), 471 were anti-vaccination (43 %), and the residual 108 were neutral or with an undefined tone toward the vaccination practice (10 %). The majority of anti-vaccination pages had radical and opposing viewpoints towards vaccinations (N = 430), whereas a minor part adopted a more moderate view on specific immunizations (anti-vaccination: reformist ⁵; N = 41). Due to the small number of moderate sites, the two anti-vaccination categories were collapsed in the comparative analysis. It has to be noted that the more or less even split between pro- and anti-vaccination webpages is a consequence of our sampling strategy and is functional at easing the comparison of the two subsets for the present analysis, but should not be taken as an indication about the prevalence of pro- and anti-vaccination views on the web.

Non-quality Attributes of Pro- and Anti-vaccination Online Information. Pro-vaccination webpages were mainly holding the following Domain Name Systems: .gov (46.7 %), .com (27.6 %), and .org (13.4 %). Strictly linked to this previous finding, half of the webpages were owned by governmental, public, and international institutions (54.1 %), followed by commercial (22.2 %), and not-for-profit organizations (13.4 %). On the other hand, anti-vaccination web sources were mainly published via the following DNSs: .com (48.8 %), .org (33.1 %), and .to (12.3 %). Slightly more than half of the anti-vaccination pages were owned by not-for-profit organizations (56.3 %), and by private individuals (14.4 %). For a consistent portion of anti-vaccination web sources the ownership type of the website was not mentioned clearly (21.7 %). Almost all the pro-vaccination webpages were inserted into general health information web portals (94 %), followed by general vaccine information websites (5.3 %), and the remaining 3 webpages were promoted via specific vaccine websites (0.6 %). Anti-vaccination webpages were mainly published either via general vaccine information websites (47.6 %) or into general health information web portals (49.7 %), and, for the remaining 13 webpages; the raters were not able to detect the scope of the websites that were including them.

For both subsets of the sample, the majority of the contents published were intended for lay people (patients or informal caregivers; 83.9 % of pro- vs. 98.9 of anti-vaccination webpages). However, roughly 30 % of the pro-vaccination pages delivered contents for healthcare professionals (29.2 %), whereas only 6 anti-vaccination pages delivered contents for this audience (1.6 %).

Approximately, 77 % of pro- and 80 % of anti-vaccination web sources offered information about more than a single injection. Barely 7 % of the pages in favor of vaccination provided contents about alternative or natural medicine (or treatments). On the other hand, slightly less than one fourth of the anti-vaccination webpages published those specific contents (23.6 %; less frequent than^[19,41]). Just about 15 % of pro- and only 5 % of anti-vaccination web sources exposed the target audience's Vaccination Recommendation Schedule (VRS). Out of the 514 webpages being in favor to vaccines, as expected, only three explained to users how to get vaccination exemptions legally (0.6 %). This information was present on slightly less than one-fifth of the anti-vaccination webpages (18.4 %; lower frequency than^[19,41]). Around 14 % of pro-vaccination pages advanced the theme of parents' rights and amplified responsibilities, especially when they decide not to get vaccinated their children. On the contrary, 37 % of anti-vaccination sources claimed the fact that parents' (or patients') rights are violated due to the strict health policies emanated by government bodies with public health executives (less frequent than^[19,41]). 43 % of the web sources opposed to the vaccination practice (lower frequency than^[41]), and only 4 % of pro-vaccination pages advanced the fact that potential conflict of interests between health professionals (e.g., physicians) and the pharmaceutical industry might be present. Both pro- and anti-vaccination webpages frequently provided contents about other immunization-related topics (e.g., vaccination information for travelers, health policy change; 56.2 % vs. 58.2 % respectively). The frequency scores for all evaluated attributes fulfilled by pro- and anti-vaccination webpages can be consulted in (see Additional file 2: Table S3 https://static-content.springer.com/esm/art%3A10.1186%2Fs12889-016-2722-9/MediaObjects/12889_2016_2722_MOESM2_ESM.docx).

Comparative Analysis: The Quality of pro- vs. Anti-vaccination Pages

Webpage Design Index. Webpages holding a favorable view of the vaccination practice showed on average more webpage design features compared to the opponent and neutral vaccination-related sources. There was a statistically significant difference (p < .01) when comparing the scores on the aggregated Webpage Design Index of the three groups based on the webpage's general tone [F(2.1090) = 15.35, p < 0.001]. Post hoc comparison using the Tukey HSD test showed that the mean Webpage Design Index score for pro-vaccination webpages (M = 3.65, SD = 0.87) was significantly higher than both the anti-vaccination pages (M = 3.27, SD = 1.36, p < 0.001), and the neutral (or undefined) set of webpages (M = 3.29, SD = 1.09, p = 0.007).

In more detail, pro-vaccination webpages offered better functioning links (p = 0.01), more often a bar menu (p < 0.001), or a search toolbar (p < 0.001). Indeed, 13 out of 15 webpages coded as having from medium to poor quality of links had an anti-vaccination tone. Roughly, one quarter of all the anti-vaccination and neutral webpages did not provide the menu bar feature, and 40 % of the total anti-vaccination subset did not have a user-friendly search toolbar (i.e., 19 % of the neutral webpages neither had this design feature). Images were present on about 60 % of both pro- and anti-vaccination online sources. Many pro- and anti-vaccination pages exposed pictures of people (42 % vs. 35 % respectively), followed by other kind of visuals (e.g., books; 28 % vs. 31.6 % respectively), and about one-fifth of both subsets showed images of drugs and medical equipment (18.7 % vs. 20.6 % respectively). Only about 12 % of pro- and around 8 % of anti-vaccination web sources provided information-supporting formats, such as graphs, tables or diagrams. Remarkably, the anti-vaccination subset provided more video formats than anti-vaccination webpages (27.6 % vs. 10.3 % respectively).

Videos, however, can be found most often on anti-vaccination webpages, p < 0.001. Almost 28 % of the webpages opposing vaccination included a video format, while only 10 % of pro-vaccination webpages incorporated video formats.

Interactivity Index. Websites holding a favorable or neutral view toward the vaccination practice provide more interactive services and tools compared to the opponent sources. A statistically significant difference (p < .05) was observed when comparing the three groups based on the webpage's general tone on the aggregated Interactivity Index [F(2.1090) = 4.43, p = 0.01]. Post hoc comparison using the Tukey HSD test showed that the mean Interactivity Index score for both pro-vaccination webpages (M = 3.71, SD = 0.96, p = 0.047), and the neutral (or undefined) subset (M = 3.86, SD = 1.74, p = 0.040) was significantly higher than the webpages which were against the vaccination practice (M = 3.49, SD = 1.70). None of the webpages evaluated fulfilled all the 6 categories grouped in this specific index. More specifically, 15 % of the anti-vaccination webpages did not provide any of the interactive tools or services coded for the present study while only 3 % of the pro-vaccination subset did not have interactivity, p < 0.001. E-mail or newsletter services were lacking in 17 % of the anti-vaccination webpages, and on 16 % of the neutral ones: p < 0.001. Interestingly, almost one quarter of the neutral web sources reviewed included forum interactions, whereas both anti- and pro-vaccination pages offered forums in only 6.6 and 4.7 % of the cases, respectively (p < 0.001). A chance to post comments, however, was more often offered on neutral (37 %) and anti-vaccination sites (31.8 %) than on the webpages of the pro-vaccination subset (11,5 %) (p < 0.001).

Health Related Content Index. Webpages holding a favorable view of the vaccination practice endorse more health information quality principles compared to the opponent and neutral vaccination-related online sources. The three groups based on the webpage's general tone presented significant differences (p < .01) on the aggregated Health Related Content Index mean score [F(2.1090) = 244.21, p < 0.001]. Post hoc comparison using the Tukey HSD test showed that the mean Health Related Content Index score for pro-vaccination webpages (M = 8.00, SD = 1.78) was significantly higher than both the neutral (or undefined) set of webpages (M = 6.32, SD = 1.77, p < 0.001), and the anti-vaccination links (M = 5.62, SD = 1.60, p < 0.001). Here the maximum score was obtained by the following pro-vaccination webpage: https://www.familydoctor.org/familydoctor/en/kids/vaccines/polio-vaccine.html, which fulfilled all the 13 content quality attributes included in the Health Related Content Index.

The specific categories which played an important role generating this significant effect were: medical ownership⁶ –80 % of the pro-vaccination webpages were affiliated with a medical organization, compared to only 5.7 % of anti-vaccination pages, and 30 % of the neutral subset – (p < 0.001); target audience – one quarter of all the pro-vaccination webpages explicitly stated their intended target audience compared to only 9 % of the anti-vaccination, and 11 % of the neutral subset – (p < 0.001); date of last update –available on half of the pro-vaccination subset, compared to only 12 % of the anti-vaccination and 13 % of the neutral webpages – (p < 0.001); accreditations – 100 out of 109 accredited webpages evaluated had a favorable view toward the vaccination practice – (p < 0.001); advertising presence – out of the total number of 114 webpages displaying unclear distinction between core contents and advertising, 107 were anti-vaccination – (p < 0.001); privacy policy – roughly 90 % of the pro-vaccination webpages properly stated the relative policies about the treatment of confidential data submitted by end-users, compared to only 60 and 77 % reported by the anti-vaccination and the neutral (or undefined) sets of the sample, respectively – (p < 0.001); and complementarity statement – available on 80 % of the pro-vaccination subset compared to only 56 % of the anti-vaccination, and 40 % of the neutral webpages – (p < 0.001). The exception is the date of creation, which was indicated more often on both neutral (or undefined) and anti-vaccination webpages (60, and 45 %, respectively) than on the pro-vaccination subset (32 %), (p < 0.001). Scientific references of the original contents disclosed were lacking on around 38 % of the pro- and on almost half of the anti-vaccination pages (49 %). Only 27 pro-vaccination pages did not offer contacts information about the owners of the website (5.3 %), while this information was lacking on roughly 17 % of the anti-vaccination subset, resulting impossible for users to contact the responsible person. Only 2 % of pro- and around 3 % of anti-vaccination web sources delivered 'easy-to-read' contents (i.e., readability score below or equal the 6^th grade levels). One-fifth of both pro- and anti-vaccination webpages showed medium levels of readability (19.1 % vs. 21 % respectively). Indeed the current study showed that the majority of the pages promoted vaccine-related contents that were proved to be difficult to read (i.e., readability score equal or above the 10^th grade levels; 79.2 % of pro- vs. 76.2 % of anti-vaccination sources). The provision of another language other than English was adopted by nearly 46 % of pro- and 38 % of anti-vaccination webpages.

Only 180 webpages were judged to target healthcare professionals. The majority of them (83.3 %) held a favorable general tone toward immunizations. Neutral (or undefined) and anti-vaccination web sources accounted for 13.3 and 3.4 % of webpages targeted to healthcare professionals respectively. Figure 2 provides a visual representation of the mean Health-Related Content index scores obtained by the three different subsets of the sample.

Vaccination-specific Content Index. Web sites holding a favorable view toward the vaccination practice provide more vaccination-specific information themes compared to the neutral vaccination-related online sources. The anti-vaccination subset, however, has proved to include vaccination-specific information and services features as much as the other two subsets without significant differences. Comparing the scores on the aggregated Vaccination Specific Content Index of the three groups based on the webpage's general tone, a statistically significant difference (p < .05) was observed [F(2.1090) = 3.59, p = 0.028]. Post hoc comparison using the Tukey HSD test showed that the mean Vaccination Specific Content Index score for pro-vaccination webpages (M = 5.95, SD = 2.06, p = 0.02) was significantly higher than the webpages that were neutral (or undefined) toward the vaccination practice (M = 5.37, SD = 2.60). Remarkably, the maximum score achievable and reported of fulfilling all the 9 vaccination-specific categories was attained by the anti-vaccination subset of webpages (e.g., https://www.prisonplanet.com/myth-busted-vaccinations-are-not-immunizations.html). In detail, neutral webpages conveyed less disease information (40 %) than both pro- and anti-vaccination subsets (58 and 52 %, respectively), p = 0.001. Treatment information was delivered more by the pro-vaccination subset (83 %), compared to both anti-vaccination and neutral (or undefined) webpages (64 and 63 %, respectively), p < 0.001. Benefits, and risks of vaccination were present in both pro-, and anti-vaccination webpages (79 and 83 %, respectively), but not as much for the subset being neutral (or undefined) in regard to vaccination (56 %), p < 0.001. As expected, benefits of vaccination were persistently treated by pro-vaccination webpages (75.3 % of the total sample's subset). About 30 % of neutral, and 10 % of anti-vaccination webpages delivered the benefits derived by immunizations, p < 0.001. Conversely, severe risks of vaccination were insistently promoted by anti-vaccination webpages (78 % of the total subset of the sample), followed by the neutral (or undefined) subset (50 %). One fifth of pro-vaccination webpages advanced the sub-topic (or theme) of the serious complications potentially caused by several immunizations, p < 0.001. The provision of terms' definitions and Q&A formats was significantly less frequent in anti-vaccination webpages (only 16 %), as compared to both pro-vaccination (29 %) and neutral subsets (35 %), p < 0.001. The inclusion of external links to other vaccination-related web sources was, as estimated, a central component for all the subsets of webpages investigated (around 85 % to 90 % each subset). In particular, a positive relationship was observed between a favorable tone toward the vaccination practice and the provision of external links to other pro-vaccination sources (77 %), p < 0.001. About half of neutral (or undefined), and one quarter of anti-vaccination webpages reviewed were externally linked with further pro-vaccination websites. On the other hand, external links to supplementary anti-vaccination websites were positively correlated with the negative tone of the webpage (83 %), p < 0.001. About 60 % of neutral (or undefined), and only 4 % of pro-vaccination webpages offered links to external anti-vaccination websites.

Total Aggregated Quality Index. Webpages holding a positive tone toward the vaccination practice fulfilled consistently more quality criteria included in the present content analysis when compared to the opponent and neutral vaccination related online sources. It has however to be noted that none of the three different subsets obtained average scores falling into the high quality range (>22 points). The analysis revealed significant differences (p < .01) when comparing the scores on the Total Aggregated Quality Index of the three groups based on the webpage's general tone [F(2.1090) = 66.49, p < 0.001]. Post hoc comparison using the Tukey HSD test showed that the mean Total Aggregated Quality Index score for pro-vaccination webpages (M = 21.31, SD = 3.64) was significantly higher than both the anti-vaccination links (M = 18.25, SD = 4.57, p < 0.001), and the neutral (or undefined) set of webpages (M = 18.84, SD = 5.30, p < 0.001). By taking a deeper look at these findings, it has to be observed that none of the webpages included in the final sample of the present study comprehensively fulfilled all the 34 information quality categories coded. The maximum score reported was 31 categories fulfilled by the following pro-vaccination webpage: https://www.netdoctor.co.uk/conditions/pregnancy-and-family/a9117/childhood-vaccinations/.

Figure 3 offers a graphical representation of the mean Total Aggregated Quality Index scores acquired by the three subsets of webpages.

BMC Public Health. 2016;16(38) © 2016  BioMed Central, Ltd.