This report is part of The Understanding “the Biggest Lie on the Internet” Project, which unpacks the reasons people agree to terms of service and privacy policies without accessing, reading, or understanding them.

When accessing terms of service and privacy policies for digital services, whose policies are the most difficult to read?

This report assesses the school-grade reading level for the privacy and terms of service policies of 70 digital services. The services includes: social media services, popular apps, Canadian internet service providers (ISPs), American ISPs, Canadian banks and American banks.

Summary of Key Findings

  • Overall, terms of service (TOS) and privacy policies (PP) tend to be quite complex.
  • For TOS, 52 of 70 digital services require a grade 12 reading level (one American bank did not have TOS).
  • 42 of 70 require a grade 13 reading level or higher (i.e. college or university education).
  • For PP, 50 of 70 require a grade 12 reading level or higher, and 38 of 70 require a grade 13 reading level or higher.
  • The most complicated policy document was J.P. Morgan Chase’s TOS with a grade score of 17.4.
  • Northwestel’s PP reading level was grade 17.3, Consolidated Communications’ PP 17.1, and Fongo Mobile’s TOS 16.6. SHAREit’s TOS requires a 16.9 grade reading level and Airbnb’s TOS was 16.4.
  • Some services have policy materials that are far less complicated.
  • Snapchat’s PP had a 9.2 grade reading level, TOS from CIBC 9.5, Rogers 9.1, Eastlink 8.7, and Cogeco 8.6.

Why Assess Digital Service Policy Complexity?

Policies requiring a college or university education can pose considerable challenges to understanding1. Individuals using apps and websites are already feeling resigned, apathetic, and overwhelmed when it comes to protecting privacy online2. Thinking policies are complicated, and upon trying them out, experiencing the expected or worse, may make policy engagement seem futile. “The biggest lie on the internet” (“I agree to the terms and conditions”)3 is said to be the idea that individuals click “agree” without accessing, engaging or understanding policies, and may be due, in part, to policy complexity4. Evaluating the complexity of privacy and terms of service policies will help determine which digital service providers are working to make policies easier to understand, and as a result, which may be helping to address “the biggest lie on the internet”.

Meaningful online consent is viewed as essential to ensuring privacy online5. Individuals make choices every day about digital service engagement, and as a result, should have access to information about those services and the implications of use. For example, individuals should know how data is collected, managed, and used. People should know if their data might, for example, be used to develop an artificial intelligence service. They should understand that data collected might be used to make eligibility decisions at a bank, at work, at a store, at an international border, and beyond6. This is a primary reason providing individuals with “notice”, or information about how digital services operate, is “fundamental”7 to privacy protections. Privacy and terms of service policies are often the mechanisms for attempting to deliver that vital information.

How is Policy Complexity Assessed?

The analysis began with the collection of the complete terms of service and the complete privacy policy for 70 digital services (see glossary below for a description of a “complete” policy). Researchers collected policy text from digital service websites via desktop computer in 2019 and at the beginning of 2020. Before completing the complexity analysis, policy word counts were assessed via a complementary policy length analysis (see this report) to provide context and support for the complexity analysis.

The analysis of complexity for each policy text was assessed through an academic measure8 for determining the school-grade reading level of a text – the Flesch-Kincaid Grade Score9 (see glossary below). The higher the grade level score, the more difficult the text is to read. Thus, text reading at a grade 10 through 12 level requires a high school reading ability. Texts scoring higher than 12 require reading levels associated with a college or university education.

Findings for all 70 Services

Social Media Services

  • Reddit and LinkedIn’s TOS are the most complicated. Grade levels are 14 and 13.7 respectively.
  • TikTok and Tumblr’s PPs are the most complicated. Grade levels are 13.3 and 13 respectively.
  • TOS tend to have higher grade levels than PP’s, with the average TOS 12.7 and PP 12.1.
  • Privacy policies are generally less-complicated than TOS. Instagram, Twitter and Wikipedia have PP grade levels higher than TOS.
  • Snapchat’s PP is the only policy lower than a grade 10 reading level.
  • 13 of 20 policies require more than a grade 12 education.

Figure 1. Policy Grade Reading Levels: Social Media Services

Table 1: Social media terms of service reading levels

Popular Apps

  • SHAREit and Airbnb’s TOS are the most complicated. Grade levels are 16.9 and 16.4 respectively.
  • SHAREit and Airbnb also have the most complicated PPs. Grade levels are 15.7 and 14.5 respectively.
  • TOS tend to have higher grade levels than PP’s, with the average TOS 14.5 and PP 12.9.
  • TOS are generally more-complicated than PP, with only Netflix having a more complicated PP.
  • None of the policies require less than a grade 10 education.
  • 15 of 20 policies require more than a grade 12 education.
  • 12 of 20 policies require more than a grade 13 education.
  • 5 policies require more than a grade 15 education.

Figure 2: Popular apps

Table 2: Reading level Popular Apps

Internet Service Providers (Canada – Major)

  • The TOS for Vidèotron and TELUS are the most complicated. Grade levels are 16.5 and 14.1 respectively.
  • TekSavvy and Cogeco’s PPs are the most complicated. Grade levels are 15.4 and 15.3 respectively.
  • While Vidèotron’s TOS has a very high score, PP’s tend to have higher grade levels than TOS, with the average PP 12.7 and TOS 11.5.
  • Comparing TOS and PP for each service, only Vidèotron and TELUS having TOS grade levels higher than PP levels.
  • Shaw, Rogers, Eastlink and Cogeco’s TOS require less than a grade 10 reading level, with Eastlink and Cogeco requiring grade eight.
  • Only Eastlink’s PP is less than a grade 10 level at 9.9.
  • 10 of 20 policies require grade 12.9 reading level or more.

RGL-Canada-ISPs-MajorTable 3: Reading level Major Canadian ISPs

Internet Service Providers (Canada – Minor)

  • Fongo and Distributel’s TOS are the most complicated. Grade levels are 16.6 and 16.1 respectively.
  • Northwestel’s PP is very complicated, with a score of 17.3, well above all other PP scores.
  • The average PP grade is 13.9 and TOS 13.
  • Comparing TOS and PP for each service, some policies are similar in scoring (e.g. Freedom Mobile, Chatr, Primus, and Koodo) while for the others there is variation in whether TOS or PP is more complicated.
  • Sasktel and Northwestel’s TOS require less than a grade 10 reading level, while the lowest grade level for PP is Koodo at 11.1.
  • 14 of 20 policies require a grade 13.4 reading level or more.
  • Three policies require more than a grade 16 education.

Fig 4-RGL-Canada-ISP-MinorTable 4: reading level Minor Canadian ISPs

Internet Service Providers (USA)

  • Cox, Suddenlink, and Spectrum’s TOS are the most complicated. Grade levels are 15.6, 15.5, and 15.3 respectively.
  • Consolidated Communication’s PP is very complicated, with a score of 17.1, well above all other PP scores.
  • Comparing TOS and PP for each service, there is variation in whether TOS or PP is more complicated.
  • Only Verizon’s TOS (grade 9.9) requires less than a grade 11 reading level.
  • 16 of 20 policies require an education of grade 12.5 or more.
  • 10 of 20 policies require a grade 14 education or more.
  • None of the PP’s require less than a grade 12 education, and 8 of 10 PPs require more than a grade 13 education.

RGL-US-ISPs

Reading-level-table5

Banks (Canada)

  • Laurentian and TD Bank’s TOS are the most complicated. Grade levels are 16.1, and 15.2 respectively.
  • Scotiabank, Tangerine and CIBC’s PPs are the most complicated, with grade level scores of 14.4, 14.3, and 14.2 respectively.
  • There is variation in policy complexity across services.
  • CIBC’s TOS is the only of the 20 policies to require less than a grade 10 reading level.
  • 15 of 20 policies require an education of grade 12.5 or more.
  • 14 of 20 policies require a grade 13 education or more.

Reading-level-table6

Banks (USA)

  • JP Morgan Chase and Citibank’s TOS are the most complicated. Grade levels are 17.4, and 16.8 respectively.
  • JP Morgan Chase and U.S. Bank’s PPs are the most complicated, with grade level scores of 14.
  • TOS for American banks are considerably more complicated than PPs, with the average TOS grade 14.7 and PP 11.9.
  • Comparing TOS and PP for each service, all TOS are more complicated than PP.
  • PNC’s PP is the only of the 20 policies to require less than a grade 10 reading level.
  • The company U.S. Bank does not have a TOS, and is tied for the most complicated PP.
  • 14 of 20 policies require an education of grade 12.4 or more.
  • 10 of 20 policies require a grade 13.5 education or more.

Reading-level-table7

Conclusion

The findings of this report suggest that across the 70 digital services assessed, terms of service and privacy policies tend to be very complicated. While a few services present policies at a grade eight or nine reading level, the majority require a grade 12 or college/university reading level. This suggests digital service providers are not doing enough to help ensure the information available in terms of service and privacy policies is supporting meaningful online consent processes.

Provide individuals with too much policy text to read and they may be overwhelmed. Limit the amount of detail provided, and essential information may be hidden. This is what Nissenbaum refers to as the “transparency paradox”10. While the transparency paradox can be understood in the context of policy length, it can also be applied to analyses of policy complexity. Provide individuals with too much policy jargon and legal language, and they will likely not understand and feel overwhelmed. Oversimplify language and the details and nuance of how digital services operate may be hidden. Furthermore, research suggests that even though policies tend to be long and complicated, information vital to understanding the implications of agreement is often missing from digital service policies11. This suggests that the work of establishing stronger connections between meaningful forms of transparency and meaningful online consent requires far more than the simplification or shortening of policies.

Digital services must assume more of a leadership role to ensure the information provided both about the services they provide as well as the implications of policy agreement supports the realization of protections online12. This goes beyond addressing the length and complexity of policies and also should address the most effective ways (i.e. user-interface designs) for communicating this information to individuals. These efforts should aim to address “the biggest lie on the internet” and as a result, support individuals as they work to ensure understanding about the realities and implications of digital engagements.

Notes

1. See: Reidenberg, J. R., Breaux, T., Cranor, L. F., French, B., Grannis, A., Graves, J. T., Liu, F., McDonald, A., Norton, T. B., Ramanath, R., Russell, N. C., Sadeh, N., & Schaub, F. (2015). Disagreeable privacy policies: Mismatches between meaning and users’ understanding. Berkeley Technology Law Journal, 30(1), 39-68.

2. See: Draper, N. A., & Turow, J. (2019). The corporate cultivation of digital resignation. New Media & Society, 21(8), 1824-39; Hargittai, E., & Marwick, A. (2016). “What can I really do?” Explaining the privacy paradox with online apathy. International Journal of Communication, 10, 3737-3757; Oeldorf-Hirsch, A., & Obar, J. A. (2019). Overwhelming, important, irrelevant: Terms of service and privacy policy reading among older adults. In Proceedings of the 10th International Conference on Social Media and Society (pp. 166-173).

3. Lannerö, P. (2012, January 27). Previewing online terms and conditions: CommonTerms alpha proposal. http://commonterms.org/commonterms_alpha_proposal.pdf; Obar, J. A., & Oeldorf-Hirsch, A. (2020). The biggest lie on the internet: Ignoring the privacy policies and terms of service policies of social networking services. Information, Communication & Society, 23(1), 128-147; TOS;DR. (2022). Terms of service; didn’t read. https://tosdr.org.

4. See: Obar, J. A. & Oeldorf-Hirsch, A. (Forthcoming). Older adults and “the biggest lie on the internet”: From ignoring social media policies to the privacy paradox. International Journal of Communication.

5. See: Office of the Privacy Commissioner of Canada. (2018). Guidelines for obtaining meaningful consent. https://www.priv.gc.ca/en/privacy-topics/collecting-personal-information/consent/gl_omc_201805/

6. Citron, D. K., & Pasquale, F. (2014). The scored society: Due process for automated predictions. Washington Law Review, 89(1), 1-33; Pasquale, F. (2015). The black box society: The secret algorithms that control money and information. Harvard University Press.

7. Federal Trade Commission. (1998). Privacy online: A report to Congress. https://www.ftc.gov/sites/default/files/documents/reports/privacy-online-report-congress/priv-23a.pdf

8. See: Fiesler, C., Lampe, C., & Bruckman, A. S. (2016). Reality and perception of copyright terms of service for online content creation. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (pp. 1450-1461); Milne, G. R., Culnan, M. J., & Greene, H. (2006). A longitudinal assessment of online privacy notice readability. Journal of Public Policy & Marketing, 25(2), 238-249; Jensen, C., & Potts, C. (2004, April). Privacy policies as decision-making tools: an evaluation of online privacy notices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 471-478).

9. Kincaid, J. P., Fishburne Jr, R. P., Rogers, R. L., & Chissom, B. S. (1975). Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Naval Technical Training Command. Research Branch Report 8-75.

10. Nissenbaum, H. (2011). A contextual approach to privacy online. Daedalus, 140(4), 32–48.

11. Clement, A., & Obar, J. A. (2016). Keeping internet users in the know or in the dark: An analysis of the data privacy transparency of Canadian internet carriers. Journal of Information Policy, 6(1), 294-331; Obar, J. A., & Pan, J. (2021). Open communication about network neutrality? Assessing the internet traffic management transparency of Canadian internet carriers. Canadian Journal of Communication, 46(3), 629-644; Obar, J. A. (2022). Defining and assessing data privacy transparency: A third study of Canadian internet carriers. International Journal of Communication, 16, 1688-1712.

12. Ananny, M., & Crawford, K. (2018). Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability. New Media & Society, 20(3), 973–989; Obar, J. A. (2020). Sunlight alone is not a disinfectant: Consent and the futility of opening Big Data black boxes (without assistance). Big Data & Society, 7(1); Schaub, F., Balebako, R., & Cranor, L. F. (2017). Designing effective privacy notices and controls. IEEE Internet Computing, 21(3), 70–77.

Glossary

  • Complete Policy: A “complete” policy includes content posted on the primary policy page, as well as supplementary content provided via links on the primary page. Complete policies are important because meaningful online consent suggests individuals understand the words in the policy as well as the implications of clicking “agree”. The hope is that via supplemental explanations of terms, scenarios, and examples linked to policy text, digital service providers are attempting to support understanding.
  • Flesch-Kincaid Grade Score: This metric is often used to analyze legal documents and other non-fiction texts (see note 8 above). While the current analysis calculated scores via the functionality built-into Microsoft Word, the traditional calculation is “.39 (words/sentence) + 11.8 (syllables/word) – 15.59” as noted in Kincaid, J. P., Fishburne Jr, R. P., Rogers, R. L., & Chissom, B. S. (1975). Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Naval Technical Training Command. Research Branch Report 8-75.

Acknowledgements

Thank you to Andrew Hatelt for the research assistance and to Valeta Wensloff for the graphic design. This project received funding from the Office of the Privacy Commissioner of Canada and from York University. Background image from istock.com: anyaberkut.

policy complexity