Executive Summary
This inaugural DTSP State-of-the-Industry Report synthesizes the results of the Safe Framework assessments conducted by ten DTSP partners during the first half of 2022. Participating partners for initial assessments were Discord, Google, LinkedIn, Meta, Microsoft, Pinterest, Reddit, Shopify, Twitter, and Vimeo. Other DTSP partners, including those that have joined the organization more recently, will participate in a future cycle of assessments.
DTSP launched in February 2021, bringing together companies of different sizes and business models, to develop industry best practices and verify their implementation through independent third-party assessments, to ensure consumer Trust & Safety when using digital services. Our goal is a safe online experience that continues to enable worldwide users of digital services to benefit from the social, economic, and political value they provide.
Our participating companies have committed to five fundamental areas of best practices (“the DTSP Commitments”) to which a digital service must adhere to promote a safer and more trustworthy internet. These five commitments are the foundation for trusted and safe products and services: product development, governance, enforcement, improvement, and transparency. They are underpinned by 35 specific best practices, known as the DTSP Best Praactices Framework, which provide concrete examples of the kinds of activities and processes that organizations will have in place to mitigate risks from harmful content and conduct.
DTSP Inventory of 35 Best Practices
Product Development
PD1 Abuse Pattern Analysis
PD2 Trust & Safety Consultation
PD3 Accountability
PD4. Feature Evaluation
PD5 Risk Assessment
PD6 Pre-Launch Feedback
PD7 Post-Launch Evuluation
PD8 User Feedback
PD9 User Controls
Product Governance
PG1 Potioes & Standards
PG2 User Focused Product Management
PG3 Community Guidelines/ Rules
PG4 User input
PG5 External Consultation
PG6 Document Interpretation
PG7 Community Self Regulation
Product Enforcement
PE11 Roles & Teams
PE12 Operational Infrastructure
PE13 Tooling
PE2 Training & Awareness
PE3 Wellness & Resilience
PE4 Advanced Detection
PE5 User Reporting
PE6.1 Enforcement Prioritization
PE6.2 Appeals
PE6.3 External Reporting
PE7 Flagging Processes
PE8 Third Parties
PE9 Industry Partners
Product Improvement
PI1 Effectiveness Testing
PI2 Process Alignment
PI3 Resource Allocation
PI4 External Collaboration
PI5 Remedy Mechanisms
Product Transparency
PT1 Transparency Reports
PT2 Notice to Users
PT3 Complaint Intakes
PT4 Research & Academic Support
PT5 In-Product Indicators
Participating partners conducted internal assessments using the DTSP assessment methodology, the Safe Framework. The assessments examined the people, processes, and technology that contribute to managing content- and conduct-related risks for participating companies. Companies assessed their practices against a common maturity scale ranging five levels, from ad hoc to optimized:
DTSP Maturity Rating Scale
1. Ad Hoc
A rating of Ad Hoc is assigned when execution of best practices is incomplete, informal, or inconsistent.
2. Repeatable
A rating of Repeatable is assigned when execution of best practices occurs without standardized processes.
Organizations aim to document more formalized practices.
3. Defined
A rating of Defined is assigned when execution of best practices occurs with defined and documented processes.
Processes are more proactive than reactive and are implemented across the organization.
4. Managed
A rating of Managed is assigned when execution of best practices is defined, documented, and managed through regular reviews.
Organizations use feedback to continuously mitigate process deficiencies.
5. Optimized
A rating of Optimized is assigned when execution of best practices promotes Trust & Safety in every aspect.
Processes are continuously improved with innovative ideas and technologies.
Safe Framework Assessments in a Nutshell
DTSP has defined goals (the DTSP Commitments), identified actions to achieve them (best practices), and established a means of measurement (the maturity scale). We have aimed to be as concrete as possible, while taking a proportionate, risk-based approach where companies focus on addressing the risks particular to their products and practices.
For example, under the Product Development commitment, there are nine listed best practices, one of which is risk assessment. A company evaluating the strength of its risk assessment practice would rank itself on the Maturity Scale anywhere from “ad hoc” to “optimized.”
This report reflects our first use of the Safe Framework in practice. We are encouraged by the results, and know that our learnings from this initial effort will help us to iterate and make our framework more effective over time. For this inaugural round, each company chose particular aspects of their operations to examine. Some focused on individual products, for example, or focused on specific kinds of prohibited content. Some assessed all five DTSP Commitments, while others focused on a subset of key commitments and best practices.
As a test run for the DTSP assessment framework, the results are encouraging. Participating companies were able to substantively assess where they stand and develop future opportunities for improvement. The results are encouraging, not only in terms of the insights shared in this report, but because they show that companies of different sizes and business models have been able to implement a common assessment framework, which will evolve as our work moves forward.
Because of the differences in approach to assessment that were deliberately built into the Safe Framework, we cannot offer sweeping conclusions about all industry practices. However, we do offer some key insights and trends that can inform future action by companies, governments, and civil society.
Key Insights
Successes: many companies reported a mature state of development for core content moderation practices
Eight best practices were assessed at an overall maturity level of Managed:
| Maturity Rating: (4) Managed | Commitment | Managed Practices |
|---|---|---|
| A rating of Managed is assigned when execution of best practices is defined, documented, and managed through regular reviews. Organizations use feedback to continuously mitigate process deficiencies. | Product Governance | Establish a team or function that develops, maintains, and updates the company’s corpus of content, conduct, and/or acceptable use policies |
| Product Governance | Develop user-facing policy descriptions and explanations in easy-to-understand language | |
| Product Enforcement | Constitute roles and/or teams within the company accountable for policy creation, evaluation, implementation, and operations | |
| Product Enforcement | Develop and review operational infrastructure facilitating the sorting of reports of violations and escalation paths for more complex issues | |
| Product Enforcement | Invest in wellness and resilience of teams dealing with sensitive materials, such as tools and processes to reduce exposure, employee training, rotations on/off content review, and benefits like counseling | |
| Product Enforcement | Implement method(s) by which content, conduct, or a user account can be easily reported as potentially violating policy (such as in-product reporting flow, easily findable forms, or designated email address) | |
| Product Enforcement | Ensure relevant processes exist that enable users or others to “flag” or report content, conduct, or a user account as potentially violating policy, and enforcement options on that basis | |
| Product Transparency | Provide notice to users whose content or conduct is at issue in an enforcement action (with relevant exceptions, such as legal prohibition or prevention of further harm) |
These practices show that Trust & Safety teams and functions across the partners have performed relatively well when it comes to core practices and activities that fall squarely within their domain and can be implemented unilaterally. These include constituting the teams responsible for content policies and developing public facing policy descriptions, as well as developing enforcement infrastructures that span people, processes, and technology, as well as notifying users whose content is subject to an enforcement action for violating policies.
Areas for improvement: many of the least mature practices relate to user feedback and external collaboration
Seven best practices were assessed at an overall level of maturity of Repeatable:
| Maturity Rating: (2) Repeatable | Commitment | Managed Practices |
|---|---|---|
| A rating of Managed is assigned when execution of best practices is defined, documented, and managed through regular reviews. Organizations use feedback to continuously mitigate process deficiencies. | Product Development | Provide for post-launch evaluation by the team accountable for managing risks and those responsible for managing the product or in response to specific incidents |
| Product Governance | Institute processes for taking user considerations into account when drafting and updating relevant Product Governance | |
| Product Governance | Create mechanisms to incorporate user community input and user research into policy rules | |
| Product Governance | Facilitate self-regulation by the user or community to occur where appropriate, for example by providing forums for community-led governance or tools for community moderation and find opportunities to educate users on policies, for example, when they violate the rules | |
| Product Improvement | Develop assessment methods to evaluate policies and operations for accuracy, changing user practices, emerging harms, effectiveness and process improvement | |
| Product Improvement | Use risk assessments to determine allocation of resources for emerging content- and conduct-related risks | |
| Product Transparency | Create processes for supporting academic and other researchers working on relevant subject matter (to the extent permitted by relevant law and consistent with relevant security and privacy standards, as well as business considerations, such as trade secrets) |
Three of the practices deemed least mature, according to the self-assessments, related to incorporating user and third-party perspectives into Trust & Safety policy and practices. This illustrates the internal focus of Trust & Safety functions. As a discipline, Trust & Safety has developed with less external engagement outside of companies until recently.
The least mature of all assessed practices is the creation of processes to support academic and other researchers working on relevant subject matter. This is an area of great interest, with pending regulations in Europe and proposed legislation in the United States. While several assessments indicated planned improvements to mature this practice in the coming year, improvements to legal frameworks to address concerns around security and privacy, among other concerns, may be worthy of consideration.
Areas of ongoing improvement: integrating Trust & Safety into product development
The majority of assessments indicated companies were in the process of formalizing the relationship between Trust & Safety and product teams to better integrate these perspectives into product development. By continuing to adopt and enhance practices in this commitment, including the “Safety by Design” approach, companies anticipate improvement across the Product Development commitment in the future, with some companies planning concrete improvements in the coming year.
Where We Go from Here
This report marks the beginning of our collective effort to evolve and evaluate Trust & Safety practices across the industry. Partner companies are using the results of their self-assessments to enhance their practices, and DTSP will facilitate learning to identify collective opportunities to mature key practices.
Looking ahead, we foresee the following lines of effort:
- Evolving the DTSP Best Practices Framework: we will use the insights generated through this process and input received externally to review and improve the DTSP Best Practices Framework;
- Moving to third-party assessment: we are working with experts with deep understanding of both Trust & Safety and assessment frameworks to articulate our approach to independent third-party assessment and will share more information on this in the coming months; and
- Engaging stakeholders globally: as we raise awareness of Trust & Safety best practices, we will put in place specific mechanisms for stakeholder input and engagement and provide opportunities for dialogue.