Document - AI@50 global project evaluation: Executive summary
� �PAGE �4� � CONTINUOUS LEARNING, CONTINUOUS IMPACT Impact study of the still separate, still unequal campaign in Slovakia �PAGE �4� AI@50 GLOBAL PROJECT EVALUATION A Global Movement Celebrates: Amnesty International’s 50th anniversary, May 2011 to May 2012
ai@50 gLOBAL pROJECT EVALUATION
a GLOBAL MOVEMENT CELEBRATES: aMNESTY INTERNATIONAL’S 50TH ANNIVERSARY, MAY 2011 TO MAY 2012
This document presents the key findings of the evaluation of Amnesty International’s AI@50 project. AI@50 was developed to celebrate the movement’s 50th anniversary. The key message of the project was a reminder of how powerful the actions of one individual can be. The strategy and planning phase of the project commenced in December 2009 and a range of campaigning activities and events took place between May 2011 and May 2012, including two global campaign moments: Shine a Light and Raise a Toast to Freedom. In addition to these global moments, the project aimed to amplify the work of five on-going campaigns, through coordinated actions around peak campaign moments and social media activities.
The purpose of the evaluation of AI@50 was to measure the outcomes of the project and ensure that these outcomes are visible to the co-owners of the project within the Amnesty International movement; and to capture learning that can inform ongoing ‘One Amnesty’ work on global projects, global reporting, global aggregation of data, which will support the success of future global projects.
This evaluation was conducted and led by the AI@50 Global Project Team, with support from work stream leaders and the Learning and Impact Unit (LIU), now Strategy and Evaluation Unit (SEU) at the International Secretariat (IS), and an external consultant. The evaluation builds on several interim evaluations, and comprises reviews of both internal processes and external outcomes. The evaluation is focused on global level processes and outcomes. National level processes and outcomes are evaluated by sections and structures.
The evaluation methodology was guided by an appreciative inquiry approach and consisted of the following main steps:
Desk review of key documents, including other global evaluations undertaken during the life of the project
Online surveys to gather feedback from sections and structures within the Amnesty International movement
Staff reflection workshops
Semi-structured in-depth staff interviews
The analysis of findings according to the project’s monitoring and evaluation framework based on Amnesty International’s Dimensions of Change and lessons from Amnesty International’s UDHR60 campaign. Lessons learnt and recommendations were drawn from trends emerging from the above
A key challenge and limitation of the evaluation was a lack of data, which means that some outcomes, such as growth objectives around increasing awareness and activism of people in the global south, cannot be fully measured or evaluated.
a) Human rights knowledge and awareness-raising
On the whole, AI@50 was successful in fulfilling this objective. The evaluation finds that there was an increase in awareness of Amnesty International and the AI@50 human rights campaign themes. Mainstream media and social and digital media of national activities at two key global moments were key contributors to this. Comparing the media impact of the two global moments side by side, success was mixed and it is clear that the global moment on 28 May 2011 (Raise a Toast to Freedom) achieved more media coverage, with human rights issues featuring prominently. Despite its weaker media performance, the global moment on the 10 December 2011 (Shine a Light) should be considered a success in terms of levels of engagement and activism at national level.
There are many positive signs that AI@50 successfully targeted younger people and those living in the global south, including the level of engagement on social media campaigns and the range of new partnerships which were formed with cultural institutions, former prisoners of conscience, schools and universities, festivals and media partnerships. The evaluation indicates that AI@50 made it possible for entities to develop more innovative collaborations. Local Amnesty International groups were cited as a valuable resource both for action and reaching out to a range of local constituencies.
b) Increased activism
There is evidence of widespread mobilization around global moments. For example, Shine a Light on 10 December 2011 - specifically designed to help amplify AI’s Letter Writing Marathon - generated 1.3 million actions, almost doubling the number of actions produced in 2010. In addition, available data suggests more than 28,000 people participated in some 580 events. However, significant numbers of sections and structures found that while AI@50 did increase activism and did increase the diversity of activists, this was to a lesser extent than they had hoped.
It remains difficult to evaluate incidences of mass mobilization at specific times of focus in support of the AI@50 campaign themes – data is very limited. However, increased international solidarity, and its benefits to local activists, was noted across all project themes.
Most sections and structures felt they achieved less than they had hoped for. The evaluation indicates that there was insufficient buy in for fundraising objectives and insufficient capacity at national level, perhaps reflecting the project’s focus on strengthening the 'Supporter Journey' at global level. That said, a positive indicator for all was the success achieved in reaching new people through raising awareness and activism and retaining existing supporters. This now provides a great opportunity to develop relationships that could lead to continued and new support.
d) Campaigning impact
AI@50 was intended to amplify and support selected current human rights campaigns at key moments throughout the year and the Letter Writing Marathon in December 2011. Achievements differ between the campaign areas and the actions, as did the level of additional ‘added value’ that AI@50 delivered. However there are some definite signs of successful change, including increased solidarity, increased awareness, changing opinions, support for local organizations, government and corporate responses and pledges, admissions of liability, and individual cases. The evaluation also identifies several areas where the AI@50 project added value across all campaign areas, including; global coordination and integration; section and structure participation; profile raising and the provision of effective and innovative tools and tactics. However, due to a crowded campaign schedule, it was not possible for the AI@50 project to react to a worsening human rights situation within a campaign thematic area.
a) Planning, strategy development and management of the project
The evaluation highlights a number of key problems during the planning phase of the project, including an inability to align strategies and priorities or develop SMART objectives and detailed action plans. The formation of a steering committee of work stream leaders with a strategic role and a project team responsible for day to day delivery is thought to have worked to some extent, but this model needs refining. For example, decision making was identified as a weakness of the project, and as a result, processes were unclear, not streamlined, or timely.
b) Level of internal participation
AI@50 aimed to get as many sections and structures engaged with the project as possible. There was a good and fairly evenly matched take up for each project objective and for each of the campaign themes (four campaign themes were adopted by more than half of the sections and structures who responded to the evaluation). Despite this, there was some frustration that it was not possible to identify a single campaign theme, despite the realities of catering for many different markets.
c) Internal communications
Coordination and support from the central AI@50 Global Project Team was considered to be of high quality, but the development and delivery of AI@50 products had mixed success. Some tools, such as the historical footage, were extremely useful, but the sheer amount of tools and materials planned may have been too ambitious. As a result, some tools which demanded highly technical input, were badly conceived or delivered, and the global messages for AI@50 were developed and shared too late. While there were several successful attempts to share skills and knowledge across sections and structures, there are also examples of missed opportunities. However, overall levels of consultation and feedback were deemed to be high.
The evaluation notes that it is still too early to assess how sustainable the gains - raising awareness; re-energising old supporters; and attracting new supporters - will prove to be. Early signs indicate AI@50 has established a good base upon which to build.
Many supporters and staff have indicated that they feel more part of a global movement and that events and actions during the AI@50 year made them feel more connected to the work of AI in other countries. That said, staff identified several significant challenges to alignment, including language; time zones; lack of effective sharing across the movement and reaping of synergies; lack of prioritisation and lack of project management skills; sheer numbers of stakeholders and little leadership and authority. These challenges are being addressed by other processes relating to wider organizational change and transition. The AI@50 project has provided opportunities to witness progress towards alignment, but it has not been the driver.
While AI@50 shows that the movement can come together around an over-arching message and specific global moments this does not mean per se that alignment has been achieved in terms of overall movement resources to stated organizational priorities.
The ‘added value’ of AI@50 was greater where it was more focused and targeted including defined campaign areas, peak activism periods, clear asks and calls to action - to the specific needs of the broader strategy of the area of work.
Despite the fact that the global project team and most sections and structures did not prioritise or plan for fundraising or membership gains at a national level, an important success highlighted in this evaluation is the potential for global projects to increase membership for smaller entities in the global south.
The global project has successfully made many staff and supporters feel part of a global movement and this was especially true in the global south (and in sections and structures looking to raise their visibility). However, the project was not prepared for increased awareness-raising having a negative impact on some smaller sections and structures which did not have the capacity to deal with the additional visibility or membership.
The global project was extremely successful in mobilizing sections and structures to participate in this project in general, and to sign up to individual campaigns they would not normally get involved with (in particular Sexual and Reproductive Rights; International Justice). The selection of multiple campaigns was considered to have caused confusion by many sections and structures and to have spread the organization’s capacity too thinly. But it is also recognised that selecting one campaign while trying to cater for many different contexts globally is extremely difficult.
The mixed results of the two global moments show that there is a need for further exploration into the key factors which made one event much more successful than the other.
The production of tools and materials for the global project are a good example of where the project was able to add value to national level plans, especially elements which sections and structures are unable to create themselves.
Examples of where the global project team enabled consultation, feedback, input and positive exchanges between sections and structures demonstrate how useful and powerful it can be.
If the AI@50 project had been based on a central project planning document with SMART objectives, many of the issues, identified by sections and structures and IS staff, could have been avoided, or rather spotted early and been overcome. As it was, a lack of a global decision making model and confusion over authority and decision making within the matrix management model of work stream leaders compounded these issues, delayed decisions and meant that processes were unclear, not streamlined, and not timely. Untimely delivery of messages and campaign tools and materials created problems for some sections and structures, and the global data management system was not delivered.
Positive steps have been taken to ensure that monitoring and evaluation is prioritised within the organisation. However, clear, globally shared metrics – SMART objectives and baseline data and a broader range of indicators - are required to fully measure project achievements. As a result the evaluation has not followed up on a number of issues – engagement in the global south, the added value of increased activism, and the qualitative successes of activities which reached new audiences.
Major surprises, successes and inconsistencies identified should be followed up (allocating extra time and resources if necessary) so that the organisation understands why these occurred and can build on or address them in the future (for example, mixed success with global moments, new audiences in the global south, an in-depth analysis of objectives and outcomes at section level, but particularly the contribution to human rights impact). This global communication project was extremely successful in breaking down internal and external barriers and removing stigma for issues such as sexual and reproductive rights. The movement should consider other human rights issues which could benefit from inclusion in future global projects.
Global project planning capacity should be built. SMART outcomes should be developed with clear signs of success at the development stage of projects and serious effort should be put into limiting the scope of project and deliverables based on this. Baseline data and data from existing campaigns on issues of awareness raising, contributions to impact, membership and fundraising, should be used to set and review indicators. In particular clear Key Performance Indicators and definitions for membership recruitment and retention, and tracking and reporting this data (all of which are the process of being developed) must be agreed upon at the beginning. Finally all future global plans should include a risk analysis – so that potential negative impacts from increased visibility, interest and awareness can be identified.
Ensure that the campaign area, peak activism period and associated ‘ask’ or ‘call to action’ is clearly defined to maximise the impact of global campaigning. However, build in the flexibility to respond to current and global and national events within the campaign perimeters (both in terms of human rights and growth outcomes). For example a mini ‘crisis and opportunities’ response element could be included within global project plans to ensure that such opportunities are not missed in the future, this could also be built into any risk assessment.
Ensure project team members have sufficient authority, outside of direct authority, and there is a clear and transparent approvals and decision-making process. Include as much of the global project work into existing projects identified within current operational plans to avoid competing priorities.
Continue to use the appreciative learning enquiry approach for evaluations which focus on organisational changes and developments as the organisation can really benefit from the resultant lessons. Ensure that these evaluations are adequately resourced.
Develop a common understanding of what alignment means with staff across the movement. During this process, ensure that the positive and successful aspects of One Amnesty in terms of human rights outcomes are well publicised.
Prioritise and facilitate the exchange of knowledge, materials and skills between sections and structures. Establish structures and tools to enable sections and structures to take the lead on this process, for example by making sections and structures part owners of products and working groups.
If the data management tool is fully delivered it will offer significant potential to understand global trends within the movement and for the movement to present itself more as ‘One Amnesty’. Resources and capacity should be aligned so that the tool can be finished and the movement can make the most of it.