CHAPTER 9
Outputs, monitoring progress and developing success criteriaThis chapter addresses the related issues of: identifying and developing effective outputs from the overview and scrutiny process; monitoring and evaluation of overview and scrutiny; and devising success criteria or performance indicators.
1. IDENTIFYING OUTPUTS AND ENSURING QUALITY OUTPUTS
The outputs from overview and scrutiny can be highly varied. The key outputs from the overview and scrutiny process are outlined below:
Use of call-in and the resultant response from the executive;
The executive’s response to the comments of overview and scrutiny on the executive’s forward plans, key plans and strategies;
Overview and scrutiny’s response to being consulted by other internal bodies (for example area committees) over key plans and strategies – and whether scrutiny’s comments are incorporated. (As external scrutiny develops, another ‘output’ will result from external bodies consulting overview and scrutiny committees);
Best value reports and recommendations;
Recommendations and reports from in-depth policy development and review work (including external scrutiny).
The success of overview and scrutiny committees in achieving quality outputs will partly rest upon the following:
Ensuring that comments, recommendations and reports are demonstrably evidence-based;
Demonstrating that recommendations and reports are founded on a robust process of gathering evidence from a wide range of internal and external stakeholders;
Developing recommendations that are SMART (specific, measurable, accurate, realistic and timely);
Producing reports that are written in clear English, have a summary, outline all the evidence gathered and use a range of visual aids to engage the reader;
Ensuring that outputs are clearly ‘owned’ by members who are closely involved in drafting recommendations and reports.
2. MONITORING PROGRESS AND SUCCESS CRITERIA
A number of authorities have begun to monitor the progress of overview and scrutiny. The key method used by authorities has been regular updates of progress against the work programmes. This often takes the form of a standing item summarising the progress against the work programme on overview and scrutiny agendas. This is an important process indicator. And authorities are also often providing annual reports of the overview and scrutiny function to full council (some examples of authorities that use this mechanism are provided below, as well as information on Bristol’s annual report).
Authorities where Overview and Scrutiny Committees provide Annual Reports
Newham London Borough
www.newham.gov.uk/democracy/scrutiny/AnnualReport.pdfBuckinghamshire County Council
Bedfordshire County Council
Carlisle City Council
Darlington Borough Council
Leeds City Council
South Ribble Borough Council
Bristol City Council: Overview and Scrutiny Annual Report 2001 – 2002
Bristol’s most recent annual report provides a wealth of relevant information about overview and scrutiny in a user-friendly, accessible form. It includes a diagrammatic representation of the relationship between full council, executive and overview and scrutiny as well as short reports on the activities and achievements of each of the Councils’ scrutiny commissions plus three case studies of the work of specific sub-committees or select committees. There are details of council spending on (and council support for) overview and scrutiny, plus a full list of the 54 cooptees attached to the various commissions and select committees. The report is laced with quotes from scrutiny committee chairs and external representatives. The tone of the report is refreshingly self-critical, with problems acknowledged as well as achievements.
Contact: The Scrutiny Team, Bristol City Council
Tel: 0117 9222742
Email: scrutiny@bristol-city.gov.ukA number of authorities have also undertaken internal and externally-facilitated evaluations of their overview and scrutiny investigations. The methods used for these evaluations include:
Member working groups;
Surveys of members and officers (and 'externals');
Interviews/focus groups with members and officers;
Observation of overview and scrutiny meetings;
Tracking detailed case studies.
These periodic evaluations are an important ‘health check’ on what are still fundamentally experiments with new ways of working. Guidance on undertaking such evaluations is provided in New Political Management Arrangements: A Self-Evaluation Toolkit (S. Snape, 2000, available via the IDeA). Interestingly, some authorities have also undertaken best value reviews of their democratic processes (Wiltshire County Council, Chichester District Council).
And authorities are also taking more interest in ‘holistic’ approaches to developing success criteria which attempt to identify and capture processes, outputs and outcomes. For example, Bedfordshire County Council is developing its own range of local performance indicators and measures (details on this system are provided below). And South Ribble Borough Council have systematically collected key data on progress over a period of two years (more details are provided below).
Performance Indicators for Overview and Scrutiny: Bedfordshire County Council
The Scrutiny Support Team at Bedfordshire County Council are in the process of developing the following performance indicators for their work:
Quantitative
Number of meetings of each committee during the year;
Number of members of committees attending Select Committee meetings (expressed as a percentage);
Number of agenda items (select committee inquiries; best value reviews; policy development; service monitoring; monitoring reports; and so on);
Number of occasions that the supporting papers to an agenda item were provided to Member Services by a deadline agreed with the Head of Member Services (expressed as a percentage) and where the team are in control of the supply of supporting papers those that do not meet this standard;
The number of press releases issued for reports of Select Committees;
Spend against budget;
Payment of invoices;
Answering telephone calls;
Responses to correspondence;
Number of occasions where there has been variance from Corporate Policies.
Qualitative
The team will conduct a range of attitudinal research, including questionnaires, to gauge its performance. The following groups will be involved:
County councillors and members of the committees who are not county councillors;
The Corporate Management Team;
Staff who participate in the overview and scrutiny process;
Witnesses;
Visitors to the authority.
The team will conduct such research on a six-monthly basis in order to secure the opportunity for continuous improvement.
Contact: Bill Hamilton, Assistant Chief Executive (Scrutiny)
Tel: 01234 228032
Email: HamiltWD@csd.bedfordshire.gov.uk
South Ribble’s Monitoring and Improvement Approach
South Ribble have continuously experimented with ways of measuring the effectiveness of their three scrutiny committees. They now have two years of data covering the following:
Whether the committees keep to their agreed programmes, actions and values;
The effectiveness of the preparation arrangements for visitors to scrutiny;
The effectiveness of the rooms/layouts used;
The effectiveness of the process at interviews;
The fairness of scrutiny reports/recommendations;
The value added by each scrutiny review;
Scrutiny members’ own satisfaction levels and suggestions for improvement.
Regular progress reports are produced and areas for improvement are addressed, for example through member workshops and training or through revising processes.
An illustration of South Ribble’s approach is provided by the mid-year progress report produced in December 2001 by the Head of Scrutiny and Improvement. This report reviews the work of the three scrutiny committees in the first half of 2001/02. Included in the report is an interesting table which sets out selected performance measures set against the position at October 2000 and the comparative position for the period May to November 2001. The targets listed are:
Scrutiny members enthusiasm for their role – with scaled collated responses shown from ‘very enthusiastic’ to ‘not enthusiastic’ gathered through two member surveys per annum;
Visitors’ opinion on whether the Committee’s values were met – with scaled collated responses from ‘fully met’ to ‘not at all met’ gathered through continuous feed-back from visitors;
Visitors’ opinion on fairness of members’ comments – with average score (scale used from 1 to 5);
Acceptability of members’ recommendations – percentage of recommendations either fully or substantially accepted provided;
Visitor’s opinion of value added by scrutiny members – with average score (scale 1 to 5);
Scrutiny members’ confidence that they receive impartial advice – with scaled collated responses from ‘very confident’ to ‘not confident’;
Scrutiny members’ satisfaction with the support they receive – with scaled collated responses from ‘very well satisfied’ to ‘not satisfied’.
And this report only lists seven of the performance targets and indicators that the Council is using – in practice information is also being gathered on a wider range of targets.
Contact: John Dakin, Head of Scrutiny and Improvement, South Ribble Borough Council
Email:jDakin@southribble.gov.ukIt is also possible to develop a range of ‘indicators’ or ‘success criteria’ which relate back to the Government’s key efficiency, transparency and accountability (ETA) criteria for new constitutions. Table 28 lists potential performance indicators against the following criteria:
- Greater Efficiency in decision-making;
- Greater Transparency in decision-making;
the ETA criteria
- Greater Accountability in decision-making;
- Encouragement of public involvement;
- Supporting effective partnership working;
- Providing satisfying and meaningful roles for councillors.
However, no authority could develop a system for performance measurement using all these measures – this would be too complex a task. Instead, authorities would need to select a range of appropriate measures and in many cases develop ‘standards’, ‘goals’ or ‘benchmarks’ based on those suggested below.
It is also the case that many indicators will be process or throughput indicators. Some output indicators are possible to devise but true outcome indicators, are, as ever, difficult to devise. This is particularly the case for overview and scrutiny committees since, as influencing and not decision-making bodies, measuring ‘influence’ is highly problematic. This means that authorities – as in the detailed examples provided above – will have to use attitudinal surveys and data to capture whether internal and external individuals and organisations perceive that they have been influenced significantly (or not).
Table 28 Criteria for considering the operation and effectiveness of overview and scrutiny
Objective
Methods/Measures/Indicators/Targets
Enhanced accountability:
- To hold the executive to account
- To support policy development and review through policy reviews
- To provide effective member oversight of the best value process
1. Holding the Executive (or Policy Committees) to Account
Number of items on cabinet agendas which are ‘scrutinised’ by overview and scrutiny committees
Number of items which are ‘referred back’/called-in to cabinet or council by overview and scrutiny
Number of items which are amended by cabinet, after challenge by overview and scrutiny
Number of comments produced by overview and scrutiny on the cabinet’s Forward Plans
Views of scrutiny councillors, cabinet members, other councillors, officers and others (where appropriate)[1]
2. Policy Reviews
Number of policy reviews which are undertaken within a given time period
Percentage of policy reviews which are completed within their original deadline
Percentage of policy reviews which are completed within budget
Percentage of reviews which are completed to their original specifications (ie fulfilling terms of reference, interviewing all those scheduled for interview, etc)
Number of reports produced by overview and scrutiny committees
Number of recommendations per policy review
Percentage of recommendations per policy review accepted by full council
Percentage of policy reviews followed up within a given timeframe (after original completion date)
Views of scrutiny councillors, cabinet members, other councillors, officers and others (where appropriate)
3. Member Oversight of Best Value (BV)
Percentage of best value reviews which go before overview and scrutiny committees
Length of time spent discussing BV items in scrutiny meetings
Time spent by councillors outside meetings on BV issues
Number of amendments by scrutiny councillors to BV reports which come in front of them
Views of scrutiny councillors, cabinet members, other councillors, officers and others (where appropriate)
Views of external bodies, such as the Best Value Inspectorate
Greater transparency
Number of meetings held in public
Percentage of meetings held in public
Ratio of ‘A’ to ‘B’ items
Number of days before meeting that the agenda is published
Number of days after meeting that the minutes are published
Number of places where agenda and minutes are distributed – both inside main council offices and outside
Publication of agenda and minutes on the council website
Publication of forward work programme for scrutiny
Number of places where work programme is available
Publication of information leaflet on how scrutiny works
Number of places where information leaflet is distributed
Awareness of overview and scrutiny amongst general public (gathered through citizens panel questions)
Awareness of overview and scrutiny amongst partners (gathered through survey)
Views of members of public who attend overview and scrutiny committees (either formal meetings or public meetings called by committee) on how clearly they understood the proceedings (gathered by short ‘exit’ questionnaires)
Views of external witnesses who are interviewed by overview and scrutiny committees on how clearly they understood the proceedings (gathered through short ‘exit’ questionnaire)
Greater efficiency in decision-making
Average number of days before scrutiny reviews cabinet decisions and takes any necessary action
Average number of days for cabinet to respond to scrutiny interventions
Number of scrutiny ‘interventions’ (ie number of times an item goes before a scrutiny committee) for a random sample of agreed full council policy items
Number of meetings through which a decision passes
Length of time an item takes to pass through new structures
Councillor time spent in meetings (and preparing for meetings)
Number of councillors involved in meetings
Number of meetings
Officer time spent in meetings
Views of scrutiny councillors, cabinet members, other councillors, officers and others (where appropriate)
Encouragement of public involvement in the scrutiny process
Seating arrangements for members of public
Provision of facilities for members of public (disabled access, etc)
Name plates for members of committees and others involved in the formal proceedings
Numbers of members of the public attending overview and scrutiny committee meetings within a given timeframe
Number of specifically targeted public meetings held by overview and scrutiny committees
Number of articles in local press relating to work of overview and scrutiny committees within a given timeframe
Percentage of meetings when a member of the local press is present
Views of members of public who attend overview and scrutiny committees (either formal meetings or public meetings called by committee) on various aspects of how the meeting was managed and proceeded (gathered by short ‘exit’ questionnaires)
Use of citizens’ panels to identify awareness of scrutiny and general opinions on scrutiny
Supporting effective partnership working
Number of scrutiny investigations undertaken within a given timeframe that are focused on cross-agency issues (such as health, crime etc)
Number of representatives of partners interviewed by overview and scrutiny committees (within a given timeframe)
Number of written responses to requests for evidence, provided by partner agencies (within a given timeframe)
Views of partner organisations (gathered through short ‘exit’ questionnaires to those representatives interviewed by committees, or more generally through surveys)
Development of influential and satisfying roles for scrutiny councillors Number of meetings in a given timeframe
Attendance of scrutiny councillors at meetings
Balance of questions asked of internal and external attendees across the membership of overview and scrutiny committees
Amount of time spent on scrutiny work by councillors. Time spent in meetings, preparing for meetings, etc., identified separately (this can be gathered through diary keeping exercises)
Views of scrutiny councillors (gathered through surveys, focus groups, ‘review’ sessions and workshops)
[1] These views can be gathered using a range of methods: surveys, focus groups, ‘review’ sessions for scrutiny committees, workshops, interviews and so on.
Published 17 October 2002
Return to Local Government Index
Return to Home Page
Web site terms
