Original language: English

Eduardo Aponte (Puerto Rico)

A first-generation higher education graduate; professor of comparative education at the University of Puerto Rico. Has been a consultant to the Puerto Rico Council on Higher Education since 1987, and currently is the president of the Puerto Rican Association for Higher Education (1998-2000).

The context of evaluation systems

Evaluation systems are influenced and defined by the political and historical contexts in which they were created and developed. The creation and refinement of an evaluation culture in higher education is a relatively new activity and it is becoming an increasingly widespread enterprise. Although there is an international trend, national-level politics very much determine evaluation culture—what, why, when and how evaluation systems work.

The existing evaluation culture in many countries is a decision-making oriented process. The culture is essentially organized and structured for making judgements about the quality (desirable goodness) of the institution's performance, and the relevance (pertinence and effectiveness) of the institutional efficiency (capacity and competency) to accomplish its mission. The evaluation system comprises: assessment processes for the collection and interpretation of information for decision-making purposes in which there is also formal and systematic testing; and quantifying or selecting criteria to define quality and determine the relevance of the courses of action (Cowen, 1996), i.e. policy-making and strategy implementation.

Accountability evaluation of higher education systems has two interrelated components: The internal accountability of how institutions are carrying out their mission, how well they are performing, what they are doing to assess their own effectiveness and what they are doing to improve performance; and the external accountability evaluations in many different forms of audits, giving the grounds for confidence and continued public support. Both types of evaluation are desirable and necessary. Currently, these processes contradict rather than complement each other. Given the quality-relevance matrix, different approaches are needed to strengthen them and to reinforce each other in the new context of higher education.

As the next century approaches, higher education institutions face a variety of challenges. As institutions have grown, they have become more complex, have taken on different functions and use greater resources. In response to recurrent fiscal crises of States, societies are questioning the mission of higher education institutions, as well as their contributions to economic, social and cultural development goals. Moreover, their efficiency and effectiveness are being debated in terms of funding and budgetary allocations in both the public and the private sectors. The objective circumstances of higher education systems have changed and there is a sense that higher education institutions are of central importance to modern societies. These factors have increased the attention given to higher education evaluation in many countries.

In the North American region, public debate about the accountability of all institutions has been escalating for the last two decades. During recent years, discontent with higher education institutions and their bureaucracies has reached unexpected levels. For many people, tuition increases have hindered access to higher education, and therefore a brighter future has become a fading dream. Both citizens and the State are expressing increasing concern about efficiency, productivity and quality in higher education and the effectiveness and relevance (pertinence) of its curriculum. Different sectors are seeking evidence and assurance about the quality and relevance of higher education, particularly in this transition period as we move towards a 'knowledge society'. This period can be defined as one in which there is a shift in the balance of employment from industrial manufacturing to the dynamic sectors of production-of-knowledge institutions and services; from factory work to professional work; a shift towards a society organized around knowledge and the information economy with a growing demand for higher education.

Accountability is being promoted by the different existing forms of evaluation in regional, State and governmental regulations, market forces, institutional self-regulation and accreditation (including peer-evaluations and the multiple forms undertaken by the institutions of higher education themselves). Institutional efforts include those carried out by trustees, administrators, academic units, presidents, chancellors, faculty, students, some ad-hoc joint committees and external funding sources.

Within this new evaluation context of 'accountable for what and to whom', institutions nowadays are trying to use evidence to assure themselves and the public that they are accomplishing their mission in an efficient and effective way. They are trying to make sure that they are meeting their own expectations and the public's quality standards. Many of their internal efforts are evident in the self-appraisal studies and the public support of these institutions expressed by continuous enrolments, funding and the external accreditation evaluation process. The higher education evaluation system does not lack accountability, rather it lacks enough of the proper kind and is burdened with too much of an unproductive kind. Therefore, questions about quality and relevance remain at the forefront of the accountability movement in terms of allocating resources to an increasing demand for access to higher education.

The quality-relevance matrix

For a political decision-maker, it is embarrassing that higher education is constantly criticized, not only for educating new generations for the wrong fields, but also for providing education that is not relevant in terms of quality. What higher education institutions are responsible and accountable for, is the quality and relevance of their educational provision. In this respect, it is the responsibility of higher education authorities and institutions to find the necessary practical solutions to these problems.

In aspiring to a knowledge-intensive society, higher education institutions are rapidly losing their earlier status as the only providers of higher education, while at the same time the status of formal qualifications is changing. In this new context, competence and performance are increasingly based on knowledge and skill regardless of where and how they have been acquired. Continuous human resource qualification through training and non-formal education in a competitive global economy is changing the concept of higher education as an institution. Governments and policy-makers expect higher education to react rapidly to the challenges of permanent lifelong learning.

The emerging knowledge-intensive society requires a response to different educational needs and to diverse growing demands for quality standards and relevance, and also entails differentiation between institutions and diversification within institutions. By and large, the response to society's educational needs has been through the establishment of new institutions alongside the traditional higher education offerings and the creation of new degree programmes to supplement the existing ones. To the question of relevance, the response has generally been two-fold: a laissez faire policy in which institutions are allowed to grow according to market needs and the traditional policy, in which expansion does not equal diversification. Anticipating and responding rapidly to new educational needs is determining and defining both quality and relevance nowadays in higher education. Hence, higher education institutions must serve societal needs and development goals.

Under the recurrent situation of fiscal crisis of the State and an ever-increasing cost of higher education, access to a quality educational opportunity has become the greatest challenge to democratic countries. In the emerging knowledge-intensive world economy and ever-increasing credentialist society, equal opportunity to meritocratic institutions and access to higher education remains at the centre of the ongoing efforts to transform policy proposals (Aponte, 1996; Dias, 1997). Limited access to a meritocratic societal structure will exacerbate social conflict and weaken liberal representative democracies.

The access/quality antinomy

In the North American region, the United States has the highest student participation rate in higher education. Yet the accountability movement has pushed policy-makers and funding sources to reconsider admission policies and the conditions for financing or investing in higher education opportunities. The debate is between those who believe inclusive policies are promoting 'academic welfare' to the lower income groups with limited social and cultural capital, and those who emphasize being more selective in order to achieve quality. Current efforts to integrate quality and access rarely reach fruition because of the inherent contradictions in the meritocratic policy approach to achieve quality at the expense of access according to measurable individual capacity for higher education academic work. Furthermore, access to unequal educational opportunities and to low-quality institutional offerings should not be considered access.

The growth of sophisticated assessment techniques and of large testing institutions testifies to the new trend towards greater selectivity in the international higher education community. As a result, although many institutions have a high percentage of successful graduates, this is not necessarily attributed to the quality of the programmes, but rather to the talents and contacts the students bring to the institutions. They would be highly successful in any institution.

Besides the equity (political) issue, another question remains: are there economic, social or cultural reasons to continue and to broaden access to higher education? Although the positive social and cultural contributions of higher education to development are well known, the need for a new workforce for the knowledge society is being underestimated. The criticism notwithstanding, indicators of confidence in higher education remain impressive. Enrolments have grown, despite the shrinking pool of traditional meritorious higher education applicants and higher-than-inflation tuition increases as the income differential between general higher education continues to grow. This demonstrates the employment market's confidence in higher education. Opinion polls and focus groups continue to place higher education as a key goal. It is estimated that two-thirds of the newly created jobs in the dynamic sectors of the economies of developed countries will require higher education training in order to meet the demand for the next decade and beyond (OECD, 1997; Reich, 1991). Both the growth of higher education graduates to meet this need and access to higher learning by the different sectors of society (women, ethnic and low-income groups) to this new 'techno-meritocracy', will be crucial for the distribution of economic (income wealth) and political power (knowledge as social and cultural capital) in democratic societies. In developing countries, a quality-relevant diversified expansion of the higher education system is a priority; limiting access to reach higher quality levels will further increase social inequalities and the impact of the 'brain drain' at the national and regional levels. A meritocracy based on more rigorous selection and exclusion will weaken the social selection and legitimization functions of education in relation to the social structure—an uneven societal development in which different constituent groups are under-represented. Therefore, the antinomy of access/quality and the new pertinence must be brought together and reconciled.

The accountability/autonomy paradox

The accountability/autonomy conflict is so complex and redundant that an additive solution would produce even worse results. This evaluation system includes State and voluntary accrediting entities, State agencies, legislative measures seeking surveillance over institutions, government regulations recognizing the accreditors and private groups attempting to 'certify' the accreditors, financial audits, institutional rating systems, bond ratings and multiple market forces, plus 'self-monitoring' by institutions themselves. The accreditation evaluations have both the potential to intrude on institutional autonomy and to induce real improvements in academic programmes. Therefore, the pertinence of the evaluation system (programme assessments, reviews, audits, etc.) is very important as a strategic policy instrument in the higher education community. The loss of institutional autonomy is both cause and consequence of the abdication of responsibility by institutions for managing their own affairs, namely assuring high-quality, relevant and effective learning and research (i.e. responsible autonomy).

Towards a new evaluation culture

In the emerging knowledge society, where knowledge has acquired more economic, social and cultural value, increasing numbers of the population will need access to higher learning, useful knowledge and new work-related skills. Higher education institutions are at the centre of this process. Therefore, quality and relevance are being aggressively pursued both inside the institutions and by the dual accountability movement.

This new context adds two new criteria in the evaluation of quality and relevance of higher education institutions. Besides resources and outcomes (input-output criteria) 'value added' and 'process' criteria are being incorporated into evaluation systems in order to determine improved institutional efficiency and effectiveness in accomplishing one's mission. While input (nature and extent of available resources to the institution, students, faculty and financial resources) and output criteria (nature and extent of institutional outcomes, alumni employment and success after graduation) once were the primary criteria for evaluating quality and relevance, the accountability movement and the new context of higher education are bringing together internal and external evaluations in a relationship where they can complement each other as an integrated system. The value-added criteria (the transformation and difference that the institution makes in the growth of all of the members of the learning community—including intellectual, moral, social, vocational and spiritual development) and the process-oriented criteria (the levels and forms of participation by all of the constituencies in the educational and administrative governance of the institution) address how they define and determine quality and relevance (Aponte, 1996; Berquist, 1995).

These new criteria and the demand for more access and higher graduation rates in higher education suggest a possible alternative scenario for the future of higher education. This scenario is being propelled by the shift from the teaching to the learning paradigm (Barr & Tagg, 1995), where higher education institutions are becoming learning organizations and are being evaluated by their performance in facilitating student learning and for producing useful knowledge for the different sectors of society. Therefore, the value-added criteria and the institutional participatory processes can be considered as a need and a condition for the development of the knowledge society.

Hence, the new context sets the stage for the access/quality antinomy—under fiscal crisis and diminishing available resources to both students and institutions, how can an education system increase access, assure quality and be more relevant, efficient and effective? How can institutions transform quantity into quality with higher graduation rates? The answer to these questions can be found in a needed transformation of the higher education institutions where evaluation systems can play an important strategic role.

Two possible scenarios for the future of higher education in terms of evaluation policy outcomes can be presented.

• Evaluation systems oriented just to the 'input-output' criteria will promote a more selective higher education system pursuing the quality-relevance matrix being promoted by accountable policies oriented to accomplish more efficiency and effectiveness. But limited graduation rates will broaden the gap between institutional quality growth and the demand for more relevant education in the emerging knowledge society.

• Evaluation systems that incorporate value-added and process criteria can improve institutional performance by increasing access and graduation rates, while at the same time strengthening responsible autonomy and increasing institutional relevance.

Quality in higher education can no longer be defined by measuring and evaluating institutional resources or outcomes. There is a need for a definition of quality based on the achievement of learning outcomes, regardless of how those outcomes are within the context, mission and purpose of higher education. For such a new context to emerge, an alternative concept of 'relevant quality' has been put forward. Relevant quality exists when adequate and appropriate resources are directed successfully towards the accomplishment of mission-related institutional outcomes, when the institutional programmes make a significant difference in the lives of the people affiliated to them, when programmes are created, conducted and modified in a participatory process according to the values and mission of the institution (Aponte, 1996; Berquist, 1995). The new evaluation culture of relevant quality and the four types of criteria can be expressed through the relationships shown in Figure 1.

FIGURE 1. The new evaluation culture

Research and evaluation are needed to develop and refine the relevant quality approach and the new culture in higher education.

Concluding remarks and recommendations

The development of new evaluation systems is not an easy task. The successful implementation of the new conceptual approach will depend on many factors:

1. The transition of higher education institutions towards student-centred, intensive knowledge learning organizations;

2. A shift in accountability evaluation policy towards a dual system of institutional self-evaluation and external audit systems to reinforce and support internal evaluation and to strengthen institutional responsibility and relevance;

3. The promotion of evaluation research to create useful knowledge in order to develop and refine the new evaluation culture in higher education institutions; and

4. The enhancement of the conditions for international comparative research and co-operation by sharing expertise and knowledge on how to develop the new evaluation system as a strategy to transform higher education institutions.

The emerging knowledge society, the paradigm shift to learning organizations and the fiscal crisis of the State in a globalizing economy make up the new context of threats and opportunities for policy options to transform higher education institutions. Evaluation policy is perhaps one of the most important strategic options to promote the necessary changes.

References

Albjerg-Graham, P.; Lyman, R.W.; Trow, M. 1995. Accountability of colleges and universities: an essay. New York, Columbia Office of Publications.

Aponte, E. 1996. Modelo de acreditación de acceso, calidad y pertinencia para la transformación de la educación superior [Model for increasing access, quality and relevance in order to transform higher education]. Paris, UNESCO. (Regional Conference: Policies and Strategies for the Transformation of Higher Education in Latin America and the Caribbean, Havana, Cuba.)

Barr, R.; Tagg, J. 1995. From teaching to learning: a new paradigm for undergraduate education. Change (Washington, DC), November-December, p. 11-26.

Berquist, W. 1995. Quality through access, access with quality: the new imperative for higher education. San Francisco, Jossey-Bass.

Cowen, R. 1996. World yearbook of education: the evaluation of higher education systems. London, Kogan Page.

Dias, M.A. 1997. Utopian realism in the reform of higher education. In: Towards a new higher education: Proceedings of the Regional Conference on Policies and Strategies for the Transformation of Higher Education in Latin America and the Caribbean, Havana, Cuba, p. 53-67. Caracas, CRESALC/UNESCO.

Organization for Economic Co-operation and Development. 1997. Towards a new global age: challenges and opportunities. Paris, OECD/CERI.

Reich, R. 1991. The work of nations: preparing ourselves for the 21^st century capitalism. New York, Alfred Knopf.